In this page:
For a number of business processes a real-time processing pattern focused around capturing small changes in the system and packaging them into messages is used. Eagle handles this with different object types which often require a specific order of processing. This order is not a thing some of the clients would like to control, so Eagle has the Generic Load workflow which follows proper dependencies allowing to package various objects into a single message and send to Eagle for processing.
Default implementation has an issue with processing many messages with a small amount of data in them. It is due to the fact that the solution creates temporary files, and lots of small files can impact file system heavily.
To resolve this situation, a new approach is suggested with the main idea of creating no files. This is achieved by a two-step process:
1. RTR processor step – System receives RTR message (in most of the cases reads them from queue) with base64 encoded archive data. This rule uses a python utility which is able to receive RTR via stdin and then parse incoming RTR, base64-decode and decompress data in memory and create a single EagleML message which contains all objects sorted in accordance with Generic Loader dependencies. This single EagleML message is then piped into stdout. MC rule is able to consume stdout to get the single EagleML message and to add special new object called CreateTSR with information for TSR creation. Then this message is sent to the next step using ProcessInStream rule element.
See an example of RTR on this page.
Rule File used is eagle_ml-2-0_cm\in\xml\realtime_rtr.xml
Example of single EagleML message, which also contains CreateTSR object (click to expand):
<EagleML eaglemlVersion="2-0" xmlns="http://www.eagleinvsys.com/2011/EagleML-2-0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:SchemaLocation="http://www.eagleinvsys.com/2011/EagleML-2-0 eagleml-ext-2-0.xsd" xsi:type="ReferenceTransactionMessage">
<referenceTransaction>
<header>
<objectType>IssuerOrganization</objectType>
</header>
<issuerOrganization>
<objectType>IssuerOrganization</objectType>
<objectId>IssuerOrganization</objectId>
<objectDescription>REFERENCE</objectDescription>
<updateTimestamp>2015-08-21T12:00:00-01:00</updateTimestamp>
<updateDate>2015-08-21</updateDate>
<sourceName>EAGLE PACE</sourceName>
<updateSource>MCADMIN</updateSource>
<effectiveDate>2015-10-05</effectiveDate>
<issuerId>TST_TEST_ISSUER_01</issuerId>
<issuerName>TST_TEST_ISSUER_01</issuerName>
<address>1 WELLS</address>
<city>NEWTON</city>
<state>MA</state>
<countryOfRiskCode>US</countryOfRiskCode>
<countryOfIncorporationCode>US</countryOfIncorporationCode>
<currency>USD</currency>
<industryCode>TECHNOLOGY</industryCode>
<countryExposure>US</countryExposure>
</issuerOrganization>
</referenceTransaction>
<referenceTransaction>
<header>
<objectType>GenericSMF</objectType>
</header>
<genericSMF>
<objectType>GenericSMF</objectType>
<objectId>SMF</objectId>
<objectDescription>SMF</objectDescription>
<sourceName>EAGLE PACE</sourceName>
<updateSource>MCADMIN</updateSource>
<effectiveDate>2016-06-14</effectiveDate>
<primaryAssetId>TST_TEST_SMF</primaryAssetId>
<primaryAssetType>INTERNAL</primaryAssetType>
<issueName>TST_TEST_SMFISSUENAME</issueName>
<issueDescription>TST_TEST_SMFSSUEDESC</issueDescription>
<primaryExchangeCode>ALL</primaryExchangeCode>
<assetCurrency>USD</assetCurrency>
<investmentType>FI</investmentType>
<processingSecurityType>DBIBTL</processingSecurityType>
<securityType>CS</securityType>
<securitySubType>COMMON STCK</securitySubType>
<altSecurityMinorType>CASH</altSecurityMinorType>
<settlementCurrency>EUR</settlementCurrency>
<incomeCurrency>USD</incomeCurrency>
<quantityType>PAR</quantityType>
<priceMultiplier>103.103</priceMultiplier>
<quantityScale>104.104</quantityScale>
<issueCountryCode>US</issueCountryCode>
<countryOfRiskCode>US</countryOfRiskCode>
<issueTaxType>EQ</issueTaxType>
</genericSMF>
</referenceTransaction>
<referenceTransaction>
<header>
<objectType>GenericCA</objectType>
</header>
<genericCA>
<objectType>GenericCA</objectType>
<objectId>GenericCA</objectId>
<objectDescription>REFERENCE</objectDescription>
<updateTimestamp>2015-08-21T12:00:00-01:00</updateTimestamp>
<sourceName>EAGLE PACE</sourceName>
<updateSource>MCADMIN</updateSource>
<effectiveDate>2016-10-05</effectiveDate>
<primaryAssetId>TST_TEST_SMF</primaryAssetId>
<primaryAssetType>INTERNAL</primaryAssetType>
<issueName>TST_TEST_SMFISSUENAME</issueName>
<issueDescription>TST_TEST_SMFSSUEDESC</issueDescription>
<primaryExchangeCode>ALL</primaryExchangeCode>
<assetCurrency>USD</assetCurrency>
<clientSpecificCAId>1</clientSpecificCAId>
<transType>CASHDIV</transType>
<sweepDate>2015-10-05</sweepDate>
<exDate>2015-10-05</exDate>
<cpStatus>PD</cpStatus>
<cpReason>Cash Dividend</cpReason>
<cpInstance>6259</cpInstance>
<splitRatio>333.592</splitRatio>
<prevSecId>TSTR820A</prevSecId>
<divAmt>833.412</divAmt>
<cpAnnounceDate>2015-10-05</cpAnnounceDate>
<cpAcquiredBy>CP_ACRED_BY</cpAcquiredBy>
<cpAddedAmt>768.954</cpAddedAmt>
<cpAddress>CP_ADDRESS</cpAddress>
<cpAgent>CP_AGENT</cpAgent>
<toAsset>
<primaryAssetId>TST_TEST_SMF</primaryAssetId>
<primaryAssetType>INTERNAL</primaryAssetType>
<issueName>TST_TEST_SMFISSUENAME</issueName>
<issueDescription>TST_TEST_SMFSSUEDESC</issueDescription>
<primaryExchangeCode>ALL</primaryExchangeCode>
<assetCurrency>USD</assetCurrency>
</toAsset>
<corpActId>754107</corpActId>
<fromFmvPrice>502.536</fromFmvPrice>
<toFmvPrice>126.918</toFmvPrice>
<cpChangeDate>2015-10-05</cpChangeDate>
<cpNotes>CP_NOTES</cpNotes>
<batchEventId>TSR80A</batchEventId>
<costAmtMethod>M</costAmtMethod>
<consentRt>364.514</consentRt>
<consentDt>2014-07-22</consentDt>
<proRation>345.541</proRation>
<cpCrncy1>USD</cpCrncy1>
<roundingPoint>170.374</roundingPoint>
<limitGlIndicator>I</limitGlIndicator>
<trmtCashGl>I</trmtCashGl>
<preference>687795</preference>
<sequence>316684</sequence>
<matchingFlag>Y</matchingFlag>
<cpReinvestRatio>723.068</cpReinvestRatio>
<cxlEventType>CXL_EVYPE</cxlEventType>
<princLossFactor>446.15</princLossFactor>
<qdiPercent>0.4</qdiPercent>
<qdiHldPeriodOvrd>I</qdiHldPeriodOvrd>
<drdHldPeriodOvrd>I</drdHldPeriodOvrd>
<qdiDrdOvrdComment>QDI_DRDENT</qdiDrdOvrdComment>
<qdiPercentEffectiveDate>2015-10-05</qdiPercentEffectiveDate>
<qdiEligibleFlag>Y</qdiEligibleFlag>
<caDetailModel>
<actionPreference>7735</actionPreference>
<actionSequence>9163</actionSequence>
<cahdAmount>453.977</cahdAmount>
<cpTkr>CP_TKR</cpTkr>
</caDetailModel>
</genericCA>
</referenceTransaction>
<referenceTransaction>
<header>
<objectType>CreateTSR</objectType>
</header>
<createTSR>
<objectType>CreateTSR</objectType>
<JMSDeliveryStream>tst_jms_stream_tsr</JMSDeliveryStream>
</createTSR>
</referenceTransaction>
</EagleML>
2 Single EagleML message processing step. This is done via the same core stream eagle_ml-2-0_default_in_xml_reference, but with Splitting parameter set to No. This setting will assure that within Single EagleML message (generated on prior step) objects will go in proper order. However, due to the fact that ProcessInStream element was used in the prior step, different messages will still be processed concurrently. The rsf was modified to be able to execute GenericSMF objects as well. Finally, new CreateTSR object is also processed and a trigger is sent to eagle_ml-2-0_default_cm_task_reporter stream to create a TSR.
Implementation details
NOTE: For now developed functionality is supported only in ORACLE environments.
In rule for processing incoming RTR it is necessary introduce:
- Check for correlation id uniqueness
- Check for correlation id length limit
- Parse task parameterss
- Creating orch state
- Send Ack (if switched on) , generate ACK, sent to configured delivery method (if NACK Update Proc Status)
- Introduce task parameter for turn on/off ACK – EnableACKSwitch. In case EnableACKSwitch is Y or omitted then send ACK, in case EnableACKSwitch = N – do not send ACK.
<taskParameter>
<name>EnableACKSwitch</name>
<dataType>S</dataType>
<value>Y</value>
</taskParameter>
Ref Inbound Stream (No Split)
In rules for load (Ref Inbound Stream No Split) it is necessary introduce:
- Add to mapping for Correlation Id alias (reference_common_header_aliases.inc)
- Collect transaction id value for each record into orch state. The transaction_id field was used as it is an index field of msgcenter_dbo.msg_detail table.
New include should be created for get_w_state for current correlation id, and updating state with update list of transaction ids. The new include will be called from set_transaction_id_content.inc
In rules for eagle_ml-2-0_default_cm_task_reporterit is necessary introduce:
- New include similar to task_reporter_generate_task_unit.inc include in order to be able to collect msg_details by list of transaction id.
- Update logic of collecting TSR
- Update report_dist_methods,inc include in order to send message instead of file for Message Queue Delivery methods.
IssuePcPrice
In memory rule has been updated and new usage case has been added, currently it works only for IssuePcPrice with one referenceTransaction in the file. If the incoming file has more than one IssuePcPrice record or another objectType, it will be resend to the default eagle_ml-2-0_default_cm_control_message stream.
This behavior can be overridden by option:
W_RESEND_TO_REQUEST_STREAM
If file has 1 IssuePcPrice transaction it will be sent to the stream: eagle_ml-2-0_default_in_xml_reference as it happens usually, but right after load TSR message will be generated and delivered in the PcPrice rule (xml-ref_issuepcprice.xml). As a result you will not see any runs of eagle_ml-2-0_default_cm_task_reporter.