In this page:
In case of messaging client is implementing quite often real-time processing pattern, which usually focused around capturing small changes in the system and packaging them into messages. In case of Eagle we have different object types, which often need to be processed in specific order. This order is not something some of the clients would like to control. So, Eagle has a generic loader workflow, which implements proper dependencies. That allows clients to package all the different objects and package them into single message and send to Eagle for processing.
Current implementation has an issue of processing many messages with small amount of data in them. That is due to the fact that solution creates temporary files, and lots of small files present tremendous problem to the file system.
So, to resolve this the new approach is suggested and the main idea is to create no files. In order to achieve this the new process was developed which consist of 2 steps:
- RTR processor – This step receives RTR message (in most of the cases reads them from queue) with base64 encoded archive data. This rule uses newly developed python utility, which is able to receive RTR via stdin and then parse incoming RTR and base64decode and un-compress data in memory and create a single EagleML message, which contains all objects that are sorted in accordance with generic loader dependencies. This single EagleML message is then piped into stdout. MC rule is able to consume stdout to get single EagleML message and in addition to add special new object called CreateTSR, which contains information for TSR creation. Then this message is sent to the next step using ProcessInStream rule element.
Example of RTR is in Appendix 1.
Rule File eagle_ml-2-0_cm\in\xml\realtime_rtr.xml is in Appendix 2.
Example of single EagleML message, which also contains CreateTSR object (click to expand):
<EagleML eaglemlVersion="2-0" xmlns="http://www.eagleinvsys.com/2011/EagleML-2-0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:SchemaLocation="http://www.eagleinvsys.com/2011/EagleML-2-0 eagleml-ext-2-0.xsd" xsi:type="ReferenceTransactionMessage">
<referenceTransaction>
<header>
<objectType>IssuerOrganization</objectType>
</header>
<issuerOrganization>
<objectType>IssuerOrganization</objectType>
<objectId>IssuerOrganization</objectId>
<objectDescription>REFERENCE</objectDescription>
<updateTimestamp>2015-08-21T12:00:00-01:00</updateTimestamp>
<updateDate>2015-08-21</updateDate>
<sourceName>EAGLE PACE</sourceName>
<updateSource>MCADMIN</updateSource>
<effectiveDate>2015-10-05</effectiveDate>
<issuerId>TST_TEST_ISSUER_01</issuerId>
<issuerName>TST_TEST_ISSUER_01</issuerName>
<address>1 WELLS</address>
<city>NEWTON</city>
<state>MA</state>
<countryOfRiskCode>US</countryOfRiskCode>
<countryOfIncorporationCode>US</countryOfIncorporationCode>
<currency>USD</currency>
<industryCode>TECHNOLOGY</industryCode>
<countryExposure>US</countryExposure>
</issuerOrganization>
</referenceTransaction>
<referenceTransaction>
<header>
<objectType>GenericSMF</objectType>
</header>
<genericSMF>
<objectType>GenericSMF</objectType>
<objectId>SMF</objectId>
<objectDescription>SMF</objectDescription>
<sourceName>EAGLE PACE</sourceName>
<updateSource>MCADMIN</updateSource>
<effectiveDate>2016-06-14</effectiveDate>
<primaryAssetId>TST_TEST_SMF</primaryAssetId>
<primaryAssetType>INTERNAL</primaryAssetType>
<issueName>TST_TEST_SMFISSUENAME</issueName>
<issueDescription>TST_TEST_SMFSSUEDESC</issueDescription>
<primaryExchangeCode>ALL</primaryExchangeCode>
<assetCurrency>USD</assetCurrency>
<investmentType>FI</investmentType>
<processingSecurityType>DBIBTL</processingSecurityType>
<securityType>CS</securityType>
<securitySubType>COMMON STCK</securitySubType>
<altSecurityMinorType>CASH</altSecurityMinorType>
<settlementCurrency>EUR</settlementCurrency>
<incomeCurrency>USD</incomeCurrency>
<quantityType>PAR</quantityType>
<priceMultiplier>103.103</priceMultiplier>
<quantityScale>104.104</quantityScale>
<issueCountryCode>US</issueCountryCode>
<countryOfRiskCode>US</countryOfRiskCode>
<issueTaxType>EQ</issueTaxType>
</genericSMF>
</referenceTransaction>
<referenceTransaction>
<header>
<objectType>GenericCA</objectType>
</header>
<genericCA>
<objectType>GenericCA</objectType>
<objectId>GenericCA</objectId>
<objectDescription>REFERENCE</objectDescription>
<updateTimestamp>2015-08-21T12:00:00-01:00</updateTimestamp>
<sourceName>EAGLE PACE</sourceName>
<updateSource>MCADMIN</updateSource>
<effectiveDate>2016-10-05</effectiveDate>
<primaryAssetId>TST_TEST_SMF</primaryAssetId>
<primaryAssetType>INTERNAL</primaryAssetType>
<issueName>TST_TEST_SMFISSUENAME</issueName>
<issueDescription>TST_TEST_SMFSSUEDESC</issueDescription>
<primaryExchangeCode>ALL</primaryExchangeCode>
<assetCurrency>USD</assetCurrency>
<clientSpecificCAId>1</clientSpecificCAId>
<transType>CASHDIV</transType>
<sweepDate>2015-10-05</sweepDate>
<exDate>2015-10-05</exDate>
<cpStatus>PD</cpStatus>
<cpReason>Cash Dividend</cpReason>
<cpInstance>6259</cpInstance>
<splitRatio>333.592</splitRatio>
<prevSecId>TSTR820A</prevSecId>
<divAmt>833.412</divAmt>
<cpAnnounceDate>2015-10-05</cpAnnounceDate>
<cpAcquiredBy>CP_ACRED_BY</cpAcquiredBy>
<cpAddedAmt>768.954</cpAddedAmt>
<cpAddress>CP_ADDRESS</cpAddress>
<cpAgent>CP_AGENT</cpAgent>
<toAsset>
<primaryAssetId>TST_TEST_SMF</primaryAssetId>
<primaryAssetType>INTERNAL</primaryAssetType>
<issueName>TST_TEST_SMFISSUENAME</issueName>
<issueDescription>TST_TEST_SMFSSUEDESC</issueDescription>
<primaryExchangeCode>ALL</primaryExchangeCode>
<assetCurrency>USD</assetCurrency>
</toAsset>
<corpActId>754107</corpActId>
<fromFmvPrice>502.536</fromFmvPrice>
<toFmvPrice>126.918</toFmvPrice>
<cpChangeDate>2015-10-05</cpChangeDate>
<cpNotes>CP_NOTES</cpNotes>
<batchEventId>TSR80A</batchEventId>
<costAmtMethod>M</costAmtMethod>
<consentRt>364.514</consentRt>
<consentDt>2014-07-22</consentDt>
<proRation>345.541</proRation>
<cpCrncy1>USD</cpCrncy1>
<roundingPoint>170.374</roundingPoint>
<limitGlIndicator>I</limitGlIndicator>
<trmtCashGl>I</trmtCashGl>
<preference>687795</preference>
<sequence>316684</sequence>
<matchingFlag>Y</matchingFlag>
<cpReinvestRatio>723.068</cpReinvestRatio>
<cxlEventType>CXL_EVYPE</cxlEventType>
<princLossFactor>446.15</princLossFactor>
<qdiPercent>0.4</qdiPercent>
<qdiHldPeriodOvrd>I</qdiHldPeriodOvrd>
<drdHldPeriodOvrd>I</drdHldPeriodOvrd>
<qdiDrdOvrdComment>QDI_DRDENT</qdiDrdOvrdComment>
<qdiPercentEffectiveDate>2015-10-05</qdiPercentEffectiveDate>
<qdiEligibleFlag>Y</qdiEligibleFlag>
<caDetailModel>
<actionPreference>7735</actionPreference>
<actionSequence>9163</actionSequence>
<cahdAmount>453.977</cahdAmount>
<cpTkr>CP_TKR</cpTkr>
</caDetailModel>
</genericCA>
</referenceTransaction>
<referenceTransaction>
<header>
<objectType>CreateTSR</objectType>
</header>
<createTSR>
<objectType>CreateTSR</objectType>
<JMSDeliveryStream>tst_jms_stream_tsr</JMSDeliveryStream>
</createTSR>
</referenceTransaction>
</EagleML>
2 step is to process Single EagleML message. This is done via the same core stream ‘eagle_ml-2-0_default_in_xml_reference’, but with Splitting parameter set to No. This setting will assure that within Single EagleML message (generated on prior step) objects will go in proper order. However, due to the fact that ProcessInStream element was used in prior step the different messages will still be processed concurrently. The rsf was modified to also be able to execute GenericSMF objects. Finally, new CreateTSR object is also possible to process and rsf has a reference to correspondent rule. Currently the trigger is sent in Create TSR to the ‘eagle_ml-2-0_default_cm_task_reporter’ stream.
Implementation details
NOTE: For now developed functionality is supported only in ORACLE environments.
In rule for processing incoming RTR it is necessary introduce:
- Check for correlation id uniqueness
- Check for correlation id length limit
- Parse task parameterss
- Creating orch state
- Send Ack (if switched on) , generate ACK, sent to configured delivery method (if NACK Update Proc Status)
- Introduce task parameter for turn on/off ACK – EnableACKSwitch. In case EnableACKSwitch is Y or omitted then send ACK, in case EnableACKSwitch = N – do not send ACK.
<taskParameter>
<name>EnableACKSwitch</name>
<dataType>S</dataType>
<value>Y</value>
</taskParameter>
Ref Inbound Stream (No Split)
In rules for load (Ref Inbound Stream No Split) it is necessary introduce:
- Add to mapping for Correlation Id alias (reference_common_header_aliases.inc)
- Collect transaction id value for each record into orch state. The transaction_id field was used as it is an index field of msgcenter_dbo.msg_detail table.
New include should be created for get_w_state for current correlation id, and updating state with update list of transaction ids. The new include will be called from set_transaction_id_content.inc
In rules for eagle_ml-2-0_default_cm_task_reporterit is necessary introduce:
- New include similar to task_reporter_generate_task_unit.inc include in order to be able to collect msg_details by list of transaction id.
- Update logic of collecting TSR
- Update report_dist_methods,inc include in order to send message instead of file for Message Queue Delivery methods.
IssuePcPrice
In memory rule has been updated and new usage case has been added, currently it works only for IssuePcPrice with one referenceTransaction in the file. If the incoming file has more than one IssuePcPrice record or another objectType, it will be resend to the default eagle_ml-2-0_default_cm_control_message stream.
This behavior can be overridden by option:
W_RESEND_TO_REQUEST_STREAM
If file has 1 IssuePcPrice transaction it will be sent to the stream: eagle_ml-2-0_default_in_xml_reference as it happens usually, but right after load TSR message will be generated and delivered in the PcPrice rule (xml-ref_issuepcprice.xml). As a result you will not see any runs of eagle_ml-2-0_default_cm_task_reporter.