Overview

This functionality requires Python to be installed. Please refer to Pypy Deployment.

Currently available only for ORA environments.

For a number of business processes a near-real time processing pattern focused around capturing small changes in the system and packaging them into messages is used. Eagle handles this with different object types which often require a specific order of processing. This order is not a thing some of the clients would like to control, so Eagle has the Generic Load workflow which follows proper dependencies allowing to package various objects into a single message and send to Eagle for processing.

Default implementation has an issue with processing many messages with a small amount of data in them. It is due to the fact that the solution creates temporary files, and lots of small files can impact file system heavily.

To resolve this situation, a new approach is suggested with the main idea of creating no files. This is achieved by a two-step process:

1. RTR processor step – System receives RTR message (in most of the cases reads them from queue) with base64 encoded archive data. This rule uses a python utility which is able to receive RTR via stdin and then parse incoming RTR, base64-decode and decompress data in memory and create a single EagleML message which contains all objects sorted in accordance with Generic Loader dependencies. This single EagleML message is then piped into stdout. MC rule is able to consume stdout to get the single EagleML message and to add special new object called CreateTSR with information for TSR creation. Then this message is sent to the next step using ProcessInStream rule element.

<EagleML
 xmlns="https://protect-us.mimecast.com/s/9QZ3C9rpDgHkAGpRIBevTv?domain=eagleinvsys.com"
 xmlns:xsi="https://protect-us.mimecast.com/s/_zz8C0R20LHGQ1LJsBl0uB?domain=w3.org"
 eaglemlVersion="2-0"
 eaglemlType="RunTaskRequest"
 xsi:schemaLocation="https://protect-us.mimecast.com/s/9QZ3C9rpDgHkAGpRIBevTv?domain=eagleinvsys.com eagleml-main-2-0.xsd"
 xsi:type="RunTaskRequest">
	<header>
		<messageId>SMF_331624_71454390-9726-41d2-95f6-b4c16e125cf0</messageId>
		<creationTimestamp>2018-01-24T18:10:44.483-06:00</creationTimestamp>
	</header>
	<taskIdentifier>
		<correlationId>LOAD_TEST_3</correlationId>
		<businessTaskId>Generic_Loader_Bundle</businessTaskId>
	</taskIdentifier>
	<taskTypeEnum>LOAD</taskTypeEnum>
	<taskParameters>
		<taskParameter>
			<name>StreamName</name>
			<dataType>S</dataType>
			<value>eagle_ml-2-0_default_cm_execwkfl</value>
		</taskParameter>
		<taskParameter>
			<name>ActionType</name>
			<dataType>S</dataType>
			<value>EXECUTE</value>
		</taskParameter>
        <taskParameter> 
			<name>EnableACKSwitch</name>
			<dataType>S</dataType>
			<value>N</value>
		</taskParameter>
        <taskParameter> 
			<name>DeliveryMethod</name>
			<dataType>S</dataType>
			<value>JMS</value>
		</taskParameter>
		<taskParameter> 
			<name>ReplyDeliveryMethod</name>
			<dataType>S</dataType>
			<value>JMS</value>
		</taskParameter>
		<taskParameter> 
			<name>ACKDeliveryMethod</name>
			<dataType>S</dataType>
			<value>JMS</value>
		</taskParameter>
		<taskParameter> 
			<name>JMSReplyDeliveryStream</name>
			<dataType>S</dataType>
			<value>tst_jms_stream_tsr</value>
		</taskParameter>
		<taskParameter>
			<name>JMSAckDeliveryStream</name>
			<dataType>S</dataType>
			<value>tst_jms_stream_tsr</value>
		</taskParameter>
		<taskParameter>
			<name>WorkflowName</name>
			<dataType>S</dataType>
			<value>eagle_wrf_generic_load</value>
		</taskParameter>
		<taskParameter>
			<name>CreateConsolidatedTSR</name>
			<dataType>S</dataType>
			<value>Y</value>
		</taskParameter>
		<taskParameter>
			<name>ConsolidatedTSROnly</name>
			<dataType>S</dataType>
			<value>Y</value>
		</taskParameter>
		<taskParameter>
			<name>WorkflowData</name>
			<dataType>S</dataType>
			<value>TST_EagleML.zip.base64</value>
		</taskParameter>
		<taskParameter>
			<name>Data</name>
			<dataType>S</dataType>
			<value>UEsDBBQAAAAIAFVgiUxelmrmhwQAAIoMAAAQAAAAX0dFTkVSSUNDQV8xLnhtbKVWW4/SQBR+1sT/
sNl3eqG0hU1tUtvuSgTEFo0+bWo7C6O94Exh2X/vmbYz04JGXRNCmO985zLnxjhhss3RcnF1KvKS
vr7e1fX+RlUfHx8VxCS4PNInqqRVoY41XVc7+mg80q5bnZsTxQO9R0OpyBbomq5+Xi7idIeKZIRL
Widliq6vGrtF/gkRiqvy9TWYEuDmaY9eX0foAREE5A1JSpqkNfCWiNJkC+rg7oY2NhdVmjDRP0XN
PY2KBJcMUE40a63Wf3Tuvnr5wiG/IDDBC2eHkgwR9hsO1ddvKK3Zhdw7VCKCU99zVIm2KmpPx9ly
3r+Y4LR51idxbEAJEE0J3jcBR+FtGIUrPwTuubBTOuyzpEYbXCAoXbF3IZXmSJuOxvpGH99oGnxG
mg7fjnpG7QzQ6kBStEoK5Ibe3SK8WnvMn4QHjuIGdpe+FyznK7ApUU5EDw8QKT6iACQs</value>
		</taskParameter>
	</taskParameters>
</EagleML>

Rule File used is eagle_ml-2-0_cm\in\xml\realtime_rtr.xml

<EagleML eaglemlVersion="2-0"  xmlns="http://www.eagleinvsys.com/2011/EagleML-2-0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:SchemaLocation="http://www.eagleinvsys.com/2011/EagleML-2-0 eagleml-ext-2-0.xsd" xsi:type="ReferenceTransactionMessage">
  <referenceTransaction>
    <header>
      <objectType>IssuerOrganization</objectType>
    </header>
    <issuerOrganization>
      <objectType>IssuerOrganization</objectType>
      <objectId>IssuerOrganization</objectId>
      <objectDescription>REFERENCE</objectDescription>
      <updateTimestamp>2015-08-21T12:00:00-01:00</updateTimestamp>
      <updateDate>2015-08-21</updateDate>
      <sourceName>EAGLE PACE</sourceName>
      <updateSource>MCADMIN</updateSource>
      <effectiveDate>2015-10-05</effectiveDate>
      <issuerId>TST_TEST_ISSUER_01</issuerId>
      <issuerName>TST_TEST_ISSUER_01</issuerName>
      <address>1 WELLS</address>
      <city>NEWTON</city>
      <state>MA</state>
      <countryOfRiskCode>US</countryOfRiskCode>
      <countryOfIncorporationCode>US</countryOfIncorporationCode>
      <currency>USD</currency>
      <industryCode>TECHNOLOGY</industryCode>
      <countryExposure>US</countryExposure>
    </issuerOrganization>
  </referenceTransaction>
  <referenceTransaction>
    <header>
      <objectType>GenericSMF</objectType>
    </header>
    <genericSMF>
      <objectType>GenericSMF</objectType>
      <objectId>SMF</objectId>
      <objectDescription>SMF</objectDescription> 
      <sourceName>EAGLE PACE</sourceName>
      <updateSource>MCADMIN</updateSource>
      <effectiveDate>2016-06-14</effectiveDate>
      <primaryAssetId>TST_TEST_SMF</primaryAssetId>
      <primaryAssetType>INTERNAL</primaryAssetType>
      <issueName>TST_TEST_SMFISSUENAME</issueName>
      <issueDescription>TST_TEST_SMFSSUEDESC</issueDescription>
      <primaryExchangeCode>ALL</primaryExchangeCode>
      <assetCurrency>USD</assetCurrency>
      <investmentType>FI</investmentType>
      <processingSecurityType>DBIBTL</processingSecurityType>
      <securityType>CS</securityType>
      <securitySubType>COMMON STCK</securitySubType>
      <altSecurityMinorType>CASH</altSecurityMinorType>
      <settlementCurrency>EUR</settlementCurrency>
      <incomeCurrency>USD</incomeCurrency>
      <quantityType>PAR</quantityType>
      <priceMultiplier>103.103</priceMultiplier>
      <quantityScale>104.104</quantityScale>
      <issueCountryCode>US</issueCountryCode>
      <countryOfRiskCode>US</countryOfRiskCode>
      <issueTaxType>EQ</issueTaxType>
    </genericSMF>
  </referenceTransaction>
  <referenceTransaction>
    <header>
      <objectType>GenericCA</objectType>
    </header>
    <genericCA>
      <objectType>GenericCA</objectType>
      <objectId>GenericCA</objectId>
      <objectDescription>REFERENCE</objectDescription>
      <updateTimestamp>2015-08-21T12:00:00-01:00</updateTimestamp>
      <sourceName>EAGLE PACE</sourceName>
      <updateSource>MCADMIN</updateSource>
      <effectiveDate>2016-10-05</effectiveDate>
      <primaryAssetId>TST_TEST_SMF</primaryAssetId>
      <primaryAssetType>INTERNAL</primaryAssetType>
      <issueName>TST_TEST_SMFISSUENAME</issueName>
      <issueDescription>TST_TEST_SMFSSUEDESC</issueDescription>
      <primaryExchangeCode>ALL</primaryExchangeCode>
      <assetCurrency>USD</assetCurrency>
      <clientSpecificCAId>1</clientSpecificCAId>
      <transType>CASHDIV</transType>
      <sweepDate>2015-10-05</sweepDate>
      <exDate>2015-10-05</exDate>
      <cpStatus>PD</cpStatus>
      <cpReason>Cash Dividend</cpReason>
      <cpInstance>6259</cpInstance>
      <splitRatio>333.592</splitRatio>
      <prevSecId>TSTR820A</prevSecId>
      <divAmt>833.412</divAmt>
      <cpAnnounceDate>2015-10-05</cpAnnounceDate>
      <cpAcquiredBy>CP_ACRED_BY</cpAcquiredBy>
      <cpAddedAmt>768.954</cpAddedAmt>
      <cpAddress>CP_ADDRESS</cpAddress>
      <cpAgent>CP_AGENT</cpAgent>
      <toAsset>
        <primaryAssetId>TST_TEST_SMF</primaryAssetId>
        <primaryAssetType>INTERNAL</primaryAssetType>
        <issueName>TST_TEST_SMFISSUENAME</issueName>
        <issueDescription>TST_TEST_SMFSSUEDESC</issueDescription>
        <primaryExchangeCode>ALL</primaryExchangeCode>
        <assetCurrency>USD</assetCurrency>
      </toAsset>
      <corpActId>754107</corpActId>
      <fromFmvPrice>502.536</fromFmvPrice>
      <toFmvPrice>126.918</toFmvPrice>
      <cpChangeDate>2015-10-05</cpChangeDate>
      <cpNotes>CP_NOTES</cpNotes>
      <batchEventId>TSR80A</batchEventId>
      <costAmtMethod>M</costAmtMethod>
      <consentRt>364.514</consentRt>
      <consentDt>2014-07-22</consentDt>
      <proRation>345.541</proRation>
      <cpCrncy1>USD</cpCrncy1>
      <roundingPoint>170.374</roundingPoint>
      <limitGlIndicator>I</limitGlIndicator>
      <trmtCashGl>I</trmtCashGl>
      <preference>687795</preference>
      <sequence>316684</sequence>
      <matchingFlag>Y</matchingFlag>
      <cpReinvestRatio>723.068</cpReinvestRatio>
      <cxlEventType>CXL_EVYPE</cxlEventType>
      <princLossFactor>446.15</princLossFactor>
      <qdiPercent>0.4</qdiPercent>
      <qdiHldPeriodOvrd>I</qdiHldPeriodOvrd>
      <drdHldPeriodOvrd>I</drdHldPeriodOvrd>
      <qdiDrdOvrdComment>QDI_DRDENT</qdiDrdOvrdComment>
      <qdiPercentEffectiveDate>2015-10-05</qdiPercentEffectiveDate>
      <qdiEligibleFlag>Y</qdiEligibleFlag>
      <caDetailModel>
        <actionPreference>7735</actionPreference>
        <actionSequence>9163</actionSequence>
        <cahdAmount>453.977</cahdAmount>
        <cpTkr>CP_TKR</cpTkr>
      </caDetailModel>
    </genericCA>
  </referenceTransaction>
  <referenceTransaction>
    <header>
      <objectType>CreateTSR</objectType>
    </header>
    <createTSR>
      <objectType>CreateTSR</objectType>
      <JMSDeliveryStream>tst_jms_stream_tsr</JMSDeliveryStream>
    </createTSR>
  </referenceTransaction>
</EagleML>

2. Single EagleML message processing step. This is done via the same core stream eagle_ml-2-0_default_in_xml_reference, but with Splitting parameter set to No. This setting will assure that within Single EagleML message (generated on prior step) objects will go in proper order. However, due to the fact that ProcessInStream element was used in the prior step, different messages will still be processed concurrently. The rsf was modified to be able to execute GenericSMF objects as well. Finally, new CreateTSR object is also processed and a trigger is sent to eagle_ml-2-0_default_cm_task_reporter stream to create a TSR.