Page Comparison

In this page:

Table of Contents

...

Info
This functionality requires Python to be installed. Please refer to Pypy Deployment

Info
Currently available only for ORA environments

For a number of business processes a near-real time processing pattern focused around capturing small changes in the system and packaging them into messages is used. In case of Eagle we have Eagle handles this with different object types , which often need to be processed in require a specific order of processing. This order is not something a thing some of the clients would like to control. So, so Eagle has a generic loader workflow, which implements proper dependencies. That allows clients to package all the different objects and package them into the Generic Load workflow which follows proper dependencies allowing to package various objects into a single message and send to Eagle for processing.

Current Default implementation has an issue of with processing many messages with a small amount of data in them. That It is due to the fact that the solution creates temporary files, and lots of small files present tremendous problem to the can impact file system heavily.

So, to To resolve this the situation, a new approach is suggested and with the main idea is to create of creating no files. In order to achieve this the new process was developed which consist of 2 steps:

...

This is achieved by a two-step process:

1. RTR processor step – System receives RTR message (in most of the cases reads them from queue) with base64 encoded archive data. This rule uses

...

a python utility

...

which is able to receive RTR via stdin and then parse incoming RTR

...

, base64-decode and decompress data in memory and create a single EagleML message

...

which contains all objects

...

sorted in accordance with

...

Generic Loader dependencies. This single EagleML message is then piped into stdout. MC rule is able to consume stdout to get the single EagleML message and

...

to add special new object called CreateTSR

...

with information for TSR creation. Then this message is sent to the next step using ProcessInStream rule element.

Example of RTR is in Appendix 1.

...

Expand

title	Example of RTR (click to expand):

Code Block

language	xml

<EagleML
 xmlns="https://protect-us.mimecast.com/s/9QZ3C9rpDgHkAGpRIBevTv?domain=eagleinvsys.com"
 xmlns:xsi="https://protect-us.mimecast.com/s/_zz8C0R20LHGQ1LJsBl0uB?domain=w3.org"
 eaglemlVersion="2-0"
 eaglemlType="RunTaskRequest"
 xsi:schemaLocation="https://protect-us.mimecast.com/s/9QZ3C9rpDgHkAGpRIBevTv?domain=eagleinvsys.com eagleml-main-2-0.xsd"
 xsi:type="RunTaskRequest">
	<header>
		<messageId>SMF_331624_71454390-9726-41d2-95f6-b4c16e125cf0</messageId>
		<creationTimestamp>2018-01-24T18:10:44.483-06:00</creationTimestamp>
	</header>
	<taskIdentifier>
		<correlationId>LOAD_TEST_3</correlationId>
		<businessTaskId>Generic_Loader_Bundle</businessTaskId>
	</taskIdentifier>
	<taskTypeEnum>LOAD</taskTypeEnum>
	<taskParameters>
		<taskParameter>
			<name>StreamName</name>
			<dataType>S</dataType>
			<value>eagle_ml-2-0_default_cm_execwkfl</value>
		</taskParameter>
		<taskParameter>
			<name>ActionType</name>
			<dataType>S</dataType>
			<value>EXECUTE</value>
		</taskParameter>
        <taskParameter> 
			<name>EnableACKSwitch</name>
			<dataType>S</dataType>
			<value>N</value>
		</taskParameter>
        <taskParameter> 
			<name>DeliveryMethod</name>
			<dataType>S</dataType>
			<value>JMS</value>
		</taskParameter>
		<taskParameter> 
			<name>ReplyDeliveryMethod</name>
			<dataType>S</dataType>
			<value>JMS</value>
		</taskParameter>
		<taskParameter> 
			<name>ACKDeliveryMethod</name>
			<dataType>S</dataType>
			<value>JMS</value>
		</taskParameter>
		<taskParameter> 
			<name>JMSReplyDeliveryStream</name>
			<dataType>S</dataType>
			<value>tst_jms_stream_tsr</value>
		</taskParameter>
		<taskParameter>
			<name>JMSAckDeliveryStream</name>
			<dataType>S</dataType>
			<value>tst_jms_stream_tsr</value>
		</taskParameter>
		<taskParameter>
			<name>WorkflowName</name>
			<dataType>S</dataType>
			<value>eagle_wrf_generic_load</value>
		</taskParameter>
		<taskParameter>
			<name>CreateConsolidatedTSR</name>
			<dataType>S</dataType>
			<value>Y</value>
		</taskParameter>
		<taskParameter>
			<name>ConsolidatedTSROnly</name>
			<dataType>S</dataType>
			<value>Y</value>
		</taskParameter>
		<taskParameter>
			<name>WorkflowData</name>
			<dataType>S</dataType>
			<value>TST_EagleML.zip.base64</value>
		</taskParameter>
		<taskParameter>
			<name>Data</name>
			<dataType>S</dataType>
			<value>UEsDBBQAAAAIAFVgiUxelmrmhwQAAIoMAAAQAAAAX0dFTkVSSUNDQV8xLnhtbKVWW4/SQBR+1sT/
sNl3eqG0hU1tUtvuSgTEFo0+bWo7C6O94Exh2X/vmbYz04JGXRNCmO985zLnxjhhss3RcnF1KvKS
vr7e1fX+RlUfHx8VxCS4PNInqqRVoY41XVc7+mg80q5bnZsTxQO9R0OpyBbomq5+Xi7idIeKZIRL
Widliq6vGrtF/gkRiqvy9TWYEuDmaY9eX0foAREE5A1JSpqkNfCWiNJkC+rg7oY2NhdVmjDRP0XN
PY2KBJcMUE40a63Wf3Tuvnr5wiG/IDDBC2eHkgwR9hsO1ddvKK3Zhdw7VCKCU99zVIm2KmpPx9ly
3r+Y4LR51idxbEAJEE0J3jcBR+FtGIUrPwTuubBTOuyzpEYbXCAoXbF3IZXmSJuOxvpGH99oGnxG
mg7fjnpG7QzQ6kBStEoK5Ibe3SK8WnvMn4QHjuIGdpe+FyznK7ApUU5EDw8QKT6iACQs</value>
		</taskParameter>
	</taskParameters>
</EagleML>

Rule File used is eagle_ml-2-0_cm\in\xml\realtime_rtr.xml is in Appendix 2.

Expand

title	Example of single EagleML message , which also contains with a CreateTSR object (click to expand):

Code Block

language	xml

<EagleML eaglemlVersion="2-0"  xmlns="http://www.eagleinvsys.com/2011/EagleML-2-0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:SchemaLocation="http://www.eagleinvsys.com/2011/EagleML-2-0 eagleml-ext-2-0.xsd" xsi:type="ReferenceTransactionMessage">
  <referenceTransaction>
    <header>
      <objectType>IssuerOrganization</objectType>
    </header>
    <issuerOrganization>
      <objectType>IssuerOrganization</objectType>
      <objectId>IssuerOrganization</objectId>
      <objectDescription>REFERENCE</objectDescription>
      <updateTimestamp>2015-08-21T12:00:00-01:00</updateTimestamp>
      <updateDate>2015-08-21</updateDate>
      <sourceName>EAGLE PACE</sourceName>
      <updateSource>MCADMIN</updateSource>
      <effectiveDate>2015-10-05</effectiveDate>
      <issuerId>TST_TEST_ISSUER_01</issuerId>
      <issuerName>TST_TEST_ISSUER_01</issuerName>
      <address>1 WELLS</address>
      <city>NEWTON</city>
      <state>MA</state>
      <countryOfRiskCode>US</countryOfRiskCode>
      <countryOfIncorporationCode>US</countryOfIncorporationCode>
      <currency>USD</currency>
      <industryCode>TECHNOLOGY</industryCode>
      <countryExposure>US</countryExposure>
    </issuerOrganization>
  </referenceTransaction>
  <referenceTransaction>
    <header>
      <objectType>GenericSMF</objectType>
    </header>
    <genericSMF>
      <objectType>GenericSMF</objectType>
      <objectId>SMF</objectId>
      <objectDescription>SMF</objectDescription> 
      <sourceName>EAGLE PACE</sourceName>
      <updateSource>MCADMIN</updateSource>
      <effectiveDate>2016-06-14</effectiveDate>
      <primaryAssetId>TST_TEST_SMF</primaryAssetId>
      <primaryAssetType>INTERNAL</primaryAssetType>
      <issueName>TST_TEST_SMFISSUENAME</issueName>
      <issueDescription>TST_TEST_SMFSSUEDESC</issueDescription>
      <primaryExchangeCode>ALL</primaryExchangeCode>
      <assetCurrency>USD</assetCurrency>
      <investmentType>FI</investmentType>
      <processingSecurityType>DBIBTL</processingSecurityType>
      <securityType>CS</securityType>
      <securitySubType>COMMON STCK</securitySubType>
      <altSecurityMinorType>CASH</altSecurityMinorType>
      <settlementCurrency>EUR</settlementCurrency>
      <incomeCurrency>USD</incomeCurrency>
      <quantityType>PAR</quantityType>
      <priceMultiplier>103.103</priceMultiplier>
      <quantityScale>104.104</quantityScale>
      <issueCountryCode>US</issueCountryCode>
      <countryOfRiskCode>US</countryOfRiskCode>
      <issueTaxType>EQ</issueTaxType>
    </genericSMF>
  </referenceTransaction>
  <referenceTransaction>
    <header>
      <objectType>GenericCA</objectType>
    </header>
    <genericCA>
      <objectType>GenericCA</objectType>
      <objectId>GenericCA</objectId>
      <objectDescription>REFERENCE</objectDescription>
      <updateTimestamp>2015-08-21T12:00:00-01:00</updateTimestamp>
      <sourceName>EAGLE PACE</sourceName>
      <updateSource>MCADMIN</updateSource>
      <effectiveDate>2016-10-05</effectiveDate>
      <primaryAssetId>TST_TEST_SMF</primaryAssetId>
      <primaryAssetType>INTERNAL</primaryAssetType>
      <issueName>TST_TEST_SMFISSUENAME</issueName>
      <issueDescription>TST_TEST_SMFSSUEDESC</issueDescription>
      <primaryExchangeCode>ALL</primaryExchangeCode>
      <assetCurrency>USD</assetCurrency>
      <clientSpecificCAId>1</clientSpecificCAId>
      <transType>CASHDIV</transType>
      <sweepDate>2015-10-05</sweepDate>
      <exDate>2015-10-05</exDate>
      <cpStatus>PD</cpStatus>
      <cpReason>Cash Dividend</cpReason>
      <cpInstance>6259</cpInstance>
      <splitRatio>333.592</splitRatio>
      <prevSecId>TSTR820A</prevSecId>
      <divAmt>833.412</divAmt>
      <cpAnnounceDate>2015-10-05</cpAnnounceDate>
      <cpAcquiredBy>CP_ACRED_BY</cpAcquiredBy>
      <cpAddedAmt>768.954</cpAddedAmt>
      <cpAddress>CP_ADDRESS</cpAddress>
      <cpAgent>CP_AGENT</cpAgent>
      <toAsset>
        <primaryAssetId>TST_TEST_SMF</primaryAssetId>
        <primaryAssetType>INTERNAL</primaryAssetType>
        <issueName>TST_TEST_SMFISSUENAME</issueName>
        <issueDescription>TST_TEST_SMFSSUEDESC</issueDescription>
        <primaryExchangeCode>ALL</primaryExchangeCode>
        <assetCurrency>USD</assetCurrency>
      </toAsset>
      <corpActId>754107</corpActId>
      <fromFmvPrice>502.536</fromFmvPrice>
      <toFmvPrice>126.918</toFmvPrice>
      <cpChangeDate>2015-10-05</cpChangeDate>
      <cpNotes>CP_NOTES</cpNotes>
      <batchEventId>TSR80A</batchEventId>
      <costAmtMethod>M</costAmtMethod>
      <consentRt>364.514</consentRt>
      <consentDt>2014-07-22</consentDt>
      <proRation>345.541</proRation>
      <cpCrncy1>USD</cpCrncy1>
      <roundingPoint>170.374</roundingPoint>
      <limitGlIndicator>I</limitGlIndicator>
      <trmtCashGl>I</trmtCashGl>
      <preference>687795</preference>
      <sequence>316684</sequence>
      <matchingFlag>Y</matchingFlag>
      <cpReinvestRatio>723.068</cpReinvestRatio>
      <cxlEventType>CXL_EVYPE</cxlEventType>
      <princLossFactor>446.15</princLossFactor>
      <qdiPercent>0.4</qdiPercent>
      <qdiHldPeriodOvrd>I</qdiHldPeriodOvrd>
      <drdHldPeriodOvrd>I</drdHldPeriodOvrd>
      <qdiDrdOvrdComment>QDI_DRDENT</qdiDrdOvrdComment>
      <qdiPercentEffectiveDate>2015-10-05</qdiPercentEffectiveDate>
      <qdiEligibleFlag>Y</qdiEligibleFlag>
      <caDetailModel>
        <actionPreference>7735</actionPreference>
        <actionSequence>9163</actionSequence>
        <cahdAmount>453.977</cahdAmount>
        <cpTkr>CP_TKR</cpTkr>
      </caDetailModel>
    </genericCA>
  </referenceTransaction>
  <referenceTransaction>
    <header>
      <objectType>CreateTSR</objectType>
    </header>
    <createTSR>
      <objectType>CreateTSR</objectType>
      <JMSDeliveryStream>tst_jms_stream_tsr</JMSDeliveryStream>
    </createTSR>
  </referenceTransaction>
</EagleML>

2 step is to process . Single EagleML message processing step. This is done via the same core stream ‘eagleeagle_ml-2-0_default_in_xml_reference’reference, but with Splitting parameter set to No. This setting will assure that within Single EagleML message (generated on prior step) objects will go in proper order. However, due to the fact that ProcessInStream element was used in the prior step the , different messages will still be processed concurrently. The rsf was modified to also be able to execute GenericSMF objects as well. Finally, new CreateTSR object is also possible to process processed and rsf has a reference to correspondent rule. Currently the trigger is sent in Create TSR to the ‘eagle_ml-2-0_default_cm_task_reporter’ stream.

Implementation details

NOTE: For now developed functionality is supported only in ORACLE environments.

In rule for processing incoming RTR it is necessary introduce:

Check for correlation id uniqueness
Check for correlation id length limit
Parse task parameterss
Creating orch state
Send Ack (if switched on) , generate ACK, sent to configured delivery method (if NACK Update Proc Status)
Introduce task parameter for turn on/off ACK – EnableACKSwitch. In case EnableACKSwitch is Y or omitted then send ACK, in case EnableACKSwitch = N – do not send ACK.

Code Block

language	xml

<taskParameter> 
	<name>EnableACKSwitch</name>
	<dataType>S</dataType>
	<value>Y</value>
</taskParameter>

Ref Inbound Stream (No Split)

In rules for load (Ref Inbound Stream No Split) it is necessary introduce:

Add to mapping for Correlation Id alias (reference_common_header_aliases.inc)
Collect transaction id value for each record into orch state. The transaction_id field was used as it is an index field of msgcenter_dbo.msg_detail table.

New include should be created for get_w_state for current correlation id, and updating state with update list of transaction ids. The new include will be called from set_transaction_id_content.inc

In rules for eagle_ml-2-0_default_cm_task_reporterit is necessary introduce:

...

task_reporter

...

IssuePcPrice

In memory rule has been updated and new usage case has been added, currently it works only for IssuePcPrice with one referenceTransaction in the file. If the incoming file has more than one IssuePcPrice record or another objectType, it will be resend to the default eagle_ml-2-0_default_cm_control_message stream.

This behavior can be overridden by option:

W_RESEND_TO_REQUEST_STREAM

If file has 1 IssuePcPrice transaction it will be sent to the stream: eagle_ml-2-0_default_in_xml_reference as it happens usually, but right after load TSR message will be generated and delivered in the PcPrice rule (xml-ref_issuepcprice.xml). As a result you will not see any runs of eagle_ml-2-0_default_cm_task_reporter. stream to create a TSR.

Versions Compared

Old Version 1

New Version Current

Key

Implementation details

IssuePcPrice