Pattern #1 - Loading Many Individual Small Data Messages (non-compressed, non-encoded, order doesn't matter)

Pattern 1 allows loading of many individual small data messages (non-compressed, non-encoded, order doesn't matter).

Note

W_USE_PARALLEL_LOAD global variable flag doesn't matter in this case.

Node <synchronousExecution> does matter. The default value of <synchronousExecution> is 'no'.

Loading thru FILE, FTP, JMS Data Delivery Channels

In this common scenario, there are large volumes of individual non-compressed, non-encoded small data messages which are needed to load. Each of these messages is valid by XSD EagleML data file (for instance: SMF, Reference, Accounting transactions, etc) with correlation ID. Order of loading these files doesn't matter. Message delivery channel is FILE (FileSystem), FTP (SFTP) or JMS (MQ).

Here is example of Reference Transaction Message with correlation ID:

<EagleML eaglemlVersion="2-0" xsi:schemaLocation="http://www.eagleinvsys.com/2011/EagleML-2-0 eagleml-main-2-0.xsd" xsi:type="ReferenceTransactionMessage"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.eagleinvsys.com/2011/EagleML-2-0">
    <referenceTransaction>
    <header>
        <objectType>IssueFxRate</objectType>
    </header>
    <taskIdentifier>
        <correlationId>IssueFxRate_sync_sample_01</correlationId>
        <businessTaskId>Test send message</businessTaskId>
    </taskIdentifier>
    <synchronousExecution>yes</synchronousExecution>
    <issueFxRate>
        <objectType>IssueFxRate</objectType>
        <objectId>IssueFxRate</objectId>
        <sourceName>EAGLE PACE</sourceName>
        <updateSource>MCADMIN</updateSource>
        <effectiveDate>2015-01-15</effectiveDate>
        <fromCurrency>
            <assetCurrency>USD</assetCurrency>
        </fromCurrency>
        <toCurrency>
            <assetCurrency>EUR</assetCurrency>
        </toCurrency>
        <exchangeRate>636.978</exchangeRate>
        <offerrate>479.072</offerrate>
        <originalSource>ORIGINAL_SOURCE</originalSource>
        <for1MnthRate>809</for1MnthRate>
        <for2MnthRate>668.875</for2MnthRate>
        <for3MnthRate>594.985</for3MnthRate>
        <for4MnthRate>734.659</for4MnthRate>
        <for5MnthRate>669.5</for5MnthRate>
        <for6MnthRate>633.966</for6MnthRate>
        <for7MnthRate>576.249</for7MnthRate>
        <for8MnthRate>827.639</for8MnthRate>
        <for9MnthRate>963.893</for9MnthRate>
        <for10MnthRate>539.896</for10MnthRate>
        <for11MnthRate>566.699</for11MnthRate>
        <for12MnthRate>527.102</for12MnthRate>
        <for13MnthRate>734.047</for13MnthRate>
        <for14MnthRate>488.872</for14MnthRate>
        <for15MnthRate>639.225</for15MnthRate>
        <for16MnthRate>375.258</for16MnthRate>
        <for17MnthRate>281.898</for17MnthRate>
        <for18MnthRate>291.081</for18MnthRate>
        <for19MnthRate>142.987</for19MnthRate>
        <for20MnthRate>435.818</for20MnthRate>
        <for21MnthRate>627.04</for21MnthRate>
        <for22MnthRate>482.984</for22MnthRate>
        <for23MnthRate>182.35</for23MnthRate>
        <for24MnthRate>541.054</for24MnthRate>
        <bidRate>549.359</bidRate>
        <fxRatesFactors>FX_RATES_FACTORS</fxRatesFactors>
        <for7dayRate>894.144</for7dayRate>
        <UDF1>200.908</UDF1>
        <UDF2>247.516</UDF2>
        <onDepositRate>315.607</onDepositRate>
        <tnDepositRate>737.886</tnDepositRate>
    </issueFxRate>
    </referenceTransaction>
</EagleML>

Important Note

Be sure that the xsi:type attribute matches the objectType of this message. Also, the correlation ID should be unique for each message and synchronousExecution node value should be "yes".

The messages should be sent to eagle_ml-2-0_default_cm_control_message stream. In this case to increase performance and payload EJM core routes the incoming data message directly to loading stream.

You will see the following in the EJM Monitoring Tool for each incoming data file. For example:
one of IssueFxRate data messages with correlation ID is sent to eagle_ml-2-0_default_in_xml_reference loading stream directly:

If we send 3 different EagleML types messages (for example: GenericSMF, IssueFxRate, AccountingTrade) at the same time to eagle_ml-2-0_default_cm_control_message stream, we can see in the EJM Monitoring Tool the following:

Each of these messages is sent to corresponding loading stream directly.

Examples of these SMF, IssueFxRate, AccountingTrade messages (click to expand)

Example of SMF:
<EagleML xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="ReferenceTransactionMessage" eaglemlVersion="2-0" eaglemlType="ReferenceTransactionMessage" actualBuild="1" xmlns="http://www.eagleinvsys.com/2011/EagleML-2-0">
    <header>
        <messageId messageIdScheme="http://www.eagleinvsys.com/coding-scheme/message-id-2-0">d123b0-7ec2-4eae-8990-42508d167414</messageId>
        <sentBy>http://www.client.com</sentBy>
        <sendTo>http://www.eagleinvsys.com/eagle_ml-2-0_default_cm_control_message</sendTo>
        <replyTo>http://www.client.com/Eagle Control Messages</replyTo>
        <creationTimestamp>2016-10-24T14:44:04.3675929-05:00</creationTimestamp>
    </header>
    <taskIdentifier>
        <correlationId>gaSMF_sync_p1_de2</correlationId>
        <businessTaskId>Test send message</businessTaskId>
    </taskIdentifier>
    <synchronousExecution>yes</synchronousExecution>
    <referenceTransaction>
        <header>
            <objectType>GenericSMF</objectType>
        </header>
        <genericSMF>
            <objectType>GenericSMF</objectType>
            <objectId>GenericSMF</objectId>
            <sourceName>EAGLE PACE</sourceName>
            <updateSource>MCADMIN</updateSource>
            <effectiveDate>2014-09-30</effectiveDate>
            <noLoadValidationFlag>N</noLoadValidationFlag>
            <accountingValidationFlag>N</accountingValidationFlag>
            <primaryAssetId>TSTSMFCUSIP_003</primaryAssetId>
            <primaryAssetType>CUSIP</primaryAssetType>
            <issueName>TSTSMFISSUENAME_003</issueName>
            <unicodeIssueName>TSTSMFUNIISSUENAME_003</unicodeIssueName>
            <issueDescription>TSTSMFISSUEDESC_003</issueDescription>
            <bloombergId>TSTSMFBLOOMBERGID_003</bloombergId>
            <internalId>TSTSMFINTERNAL_003</internalId>
            <xrefIdentifiers>
                <xId1>
                    <xId>TSTSMFXREFID_003</xId>
                    <xIdType>TSTSMFXID1</xIdType>
                </xId1>
            </xrefIdentifiers>
            <primaryExchange>US</primaryExchange>
            <primaryExchangeCode>US</primaryExchangeCode>
            <assetCurrency>USD</assetCurrency>
            <assetResolutionOption>MATCH_ALL</assetResolutionOption>
            <battingOrder>cusip,internalId,bloombergId,xId1</battingOrder>
            <investmentType>CASH</investmentType>
            <processingSecurityType>EQCSCS</processingSecurityType>
            <incomeCurrency>USD</incomeCurrency>
            <issueCountry>CND</issueCountry>
            <settlementCurrency>USD</settlementCurrency>
            <priceMultiplier>1.0</priceMultiplier>
            <quantityScale>1.0</quantityScale>
            <releaseStatus>1</releaseStatus>
            <maturityDate>2015-06-30</maturityDate>
        </genericSMF>
    </referenceTransaction> 
</EagleML>

Example of IssueFxRate:
<EagleML xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="ReferenceTransactionMessage" eaglemlVersion="2-0" eaglemlType="ReferenceTransactionMessage" actualBuild="1" xmlns="http://www.eagleinvsys.com/2011/EagleML-2-0">
    <header>
        <messageId messageIdScheme="http://www.eagleinvsys.com/coding-scheme/message-id-2-0">d123b0-7ec2-4eae-8990-42508d167414</messageId>
        <sentBy>http://www.client.com</sentBy>
        <sendTo>http://www.eagleinvsys.com/eagle_ml-2-0_default_cm_control_message</sendTo>
        <replyTo>http://www.client.com/Eagle Control Messages</replyTo>
        <creationTimestamp>2016-10-24T14:44:04.3675929-05:00</creationTimestamp>
    </header>
    <taskIdentifier>
        <correlationId>gaIssueFxRate_sync_p1_de7</correlationId>
        <businessTaskId>Test send message</businessTaskId>
    </taskIdentifier>
    <synchronousExecution>yes</synchronousExecution>
    <referenceTransaction>
        <header>
            <correlationId>gaIssue_test_001</correlationId>
            <objectType>IssueFxRate</objectType>
        </header>
        <issueFxRate>
            <objectDescription>REFERENCE</objectDescription>
            <objectType>IssueFxRate</objectType>
            <objectId>IssueFxRate</objectId>
            <updateTimestamp>2015-08-21T12:00:00-01:00</updateTimestamp>
            <sourceName>EAGLE PACE</sourceName>
            <updateSource>MCADMIN</updateSource>
            <effectiveDate>2015-10-05</effectiveDate>
            <fromCurrency>
                <assetCurrency>USD</assetCurrency>
                <issueName>US Dollar</issueName>
                <primaryAssetType>CASH</primaryAssetType>
                <primaryAssetId>CASHUSD</primaryAssetId>
                <issueDescription>US Dollar</issueDescription>
                <securityAlias/>
            </fromCurrency>
            <toCurrency>
                <assetCurrency>EUR</assetCurrency>
                <issueName>EURO</issueName>
                <primaryAssetType>CASH</primaryAssetType>
                <primaryAssetId>CASHEUR</primaryAssetId>
                <issueDescription>Euro</issueDescription>
                <securityAlias/>
            </toCurrency>
            <exchangeRate>636.978</exchangeRate>
            <offerrate>479.072</offerrate>
            <originalSource>Eagle Pace</originalSource>
            <for1MnthRate>809.1</for1MnthRate>
            <for2MnthRate>668.875</for2MnthRate>
            <for3MnthRate>594.985</for3MnthRate>
            <for4MnthRate>734.659</for4MnthRate>
            <for5MnthRate>669.5</for5MnthRate>
            <for6MnthRate>633.966</for6MnthRate>
            <for7MnthRate>576.249</for7MnthRate>
            <for8MnthRate>827.639</for8MnthRate>
            <for9MnthRate>963.893</for9MnthRate>
            <for10MnthRate>539.896</for10MnthRate>
            <for11MnthRate>566.699</for11MnthRate>
            <for12MnthRate>527.102</for12MnthRate>
            <for13MnthRate>734.047</for13MnthRate>
            <for14MnthRate>488.872</for14MnthRate>
            <for15MnthRate>639.225</for15MnthRate>
            <for16MnthRate>375.258</for16MnthRate>
            <for17MnthRate>281.898</for17MnthRate>
            <for18MnthRate>291.081</for18MnthRate>
            <for19MnthRate>142.987</for19MnthRate>
            <for20MnthRate>435.818</for20MnthRate>
            <for21MnthRate>627.04</for21MnthRate>
            <for22MnthRate>482.984</for22MnthRate>
            <for23MnthRate>182.35</for23MnthRate>
            <for24MnthRate>541.054</for24MnthRate>
            <bidRate>549.359</bidRate>
            <fxRatesFactors>FX_RATES_F</fxRatesFactors>
            <for7dayRate>894.144</for7dayRate>
            <UDF1>200.908</UDF1>
            <UDF2>247.516</UDF2>
            <onDepositRate>315.607</onDepositRate>
            <tnDepositRate>737.886</tnDepositRate>
        </issueFxRate>
    </referenceTransaction>
</EagleML>

Example of AccountingTrade:
<EagleML xmlns="http://www.eagleinvsys.com/2011/EagleML-2-0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" eaglemlVersion="2-0" eaglemlType="AccountingTransactionMessage" xsi:schemaLocation="http://www.eagleinvsys.com/2011/EagleML-2-0 eagleml-main-2-0.xsd" xsi:type="AccountingTransactionMessage">
    <header>
        <messageId messageIdScheme="http://www.eagleinvsys.com/coding-scheme/message-id-2-0">d123b0-7ec2-4eae-8990-42508d167414</messageId>
        <sentBy>http://www.client.com</sentBy>
        <sendTo>http://www.eagleinvsys.com/eagle_ml-2-0_default_cm_control_message</sendTo>
        <replyTo>http://www.client.com/Eagle Control Messages</replyTo>
        <creationTimestamp>2016-10-24T14:44:04.3675929-05:00</creationTimestamp>
    </header>
    <taskIdentifier>
        <correlationId>gaAcctTrade_sync_p1_de7</correlationId>
        <businessTaskId>Test send message</businessTaskId>
    </taskIdentifier>
    <synchronousExecution>yes</synchronousExecution>
    <accountingTransaction>
        <header>
            <identifier>
                <transactionId>DEQOPENFRKIENG23</transactionId>
            </identifier>
            <objectType>AccountingTrade</objectType>
        </header>
        <accountingTrade>
            <buySellIndicator>BUY</buySellIndicator>
            <xrefAccountId>ITEMLXREFTST</xrefAccountId>
            <xrefAccountIdType>PORTM</xrefAccountIdType>
            <cusip>546475PS8</cusip>
            <isin>US546475PS74</isin>
            <bloombergId>FF000ED6110</bloombergId>
            <assetResolutionOption>ALL</assetResolutionOption>
            <tradeDate>2014-07-04</tradeDate>
            <settlementDate>2014-07-06</settlementDate>
            <sharesParQuantity>4700.00</sharesParQuantity>
            <price>93.12</price>
            <taxAmount>10</taxAmount>
            <impliedCommissionIndicator>Y</impliedCommissionIndicator>
            <commission>663.968</commission>
            <secFee>875.645</secFee>
            <stampDutyTax>9.99</stampDutyTax>
            <localToBaseFXRate>335.617</localToBaseFXRate>
            <otherFee>10.1</otherFee>
            <autoSettleIndicator>YES</autoSettleIndicator>
            <custodianBankCode>1</custodianBankCode>
            <custodianBankName>CUSTODIAN_BANK_NAME</custodianBankName>
            <custodyBankAccount>11</custodyBankAccount>
            <custodyBankSubaccount>CUSTODY_BANK_SUBACCOUNT</custodyBankSubaccount>
            <cashAccountName>CASH</cashAccountName>
            <cashAccountNumber>1</cashAccountNumber>
            <exchangeCode>236</exchangeCode>
            <brokerCode>1</brokerCode>
            <brokerName>1</brokerName>
            <clearingBrokerCode>BONY</clearingBrokerCode>
            <clearingBrokerName>BANK OF NY</clearingBrokerName>
            <settleLocation>SETTLE_LOCATION</settleLocation>
            <settleInstructions1>SETTLEINSTRUCTION</settleInstructions1>
            <settleInstructions2>SETTLE_INSTRUCTIONS2</settleInstructions2>
            <settleInstructions3>SETTLE_INSTRUCTIONS3</settleInstructions3>
            <settleInstructions4>SETTLE_INSTRUCTIONS4</settleInstructions4>
            <transactionNotes>Other</transactionNotes>
            <originalTradeTicketNumber>544828</originalTradeTicketNumber>
            <orderNumber>157.096</orderNumber>
            <userDefinedPriority>441061</userDefinedPriority>
            <fas115Code>Trading</fas115Code>
            <regulatoryIntent>HFT</regulatoryIntent>
            <holdingTerm>CE</holdingTerm>
            <calculateLocalAmountsFlag>Y</calculateLocalAmountsFlag>
            <calculateBaseAmountsFlag>Y</calculateBaseAmountsFlag>
        </accountingTrade>
    </accountingTransaction>
</EagleML>

Loading Through WS Delivery Channel

In this common scenario, there are large volumes of individual non-compressed, non-encoded small data messages which are needed to load. Each of these messages is valid by XSD EagleML data file (for instance: SMF, Reference, Accounting transactions, etc) with correlation ID. The order of loading these files doesn't matter. Message delivery channel is WS.

You need to wrap the EagleML message into the SOAP envelope.

Here is example of Reference Transaction Message with correlation ID wrapped into the SOAP envelope:

<soap:Envelope xmlns:addr="http://www.w3.org/2005/08/addressing" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
    <soap:Body>
        <runTaskRequest xmlns="http://www.eagleinvsys.com/2011/wsdl/EagleML-2-0">
            <EagleML xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="ReferenceTransactionMessage" eaglemlVersion="2-0" eaglemlType="ReferenceTransactionMessage" actualBuild="1" xmlns="http://www.eagleinvsys.com/2011/EagleML-2-0">
                <header>
                    <messageId messageIdScheme="http://www.eagleinvsys.com/coding-scheme/message-id-2-0">d123b0-7ec2-4eae-8990-42508d167414</messageId>
                    <sentBy>http://www.client.com</sentBy>
                    <sendTo>http://www.eagleinvsys.com/eagle_ml-2-0_default_cm_control_message</sendTo>
                    <replyTo>http://www.client.com/Eagle Control Messages</replyTo>
                    <creationTimestamp>2016-10-24T14:44:04.3675929-05:00</creationTimestamp>
                </header>
                <taskIdentifier>
                    <correlationId>gaIssueFxRate_sync_p1_de12</correlationId>
                    <businessTaskId>Test send message</businessTaskId>
                </taskIdentifier>
                <synchronousExecution>yes</synchronousExecution>
                <referenceTransaction>
                    <header>
                        <objectType>IssueFxRate</objectType>
                    </header>
                    <issueFxRate>
                        <objectDescription>REFERENCE</objectDescription>
                        <objectType>IssueFxRate</objectType>
                        <objectId>IssueFxRate</objectId>
                        <updateTimestamp>2015-08-21T12:00:00-01:00</updateTimestamp>
                        <sourceName>EAGLE PACE</sourceName>
                        <updateSource>MCADMIN</updateSource>
                        <effectiveDate>2015-10-05</effectiveDate>
                        <fromCurrency>
                            <assetCurrency>USD</assetCurrency>
                            <issueName>US Dollar</issueName>
                            <primaryAssetType>CASH</primaryAssetType>
                            <primaryAssetId>CASHUSD</primaryAssetId>
                            <issueDescription>US Dollar</issueDescription>
                            <securityAlias/>
                        </fromCurrency>
                        <toCurrency>
                            <assetCurrency>EUR</assetCurrency>
                            <issueName>EURO</issueName>
                            <primaryAssetType>CASH</primaryAssetType>
                            <primaryAssetId>CASHEUR</primaryAssetId>
                            <issueDescription>Euro</issueDescription>
                            <securityAlias/>
                        </toCurrency>
                        <exchangeRate>636.978</exchangeRate>
                        <offerrate>479.072</offerrate>
                        <originalSource>Eagle Pace</originalSource>
                        <for1MnthRate>809.1</for1MnthRate>
                        <for2MnthRate>668.875</for2MnthRate>
                        <for3MnthRate>594.985</for3MnthRate>
                        <for4MnthRate>734.659</for4MnthRate>
                        <for5MnthRate>669.5</for5MnthRate>
                        <for6MnthRate>633.966</for6MnthRate>
                        <for7MnthRate>576.249</for7MnthRate>
                        <for8MnthRate>827.639</for8MnthRate>
                        <for9MnthRate>963.893</for9MnthRate>
                        <for10MnthRate>539.896</for10MnthRate>
                        <for11MnthRate>566.699</for11MnthRate>
                        <for12MnthRate>527.102</for12MnthRate>
                        <for13MnthRate>734.047</for13MnthRate>
                        <for14MnthRate>488.872</for14MnthRate>
                        <for15MnthRate>639.225</for15MnthRate>
                        <for16MnthRate>375.258</for16MnthRate>
                        <for17MnthRate>281.898</for17MnthRate>
                        <for18MnthRate>291.081</for18MnthRate>
                        <for19MnthRate>142.987</for19MnthRate>
                        <for20MnthRate>435.818</for20MnthRate>
                        <for21MnthRate>627.04</for21MnthRate>
                        <for22MnthRate>482.984</for22MnthRate>
                        <for23MnthRate>182.35</for23MnthRate>
                        <for24MnthRate>541.054</for24MnthRate>
                        <bidRate>549.359</bidRate>
                        <fxRatesFactors>FX_RATES_F</fxRatesFactors>
                        <for7dayRate>894.144</for7dayRate>
                        <UDF1>200.908</UDF1>
                        <UDF2>247.516</UDF2>
                        <onDepositRate>315.607</onDepositRate>
                        <tnDepositRate>737.886</tnDepositRate>
                    </issueFxRate>
                </referenceTransaction>
            </EagleML>
        </runTaskRequest>
    </soap:Body>
</soap:Envelope>

Important Note

The xsi:type attribute must match the objectType of this message. Also , the correlation ID should be unique for each message and synchronousExecution node value should be "yes". The delivery stream name is defined in the node <sendTo>.

The messages should be sent to eagle_ml-2-0_default_cm_control_message stream (<sendTo>http://www.eagleinvsys.com/eagle_ml-2-0_default_cm_control_message</sendTo>) . In this case to increase performance and payload EJM core routes the incoming data message directly to loading stream.

For example, sending IssueFxRate data message using SoapUI:

In the EJM Monitoring Tool:

In this case SoapUI shows the final TSR and message is sent to loading stream directly.