EagleML Release Notes - December 2018

ESP-1016 Credit Risk files were received with future timestamp in Eagle Production

The unpacker was failing with the error: "time stamp ... is ... in the future" if a TAR or TARGZ was sent for processing with a timestamp that was in the future.

There was a change made that stopped the timestamp check in the unpacker. This allowed TAR and TARGZ archives to be processed correctly despite a future timestamp.

Files changed:
wrf_unzip.prf

ESP-1121 Sending messages through AciveMQ, removing ReplyTo element in order to see messages in the queue

The new W_MESSAGING_COMPATIBILITY_REPLY_TO_CHECK parameter was added to allow omission of the ReplyTo and AckTo fields.

If the parameter is set to “Y” the values will not be populated for the two fields.

Files changed:
eagleml_load.xml
realtime_rtr.xml
resolve_inbound_stream_name.inc
parse_url.inc

ESP-1643 Changes to _schd stream-III - Development

Workflow restart logic was not working correctly when there was an active instance of the workflow attempted to be restarted.

The ExcludeCurrentInstance parameter was added to be able to cancel the current instance and then restart the workflow.

Files changed:
execute_wrkfl_event.xml
cancel_workflow.inc

ESP-1836 EagleML IssuerOrganization Interface cannot process correctly historical Issuer data

There was an issue with duplicate records being loaded through the IssuerOrganization interface. If two records with the same issuer id but different effective dates are loaded in parallel they sometimes will both get inserted which creates duplicates.

The load through this interface was changed to be specifically sequential so that no duplicates can be loaded in parallel allowing the edit check to catch all of the duplicates.

Files changed:
xml-ref_objects.spl
xml-ref_objects.rsf

ESP-3004 Update XSD for the WRHSTRADEACTIVITYEXTRACT to apply DB values for missing fields causing IWS Element Reduction issues

The following fields would successfully load through the warehouse trade inbound interface but were not being extracted in the WRHSTRADEEXTRACT: cpSwapRatio, brokerCode, eventIdSwitch.

The database paths were added to the XSD allowing them to appear in extracts.

Files changed:
eagleml-wh-shared-2-0.xsd
eagleml-shared-2-0.xsd

ESP-3007 Entity Extract should contain all Entity references under entityXrefs node

The entity xreferences can now be stored under an entityXrefs node instead of the GenericEntity model.

This functionality is controlled by the EntityXrefsGroupOnly RTR parameter. There is no global variable for this functionality in the w_config files. The default for the parameter is ‘N’ so if the parameter is not present in the RTR then the xreferences will be extracted regularly.

Example of use in an RTR:

<taskParameter>
 <name>EntityXrefsGroupOnly</name>
 <dataType>S</dataType>
 <value>Y</value>
 </taskParameter>

Files changed:

extract_entity_fields.inc
t_entity_common_ids_fields_streaming.inc

ESP-3417 issueTaxType, avoid default T value if incoming value is not NULL

If the issueTaxType field is not on the inbound SMF record and can’t be resolved in the database and the processing security type is one of the following (DBIBMU, DBFBIO, DBFBAL, DBFBCC, DBFBFB) then the issueTaxType is set to “T”.

If the element’s value is present on the incoming file or can be resolved from the database (regardless of processing security type) then one of those values are taken.

File changed:
generic_smf_fields_logic.inc

ESP-3419 Please add "ORDER BY" to BANKCASHSETTLEMENTEXTRACT

The SQL query used in the BANKCASHSETTLEMENTEXTRACT was modified to order the results by the BCS_INSTANCE field.

This change was made to make it easier for the streaming translator to run the transformation since the grouping in this rule is done by that same field.

File changed:
extract_bank_cash_settlement.inc

SDP-9279 Credit Risk Process: add support to process Issuer Organization in XMLDBAPI format

Added support for loading the Issuer Organization EagleML object in XMLBDAPI format. The issuerAlias is resolved by the in_issuer_id and

File example:

<INSERT_ISSUER_ORG>

<MSG_TYPE>MTADMIN</MSG_TYPE>

<SUB_MSG_TYPE>INSERT_ISSUER_ORG</SUB_MSG_TYPE>

<INPUT_PAR>

EAGLE PACE

MCADMIN

20171123

SPD9279_ISR_ORG_1

SPD9279_ISR_ORG_1 NAME

SPD9279_ISR_ORG_1

CUSIP

US

US

N

US

N

LEI 1

LEI 1 name

ACTIVE

LEI 1 form

20140630

20170614

LEI 1 street

LEI 1 city

LEI 1 zip

US

NO

INSERT

</INPUT_PAR>

</REQUEST>

<MSG_TYPE>MTADMIN</MSG_TYPE>

<SUB_MSG_TYPE>INSERT_ISSUER_ORG</SUB_MSG_TYPE>

<INPUT_PAR>

EAGLE PACE

MCADMIN

20171123

SPD9279_ISR_ORG_2

SPD9279_ISR_ORG_2 NAME

SPD9279_ISR_ORG_2

CUSIP

US

US

N

US

N

LEI 2

LEI 2 name

ACTIVE

LEI 2 form

20140520

20150624

LEI 2 street

LEI 2 city

LEI 2 state

LEI 2 zip

US

NO

INSERT

</INPUT_PAR>

</REQUEST>

</EagleDBMLRequest>

</INSERT_ISSUER_ORG>

Files changed:

xml-dbdirect_common.inc

xml-dbdirect_invar.xml

xml-warehouse_v2.xml

Files added:

xml-warehouse_common.inc

SDP-9297 Improve cancel process so that it does not depend on the logs

If the proctasks and proclog logs were disabled for a workflow that workflow could not be cancelled.

The cancel logic was based on reading the logs, but this was changed to a SQL based approach which allowed for discontinuation of the logging functionality.

Files changed:
• cancel_subtasks.inc
• execute_wrkfl_event.xml
• remove_process_wait_states.inc

SDP-9302 (ESP-3256) Update Security Resolution for First Match mode not to use Primary Asset Id if Primary Asset Id type isn't present in assetResolutionOption

The SKIP_PRIMARY_ID option was added to the assetResolutionOption parameter. This allows for skipping the primary identifiers in case the resolution is running in First Match mode.

Sample file with parameter included:

<EagleML xmlns="http://www.eagleinvsys.com/2011/EagleML-2-0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:SchemaLocation="http://www.eagleinvsys.com/2011/EagleML-2-0 eagleml-ext-2-0.xsd" xsi:type="ReferenceTransactionMessage">
<referenceTransaction>
<header>
<objectType>GenericSMF</objectType>
</header>
<genericSMF>
<sourceName>BBPS</sourceName>
<updateSource>BBPS</updateSource>
<effectiveDate>2018-10-25</effectiveDate>
<preserveNull>1</preserveNull>
<primaryAssetId>ESP3265_1</primaryAssetId>
<primaryAssetType>TICKER</primaryAssetType>
<issueName>ESP3265_2 OPTN Dec18C 500</issueName>
<issueDescription>ESP3265_2 OPTN Dec18C</issueDescription>
<processingSecurityType>DBFBFB</processingSecurityType>
<ticker>ESP3265_1</ticker>
<xrefIdentifiers>
<xId1>
<xId>ESP3265_2_BBG</xId>
<xIdType>BBGID</xIdType>
</xId1>
<xId4>
<xId>ESP3265_2_BB</xId>
<xIdType>BBID</xIdType>
</xId4>
<xId8>
<xId>ESP3265_2 500 COMB Comdty</xId>
<xIdType>BBIDFUTOPT</xIdType>
</xId8>
</xrefIdentifiers>
<primaryExchangeCode>XCBT</primaryExchangeCode>
<assetCurrency>USD</assetCurrency>
<assetResolutionOption>BBGCID,BBID,CUSIP,SEDOL,ISIN,BBIDFUTOPT,ASSET_CURRENCY,SKIP_PRIMARY_ID</assetResolutionOption>
<battingOrder>xId2,xId3,xId4,xId5,xId6,xId7,xId8,xId9,xId10,</battingOrder>
<historyOnlyFlag>HISTORY_ONLY</historyOnlyFlag>
<validationProcessFlag>Y</validationProcessFlag>
</genericSMF>
</referenceTransaction>
</EagleML>

Changed files:
security_resolution_generic.inc

SDP-9316 EagleML classic MC - Generic Extractor execution - ENTITYCOMPOSITEEXTRACT appearing as part of the Entity Feed Type and for MC2 as well.

The ENTITYCOMPOSITEEXTRACT had been removed from EagleML but there was still a task to extract this object in the generic extract workflow. This was causing an unknown feed type error to show up on the TSR.

In order to fix this issue the task was removed from the generic extract workflow.

Files changed:
Generic Extract workflow files (ejmwf, sln, wrf)

SDP-9413 Fix issue with missing some records in File Statistic tab for In-memory bundle load

There was an issue with records getting inserted into the MSGCENTER_DBO.MSG_DETAIL table with a delay and causing the incorrect number of entries in the FileStatistics tab.

The re-try logic was changed to increase the number of attempts. In the case that the number of entries in the FileStatistics tab is still lower than the number of records loaded the following error will be generated:

Not all MSG_DETAIL_ID were found for DATA. With CorrId = [value]

The fix was only implemented for In-Memory processing.

The file statistics records will still be generated to show the user something.

Files changed:
realtime_ejm_stat.xml

SDP-9429 - TRADEDCAEXTRACT and SETTLEDCAEXTRACT to include new switch to append the CANCEL value to the STAR event type

New logic for the starEventType was added for the SETTLEDCAEXTRACT and TRADEDCAEXTRACT extracts.

The existing EnhanceCalcTransactionId parameter was used to control the calculation of the value for this field.

In case the parameter is “Y” extra logic is applied when retrieving the field.

If the parameter is set to “N” then the default value is used.

Files changed:
- extract_acct_settled_cash_activity.inc
- extract_acct_traded_cash_activity.inc

SDP-9433- WRHSTRADEACTIVITY, SETTLEDCAEXTRACT and TRADEDCAEXTRACT - add / review calcMasterTransId

New logic for the calcMasterTransactionId was added for the WRHSTRADEEXTACT, SETTLEDCAEXTRACT and TRADEDCAEXTRACT extracts.

The existing EnhanceCalcTransaction parameter was used to control the concatenation of the Entity Id, account basis, security alias, long short indicator and event id and storage of this concatenated value in the above field.

This logic will be applied if the parameter is set to “Y” in the RTR.

If the parameter is set to “N”, the field calcMasterTransactionId will be filled with the default value.

Files changed:
- extract_warehouse_trade.inc
- extract_acct_settled_cash_activity.inc
- extract_acct_traded_cash_activity.inc

SDP-9448 Cash Journal Extract - Usage of Run Panel requires additional field elements to be passed / default logic issues

Two new filters were added to the CASHJOURNALEXTRACT so that the extract can be properly run by the panel.

processCodeTransaction and codeMapName are now being passed with the following default values:

processCodeTranslation = ‘Y’ and
codeMapName = ‘STAR REPORT’.

If processCodeTranslation is ‘N’, then tag 10131 will not be populated and the codeMapName will not be sent to the panel.

To run this extract through the panel the AccountingValidationFlag must either not be present or be set to ‘Y’

File changed
\eagle_ml-2-0_cm\out\tagvalue\extract_acct_cash_journal.inc

SDP-9485 Create a new EagleML Object to expose data from the ESTAR.spd_delta_trigger_log

The new SpdDeltaTriggerLog EagleML object was created to extract data from the ESTAR.SPD_DELTA_TRIGGER_LOG table.

Task parameter value for the extract:
<taskParameter>
<name>FeedType</name>
<dataType>S</dataType>
<value>SPDDELTATRIGGERLOGEXTRACT</value>
</taskParameter>

Requested filters are processed by the interface:
• Entity Filter
• Entity Filter with TraverseHierarchy Parameter
• Security Master File (SMF) Filter
• Date Filter
• Sources Filter
• Common Filter
• Delta Based Filter,

The interface also supports some of its own filters:
• eventId Filter,
• messageSubType Filter,
• updateSource Filter,
• localCurrency Filter,
• accountingBasis Filter.

File changed
• eagle_ml-2-0_cm/w_config.inc
• eagle_ml-2-0_cm/out/tagvalue/extracts.rsf
• eagle_ml-2-0_cm/out/xml/t_eagleml-2-0_streaming.rsf
• eagle_ml-2-0_cm/out/xmlt_spd_delta_trigger_log_streaming.inc
• eagle_ml-2-0_cm/out/xml/t_spd_delta_trigger_log_streaming.xml
• eagle_ml-2-0_cm/out/tagvalue/extract_spd_delta_trigger_log.inc
• eagle_ml-2-0_cm/out/tagvalue/extract_spd_delta_trigger_log.xml

SDP-9492 Increase the number of possible failed records for ConsolidatedTSR - II

There is a 1000 error limit on TSR messages since TSRs are stored in the database and having values that are too large would create an issue with performance.

There are cases where the TSR is used as an incoming message for the following task in a workflow which requires the TSR to be complete.

Support for the AllowAllErrorsTSR RTR parameter and the W_ALLOW_ALL_ERRORS_TSR was added.

Logic for allowing the fill RTR to be sent only to this task was added. In EJM tasks an abridged version will be shown and same with the TSR sent to FTP, MQ, etc.

Using this option will still have some impact on performance since there are additional queries that have to be made to send the full TSR.

Additionally the failure tolerance initially had to be individually queried but this logic was changed since the information is actually contained in the SUMMARY query.

Files changed:
task_reporter.xml
task_reporter_generate_task_unit.inc
task_reporter_gen_step_summary.inc
task_reporter_gen_step_detail.inc

Created a new interface to support loading SCIM 2.0 objects which share logic with the User Group object in EagleML.

File changed
• /eagle_default/in/xml/include/user_group_common.inc
• /eagle_default/in/xml/xml-user_group.xml
• /eagle_default/in/xml/xml-user_group_scim.xml
• /eagle_default/in/xml/xml-user_objects.rsf

SDP-9922 Add True/Rebook and Cancel logic for multipleLegAccountingTrade

Introduced logic to automatically re-book Multi-leg Trade records.

Files changed:

estar\tpe\dynamic\msgcenter\eagle_default\in\include\accttrade_true_rebook.inc
estar\tpe\dynamic\msgcenter\eagle_default\in\xml\xml-accttrade_multipleleg.xml
estar\tpe\dynamic\msgcenter\eagle_default\in\xml\include\multiplelegtrade_aliases.inc

SDP-9926 EagleML: Pass the value from tag4590 into the get_smf_all_xrefs_list for correct SWAPID logic

There was initially no logic that would allow checking for an existing swap type in the database when loading a record.

Logic to check for an existing swap type was added and now if a value is found in the database the load will use that value. If no swap type is found in the database then the value on the incoming file will be used.

File changed:
\dynamic\msgcenter\eagle_default\in\xml\xslt\get_smf_all_xrefs_list.xsl
\dynamic\msgcenter\eagle_default\in\xml\include\get_xrefs_list.inc

SDP-9956 EagleML Populate and Extract Eagle Administration Settings - developing rules

New EagleML inbound and outbound interfaces and a new object in the XSD schema have been created.

XSD EagleML object: EagleAdministrationSettings
<xsd:complexType name="EagleAdministrationSettings">
<xsd:annotation>
<xsd:documentation xml:lang="en">A type describing single User Eagle Administration Settings record.</xsd:documentation>
</xsd:annotation>
<xsd:complexContent>
<xsd:extension base="UserObject">
<xsd:sequence>
<xsd:element name="webSiteSettings" type="WebSiteSettingsModel" minOccurs="0" maxOccurs="1"/>
<xsd:element name="passwordSettings" type="PasswordSettingsModel" minOccurs="0" maxOccurs="1"/>
<xsd:element name="ldapSettings" type="LdapSettingsModel" minOccurs="0" maxOccurs="1"/>
</xsd:sequence>
</xsd:extension>
</xsd:complexContent>
</xsd:complexType>

The new type contains the following User Administration settings displyed in User Administration -> Setup tabs:
- Web Site Setting tab;
- Password Settings tab;
- LDAP Settings tab.

Inbound interface: Incoming message can only update existing settings. New ones cannot be added and existing ones cannot be deleted.
Example of incoming message:
<EagleML xmlns="http://www.eagleinvsys.com/2011/EagleML-2-0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" eaglemlVersion="2-0" eaglemlType="UserAdministrationTransactionMessage" xsi:schemaLocation="http://www.eagleinvsys.com/2011/EagleML-2-0 eagleml-main-2-0.xsd" xsi:type="UserAdministrationTransactionMessage">
<header>
<messageId messageIdScheme="http://www.eagleinvsys.com/coding-scheme/message-id-2-0">TSTR8210A</messageId>
<sentBy>http://www.eagleinvsys.com/</sentBy>
</header>
<userAdministrationTransaction>
<header>
<objectType>EagleAdministrationSettings</objectType>
</header>
<eagleAdministrationSettings>
<objectType>EagleAdministrationSettings</objectType>
<updateSource>MCADMIN</updateSource>
<webSiteSettings>
<closeClientBrowserOnLogOff>N</closeClientBrowserOnLogOff>
<showInaccessibleActions>Y</showInaccessibleActions>
<showUserSessionsForAllUsers>Y</showUserSessionsForAllUsers>
<openMenuInNewWindow>N</openMenuInNewWindow>
<showWebServerInWindowTitle>Y</showWebServerInWindowTitle>
<showServerStatus>Y</showServerStatus>
<maximumUnsuccessfulLogonAttempts>5</maximumUnsuccessfulLogonAttempts>
<idlePeriod>120</idlePeriod>
<minimumUserNameLength>4</minimumUserNameLength>
</webSiteSettings>
<passwordSettings>
<passwordExpiresEvery>60</passwordExpiresEvery>
<lockUserAccountsAfter>30</lockUserAccountsAfter>
<disableUserAccountsAfter>90</disableUserAccountsAfter>
<minimumPasswordLength>4</minimumPasswordLength>
<strongPasswordEnforced>N</strongPasswordEnforced>
<passwordHistory>0</passwordHistory>
<minimumPasswordAge>0</minimumPasswordAge>
</passwordSettings>
<ldapSettings>
<allowImportFromNT>Y</allowImportFromNT>
<allowImportFromLDAP>Y</allowImportFromLDAP>
<ldapConnectionParameters>
<domain>eagleinvsys</domain>
<baseDN>DC=eagleinvsys,DC=com</baseDN>
</ldapConnectionParameters>
</ldapSettings>
</eagleAdministrationSettings>
</userAdministrationTransaction>
</EagleML>

Outbound rule: extract from DB table pace_masterdbo.starsec_group_detail.
Feedtype = EAGLEADMINISTRATIONSETTINGSEXTRACT.
Example of RTR to run the extract:
<EagleML xmlns="http://www.eagleinvsys.com/2011/EagleML-2-0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="RunTaskRequest" eaglemlVersion="2-0" xsi:schemaLocation="http://www.eagleinvsys.com/2011/EagleML-2-0 eagleml-main-2-0.xsd" eaglemlType="RunTaskRequest">
<header>
<messageId>USEREASETTINGSEXTRACT</messageId>
<sentBy>http://www.client.com</sentBy>
<sendTo>http://www.eagleinvsys.com/eagle_ml-2-0_default_cm_control_message</sendTo>
<replyTo>http://www.client.com/Eagle Control Messages</replyTo>
<creationTimestamp>2012-08-02T22:13:32.272-05:00</creationTimestamp>
</header>
<taskIdentifier>
<correlationId correlationIdScheme="correlationIdScheme">asEASETTINGS_281118_01</correlationId>
<businessTaskId correlationIdScheme="businessTaskIdScheme">EAGLEADMINISTRATIONSETTINGSEXTRACT</businessTaskId>
</taskIdentifier>
<taskTypeEnum>NEW</taskTypeEnum>
<taskParameters>
<taskParameter>
<name>ActionType</name>
<dataType>S</dataType>
<value>EXTRACT</value>
</taskParameter>
<taskParameter>
<name>StreamName</name>
<dataType>S</dataType>
<value>eagle_ml-2-0_default_out_q</value>
</taskParameter>
<taskParameter>
<name>FeedType</name>
<dataType>S</dataType>
<value>EAGLEADMINISTRATIONSETTINGSEXTRACT</value>
</taskParameter>
</taskParameters>
</EagleML>

Added files:
\dynamic\msgcenter\eagle_default\in\xml\xml-user_eagle_administration_settings.xml
\dynamic\msgcenter\eagle_ml-2-0_cm\out\tagvalue\extract_user_eagle_administration_settings.xml
\dynamic\msgcenter\eagle_ml-2-0_cm\out\tagvalue\extract_user_eagle_administration_settings.inc
\dynamic\msgcenter\eagle_ml-2-0_cm\out\xml\t_user_eagle_administration_settings_streaming.xml
\dynamic\msgcenter\eagle_ml-2-0_cm\out\xml\t_user_eagle_administration_settings_streaming.inc
Changed files:
\dynamic\msgcenter\eagle_default\in\xml\xml-user_objects.rsf
\dynamic\msgcenter\eagle_ml-2-0_cm\out\tagvalue\extracts.rsf
\dynamic\msgcenter\eagle_ml-2-0_cm\out\xml\t_eagleml-2-0_streaming.rsf

SDP-9994 Update Editcheck after ToAssetId Security Resolution eagle_default/in/xml/xml-ref_ca.xml

A new edit check has been added after the ToAsset resolution in the Accounting CA inbound rule.

If the ToAsset has not been resolved based on security xreference identifiers the record will now fail with the following error:

'Security for the identifier(s) does not exist in Eagle'.

File changed:
\dynamic\msgcenter\eagle_default\in\xml\xml-ref_ca.xml

SDP-10008 True Rebook logic support to acct. trades process (FWD)

Introduced logic to automatically re-book Trade FWD records.

Files changed:

estar\tpe\dynamic\msgcenter\eagle_default\in\include\accttrade_true_rebook.inc
estar\tpe\dynamic\msgcenter\eagle_default\in\xml\xml-accttrade_fxfwd.xml
estar\tpe\dynamic\msgcenter\eagle_default\in\include\csv-fxfwd_columns.inc

SDP-10024 BNY: Update logic for ForceExtractFileCreation task parameter and W_FORCE_EMPTY_EXTRACTFILE config setting

Added support for the ForceExtractFileCreation task parameter in the case the RTR that is sent by the workflow contains it.

Fixed an issue with the ExactExtractFileName parameter from the RTR.

Fixed an issue with the values from the ExactExtractFileName and EXtractNamePatternEx showing up if there was no data returned while ForceExtractFileCreation was set to “N”

Files changed:

estar\tpe\dynamic\msgcenter\eagle_ml-2-0_cm\reporter_dist.inc

estar\tpe\dynamic\msgcenter\eagle_ml-2-0_cm\extract_eofw.inc

SDP-10051 EagleML stream unable to process trade cancel with current date accounting date; receiving error cancel Accounting Date: This accounting date is closed

There was an issue with loading trade cancels through the panel.

Fixed the FORCEADDTAG logic for tag 2283 which was causing the issue with loading through the panel.

Files changed:

estar\tpe\dynamic\msgcenter\eagle_default\in\include\csv-cancel_panels.inc