EagleML Release Notes - December 2021

Enhancements

SDP-51688 Revised the Receive and Deliver Inbound messages to include additional tags and resolve date formatting issues

For the Accounting Trade Receive and Deliver EagleML load messages, a series of elements were added.

For Receives, the following elements were added:

currentILBQuantity - Tag 965
tradeYield - Tag 9430
commission - Tag 47
secFee - Tag 48
taxAmount- Tag 46
stampDutyTax - Tag 51
otherFee - Tag 3752
clearingBrokerCode - Tag 1237

For Delivers:

clearingBrokerCode - Tag 1237

For both the Receive and Deliver Inbound messages, the ability to pass in a date with a “-” (i.e. YYYY-MM-DD) into the originalSettlementDate - Tag 154; holdingPeriodDate - Tag 164 and the originalAcquisitionDate - Tag 216 was revised as it previously required the user to always send the YYYYMMDD format only.

Files changed:
eagle_default/in/include/csv-deliver.inc
eagle_default/in/include/csv-receive.inc

SDP-53594 Send alert timeouts functionality in IWS has been revised to properly send alerts

The send_alerts feature in IWS utilized by both PACE events and STAR events was updated for when the AlertTimeout is reached. This is accomplished by using the default eagle_ml-2-0_default_cm_alerts_distributor stream for the alerts distribution.

Files changed:
eagle_ml-2-0_cm/send_alert.inc
eagle_ml-2-0_cm/w_config.inc

SDP-56998 IssuerXrefIdentifiers returned in the Issuer Organization element extract

The issuerOrganization element extract from the Reference Object Data Maps was modified to return the issuerXrefIdentifiers.

Files changed:
eagle_ml-2-0_cm/out/xml/t_issuer_org_streaming.inc
eagle_ml-2-0_cm/out/tagvalue/extract_issuer_org.inc

SDP-50779 The genericEntity inbound interface supports the marginPriceSource element when the Accounting Validation flag is enabled.

The genericEntity inbound interface has been enhanced to support the marginPriceSource element - tag 18041 for when the accountingValidationFlag = Y in the inbound load message.

The marginPriceSource was also added to the genericEntity extract for MC and MC2.

Files changed:
eagle_default/in/include/entity_acct.inc
eagle_default/in/include/entity_aliases.inc
eagle_default/in/include/entity_panels.inc
eagle_ml-2-0_cm/out/xml/t_entity_streaming.inc
eagle_ml-2-0_cm/out/xml/t_entity_streaming_eml.inc

SDP-56993 New Position Cost Lot fields are supported in the warehouse position extract for TRRESET

The Warehouse Position outbound interface was updated to support the following additional fields supported in v17.R2.30:

HOLDINGDBO.POSITION_COST_DETAIL.TRRRESET_RCVBL_L - tag 18328
HOLDINGDBO.POSITION_COST_DETAIL.TRRRESET_RCVBL_B - tag 18329
HOLDINGDBO.POSITION_COST_DETAIL.TRRRESET_RCVBL_S - tag 18330

Xpath:
EagleML/warehouseTransaction/warehousePosition/totalReturnResetReceivableLocal
EagleML/warehouseTransaction/warehousePosition/totalReturnResetReceivableBase
EagleML/warehouseTransaction/warehousePosition/totalReturnResetReceivableSys

Files changed:
eagle_ml-2-0_cm/out/xml/t_warehouse_position_streaming.inc

SDP-56033 The Cost Basis Adjustment inbound interface was updated to be in sync with the Open Lot Cost Adjustment panel logic changes

The eagle/star/reorg/pan-openlotcostadj.htm panel logic was updated in the core V17 R2.30 release. As a result, several tags cannot be resolved on the panel side due to existing EagleML rules.

The following tags were added in the Cost Basis Adjustment inbound interface in order to be used in the panel logic.

3931 - Processing Security Type,
961 - Issue Name,
85 - Issue Currency,
11 - Instrument Type,
12 - Quantity Type,
18 - Price Factor,
19 - Quantity Scale.

Files changed:
eagle_default/in/xml/xml-costbasisadj.xml

SDP-56564 Enhanced Corporate Actions to support amortizing notional swaps additional fields in V17 R2.30

New tags were added to the amortizing notional swap corporate actions panels in V17 R2.30. As a result they were added to EagleML to support those new fields.

Panels for processing:
/eagle/star/reference/addnotionalamort.htm
/eagle/star/reference/chgnotionalamort.htm

File changed:
eagle_default/in/xml/xml-ref_ca.xml

SDP-55954 Provided a global configuration override to apply the payment frequency code when null

Provided a global configuration parameter for Generic SMF load when paymentFrequencyCode is not populated in the incoming message, thus the value is taken from the paymentFrequency field.

To apply, the w_config_custom.inc requires the following entry. <CODE>:W_USE_PAYMENT_FREQUENCY_AS_PAYMENT_FREQ_CODE: := 'Y';</CODE>

SDP-55846 Issue Rating supports Source Description and Release Status fields.

The IssuerRating extract was enhanced to include the source_description and release_status fields.

Files changed:

eagle_ml-2-0_cm/out/tagvalue/extract_issuer_rating.inc
eagle_ml-2-0_cm/out/xml/t_issuer_rating_streaming.inc
eagleml/schema/eagleml-issuer-2-0.xsd

SDP-57058 The unpacker step of the generic load workflow has been improved to bypass an unnecessary preprocessor step when the Load Profile Name is null.

The generic_load_unpacker step has been enhanced to address when the variable LoadProfileName is null it will bypass an unnecessary preprocessor trackback error message. This will reduce any unnecessary processing due to this error and result in improved processing times of the generic load workflow.

Files changed:
eagle_ml-2-0_cm/unpacker/custom_routing.inc

SDP-57396 Added a folder and an RTR message to regenerate EagleML binds every time the MC engine recycles

Added the ability to regenerate the EagleML binds every time the MC engines recycle by adding a folder and RTR message that utilizes the AutoStart feature. By adding the parameter "AutoStartStreams=eagle_ml-2*-*0_default_cm_control_message" parameter in the [Service] section of the msgcenter.ini on every MC instance, the folder and RTR will run via the control message stream and regenerate the binds. If the Eagle Core version has not changed, the check will complete without any regeneration. This feature is available starting from Eagle Core version 17 R2.19. It can be utilized after v17 R2.19 by locally adding the folder path and RTR if using a version of EagleML prior to the December 2021 release.

Files added:
dynamic/msgcenter/autostart/eagle_ml-2-0_default_cm_control_message

SDP-57169 Issuer Resolution logic supports resolution by just Issuer Xreference even if the IssuerID is available

The ability to resolve an Issuer by using just the Issuer Xreferences was previously unavailable when the IssuerId field is present in the incoming file. As a result, the USE_XREF_ONLY Issuer Resolution option was introduced to run the Issuer Resolution by using the Issuer Xreferences even if the IssuerId field is populated.

Within the EagleML block:
<issuerResolutionOption>USE_XREF_ONLY</issuerResolutionOption>

Files changed:
eagle_default/in/xml/xml-ref_issuerorganization.xml
eagle_default/in/xml/include/issuer_xref_resolution.inc

SDP-54665 Rebook logic was implemented for Price Classification to minimize updates

To reduce unnecessary updates at the database level, a change was introduced to address when the Price Classification data was loaded twice without changes. The process will validate that the original record exists and it will ignore the message.
The comparison logic includes the list of fields that will be validated to skip a record update:
SECURITYDBO.SMF_PRICE_CLASSIFY.PRICE_CLASS1 (Eagle/referenceTransaction/priceClassification/priceClass1)
SECURITYDBO.SMF_PRICE_CLASSIFY.PRICE_CLASS2 (Eagle/referenceTransaction/priceClassification/priceClass2)
SECURITYDBO.SMF_PRICE_CLASSIFY.PRICE_CLASS3 (Eagle/referenceTransaction/priceClassification/priceClass3)

The new logic is only available when the processing option PRICECLASSIFICATION_TRUE_REBOOK is set in the incoming file or as a global parameter W_PRICECLASSIFICATION_TRUE_REBOOK in the w_config_custom.inc.

Usage:
<header>
<objectType>PriceClassification</objectType>
<processingOptions>PRICECLASSIFICATION_TRUE_REBOOK</processingOptions>
</header>
or
<COL TAG="W_PRICECLASSIFICATION_TRUE_REBOOK" EXPRESSION= "'Y'"/>

Files changed:
eagle_default/in/xml/xml-ref_price_classification.xml 

SDP-57446 Field mapping was corrected for ENTITYANALYTICEXTRACT

Field mapping was corrected for EDS ENTITYANALYTICEXTRACT. The entityAnalyticModel now has the correct reference to the database field ENTITY_ANALYTICS.USER_FLOAT1 instead of the database field ENTITY_PEER_GRP_ANALYTICS.USER_FLOAT1.
 
File changed:
dynamic/metadata/core/ontology/entity/entityanalytic-models.json

SDP-57367 Ontology generator was improved to allow for the custom ontology file to not have any limitations on the file name

Processing rules were generated incorrectly due to a custom ontology file name that began with the client id. The ontology generator was therefore improved to allow for the custom ontology file to not have any limitations on that file name.

Files changed:
eagleinvsys/rdf_metadata/rdf_meta_graph_utils.py

SDP-57303 EDS Warehouse data rules support Batch and Delta mode for load

The EDS data rules were enhanced for all the resources listed below. The Warehouse resources were also updated to use Batch mode for load by default, except for warehousependingtrade, which is using Delta mode by default.

The update mode or deltaBased load is supported for EDS resources and can be turned on via a parameter:

warehouseposition
warehousecloselot
warehousecashactivity
warehousependingtrade
warehousetrade
warehousenav

<taskParameter>
<name>deltaBased</name>
<value>Y</value>
</taskParameter>

Delta Load mode was not implemented for warehouseopenlot and warehouseperformance.

Sample Rest API request for Azure:
{ "resourcename": "warehousetrade", "dbprovider": "snowflake", "deltaBased": "Y" }

Files changed:
warehouse/ warehouseposition_in.json
warehouse/ warehousecloselot_in.json
warehouse/ warehousecashactivity_in.json
warehouse/ warehousetrade_in.json
warehouse/ warehousenav_in.json
warehouse/ warehousenavmulti_in.json

SDP-57133 Added the ability to generate JSON schema for ontology resource

Metadata Service supports schema generation for the resource. The following schema types are now available: XSD, JSON, AVRO.
The schema generation can be executed only for on-premise environments. For custom resources, only the JSON schema is currently supported.

The schema can be generated via RTR, there are two required parameters:
ResourceName
SchemaFormat: XSD, JSON, AVRO

<EagleML xmlns="[http://www.eagleinvsys.com/2011/EagleML-2-0%22|http://www.eagleinvsys.com/2011/EagleML-2-0%22|smart-link ] xmlns:xsi="[http://www.w3.org/2001/XMLSchema-instance"|http://www.w3.org/2001/XMLSchema-instance%22] xsi:type="RunTaskRequest" eaglemlVersion="2-0" xsi:schemaLocation="[http://www.eagleinvsys.com/2011/EagleML-2-0|http://www.eagleinvsys.com/2011/EagleML-2-0|smart-link ] eagleml-main-2-0.xsd" eaglemlType="RunTaskRequest">
<header>
<messageId>B70DHO72GCLFRHGI</messageId>
<sentBy><[http://www.eagleinvsys.com </sentBy>>|http://www.eagleinvsys.com%3C/sentBy%3E%3E]
<sendTo><[http://www.eagleinvsys.com/eagle_default_ruleservice%3C/sendTo%3E%3E|http://www.eagleinvsys.com/eagle_default_ruleservice%3C/sendTo%3E%3E|smart-link ]
</header>
<taskIdentifier>
<correlationId>whspos_xsd</correlationId>
<businessTaskId>whspos_xsd</businessTaskId>
</taskIdentifier>
<taskTypeEnum>NEW</taskTypeEnum>
<synchronousExecution>yes</synchronousExecution>
<taskParameters>
<taskParameter>
<name>ServiceName</name>
<value>Metadata</value>
</taskParameter>
<taskParameter>
<name>ServiceMethodName</name>
<dataType>S</dataType>
<value>Schema</value>
</taskParameter>
<taskParameter>
<name>ResourceName</name>
<value>warehouseposition</value>
</taskParameter>
<taskParameter>
<name>SchemaFormat</name>
<value>XSD</value>
</taskParameter>
</taskParameters>
</EagleML>

The schema file that has been generated will be stored under the following directories:

Core resource:
metadata/core/schema/{schema_type}

Custom resource:
metadata/custom/schema/{client_id}\{schema_type}

SDP-56825 The ontology parsing methods were adjusted to work with resource version

The ontology deployment and reading from the blob storage logic were improved to work with a specified version. Previously, only the latest ontology version was supported. This change is required for implementation of the ALTER mode for DLL generation, where it is essential to support multiple versions and to be able to read a specific version. For example, the method to get the AzureContainerWrapper was improved to handle the version_id parameter.
def get(self, version_id=None):
This version is stored in eagle-main.json - "owl:versionInfo": "2.0.7"

Files changed:
src/eagleinvsys/code_deployment/code_deployment_manager.py
src/eagleinvsys/mc2py/azure_blob_wrapper.py

SDP-56387 Extended the list of core cache policies

The Delta Cache Update Policies are now available for additional core resources by default. Those enhancements are listed below.

  • smfidentifiers - check for any change or update every 60 seconds;

  • entityidentifiers - check for any change or update every 60 seconds;

  • codevalue - check for any change or update every 30 seconds;

  • codecategory - check for any change or update every 30 seconds;

  • sources - check for any change or update every 300 seconds.

If new records were loaded or existing records were updated for the additional resources in the In-Market region, then after 30-60 seconds these changes will be automatically uploaded to the Redis Cache. The end users are no longer required to intervene and manually send the RTR message to refresh the dataset cache in the In-Market region. However, it is still required to refresh the Redis cache manually for Azure processing.

The Cache Update management process was also enhanced to check and update both possible locations for the dataset definitions, to keep that data intact. Those updates are listed below.

  • The Application server folder: estar/tpe/dynamic/metadata/core/datasets

  • The Redis cache.

Files changed:

eagle/estar/tpe/dynamic/metadata/core/cachepolicies/codecategory.json
eagle/estar/tpe/dynamic/metadata/core/cachepolicies/codevalue.json
eagle/estar/tpe/dynamic/metadata/core/cachepolicies/entityidentifiers.json
eagle/estar/tpe/dynamic/metadata/core/cachepolicies/smfidentifiers.json
eagle/estar/tpe/dynamic/metadata/core/cachepolicies/sources.json
src/eagleinvsys/ebs/ebs_interfaces_manager.py

SDP-55765 Reprocess functionality was implemented at the record level for EDS Data Rules

Reprocess functionality was implemented at the record level in the event of an unsuccessful load via the EDS data rules. The Reprocess functionality is unavailable in case of failed batch.

Files changed:
eagleinvsys/mc2py/dbdescriptor.py
eagleinvsys/mc2py/dbdescriptor_sf.py
eagleinvsys/mc2py/destination.py
eagleinvsys/mc2py/dscacheparser.py
eagleinvsys/mc2py/dsparser.py
eagleinvsys/mc2py/format_enums.py
eagleinvsys/mc2py/genetl.py
eagleinvsys/mc2py/idsdataframe.py
eagleinvsys/mc2py/igenetl_items.py
eagleinvsys/mc2py/proc_log_stru.py
eagleinvsys/mc2py/processing_context.py
eagleinvsys/mc2py/sqlbase_enum.py
eagleinvsys/mc2py/string_utils.py
eagleinvsys/mc2py/t_eagle_xj.py
estar/tpe/dynamic/metadata/core/ebs/eds_reprocess.json

SDP-54590 The genericsmf EDS resource was enhanced to support the MBSExtension model for load and extract for In-Market/Oracle regions

The genericsmf EDS resource has been updated to support all fields as part of the MBSExtension model for the SECURITYDBO.MBS_DETAIL_EXT table for load and extract.
MBS Extension fields that are available from V17R2.28 and on are fully supported for load and extract for In-Market/Oracle regions.

Files changed:
metadata/core/ontology/reference/genericsmf-models.json

SDP-46710 EDS Load - Security resolution was updated to support batting order and resolution options

The security resolution was updated to support the batting order and resolution options. It now supports any available incoming Xref identifier to create a list of up to 20 identifiers for the security resolution request. This list is based on the security resolution option and batting order.

Default batting order (if no batting order is set): cusip,cusipPay,cusipReceive,bcusip,isin,sedol,reuters,bloombergId,sicovmId,valorenId,cedel,internalId,cinsId,ticker,xId1,xId2,xId3,xId4,xId5,xId6,xId7,xId8,xId9,xId10,uniqueProductId,uniqueSwapId

Default resolution option (if no resolution option is set): ALL

Files changed:

src/eagleinvsys/mc2py/exprbase.py
src/eagleinvsys/mc2py/genetl_loop.py

SDP-53837 EDS Load - Security Resolution was enhanced to support the USE_FIRST_AS_MAIN option

USE_FIRST_AS_MAIN can be added as an option to assetResolutionOption to activate USE_FIRST_AS_MAIN mode. This option is turned off by default.
If USE_FIRST_AS_MAIN option is set, then only the first available XREF Identifier will be used and only the identifier that is suitable according to the batting order and resolution option.

Example assetResolutionOption : <assetResolutionOption>ALL,USE_FIRST_AS_MAIN</assetResolutionOption>

Files changed:

src/eagleinvsys/mc2py/exprbase.py
src/eagleinvsys/mc2py/genetl_loop.py
metadata/core/ebs/resolution_service_new.json
core/ontology/templates/includes/defaultassetresolutionservice.json

Production Bugs

SDP-54508 The Task Reporter rule was fixed to show the correct details for Pace Event task in the Consolidated TSR

The Consolidated TSR contained incorrect file names for the PACE Event task when the parameter IncludeErorrDetails = N was used. The issue occurred due to a corrupted SQL query that gathers information about files from the Database. The SQL query in the Task Reporter rule was therefore fixed in order to assemble all the necessary information about the Pace Event task including the file name and any error details associated with it.

Before fix:

<taskUnit> <loadDetails> <URI>file://20211207-120951-_C~28KF599ICU1SA005_C~PE_SDP54508#executecm.dat</URI> <details> <file> <fileName>20211207-120951-_C~28KF599ICU1SA005_C~PE_SDP54508#EXECUTECM.DAT</fileName> <type>CUSTOM</type> <summary> <eagleStatId>199840391</eagleStatId> <failedRecords>0</failedRecords> <totalRecords>1</totalRecords> <taskType>Stream</taskType> <taskName>eagle_ml-2-0_default_cm_execute_pace_event</taskName> </summary> </file> </details> </loadDetails> </taskUnit>

After fix:

<taskUnit> <loadDetails> <URI>file:///apps/eagle/data/pace/uploader/EAGLEPACE/CDV1/PE_Code_Value_Test_19.txt</URI> <details> <file> <fileName>/apps/eagle/data/pace/uploader/EAGLEPACE/CDV1/PE_Code_Value_Test_19.txt</fileName> <type>CUSTOM</type> <summary> <eagleStatId>19185292</eagleStatId> <failedRecords>1</failedRecords> <totalRecords>1</totalRecords> </summary> <record> <eagleDetailId>5830866</eagleDetailId> <indentifier> <indentifierName>Field Not Found</indentifierName> <indentifierValue>CODE1,CODE2,CODE3,CODE4</indentifierValue> </indentifier> <reason> <reasonTypeEnum>ERROR</reasonTypeEnum> <reasonCode>1</reasonCode> <description>CDVSRCNOTFND: Code Value Source Interface not found</description> </reason> </record> </file> </details> </loadDetails> </taskUnit>

Files changed:
eagle_ml-2-0_cm/out/xml/task_reporter_generate_task_unit.inc

SDP-57393 The Issuer Rating outbound interface was fixed to create valid records

The issuerRating outbound interface had an incorrect grouping key which did not correspond to the DB table primary key. This caused some Issuer Rating records to be incorrectly formed by the Processing Rule.

The grouping key was fixed for the interface.
Files changed:
eagle_ml-2-0_cm/out/tagvalue/extract_issuer_rating.inc
eagle_ml-2-0_cm/out/xml/t_issuer_rating.xml
eagle_ml-2-0_cm/out/xml/t_issuer_rating_streaming.inc

SDP-56958 WRHSTRADEEXTRACT - Expand Settlement Amount Base and Local logic
SDP-56407 WRHSTRADEEXTRACT - Signage changes for additional complex transaction types

The WRHSTRADEEXTRACT - Warehouse Trade is extended to revise signage for a number of elements for complex transactions supporting newly identified scenarios.
When the Transaction Code is equal to TRRESETCOST, the data reflects a value. For reporting purposes, this value should be 0 and as a result, the SETTLEMENT_AMOUNT and SETTLE_AMOUNT_BASE fields were modified to set to 0 when a TRRESETCOST. The targeted fields have had expanded logic for when EnabledSignageLogic is set to 'Y' as well the SETTLEMENT_AMOUNT field.

Fields Affected by Signage Logic

Revised Fields
ACCRUED_INTEREST
EagleML/WarehouseTransaction/warehouseTrade/accruedInterest
BASEINTERESTADJ
EagleML/WarehouseTransaction/warehouseTrade/interestAdjBase
NETSETTLEAMT
EagleML/WarehouseTransaction/warehouseTrade/netsettleamt
SETTLE_AMOUNT_BASE
EagleML/WarehouseTransaction/warehouseTrade/settleAmountBase
SETTLEMENT_AMOUNT
EagleML/WarehouseTransaction/warehouseTrade/settlementAmount
TRADE_AMOUNT
EagleML/WarehouseTransaction/warehouseTrade/tradeAmount

Signage logic for the Settlement Amount Base and Settlement Amount Local is enabled through an opt-in element. The SettlementSignageFlag is used in tandem with the EnableSignageLogic set to Y. When the SettlementSignageFlag is set to Y, signage logic is applied to the SETTLEMENT_AMOUNT and SETTLE_AMOUNT_BASE fields. The SETTLE_AMOUNT_BASE field is supported in v17.R2.30 forward.

Signage is enabled when the following parameters are included:
<taskParameter>
<name>EnableSignageLogic</name>
<dataType>S</dataType>
<value>Y</value>
</taskParameter>
<taskParameter>
<name>SettlementSignageFlag</name>
<dataType>S</dataType>
<value>Y</value>
</taskParameter>

Files changed:
eagle_ml-2-0_cm/out/tagvalue/extract_wrhs_trade_signage_logic.inc