EagleML Release Notes - May 3rd 2022

Owned by Jeffrey Santora (Deactivated)
Last updated: Jun 01, 2022
5 min read

Enhancements

SDP-59862 Accounting mode for Generic SMF Inbound interface was updated to support new elements

The Generic SMF interface for Accounting load was updated to support the ability insert into the SECURITYDBO.SECURITY_REGULATORY database table as well as several new fields from other data models.

Xpath:
EagleML/referenceTransaction/genericSMF/FixedIncomeModel/callAnnounceDate
EagleML/referenceTransaction/genericSMF/FixedIncomeModel/firstCouponPeriodType
EagleML/referenceTransaction/genericSMF/FixedIncomeModel/isStillCallableIndicator
EagleML/referenceTransaction/genericSMF/FixedIncomeModel/seriesAmountOutstanding
EagleML/referenceTransaction/genericSMF/FixedIncomeModel/stripType
EagleML/referenceTransaction/genericSMF/FixedIncomeModel/surplusNoteIndicator
EagleML/referenceTransaction/genericSMF/MBSExtension/wamNextResetTerm
EagleML/referenceTransaction/genericSMF/MBSExtension/prepayPenaltyTerm
EagleML/referenceTransaction/genericSMF/MBSExtension/prepayPenaltyType
EagleML/referenceTransaction/genericSMF/classificationLevel1Name
EagleML/referenceTransaction/genericSMF/classificationLevel2Name
EagleML/referenceTransaction/genericSMF/classificationLevel3Name
EagleML/referenceTransaction/genericSMF/classificationLevel4Name
EagleML/referenceTransaction/genericSMF/securityRegulatoryModel/sanctionedIndicator
EagleML/referenceTransaction/genericSMF/securityRegulatoryModel/confidenceLevel
EagleML/referenceTransaction/genericSMF/securityRegulatoryModel/sanctionReferenceCapital
EagleML/referenceTransaction/genericSMF/securityRegulatoryModel/sanctionActualCapital
EagleML/referenceTransaction/genericSMF/securityRegulatoryModel/europeanSanctionDirective
EagleML/referenceTransaction/genericSMF/securityRegulatoryModel/unitedStatesSDNSanctionDirective
EagleML/referenceTransaction/genericSMF/securityRegulatoryModel/unitedStatesNonSDNSanctionDirective
EagleML/referenceTransaction/genericSMF/securityRegulatoryModel/unitedNationsSanctionDirective
EagleML/referenceTransaction/genericSMF/securityRegulatoryModel/hongKongSanctionDirective
EagleML/referenceTransaction/genericSMF/securityRegulatoryModel/unitedKindomSanctionDirective
EagleML/referenceTransaction/genericSMF/securityRegulatoryModel/chinaSanctionDirective
EagleML/referenceTransaction/genericSMF/securityRegulatoryModel/canadaSanctionDirective
EagleML/referenceTransaction/genericSMF/securityRegulatoryModel/australiaSanctionDirective
EagleML/referenceTransaction/genericSMF/securityRegulatoryModel/singaporeSanctionDirective
EagleML/referenceTransaction/genericSMF/securityRegulatoryModel/sanctionTargetCountry
EagleML/referenceTransaction/genericSMF/securityRegulatoryModel/instrumentStatus
EagleML/referenceTransaction/genericSMF/securityRegulatoryModel/sanctionChangeReasonMemo
EagleML/referenceTransaction/genericSMF/securityRegulatoryModel/sanctionRationaleMemo

Files added & changed:

eagle_default/in/xml/include/fixedincome_extension.inc
eagle_default/in/xml/include/mbs_extension.inc
eagle_default/in/xml/include/sector_detail.inc
eagle_default/in/xml/include/securityregulatory_model.inc
eagle_default/in/xml/include/sm_extension.inc
eagle_default/in/xml/include/smf_aliases.inc
eagle_default/in/xml/xml-smf.xml

SDP-60873 Corporate Action logic was corrected due to unexpected records on the voluntary election table

Corporate Action Rebook functionality was updating records when they should be ignored due to tag 3960 value incorrectly being passed to panels. The rebook logic was adjusted and therefore the issue has been fixed.

File changed:

eagle_default/in/xml/xml-ref_ca.xml

SDP-59961 Added a lookup to check if an entity already exists in the Database for the Entity Analytics object

It was possible to load entity analytics records for an entity that does not exist in the database. The issue has been resolved by interrupting the processing if an entity ID does not exist in the database. An error message will be displayed as follows. 'Entity ID is empty or not found in the database.'

File changed:
xml-ent_entity_analytics.xml

SDP-60858 Unpacker step has been enhanced to support a dynamic timeout feature

The unpacker step has been enhanced to support the ability to dynamically timeout by a user defined value rather than the default value of 300 seconds. This can be set by using either of the two new variables below.

  • UnpackerTimeout – A local variable that can be set in the IWS task unpacker and will have the highest priority.

  • W_UNPACKER_TIMEOUT – A global variable that can be set in w_config_custom config and will have a priority lower than the UnpackerTimeout variable.

Usage:

  • In the task_unpacker step in IWS, set an UnpackerTimeout variable in the Custom Parameters of the task_unpacker

  • <COL TAG="W_UNPACKER_TIMEOUT" EXPRESSION="'500'"/>

File changed:
dynamic/msgcenter/eagle_ml-2-0_cm/unpacker/wrf_unzip.prf

SDP-46913 EntityComposite and EntityToComposite interfaces were updated to support Control Center Composite Entity

Control Center Composite Entity (CTRC Entity) was not fully supported in EntityComposite and EntityToComposite interfaces.

Three issues were therefore fixed.

  1. During CTRC Entity Insert by the EntityComposite interface, there were errors from the panel that required tags are missed. The mapping for four tags was added.
    EagleML/entityTransaction/entityComposite/ccProcessCtr - tag 11407
    EagleML/entityTransaction/entityComposite/ccDailyMapName - tag 3975
    EagleML/entityTransaction/entityComposite/portCompType - tag 3953
    EagleML/entityTransaction/entityComposite/ccMeMapName - tag 80

  2. During CRTC Entity Update via EntityComposite interface there was an error from the panel that process_center_id was missed. The logic to resolve tag 4896 was added.

  3. Entity wasn't being removed from CTRC Entity via EntityToComposite interface. The EntityToComposite interface was updated to support not only COMP but CTRC Entity as well.

Files changed:
eagle_default/in/xml/xml-ent_entitycomposite.xml
eagle_default/in/xml/xml-ent_entitytocomposite.xml

SDP-60999 EDS genericsmf resource was updated to support instrumentStatus

EagleML/referenceTransaction/genericSMF/securityRegulatoryModel/instrumentStatus

Files changed:
metadata/core/ontology/reference/genericsmf-models.json

SDP-60287 Support for new SMF fields for eagle ml generic SMF stream

GenericSMF load and extract interfaces were enhanced to support new elements contract1256Indicator and derivativeType for tags 18400 and 18385 respectively.

contract1256Indicator is used for OPOPEQ (processingSecurityType = OPOPEQ) securities (panels restriction) and can be set to N or Y.

derivativeType is used for Credit Default Swap SMF (processingSecurityType = SWCDCO or SWCDCX or SWLOCK) and can be set to C (Corporate), O (Other) or S (Structured).

Rule files changed:
eagle_default/in/xml/include/smf_aliases.inc
eagle_default/in/xml/include/derivative_ext.inc
eagle_default/in/xml/include/smf_panels.inc
eagle_default/in/xml/xslt/smf_egl_gen_interface_template.xsl
eagle_default/in/xml/xslt/smf_egl_gen_interface_template_mssql.xsl
eagle_ml-2-0_cm/out/xml/t_smf_streaming_eml.inc
eagle_ml-2-0_cm/out/xml/t_smf_streaming.inc
eagle_ml-2-0_cm/out/xml/t_smf.xml

XSD files changed:
schema/eagleml-enum-2-0.xsd
schema/eagleml-sec-shared-2-0.xsd

SDP-60096 Support for new basis level election for 1256

GenericEntity in accounting mode and EntityAccountingBasis load and extract interfaces were enhanced to support the new element contract1256Processing with tag 18401. Please note that contract1256Processing has a default value of N and can be set to Y only when ledgerProcessingFlag is N (panels restrictions).
If attempting to load contract1256Processing and ledgerProcessingFlag as Y, there will be an error flagged from the panel.

Files changed:
eagle_default/in/include/entity_aliases.inc
eagle_default/in/include/entity_acct.inc
eagle_default/in/include/entity_panels.inc
eagle_default/in/xml/xml-ent_entityaccountingbasis.xml
eagle_ml-2-0_cm/out/xml/t_entity.xml
eagle_ml-2-0_cm/out/xml/t_entity_streaming.inc
eagle_ml-2-0_cm/out/xml/t_entity_streaming_eml.inc
eagle_ml-2-0_cm/out/xml/t_entity_accounting_basis.xml
eagle_ml-2-0_cm/out/xml/t_entity_accounting_basis_streaming.inc

XSD files changed:
eagleml-entity-shared-2-0.xsd

SDP-61633 WRHSTRADEEXTRACT - Updated signage logic for DL_OPEN_COST_BASE

Signage logic was adjusted for the openCostBase field. This change will have one row per TRADE.TRADE_ID on the extract where the DL_OPEN_COST_BASE will return the SUM of all DISPOSAL_LOTS.OPEN_COST_BASE records with the matching DISPOSAL_LOTS.TRADE_ID. This SUM or rolled up value will equal the same value as TRADE.COST_BASE.

DISPOSAL_LOTS.OPEN_COST_BASE - DL_OPEN_COST_BASE
EagleML/WarehouseTransaction/warehouseTrade/openCostBase

Files changed:
eagle_ml-2-0_cm/out/tagvalue/extract_wrhs_trade_signage_logic.inc
eagle_ml-2-0_cm/out/tagvalue/extract_warehouse_trade.inc
eagle_ml-2-0_cm/out/xml/t_warehouse_trade_streaming.inc

SDP-61634 WRHSTRADEXTRACT - Update signage logic for ORIGINAL_FACE_VALUE

Signage logic was adjusted for the originalFaceValue field. This change will have TRADE.ORIGINAL_FACE_VALUE when equal to zero, displayed as <originalFaceValue>0</originalFaceValue>.

ORIGINAL_FACE_VALUE
EagleML/WarehouseTransaction/warehouseTrade/originalFaceValue

Files changed:
eagle_ml-2-0_cm/out/tagvalue/extract_wrhs_trade_signage_logic.inc

SDP-61166 Loading FX Rates through RDC

FX rates can be loaded to both old Pricing FX Rates or RDC FX Rates using a dynamic switch. The switch is useRdcFx != 'Y' or useRdcFx = 'Y'. This feature will support the build out to support FX Rates in RDC.

  1. Added new flag  useRdcFx (possible values is ‘Y’), it will define the name of the run panels.

  2. Added the condition of using flag useRdcFx or config parametr W_USE_RDC_FX.

  3. Added status, helped user to check the processing information.

  4. Added condition for calling pan-rdc_fx_upload.htm panel .

Usage:

w_config_custom.inc:
<COL TAG="W_USE_RDC_FX" EXPRESSION="'Y'"/>

Files changed:
eagle_default/in/xml/xml-ref_fxpcrate.xml

SDP-60979 Ontology converter was enhanced to support slashes in the header for files in CSV format

Ontology converter was improved to handle slashes in the CSV file header and allow it to generate a valid processing rule.

The logic for load data was also improved and a new flag was added in the source format dialect which is shown as below.

"encode_header": false

The default value is false:

"source_format_dialect": \{
"is_single_row_txn": true,
"rootnodename": "EagleML",
"rownodename": "clientRelationship",
"txnheadernodename": "header",
"txnnodename": "referenceTransaction",
"use_raw_parsing": true,
"encode_header": false

SDP-60663 EDS genericca resource was enhanced to improve data load to Oracle and Snowflake

Security resolution for genericca was enhanced.
Enhanced DDL for CORP_ACT_ANNOUNCEMENT_HIST.

Additional fields are supported for TRADESDBO.CORP_ACT_HIST as shown below:
MAKE_WHOLE_CALLED
BYPASS_SINK_CHECK
CP_STOCK_OPTION

The tables listed below are supported for load in composite mode:
CORP_ACT_HIST
CORP_ACT_HIST_DETAIL
CORP_ACT_HIST_EXTENSION

The tables listed below are supported to load in ASOF mode:
CORP_ACT_ANNOUNCEMENT_HIST
CORP_ACT_HIST_DETAIL_HIST
CORP_ACT_HIST_EXTENSION_HIST

Files changed:

metadata/core/ontology/templates/eagle_ref_genericca_insertupdate.txt
metadata/core/ontology/definition/eagle-vocab-definition-ref-shared.json
src/eagleinvsys/rdf_metadata/rdf_meta_enum.py
src/eagleinvsys/rdf_metadata/rdf_meta_templates_processor.py
src/eagleinvsys/rdf_metadata/rdf_meta_utils.py

SDP-55087 EDS fieldattribute extract as part of Metadata lineage functionality

THe EDS fieldattribute extract is similar to EagleML extract in a sense that the extract contains meta data information about Eagle fields like field type, name, data base table, description and etc. This implementation supports information for core, custom and derived fields.

New logic for the calculated fields was added and now all user defined expressions for the calculated fields will be part of the extract.

There will be two additional fields:
fieldExpression - user friendly expression which is available from MDC
userData - field which contain RAW definition or the value stored in the database

New fields are included by default, but they can be excluded by setting RTR parameter _includederivedattributes_ to N.

EQL REST API is supported - api/eagle/v2/eql

The body of the request is as follows:

{
"ResourceName": "fieldattribute",
"ServiceName": "EBS",
"ServiceMethodName": "extract",
"savesqlquery": "Y"
}

The filters listed below are now supported:

fieldattributeid
fieldtype
fieldname

The initial configuration steps are required. Please see the confluence page for additional details.

Files changed:
src/eagleinvsys/ebs/filters/cursor_filter_registry.py
src/eagleinvsys/ebs/filters/fieldattribute_filter/fieldattribute_filter_cursor_handler.py
src/eagleinvsys/ebs/filters/fieldattribute_filter/fieldattribute_filter_sql.py
src/eagleinvsys/mc2py/cm_enum.py
src/eagleinvsys/rdf_metadata/rdf_meta_runtime_utils.py

SDP-52602 EDS resource rating

EDS security rating resource was implemented to support load and extracts. Both Oracle and Snowflake database providers are supported.

Files changed:

src/eagleinvsys/rdf_metadata/rdf_meta_templates_processor.py
metadata/core/ontology/templates/eagle_ref_rating_insertupdate.txt

Production Bugs

SDP-61112 Duplicate ID’s for issuerResolutionOption and addressLine2 aliases are fixed for the issuerOrganization inbound interface.

There were duplicate ID’s corresponding to issuerResolutionOption and addressLine2 aliases. Both had the same ID for tag 170 in the issuerOrganization inbound interface. This resulted in the issuerResolutionOption alias being overridden by addressLine2 alias. AddressLine2 alias was updated and now corresponds to ID tag 701 which resolved the issue.

File changed:
eagle_default/in/xml/xml-ref_issuerorganization.xml 

SDP-61374 Accounting Cash inbound interface was fixed to Cancel Trades

Lower and upper case values are now supported for trade resolution when using the Accounting Cash inbound interface. Clients who passed lower case values for the originalTradeTicketNumber would have failures due to the data being stored in the database as an upper case value.

Files changed:
eagle_default/in/xml/xml-acct_cash_cancel.xml

SDP-61288 ENTITYEXRACT Entity extension appears in the entity node.

Entity extension data was appearing in the wrong entity node when running the ENTITYEXTRACT through MC2. In order to properly create an extension via MC2, logic for calling genericentity_streaming_init.inc and genericentity_streaming_onend.inc was improved.

Files changed:
eagle_ml-2-0_cm/out/xml/t_entity_streaming.inc