EagleML Release Notes - March 7, 2023

Enhancements

SDP-70446 Conversion Workflow Logic for Date Filters was updated to support the Datetime type

With the Conversion Workflow, the logic for Date filters was improved to support a timestamp. The datetime type is supported now for date filters:

<taskParameter><name>fromdate</name><value>2010-01-01 00:00:00</value></taskParameter>
<taskParameter><name>todate</name><value>2010-10-25 23:59:59</value></taskParameter>

There was no change for the effectivedate filter.

Files changed:
src/eagleinvsys/etl/conversion_main.py

SDP-70197 Conversion Workflow Delete Processing Improvements

The Conversion Workflow Delete Logic has been updated. For all resources with a single table, the delete operation is executed. For resources where multiple tables are involved, a temporary table is used. This table is created ad-hoc, then inserts key fields from the master table and uses this table to execute the delete from the related table first. At the end of the process, the master table is truncated. Three resources - entitypolicydetail, entitypolicymap,reportdictionarylevel, have no date fields. As a result, there is no way to split result file on chunks and there will be a single extract file that contains all data from the Eagle system. On the load to Snowflake, the entire table will be truncated for these resources.

For resources:

• entitypolicydetail

• entitypolicymap

• reportdictionarylevel

Full table will be extracted every time and a SINGLE extract file created because there is no date field that can be used for delete. Therefore, the delete will PURGE the entire table without conditions and load the extracted data.

For example:

CREATE TEMPORARY TABLE IF NOT EXISTS ESTAR.TEMP_TABLE_5894(ENTITY_ID VARCHAR(200) ,EFFECTIVE_DATE VARCHAR(200));
INSERT INTO ESTAR.TEMP_TABLE_5894 SELECT ENTITY_ID,EFFECTIVE_DATE FROM RULESDBO.ENTITY_HIST WHERE EFFECTIVE_DATE BETWEEN '2099-01-02' AND '2099-01-01';
DELETE FROM RULESDBO.ENTITY_EXTENSION_HIST USING ESTAR.TEMP_TABLE_5894 WHERE ESTAR.TEMP_TABLE_5894.ENTITY_ID = RULESDBO.ENTITY_EXTENSION_HIST.ENTITY_ID AND ESTAR.TEMP_TABLE_5894.EFFECTIVE_DATE = RULESDBO.ENTITY_EXTENSION_HIST.EFFECTIVE_DATE;
DELETE FROM RULESDBO.ENTITY_EXTENSION_DETAIL_HIST USING ESTAR.TEMP_TABLE_5894 WHERE ESTAR.TEMP_TABLE_5894.ENTITY_ID = RULESDBO.ENTITY_EXTENSION_DETAIL_HIST.ENTITY_ID AND ESTAR.TEMP_TABLE_5894.EFFECTIVE_DATE = RULESDBO.ENTITY_EXTENSION_DETAIL_HIST.EFFECTIVE_DATE;
DELETE FROM RULESDBO.ENTITY_OVERRIDES_HIST USING ESTAR.TEMP_TABLE_5894 WHERE ESTAR.TEMP_TABLE_5894.ENTITY_ID = RULESDBO.ENTITY_OVERRIDES_HIST.ENTITY_ID AND ESTAR.TEMP_TABLE_5894.EFFECTIVE_DATE = RULESDBO.ENTITY_OVERRIDES_HIST.EFFECTIVE_DATE;
DELETE FROM RULESDBO.ENTITY_STAR_PROCESSING_HIST USING ESTAR.TEMP_TABLE_5894 WHERE ESTAR.TEMP_TABLE_5894.ENTITY_ID = RULESDBO.ENTITY_STAR_PROCESSING_HIST.ENTITY_ID AND ESTAR.TEMP_TABLE_5894.EFFECTIVE_DATE = RULESDBO.ENTITY_STAR_PROCESSING_HIST.EFFECTIVE_DATE;
DELETE FROM RULESDBO.ENTITY_HIST USING ESTAR.TEMP_TABLE_5894 WHERE ESTAR.TEMP_TABLE_5894.ENTITY_ID = RULESDBO.ENTITY_HIST.ENTITY_ID AND ESTAR.TEMP_TABLE_5894.EFFECTIVE_DATE = RULESDBO.ENTITY_HIST.EFFECTIVE_DATE;

Files changed:

src/common_core/core_utils/rdf_meta_runtime_utils.py
core/ontology/entity/entitypolicydetail.json
core/ontology/entity/entitypolicymap.json
core/ontology/entity/entityshareclass.json
core/ontology/reference/issuerregulatory.json
core/ontology/reference/reportdictionarylevel.json

SDP-69904 EDS Resource exchangeprice was enhanced to support History mode

The EDS Resource exchangeprice was enhanced to support the extract from the SECURITYDBO.PRICE_EXCHANGE_ARCH table and upload the data to the corresponding Snowflake table in History mode.

It is necessary to use the historyOnlyFlag to extract data in History mode.

<taskParameter>
<name>historyOnlyFlag</name>
<value>Y</value>
</taskParameter>

Files changed:
exchangeprice_in.json.
exchangeprice-models.json

SDP-68648 EDS washsaleopenlot Resource

The EDS washsaleopenlot resource was created to support the Eagle - Vault pipeline, to extract data from Eagle and upload to Snowflake.

Table: HOLDINGDBO.LOT_LEVEL_WS_POSITION
Key: LOT_LEVEL_POSITION

Extract Filters:
sourcename
entityfilter, entitytype
processingsecuritytype, securitytype, investmenttype, releasestatus, primaryassetid, xrefid, ticker, securityalias
fromdate, todate,
tradedate, fromtradedate, totradedate,
effectivedate, fromeffectivedate, toeffectivedate
postdate, frompostdate, topostdate
accountingdate, fromaccountingdate, toaccountingdate

Files changed:
metadata/core/ontology/eagle-vocab-enumerations.json

Files added:
metadata/core/ontology/warehouse/washsaleopenlot.json
metadata/core/ontology/warehouse/washsaleopenlot-models.json
metadata/core/ontology/warehouse/washsaleopenlot_in.json

SDP-70921 Group By Key Expression was enhanced for EDS resources

The Ontology was updated for all resources where the SourceName is used. The hasGroupByKeyExpression was improved: |sourceName| + |srcIntfcInst|
The sourceName element should always be part of the source_instance resolution. There are some cases when data is migrated from the spoke to the hub and data is uploaded as is. In that case, the source_instance is already part of the file and should be used if the sourceName is missing in the incoming file. Resources which reference sourceName in grouping were updated to handle cases when the sourceName is missing, but the source_instance exists.

SDP-70422 The ontology generator was improved to support mashup with indexes

The ontology generator was enhanced to handle incoming mashup definitions with index and generate appropriate structure in ontology.

Files changed:
src/eagleinvsys/rdf_metadata/rdf_ontology_generator.py
src/eagleinvsys/rdf_metadata/rdf_ontology_generator_templates.py

SDP-71079 client, clientaddress, clientrelationship resources were updated to support Update Date and Update User

The Update Date and Update User fields were not loaded into the client related tables.

• Added mapping for upadteTimeStamp and updated mapping for the effectiveDate for the client resource

• Mapping for Update Date and Update User were added for the following fields for the client resource:

RULESDBO.INSTITUTION.UPD_DATE
RULESDBO.INDIVIDUAL.UPD_DATE
RULESDBO.INSTITUTION.UPD_USR
RULESDBO.INDIVIDUAL.UPD_USR

• Set type as DATETIME instead of DATE for the following fields:

RULESDBO.CLIENT.UPDDATETIME
RULESDBO.INSTITUTION.UPD_DATE
RULESDBO.INDIVIDUAL.UPD_DATE
RULESDBO.CLIENT_ADDRESS.UPDATE_DATE
RULESDBO.CLIENT_RELATIONSHIPS.UPDATE_DATE
RULESDBO.CLIENT_RELATIONSHIPS_DETAIL.UPDATE_DATE

Files changed:

metadata/core/ontology/eagle-vocab-common.json
metadata/core/ontology/reference/client-models.json
metadata/core/ontology/reference/clientaddress-models.json

SDP-68346 EagleML changes to Generic SMF and Multileg SMF for supporting XREF enumeration

New functionality was added into core EagleML for If when incoming message includes xreferences with numbers greater than 10, EagleML can remap them into xIds with lower number consecutive with regular ones. This change is available for genericSMF and multilegSMF both on MC and MC2 engines. For example, if xid1, xid3, xid9 and xid20 are sent in, when applied the values will be recast as xid1, xid2, xid3, and xid4 and will post successfully.

This function can be enabled by adding in the processing options GLOBAL_SWITCH flag.
<processingOptions>GLOBAL_SWITCH</processingOptions>
Files changed:
eagle_default/in/xml/include/identifier_reenumeration.inc
eagle_default/in/xml/include/smf_generic_remap.inc
eagle_default/in/xml/include/smf_multileg_remap.inc
eagle_default/in/xml/include/smf_aliases.inc
eagle_default/in/xml/include/additional_multilegsmf_xreference_aliases.inc
eagle_default/in/xml/xml-smf.xml
eagle_default/in/xml/xml-smf_generic.xml
eagle_default/in/xml/xml-smf_multileg_swap.xml

SDP-69190 SPOT Accounting Trade interface was enhanced to support pending trade flag and status

The SPOT Accounting Trade interface was enhanced to support the pendingTradeFlag - Tag 3695 and pendingTradeStatus - Tag 3720

PENDING_TRADE_FLAG - Flag to indicate that a transaction has been pended in the warehouse pending trades table
PT_TRADE_STATUS - Classification status of the Pending Trade (This field may be null)

Files changed:
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/include/csv-fxspot.inc

SDP-67492 Line of Credit related EagleML objects were updated to use Eagle core stored procedures

The Line of Credit related EagleML objects were updated to use Eagle core stored procedures instead of the auto-generated to remain in sync. The core stored procedures work beginning with the Eagle core release v17.2.38.

The ability to delete records for Line of Credit related interfaces has been added as well. To delete, the record element ‘action’ value DELETE should be passed in the header of the incoming message. If it is passed in the lineOfCreditFacility object, all related records will be deleted (Issuer relationship, Entity relationship) along with lineOfCreditFacility.

Files changed:
dynamic/msgcenter/eagle_default/in/xml/eagleml_ref_lineofcreditfacility.xml
dynamic/msgcenter/eagle_default/in/xml/eagleml_ref_locentity.xml
dynamic/msgcenter/eagle_default/in/xml/eagleml_ref_locissuer.xml
dynamic/msgcenter/eagle_ml-2-0_cm/generate_sp_on_fly.inc
dynamic/metadata/bind/xml-ref_line_of_credit.xml
dynamic/metadata/bind/xml-ref_loc_entity.xml
dynamic/metadata/bind/xml-ref_loc_issuer.xml

SDP-66988 AccountingCash enhancement to include MISCEXPADJ and MISCINCADJ events

The MISCEXPADJ and MISCINCADJ events were added to the AccountingCash inbound interface. These events support the ability to load the Miscellaneous Expense and Miscellaneous Income as Class Specific actions.

Two new elements were added to the AccountingCash object:
EagleML/accountingTransaction/accountingCash/shareClass - Tag 34
EagleML/accountingTransaction/accountingCash/ledgerReplicationSwitch - Tag 4971

Files changed:
eagle_default/in/include/cash_panels.inc
eagle_default/in/include/csv-cash.inc
eagle_default/in/include/cash_aliases.inc

SDP-71175 eagle_default_cm_file_distribution marked as FAILED when status is "NO DATA'

The eagle_default_cm_file_distribution object was modified to not raise an error from the LISTFILES step when it returns “NO DATA" status.

Files Changed:
eagle_ml-2-0_cm/out/tagvalue/eagle_default_cm_file_distribution.xml

Production Bugs

SDP-70832 EDS client resource was improved

The mapping for the CLIENT.INSTANCE field was improved as the distinct expression was added to exclude multiple records with the same instance.

Files changed:
ontology/reference/client-models.json
ontology/reference/client_in.json

SDP-70476 Conversion Workflow: Additional Changes were implemented to improve the Workflow Execution

The following changes were completed to improve the data conversion process from Oracle to Snowflake:

• The issuerregulatory conversion query was corrected to use the effective_date instead of update_date

• The entityshareclass conversion query was corrected to use the update date instead of a null effective_date

• The conversion query and delete logic were updated for the entitypolicydetail, entitypolicymap and reportdictionarylevel resources

• The disabledatasetcache flag is set to Y by default for all generated RTR’s

• Updated the query to pull all data from the tables and improved the update date filter logic for entitypolicydetail, entitypolicymap, reportdictionarylevel resources

• The accountingbasis conversion query was improved to calculate accurate number of records

• The clientrelationship conversion query was adjusted to support the effective date filter

Files changed:
src/common_core/core_utils/rdf_meta_runtime_utils.py
src/eagleinvsys/etl/conversion_main.py
metadata/core/ontology/entity/entitypolicydetail.json
metadata/core/ontology/entity/entitypolicymap.json
metadata/core/ontology/reference/reportdictionarylevel.json
metadata/core/ontology/reference/clientrelationship.json

SDP-70465 EDS warehouseperformance resource was updated to correct the keys used for data injection to Snowflake

The UpdateSource field was defined with incorrect keys as part of the Warehouse Performance metadata. As a result, empty records were occasionally created in the PERF_SEC_RETURNS_EXT or PSR_ANALYTIC tables even though there is no data in the source file. The Ontology for warehouseperformance resource was corrected. The key isShellRecordColumn set to true was added to the field UpdateSource for both tables - PERFORMDBO.PERF_SEC_RETURNS_EXT and PERFORMDBO.PSR_ANALYTICS. It allows to avoid extra records created on upload to Snowflake.

Files changed:
warehouseperformance-models.json

SDP-71169 corporateActionResetIndicator value correctly loads into SECURITYDBO.SECMASTER_HISTORY.CA_RESET_INDICATOR

The corporateActionResetIndicator element for the multilegSMF is now supported. Values are now correctly passed into the history table as expected and RDC will not delete these values from composite table. This works with and without accounting validation.

For consistency with the XSD, a new XML path was added and may now can be placed in MultipleLegSwapModel:

The existing definition, when corporateActionResetIndicator is set in the root node of leg, is also supported: