EagleML Release Notes - July 26th 2022

Owned by Jeffrey Santora (Deactivated)
Last updated: Aug 15, 2022 by Olga Galis (Deactivated)
4 min read

Enhancements

SDP-61931 The Generic Issue Analytic Inbound and Outbound interfaces were enhanced to support new fields

The XSD and Generic Issue Analytic interface were updated to support additional Bloomberg fields listed below.

SECURITYDBO.SECURITY_ANLYTICS_EXT.WORKOUT_DATE (tag 18394)
SECURITYDBO.SECURITY_ANLYTICS_EXT.WORKOUT_PRICE (tag 18395)

Xpath:
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/workoutDate
EagleML/referenceTransaction/genericIssueAnalytic/analytics/analyticExtModel/workoutPrice

Files changed:
eagle_default/in/xml/xml-ref_genericissueanalytic.xml
metadata/bind/eagleml_ref_genericissueanalytic.xml
eagle_ml-2-0_cm/out/xml/t_generic_issue_analytic_streaming.inc

SDP-59347 The Warehouse Position Inbound and Outbound interfaces were enhanced to support new fields

The XSD and Warehouse Position interface was updated to support additional fields listed below.

HOLDINGDBO.POSITION_COST_DETAIL.ORIG_ACCRUED_INCOME_BASE (18342)
HOLDINGDBO.POSITION_COST_DETAIL.TAX_PAYABLE_LOCAL (tag 18343)
HOLDINGDBO.POSITION_COST_DETAIL.TAX_PAYABLE_BASE (tag 18344)
HOLDINGDBO.POSITION_COST_DETAIL.TAX_PAYABLE_SYS (tag 18345)
HOLDINGDBO.POSITION_COST_DETAIL.ORIG_TAX_PAYABLE (tag 18346)

Xpath:
EagleML/warehouseTransaction/warehousePosition/originalAccruedIncomeBase
EagleML/warehouseTransaction/warehousePosition/taxPayableLocal
EagleML/warehouseTransaction/warehousePosition/taxPayableBase
EagleML/warehouseTransaction/warehousePosition/taxPayableSys
EagleML/warehouseTransaction/warehousePosition/originalTaxPayable

Files changed:
eagle_default/in/xml/xslt/wrh_egl_gen_interface3.xsl
eagle_ml-2-0_cm/out/xml/t_warehouse_position_streaming.inc
eagle_ml-2-0_cm/out/xml/t_warehouse_position_xmldbapi_streaming.inc

SDP-63525 GenericSMF inbound interface was enhanced to support History mode for initial security load

During initial load of a security, tag 1257 (HISTORY ONLY) is not sent to the Stored Procedure even when it is populated in incoming file. As a result, the entire record is loaded into the security master table.

A new flag was introduced to load the initial security in History mode. As a result, only a specific list of tags will be inserted into the master table.

Usage:
<header>
<objectType>GenericSMF</objectType>
<processingOptions>SEC_SHELL_RECORD_ONLY</processingOptions>
</header>
or the following parameter in w_config_custom.inc
<COL TAG="W_SEC_SHELL_RECORD_ONLY" EXPRESSION= "'Y'"/>

Files changed:
eagle_default/in/xml/xslt/eagleml_smf_gen_fields.xsl
eagle_default/in/xml/include/security_resolution_generic.inc

SDP-63538 Roll Up in Warehouse Position and Warehouse Open Lot interfaces was enhanced to support a new parameter

A new parameter called in_update_source was added into the EGL_GEN_HOLDNG.POSFINAL_EGL_GEN and EGL_GEN_HOLDNG.LOTFINAL_EGL_GEN Stored Procedures.

In EagleML rules we call these Stored Procedures with specific list of parameters and an incorrect number of parameters can lead to an error during the POSFINAL_EGL_GEN or LOTFINAL_EGL_GEN call. Warehouse Position and Warehouse Open Lot interfaces were therefore enhanced to support the new parameter in_update_source in the POSFINAL_EGL_GEN and LOTFINAL_EGL_GEN Stored Procedure calls.

Files changed:
eagle_default/in/xml/xml-dbdirect_invar_eofp.xml
eagle_default/in/xml/xml-dbdirect_invar_rollup.xml
eagle_default/in/xml/include/xml-warehouse_common.inc

SDP-64139 WRHSTRADEEXTRACT - Updated signage logic for list of fields

Signage Key and Conditions were adjusted for the following fields:

SETTLEMENT_AMOUNT - EagleML/WarehouseTransaction/warehouseTrade/settlementAmount
SETTLE_AMOUNT_BASE - EagleML/WarehouseTransaction/warehouseTrade/settleAmountBase
TRADE_AMOUNT - EagleML/WarehouseTransaction/warehouseTrade/tradeAmount
NETSETTLEAMT - EagleML/WarehouseTransaction/warehouseTrade/netsettleamt

Affected signage key:
180MTOPENLNNBUYY

Files changed:
eagle_ml-2-0_cm/out/tagvalue/extract_wrhs_trade_signage_logic.inc

SDP-58789 Added filters for Warehouse Performance DataSet Extract

DictionaryId and PerfFreqCode filters were previously added for EDS ontology based extract but the same filters still did not work for the Warehouse Performance EDS dataset based extract. The filters were added and the issue was resolved.

Additional supported filters:

Parameter | Type | Filter SQL statement
dictionaryid default "MASTERTABLE.DICTIONARY_ID = :dictionaryid"
endeffectivedate DATE "MASTERTABLE.END_EFFECTIVE_DATE = :endeffectivedate"
perfsummarytype CHAR(2) "MASTERTABLE.PERF_SUMMARY_TYPE = :perfsummarytype"
perffreqcode CHAR(8) "MASTERTABLE.PERF_FREQ_CODE = :perffreqcode"
perfrollupreturnsid default "PERF_SEC_RETURNS.PERF_ROLLUP_RETURNS_ID = :perfrollupreturnsid"

Files changed:
core/datasets/warehouseperformance.json

SDP-56042 EDS Schedule Resource

EDS Schedule Resource was implemented to support load and extract to SECURITYDBO.SCHEDULE for both Oracle and Snowflake database providers.

Files changed:
metadata/core/ontology/reference/schedule_in.json
metadata/core/ontology/templates/eagle_ref_schedule_insertupdate.txt
metadata/core/ontology/reference/schedule.json
metadata/core/ontology/reference/schedule-models.json

SDP-60366 Implemented DeltaBatch extract for EDS warehouse objects

New parameter supported for the following EDS extracts:

  • warehousecashactivity

  • warehousetrade

New filter requires 3 parameters:

  • usedeltabatch

  • fromdate

  • todate

<taskParameter>
<name>usedeltabatch</name>
<dataType>S</dataType>
<value>Y</value>
</taskParameter>
<taskParameter>
<name>fromdate</name>
<value>2017-04-16 11:00:00</value>
</taskParameter>
<taskParameter>
<name>todate</name>
<value>2022-04-17 13:00:00</value>
</taskParameter>

As a result, the extract for EDS warehouse objects will contain only batches (Entire batch Entity + Effective Date + Source) where at least one record has update_date in the specified range.

A new extract mode 'Delta-Batch' was added for Warehouse Resources to exclude duplicates. The 'Delta-Batch' mode will extract all records for a defined batch if one or more records in a batch is updated in Eagle.

To resolve issue with duplicates, the following steps should be executed:

  • EDS Extract in Delta-Batch mode should be executed on in-market region.

  • Batch load mode should be used to upload data to Snowflake/Vault

Files changed:
common_core/dataset/ds_filters/delta_batch_filters/delta_batch_filter_cursor_handler.py

SDP-63514 EDS code was updated to support new parameters for the ROLLUP procedures

New parameters were added to the core Rollup procedures for Warehouse Position and Openlot for Eagle DB version = 2017.2.35.0. The new parameters are supported for both in-market and Vault.

Core procedures:
EGL_GEN_HOLDNG.POSFINAL_EGL_GEN
EGL_GEN_HOLDNG.LOTFINAL_EGL_GEN

Files changed:
ontology

warehouseposition_in.json
warehouseopenlot_in.json

ontology
templates
eagle_wrhs_warehouseposition_rollup.oracle.txt
eagle_wrhs_warehouseposition_rollup.snowflake.txt
eagle_wrhs_warehouseopenlot_rollup.oracle.txt
eagle_wrhs_warehouseopenlot_rollup.snowflake.txt

ontology/templates/db_oper
snowflake
warehouseposition.reinit_batch.sql
warehouseopenlot.reinit_batch.sql

SDP-63581 Data Rules metadata was updated to remove BKUP_UPDATE_SOURCE field

Ontology models were corrected for the following EDS resources - issuerrating, smfidentifiers, smfidentifiershist.

The database fields listed below were removed:
XREFERENCE.BKUP_UPDATE_SOURCE
XREFERENCE_HIST.BKUP_UPDATE_SOURCE
ISSUER_RATINGS.BKUP_UPDATE_SOURCE

DDL should be regenerated in Create mode for above resources.

Files changed:

ontology\\reference\\issuerrating-models.json
ontology\\reference\\smfidentifiers-models.json
ontology\\reference\\smfidentifiershist-models.json

SDP-64059 EDS genericsmf resource was enhanced to support History mode on load and update

The historyOnlyFlag flag was previously supported for extract only. As a result, history only mode was not available for EDS genericsmf resource.
The history only mode is now available for the initial security load and update.

Files changed:
metadata/core/ontology/reference/genericsmf-models.json
metadata/core/ontology/reference/genericsmf-in.json

SDP-62741 EDS issuerrole resource was improved to load data in history mode

The issuerrole model was updated with additional mapping for ASOF or history mode. Now processing of elements in rdf_meta_utils checks if ASOF mode is supported by resource and includes such elements in method definition.

Files changed:

rdf_metadata/rdf_meta_utils.py
ontology/reference/issuerrole-models.json
expected-results/issuerrole_resource_output.json

SDP-63113 EDS issuerrating extracts was enhanced to support a new filter

A new filter called “reason” was added to exclude Canceled Ratings or any other unwanted records from issuerrating extract. It appeared there was no such filter neither as part of resource nor in eagledb-crud-filters, It is supported now.

Usage:

<taskParameter>
<name>reason</name>
<dataType>S</dataType>
<value>E(LzlsEnHa)</value>
</taskParameter>

Files changed:

ontology/eagledb-crud-filters.json
ontology/reference/issuerrating.json
expected-results/issuerrating_resource_output.json

Production Bugs

SDP-63892 The RelativeDelta Function was restored

The RelativeDelta function and import were added to exprbase.py

The Function was imported from external library and our own function wrapper was added to prevent code smell: “unused import”

Files changed:
common-core/lang_expr/exprbase.py

SDP-63877 EDS genericsmf was updated to improve update_date logic

The UPDATE_DATE logic is in sync with EagleML now.

SYSDATE is used to populate UPDATE_DATE field on insert and update.

Files changed:

metadata/core/ontology/reference/genericsmf-models.json
metadata/core/ontology/reference/genericsmf.json
metadata/core/ontology/reference/genericsmf_in.json

SDP-63728 The Issuer Rating Inbound interface was fixed to allow empty submittedDate field

The empty submittedDate field in the incoming file caused an error in the Stored Procedure due to an incorrect pasring of the field. The logic for submittedDate field was fixed to allow an empty value.

Files changed:
eagle_default/in/xml/xml-ref_issuerrating.xml

SDP-62892 Error is not being raised when extension data is not passed

There was an issue when an empty extension model was passed in incoming message like this as shown below.

<customExtensions>
                <custom_wrhscashactExt>
                </custom_wrhscashactExt>
</customExtensions>

This issue is now fixed and the error is not being raised.

File changed:
eagle_default/in/xml/include/wrhs_ext_custom_fields.inc

SDP-64226 The Accounting Cash interface was updated to improve performance for Cancel Trades

The changes in SDP-61374 resulted in a performance degradation due to an incorrect query which is used to check if a record exists in the database. This query was fixed and bind parameters were added which resulted in equal and improved performance.

Files changed:
eagle_default/in/xml/xml-acct_cash_cancel.xml

SDP-63714 The spdDeltaTriggerLog extract sql query was fixed for data from different partitions

The extract with entityId filter was not working as expected for entities which are from partitions that are not the primary partitions. As a result, multiple objects could have been extracted from the PRICE_FX_EXCEPTION table. This issue was caused by query generation that works incorrectly for data from nonprimary partitions.

Related file:
eagle_ml-2-0_cm/out/tagvalue/extract_spd_delta_trigger_log.inc

SDP-64275 Warehouse Trade outbound interface supports extensions extract

Database fields were being incorrectly collected by the extension table due to issues with the building of those queries.

File changed:
eagle_ml-2-0_cm/extract_extension_table_fields.inc