Enhancements

SDP-79796 Data Rules logic was enhanced to support SLA for all data rules models – History and Composite

The Data rules logic was enhanced to allow a smooth integration with SLA’s for any DYNSQL model. The file name pattern for the data rules history extracts was changed on the Eagle side and the history files will have the ASOF suffix added to the resource name.
For example:
GENERICSMFASOF.* - the file mask for history data
GENERICSMF.* - the file mask for composite data
The schema_mappings and feed_resolvers have to be updated via the Settings User Interface to support the new file masks. The new file masks should be added under the new feeds with suffix _hist, for example genericsmf_hist. New feeds are linked to the same resourcename via the schema_mappings, enabling the genericsmf_hist feedtype to trigger the genericsmf resource to upload data. The EDS Service logic was changed to not resolve the resourcename as a feed-type from the incoming payload and instead to save the feed-type as is and add the resourcename field to preserve the original resourcename. The Message to SLA was updated and the feed-type field will be the same with other services. As a result, all history loads will be reported in all pipeline services as _hist feed-type, e.g. genericsmf_hist, while composite loads will be reported as usual, e.g. genericsmf.

Important Note:

The latest DYNSQL HISTORY models should be:

deployed
republished
previously setup SLA Jobs for Data Rules History models have to be resaved

Files changed:
msgcenter/eagle_ml-2-0_workflows/eagle_wrf_cloud_extract.wrf
msgcenter/eis/ejmwf/eagle_wrf_cloud_extract.ejmwf
svc-eds

job_service.py
asset_catalog_service.py
file_load_utils.py

sd-component-eds

eagleinvsys/etl/conversion_main.py

ce-common-core-py

common_core/enums/cm_enum.py
common_core/enums/def_enums.py
common_core/lang_expr/exprbase.py
common_core/processing_env/instructions.py

SDP-79331 SECURITY_TENOR table is removed from genericSMF extract

The SECURITY_TENOR table was removed from the genericSMF extract as this join was causing performance issues. This applied both for MC and MC2 extracts. It is recommended to use EDS instead of EagleML if the client requires to extract data from SECURITY_TENOR table. Changes for EDS were done in the June 29, 2023 Release as part of SDP-74719: GenericSMF Extract - Duplicate records extracted due to join to security_tenor tableCLOSED.

Files changed:
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/tagvalue/extract_smf.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/xml/t_smf.xml
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/xml/t_smf_streaming.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/xml/t_smf_streaming_eml.inc

SDP-79271 The exchangeprice history mode was disabled by default

The BNYM_EAGLE_DYNSQL_EXCHANGEPRICE_HISTORY model is no longer generated. The eagle_wrf_cloud_extract workflow was updated with a new flag, usearchtables, set to N to skip the history extract for the exchangeprice resource by default.

Files changed:
eagle_ml-2-0_workflows/eagle_wrf_cloud_extract.wrf
metadata/core/ontology/vault/eagle-vault-model.json
rdf_metadata/rdf_logical.py

SDP-79226 EntityAccountingPeriods enhancement for SP mode

Two elements were added to the EntityAccountingPeriods load interface for SP mode.

EagleML/entityTransaction/entityAccountingPeriods/acctPeriodFreq - Tag 624 - in_acct_period_freq flag in SP.
EagleML/entityTransaction/entityAccountingPeriods/acctPeriodWeekEndDay - Tag 1943 - in_acct_period_week_end_day flag in SP.

The acctPeriodWeekEndDay applies weekdays as values like 'Monday' and works only when acctPeriodFreq is set to 'W'.

Files changed:
metadata/bind/eagleml_ent_accountingperiods.xml
msgcenter/eagle_default/in/xml/xml-ent_entityaccountingperiods.xml

SDP-79076 Added support for config variables in the server rules for the amq monitor for EA Hosted regions only

Enhanced link within configuration settings to support amq monitor parameters to support virtual Ips. The variable SMTPSERVER was changed to support the configured value from the custom configuration. Additionally, for amq monitor rule variables, the SENDMAILTO and SMTPPORT have been changed similarly.

These variables can be set in the w_config_custom.inc:

Files changed:
eagle_mc/in/tagvalue/eagle_mc_server_support_amq_monitor.xml
eagle_mc/installer/out/csv/eagle_mc_server_support_environment_include.xml

SDP-79025 The warehousenav resource was enhanced to support load for Oracle

The warehousenav EDS resource has been enhanced to now support loading to Oracle using the core stored procedure HOLDINGDBO.EGL_GEN_HOLDING.MFP_NAV_PERF_EGL_GEN.

Files changed:
src/eagleinvsys/rdf_metadata/rdf_meta_utils.py

SDP-78520 Update to the AccountingTrade rebook logic for Forward Trades to prevent duplicates

The rebook logic for FWD trades has been enhanced in conjunction with the allowAutoRebookFlag flag and new keys specific to forwards. The new rebook logic will look to the allowAutoRebookFlag option and if not set, will create a new trade even if the keys on the data file are the same as a trade in the database. If this flag is set to Y, the buyQuantity for the Buy leg - Tag 316 and sellQuantity for the Sell leg - Tag 366 will be used instead of the sharesParQuantity - Tag 40 during the comparison to know if the trade already exists and if it already exists will trigger a cancel and rebook.

EagleML elements used in rebook comparison logic for FWD trades:
EagleML/accountingTransaction/accountingTrade/tradeDate
EagleML/accountingTransaction/accountingTrade/commission
EagleML/accountingTransaction/accountingTrade/otherFee
EagleML/accountingTransaction/accountingTrade/otherFeeSettlementDate
EagleML/accountingTransaction/accountingTrade/buySellIndicator
EagleML/accountingTransaction/accountingTrade/brokerCode
EagleML/accountingTransaction/accountingTrade/settlementCurrency
EagleML/accountingTransaction/accountingTrade/forward/buyQuantity
EagleML/accountingTransaction/accountingTrade/forward/sellQuantity

Also, tag mapping is fixed for otherFeeSettlementDate - Tag 9357.

Files changed:
eagle_default/in/include/accttrade_true_rebook.inc
eagle_default/in/include/csv-fxfwd_columns.inc

SDP-76576 Added mc2 load support for the eagle_ml-2-0_default_cm_acquire_data stream

The load via the eagle_ml-2-0_default_cm_acquire_data stream was enhanced to support processing via mc2 engines. For use, the MC2Load RTR parameter was introduced
<taskParameter>
<name>MC2Load</name>
<value>Y</value>
</taskParameter>

A global configuration option, W_MC2_LOAD, may be specified as well in the w_config_custom.inc.

Files changed:
eagle/estar/EAGLE_EXTRACTSERVICE.pkg
eagle/estar/EAGLE_ML_2_0.pkg
eagle/estar/eagle_default_runstream_mc2.pkg
eagle_ml-2-0_cm/mc2/runstream/runstream.rsf
msgcenter/eagle_ml-2-0_cm/mc2/runstream/runstream.rsf
eagle_ml-2-0_cm/out/tagvalue/parallel_exec.xml
eagle_ml-2-0_cm/out/xml/task_reporter_generate_task_unit.inc

Production Bugs

SDP-79835 The Conversion Workflow was improved to use the fromdate/todate parameters in the SQL query extract

The SQL extract was using the fromeffectivedate and toeffectivedate parameters which was incorrect since the reportdictionarysummary resource did not have the effectivedate field.

The logic was corrected to use the fromdate/todate instead when applying the date time filter as part of the SQL extract for the below resources:

entitylist
entitycharacteristics
entityshareclass
languagetranslation
reportdictionarysummary

Files changed:
metadata/core/ontology/entity/entitycharacteristic.json
metadata/core/ontology/entity/entitylist.json
metadata/core/ontology/entity/entityshareclass.json
metadata/core/ontology/reference/languagetranslation.json
metadata/core/ontology/reference/reportdictionarysummary.json

SDP-79373 Fixed an issue with mc_installer

There was an issue with the Eagle Installer as the installation process hung due to a missing file in the package. The referenced include, waiting_task_reporter.inc, was not referenced in the master packages, such as eagle_mc_installer.pkg
The packages were fixed to list this include. The manual import of the Installer package for a first-time installation now consists of all needed files and references.

Files changed:
eagle/estar/EJM.pkg
eagle/estar/eagle_mc_installer.pkg

SDP-79774 Data Rules SLA was breached for warehouseposition load after the latest release deployment

Unmatched records were detected in PROD for the HOLDINGDBO.POSITION_DETAIL on ingestion to Snowflake. This regression issue was introduced by new logic added for the field POSITION_DETAIL.BASE_CURRENCY resulting in the warehouseposition extract working incorrectly. The issue was identified and corrected.

Files changed:
metadata/core/ontology/warehouse/warehouseposition-models.json

SDP-79705 GenericEntity load interface enhancement for mfpTaxableIncomeFlag

The element, mfpTaxableIncomeFlag - Tag 2447, was loading directly to the panel or SP without changing values Y to 1 and N to 0 which led to issues because values are expected to be 0 or 1 in database SP logic. Also, there is a transformation on the extract where the change from 1 to Y and 0 to N are applied during the extract. To be consistent between the load and extract, the tag mapping in load interface was changed for accounting and SP modes and now it changes values Y to 1 and N to 0 during load.

Files changed:
dynamic\metadata\bind\eagleml_ent_genericentity.xml
eagle_default\in\include\csv-entity_columns.inc

SDP-78477 The entitypolicy resource was corrected to use correct dates for data conversion

Previously the fromeffectivedate and toeffectivedate were used for data conversion. The execution plan for entitypolicy is currently based on the fromdate and todate which is correct.

The logic was corrected to use the fromdate/todate instead when applying the date time filter as part of the SQL extract.

Files changed:
metadata/core/ontology/entity/entitypolicy.json

EagleML and EJM

EagleML Release Notes - November 16, 2023