EagleML Release Notes - January 2025

Enhancements

SDP-85645 Logging System Errors on Ingestion To Snowflake

Added log for the systems errors.

Files Changed:
sqlbase.py

SDP-86234 Data Rules genericentity resource was updated to support new fields

The genericentity resource was updated to support the load and the extract for the additional fields.

Database Field Canonical Element
EQUALIZATION_CALCULATION equalizationCalculation
SWAP_PRICING_METHOD swapPricingMethod

Files Changed:
eagle-vocab-entity.json
eagle-vocab-definition-entity.json
eagle-vocab-enumerations.json
genericentity-models.json

SDP-86238 Enhanced Data Rules exchangeprice resource

Added mapping for the following fields:
SECURITYDBO.PRICE_EXCHANGE.PX_SETTLE_ACTUAL - actualSettlementPrice
SECURITYDBO.PRICE_EXCHANGE.PX_SETTLE_ACTUAL_DATE - actualSettlementPriceDate
SECURITYDBO.PRICE_EXCHANGE.MARKET_STATUS - marketStatus
SECURITYDBO.PRICE_EXCHANGE.FII_LIMIT - foreignInstitutionalInvestorLimit
SECURITYDBO.PRICE_EXCHANGE.CLEAN_DIRTY_FLAG - cleanDirtyFlag
SECURITYDBO.PRICE_EXCHANGE.EXCHANGE_LISTED_FLAG - exchangeListedFlag
SECURITYDBO.PRICE_EXCHANGE.HOLIDAY_FLAG - holidayFlag

Files changed:
metadata/core/ontology/reference/exchangeprice-models.json
metadata/core/ontology/warehouse/warehousetrade-models.json
metadata/core/ontology/templates/eagle_ref_exchangeprice_insertupdate.txt
metadata/core/ontology/definition/eagle-vocab-definition-ref.json
metadata/core/ontology/eagle-vocab-enumerations.json
metadata/core/ontology/eagle-vocab-ref.json

SDP-86366 Data Rules genericsmf resource updated to support new fields

The genericsmf resource was updated to support the load and the extract for the additional field and altered the column size for two existing columns: SECURITYDBO.FIXED_INCOME_EXT.CALL_TYPE and SECURITYDBO.MBS.POOL_TYPE.

Database Field Canonical Element
LAST_CPN_PERIOD_TYP lastCouponPeriodType

Files Changed:
eagle-vocab-sec-shared.json
eagle-vocab-definition-sec-shared.json
genericsmf-models.json

SDP-87863 DBT SQL enhancements

Enhanced DBT generator for UPDATE_DATEs
Fixed securityAlias mapping for custombenchmark resource.

Files Changed:

ontology/entity/custombenchmark-models.json

SDP-90571 Batch Identifier Added to the Contribution and Withdrawal Cash Transaction Types

The AccountingCash inbound interface was enhanced with the addition of a new, optional batchEventId element - Tag 701. This change ensures alignment with the Cash Contribution and Cash Contribution panels, corresponding to the CONTRIBUTION and WITHDRAWAL cashTransactionType respectively.

Files changed:
eagle_default/in/include/cash_aliases.inc
eagle_default/in/include/csv-cash.inc

SDP-90639 GenericSMF Inbound Interface Enhanced With New Elements

New elements were added to the GenericSMF Inbound Interface, for both Accounting and Stored Procedure modes:
referenceTransaction/genericSMF/MBSModel/originationOvercollateralPercentage - Tag 18633 Database: MBS.ORIG_CREDIT_SUPPORT
referenceTransaction/genericSMF/MBSModel/originationBalloonPaymentPercentage - Tag 18634 Database: MBS.IO_FULL_ORIG
referenceTransaction/genericSMF/MBSModel/currentOvercollateralPercentage - Tag 18635 Database: MBS.MTG_CURR_DEAL_OVER_COLLAT
referenceTransaction/genericSMF/FixedIncomeModel/muniIssueType - Tag 18636 Database: MUNI.MUNI_ISSUE_TYP
referenceTransaction/genericSMF/futuresLastDeliveryDate - Tag 18638 Database: DERIVATIVES_EXT.FUT_DLV_DT_LAST

Files changed:
eagle_default/in/xml/include/derivative_ext.inc
eagle_default/in/xml/include/fixedincomesmf_model.inc
eagle_default/in/xml/include/ins_upd_smf_req.inc
eagle_default/in/xml/include/mbs_model.inc
eagle_default/in/xml/include/smf_aliases.inc
eagle_default/in/xml/include/smf_panels.inc
eagle_default/in/xml/xslt/smf_egl_gen_interface_template.xsl
eagle_default/in/xml/include/derivative_ext.inc

SDP-90924 New Eagle Delta storage was introduced in Data Rules to store delta in Eagle Database

New storage and new logic was added to save delta information into the Eagle database for Data Rules. The delta info is saved into the PACE_MASTERDBO.ORCH_REQUEST_DEF and PACE_MASTERDBO.ORCH_REQUEST_PARAMS tables in Oracle (similar to EagleML).
By default, delta is stored in the Eagle database instead of Redis now. The new parameter, useorchfordelta, allows to switch delta storage from the Eagle database to Redis storage.
New logic is supported only for Data Rules extracts using the eagle_wrf_generic_extract and eagle_wrf_cloud_extract workflows (with ExtractLevelDeltaMode=Y flag) and direct extract via EQL.

Example of usage in profile for the eagle_wrf_generic_extract workflow:
ebs:default:DeltaBased - Y
ebs:default:ExtractLevelDeltaMode - Y
ebs:default:useorchfordelta - N

Example of usage in profile for the eagle_wrf_cloud_extract workflow:
DeltaBased - Y
ExtractLevelDeltaMode - Y
useorchfordelta - N

Files changed:
pyrules/eagleinvsys/common_core/dataset/dataset_cache/delta_extract_info_store.py
pyrules/eagleinvsys/common_core/dataset/idsdataframe_delta.py
pyrules/eagleinvsys/common_core/enums/cm_enum.py

Files added:
pyrules/eagleinvsys/common_core/storage/db_delta_storage.py
pyrules/eagleinvsys/common_core/storage/eagle_delta_sql.py

Workflows changed:
eagle_wrf_generic_extract
eagle_wrf_cloud_extract

SDP-91138 Expanded support for Private Equity Processing - EQCBFI and EQSBFI

Added support for processing security type EQCBFI and EQSBFI within the EagleML SMF stream.

Files changed:
eagle_default/in/xml/include/smf_panels.inc

SDP-91226 Added Ability to Include the Business Task Id as part of the Email Subject

Enabled the ability to add the Business Task Id to the Email Subject through a configuration.

It may be applied by adding the parameter AddBusinessTaskIdToEmailSubject within the RTR or setting the W_ADD_BUSINESS_TASK_ID_TO_EMAIL_SUBJECT attribute to Y within the w_config_custom.inc.

<COL TAG="W_REPLACE_WITH_XID" EXPRESSION="'Y'"/>

or

Example of subject: EJM Notification: ExtractData. Business Task Id: TEST_ENTITY_EXTRACT#eagle_ml20_default_cm_file_distribution Process Correlation Id: TEST_ENTITY_EXTRACT_N4TM8F7DFVV8CRC5 Task TR7GRIIN2SAC1029

Files changed:
eagle_ml-2-0_cm/collect_task_parameters.inc
eagle_ml-2-0_cm/ejm_workflow_config.inc
eagle_ml-2-0_cm/get_w_state.inc
eagle_ml-2-0_cm/mc2/ejm/distribution/mc2_email_distribution.inc
eagle_ml-2-0_cm/mc2/mc2_check_task_param.inc
eagle_ml-2-0_cm/mc2/mc2_create_w_state_str.inc
eagle_ml-2-0_cm/out/tagvalue/execute_wrkfl_event.xml
eagle_ml-2-0_cm/out/xml/task_distributor_email.xml
eagle_ml-2-0_cm/run_wrf_instance.inc
eagle_ml-2-0_cm/upd_w_state.inc

SDP-91417 New logic for Data Rules Delta Extract was introduced

The resetdeltatime Service Method was introduced to reset the start time for the next delta extract. If the fromdate parameter is specified, it will be used as the new start time, otherwise the current time will be used.

The resetdeltatime parameter was introduced to reset the start time of current delta extract.

The timeoffset parameter was introduced to allow setting the time offset in seconds for delta extracts. Using this parameter, it is possible to move the start time of an extract back.

The new logic is supported only for Data Rules extracts using the eagle_wrf_generic_extract and eagle_wrf_cloud_extract workflows (with ExtractLevelDeltaMode=Y flag) and direct extract via EQL.

Example of timeoffset usage in RTR:

<taskParameter>
<name>timeoffset</name>
<dataType>S</dataType>
<value>600</value>
</taskParameter>

Example of resetdeltatime usage in RTR:

<taskParameter>
<name>resetdeltatime</name>
<dataType>S</dataType>
<value>Y</value>
</taskParameter>
<taskParameter>
<name>fromdate</name>
<dataType>S</dataType>
<value>2024-10-11 23:00:00</value>
</taskParameter>

Example of resetdeltatime Service Method usage:

<taskParameter>
<name>ServiceName</name>
<value>orchestration</value>
</taskParameter>
<taskParameter>
<name>ServiceMethodName</name>
<value>resetdeltatime</value>
</taskParameter>
<taskParameter>
<name>ResourceName</name>
<value>genericsmf</value>
</taskParameter>
<taskParameter>
<name>DeltaBasedUID</name>
<dataType>S</dataType>
<value>SDP91417_CLOUD</value>
</taskParameter>
<taskParameter>
<name>fromdate</name>
<dataType>S</dataType>
<value>2024-09-01 23:10:10</value>
</taskParameter>

Example of usage in a profile for the eagle_wrf_generic_extract workflow:
ebs:default:DeltaBased - Y
ebs:default:ExtractLevelDeltaMode - Y
ebs:default:resetdeltatime - Y
ebs:default:timeoffset - 600

Example of usage in a profile for the eagle_wrf_cloud_extract workflow:
DeltaBased - Y
ExtractLevelDeltaMode - Y
resetdeltatime - Y
timeoffset - 600

Files changed:
pyrules/eagleinvsys/common_core/dataset/ds_filters/filters.py
pyrules/eagleinvsys/common_core/dataset/idsdataframe_delta.py
pyrules/eagleinvsys/common_core/enums/cm_enum.py
pyrules/eagleinvsys/common_core/enums/def_enums.py
src/eagleinvsys/etl/etl_main.py

Workflows changed:
eagle_wrf_generic_extract
eagle_wrf_cloud_extract

SDP-91447 Refactored concurrency for the Conversion Process

The Conversion process was improved to use more connections to the database to run queries for analysis in parallel. Previously, only one connection was used and all queries were executed on this single connection, which could be slow. The change now establishes up to 4 connections to speed up the execution.

Files Changed:
common_core/dataset/metadata/eagledb_query_generator.py
eagleinvsys/etl/db_physical_info.py

SDP-91865 RDC Pricing Report Extract - PRICINGRDCREPORTEXTRACT Updated with New Pricing Status Table

As part of the changes in RDC with RDC Pricing now using the RDC_PRICE_STATUS table in place of the PRICE_STATUS, the RDC Pricing Report extract - PRICINGRDCREPORTEXTRACT was updated. Changes are applied from v17.R2.51

Files changed:
eagle_ml-2-0_cm/out/tagvalue/extract_pricing_rdc_report.inc

SDP-91936 Conversion Build Plan for Physical Tables was Improved for Warehouse Performance

The conversion build plan for physical tables worked slowly when master tables were connected with two levels of detail tables.
FROM PERFORMDBO.PERF_SUMMARY MASTERTABLE
INNER JOIN PERFORMDBO.PERF_SEC_RETURNS on MASTERTABLE.PERF_SUM_INST = PERF_SEC_RETURNS.PERF_SUM_INST
INNER JOIN PERFORMDBO.PERF_SEC_RETURNS_EXT on
PERF_SEC_RETURNS_EXT.PERF_SUM_INST = MASTERTABLE.PERF_SUM_INST and
PERF_SEC_RETURNS_EXT.SECURITY_ALIAS = PERF_SEC_RETURNS.SECURITY_ALIAS and
PERF_SEC_RETURNS_EXT.PERF_ROLLUP_RETURNS_ID = PERF_SEC_RETURNS.PERF_ROLLUP_RETURNS_ID and
PERF_SEC_RETURNS_EXT.STRATEGY_DETAIL_ID = PERF_SEC_RETURNS.STRATEGY_DETAIL_ID

To improve the performance, the master table is revised to connect to the last detail table using the primary key of the master table (PERF_SUM_INST).
FROM PERFORMDBO.PERF_SUMMARY MASTERTABLE
INNER JOIN PERFORMDBO.PERF_SEC_RETURNS_EXT on MASTERTABLE.PERF_SUM_INST = PERF_SEC_RETURNS_EXT.PERF_SUM_INST

Files changed:
rdf_physical_outflow.py

SDP-92484 Expanded Support for Private Equity Processing

Expanded support in the Accounting Trade inbound interface for Private Equity Processing - EQCBFI by including additional elements to directly adjust the funded commitment and to directly specify the balance of accumulated capitals calls at the time of the initial commitment. The tags are mapped directly without calculations:

EagleML Xpath:
EagleML/accountingTransaction/accountingTrade/fundedCommitmentLocal - Tag 18386
EagleML/accountingTransaction/accountingTrade/fundedCommitmentBase - Tag 18387
EagleML/accountingTransaction/accountingTrade/capitalCallsBase - Tag 18449
EagleML/accountingTransaction/accountingTrade/capitalCallsLocal - Tag 18450

Files changed:
eagle_default/in/include/trade_aliases.inc
eagle_default/in/include/csv-capfundinvestments_columns.inc

SDP-92547 Updated mtgStartPrincplWndwMatu Element From String to Date Format

The element EagleML/referenceTransaction/genericSMF/MBSExtension/mtgStartPrincplWndwMatu was changed from String to Date to align core changes in 2017R2.51 for the SECURITYDBO.MBS_DETAIL_EXT.MTG_START_PRINC_WNDW_MTY and MBS_DETAIL_EXT_HIST.

Files changed:
eagle_default/in/xml/include/mbs_extension.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/xml/xslt/smf_egl_gen_interface_template.xsl
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/xml/xslt/smf_egl_gen_interface_template_mssql.xsl
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/xml/xslt/smf_genericsmf_debug.xsl

SDP-92561 Data Rules genericsmf Datatype Update for MTG_START_PRINCIPAL_WNDW_MATU

The Data Rules Resource genericsmf column SECURITYDBO.MBS_DETAIL_EXT.MTG_START_PRINCIPAL_WNDW_MATU now supports DATE values and was updated from Varchar2(30) to DATE format.

Files changed:
ontology/reference/genericsmf-models.json

SDP-92754 Special parameters to include NAIC elements into SMFEXTRACT

The SMFEXTRACT interface was enhanced and only returns data from the SECURITYDBO.NAIC when both the RTR and global parameters are enabled (set to 'Y'):

• W_SMFEXTRACT_INCLUDE_NAIC - global parameter that should be set in w_cofig_custom.inc file.

• includenaic - RTR parameter that should be set in extract RTR‌

Files changed:
eagle_ml-2-0_cm\out\tagvalue\extract_smf.inc
eagle_ml-2-0_workflows\eagle_wrf_generic_extract.wrf
eis\ejmwf\eagle_wrf_generic_extract.ejmwf

Production Bugs

SDP-88497 Control TSR Messages Delivery Options for MC2 Single Extracts

Revised the processing of TSR messages for MC2 single extracts to allow the management of the TSR messages delivery. Applying the parameter, w_default_send_notifications, may be set to ONFAILURE, NEVER, or ALWAYS within the w_custom_config.inc or be sent in the RTR.

Options include:
‘ALWAYS’ or ‘1’ to enable
‘NEVER’ or ‘0’ to disable
‘ONFAILURE’ to send notification on failure case only \(TSR has ‘FAILED’ final status\).
Sample examples:
Within the w_custom_config.inc:

RTR parameter SendNotification:

Files changed:
eagle_ml-2-0_cm/mc2/ejm/distribution/mc2_distribution.inc

SDP-89481 EagleML Delta Extracts Enhanced with Restoring the Schedule Record

For all the feeds that support Delta processing, there was a delay after the purge process. The delay issue was caused by broken data integrity as the purge process eliminated the schedule record for the Delta ID of extracts. Attempts to update the record raised a database error and cyclic retrial. To resolve, the delta functionality has been enhanced to recreate the schedule record if it does not exist.

Files changed:
eagle_ml-2-0_cm/out/tagvalue/extract_delta_orch.inc

SDP-90772 Added support for ForceExtractFileCreation parameter in MC2

Added support for the ForceExtractFileCreation parameter in MC2 which enables the creation of a file when no data is returned.

Files changed:
estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/xml/extract_service_result_ejm.inc

SDP-90827 Routing Logic for TBA's for BUYCVR Transactions

Added routing logic in the eagle_ml-2-0_default_in_xml_acct_trades stream to support the event type BUYCVR for TBA's enabling the DBTATA Processing Security Type to post to the Sell / Close Trade panel. Support already existed for the SELL and the TBACLOSE event types.

Files changed:
eagle_default/in/include/accttrade_routing_check.inc

SDP-91095 EagleML EntityToComposite Inbound Interface Modified to Check if only one Record Exists

When loading the EntityToComposite EagleML object, the error ‘More than one record exist in RULESDBO.ENTITY_DETAIL table with such key as entity_id, entity_detail_id.’ occurred. The issue was a result of the EntityToComposite lookup returning a full list of entities linked to the composite.

The lookup was modified to return only one record for specified Composite ID and Entity ID.

File changed:
eagle_default/in/xml/xml-ent_entitytocomposite.xml

SDP-91140 Entity Bank Resolution redesigned for the EagleML Voluntary CA Election inbound interface

The custodianBankCode, custodyBankAccount and custodyBankSubaccount are required elements for Entity Bank relationship resolution. It was needed to pass all three elements to resolution take attempt.

Resolution was redesigned to attempt to resolve if the custodianBankCode or custodyBankAccount are passed only. If just one of these elements is passed and no matches are found or if the lookup returns multiple records, it will fail by design.

Files changed:
eagle_default/in/xml/include/xml-ref_voluntarycaelections.inc

SDP-91251 Mapping xreferences with special name to named tags.

Added ability to map Xreference with specific names (e.g. ISIN, SEDOL) related to an XML tag. The TICKER Xreference is also mapped to Tag 13.

This functionality can be enabled by either of two ways:

1. Set flag W_REPLACE_WITH_XID = Y in the w_config_custom.inc

2. Add processing option REPLACE_WITH_XID in the incoming message

The variable for the configuration may be set up similar to below:

When using the processing option, the incoming message should include in the header:

Files changed:
eagle_default/in/xml/include/generic_smf_fields_logic.inc
eagle_default/in/xml/include/identifier_reenumeration.inc

SDP-91502 Pricing Report extract - PRICINGREPORTEXTRACT modified to clear Database Bind Parameters

When executing the PRICINGREPORTEXTRACT, the error “\Data query is not executed. ErrorCode: -2. ErrorMessage: ORA-01036: illegal variable name/number”\ was occurring. The issue was that bind parameters used as part of the process were reused within the extract query. During processing, database bind parameters are now cleared out per execution to avoid this contention.

Additionally, the Pricing Report swagger was enhanced with Entity and Usergroups filters

Files changed:
eagle_ml-2-0_cm/out/tagvalue/extract_save_and_execute_query.inc
eagle_ml-2-0_cm/out/tagvalue/extract_pricing_report.inc
eagle_ml-2-0_cm/out/tagvalue/extract_pricing_rdc_report.inc
eagle/estar/tpe/dynamic/eagleml/swagger/swagger-extracts.json

SDP-91630 dLogic for dbcall Element was Fixed

There was an error when the dbcall element process included INSERT or UPDATE statements without output parameters.
The logic for the dbcall element was fixed to not fail when INSERT or UPDATE statements are processed without output parameters.

Files changed:
src/eagleinvsys/etl/core_etl/genetl_dbcall.py