EagleML Release Notes - September 19, 2024

Owned by Larry Ackerman
Last updated: Nov 14, 2024 by Kinga Huszno
6 min read

Enhancements

SDP-81873 / SDP-87341 New Data Rules Resource to Support PERFORMDBO.PERF_APPROVAL_JOURNAL database table

New resource, warehousePerformanceSummaryStatus was added to support data extract from PERFORMDBO.PERF_APPROVAL_JOURNAL.
Extract will pull records updated in the table PERF_APPROVAL_JOURNAL via Performance Wizard User Interface.
The ingestion to Vault process will update the field STATUS_FLAG in the PERFORMDBO.PERF_SUMMARY table by using data from table PERF_APPROVAL_JOURNAL.
The DeltaBatch filter works with Performance Journal UpdateDate field - PERF_APPROVAL_JOURNAL.UPDATE_DATE instead of UpdateDate field from the main table - PERF_SUMMARY.UPDATE_DATE. It allows to select changes reflected in journal table only.
Load interface will be added later.

Files changed:
eagle-vocab-enumerations.json
eagledb-inventory.json
vault\eagle-vault-model.json
eagle-vocab-wh.json
definition/eagle-vocab-definition-wh.json
warehouseperformancesummarystatus.json
warehouseperformancesummarystatus-models.json

SDP-86216 Required fields mapped in Generic Transform Workflow

Generic Transform Workflow is updated for the Generic SMF. Filemask for incoming csv files is

_CSV2XML_GENERICSMF_

Fields added to the workflow:

  • noLoadValidationFlag (F1081): if set to 'NONE' in csv, xml will have noLoadValidationFlag = ‘Y' , else 'N’

  • action (F1076): if set to ‘Y' or ‘B’ in csv, xml will have action = ‘UPDATE’, else 'ADD’

  • security resolution logic was enhanced to replicate csv processing logic

    • F70XX tags is used for resolution in separate security lookup before transform

    • security resolution lookup supports all resolution options used in csv rulefile

    • securityAlias is passed into transformed data

    • F19XX tags are mapped into corresponding xreference tags in transformed xml data

    • to avoid double resolution, assetResolutionOptions is set to SKIP

Files changed:

/eagle_ml-2-0_workflows/eagle_wrf_csv_smf_generic.wrf
/eagle_ml-2-0_workflows/eagle_wrf_csv_smf_generic.map
/eis/ejmtsk/eagle_wrf_csv_smf_generic_tsk/eagle_wrf_csv_smf_generic_tsk.ejmstr
/eis/ejmwf/eagle_wrf_csv_smf_generic.ejmwf
/pyrules/custom/eagle_wrf_csv_smf_generic_tsk/eagle_wrf_csv_smf_generic_tsk.py
/eagle_wrf_csv_smf_generic_tsk.pkg

SDP-85599 EagleML Global Tax Rate interface was enhanced with the doubleTaxReclaimRate element for both Load and Extract

New element doubleTaxReclaimRate supported within the Global Tax interface for load - GlobalTaxRate and extract - GLOBALTAXRATEEXTRACT.

EagleML Xpath:
EagleML/referenceTransaction/globalTaxRate/doubleTaxReclaimRate

DB field:
RULESDBO.GLOBAL_TAX_RATE - Tag 18597
RULESDBO.GLOBAL_TAX_RATE_HIST

Files changed:
eagleml/schema/eagleml-sec-shared-2-0.xsd
eagle_default/in/xml/xml-ref_global_tax_rate.xml
eagle_ml-2-0_cm/out/xml/t_globaltaxrate_streaming.inc
eagle_ml-2-0_cm/out/tagvalue/extract_globaltaxrate.inc

SDP-87774 Data Rules Security Resolution Service was Enhanced

The logic for default asset resolution mode ALL and for INVESTMENT_TYPE option was fixed.

Files changed:
metadata/core/ontology/templates/includes/defaultassetresolution.json
metadata/core/ontology/eagle-resolution.json

SDP-87658 Support of SHARE_CLASS Added for WarehouseTrade resource

Support of the field TRADE_DETAIL.SHARE_CLASS was added for Eagle DB Model version >= '2017.2.44.0'.
XSD name is: shareClass.

Files changed:
warehousetrade-models.json

SDP-87458 Update generic load workflow to be able to control Processing statuses message limit

procstatusmsgsize - parameter was added to the all generic load steps, in order to be able to control processing status message limit. It can be useful for troubleshooting tasks, when error message is too big and there is a need to extend it.

Files Changed:
estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/set_eds_params.inc
estar/tpe/dynamic/msgcenter/eagle_ml-2-0_workflows/eagle_wrf_generic_load.wrf

SDP-86848 Data Rules Profile Templates for Conversion and Cloud Workflows for New Resources

Added Data Rules profile templates for the Conversion and Cloud Workflows for warehousetradecharges and warehouseperformancesummarystatus resources.

Files added:
msgcenter/profile/workflows/EagleToVault-cloud-warehousetradecharges_tmp.profile
msgcenter/profile/workflows/EagleToVault-cloud-warehouseperformancesummarystatus_tmp.profile
msgcenter/profile/workflows/EagleToVault-conversion-warehousetradecharges_tmp.profile

Files changed:
eagle/estar/EAGLE_EXTRACTSERVICE.pkg
eagle/estar/eagle_default_workflows.pkg

SDP-87458 Update generic load workflow to be able to control Processing statuses message limit

Added Data Rules profile templates for the Conversion and Cloud Workflows for warehousetradecharges and warehouseperformancesummarystatus resources.

Files added:
msgcenter/profile/workflows/EagleToVault-cloud-warehousetradecharges_tmp.profile
msgcenter/profile/workflows/EagleToVault-cloud-warehouseperformancesummarystatus_tmp.profile
msgcenter/profile/workflows/EagleToVault-conversion-warehousetradecharges_tmp.profile

Files changed:
eagle/estar/EAGLE_EXTRACTSERVICE.pkg
eagle/estar/eagle_default_workflows.pkg

SDP-87771 EagleML Entity History outbound interface was enhanced to support delta mode

The EagleML DeltaBased ENTITYEXTRACT in history mode is now supported.

Files changed:
eagle_ml-2-0_cm/out/tagvalue/extract_delta_based_param.inc

SDP-87546 EagleML MultilegSMF load updates to support accurate load of xreference identifiers for all legs

Previously, the xreference identifiers were not mapped into XMLDBAPI for Pay and Receive legs and incorrectly reset before processing Pay and Receive legs.

The include file, eagle_default/in/xml/tag_purge.inc, was moved in the multilegSMF rulefile to clean xreference variables after insert when the noLoadValidationFlag is set to Y or N. Now xreference identifiers are loaded correctly for all legs.

Files changed:
eagle_default/in/xml/xml-smf_multileg_swap.xml

SDP-87502 EagleML Voluntary Election inbound interface was enhanced with Entity Bank relationship resolution for update and delete operations

The EagleML Voluntary Election inbound interface was enhanced with the Entity Bank relationship resolution for update and delete operations.

The voluntaryCAElection feedtype support has been added to the eagle_wrf_genereic_delete workflow

An update to the ENTITY_BANK_ALIAS is not supported because it is part of the unique key for the Voluntary Corporate Action Election. If the alias is changed, then a new record is created.

Files added:
eagle_default/in/xml/include/xml-ref_voluntarycaelections.inc

Files changed:
eagle_default/in/xml/xml-ref_voluntarycaelections.xml
eagle_default/in/xml/t_eagle_ml-2-0_default_in_xml_delete_voluntary_ca_election.xml

SDP-87160 EagleML Warehouse Open Lot outbound interface was enhanced with custodian bank related elements

Custodian related elements were added to the EagleML Warehouse Open Lot - WRHSOPENLOTEXTRACT outbound interface:

warehouseTransaction/warehouseOpenLot/custodianBankCode
warehouseTransaction/warehouseOpenLot/custodyBankAccount
warehouseTransaction/warehouseOpenLot/custodyBankSubaccount

Files changed:
eagle_ml-2-0_cm/out/tagvalue/extract_warehouse_openlot.inc
eagle_ml-2-0_cm/out/xml/t_warehouse_openlot_streaming.inc

SDP-88254 EagleML Working Trial Balance outbound interface was enhanced with new date parameters

Four new date parameters have been added to comply with the Eagle Standard Report for the Working Trial Balance - WORKINGTRIALBALANCEEXTRACT:

Accounting Date Report type:
reportstartdate - is required parameter for this report type
reportenddate - is not required. If not provided it is defaulted as the current day

Month End Date Report type:
monthenddate - is required parameter for this report type
monthendreportstartdate - is not required. If not provided it is defaulted as the first day of the month passed in the ‘monthenddate’ parameter

Swagger UI has been modified accordingly.

Files changed:
dynamic/msgcenter/eagle_ml-2-0_cm/out/tagvalue/extract_acct_working_trial_balance.inc
dynamic/eagleml/swagger/swagger-extracts.json

SDP-88126 Data Rules resource vendorcashflow was updated to support UpdateDate filter

The UpdateDate filter is supported now for the Data Rules vendorcashflow resource and the Delta process is operational now.
Additional supported parameters are todate and fromdate.

Files changed:
vendorcashflow.json

SDP-87194 Data Rules Security Resolution Service was Enhanced

The logic for the tagNochangecase element and CASE_INSENSITIVE asset resolution option was synchronized to act the same way as for EagleML.
For the tagNochangecase element, a user can specify tags that are case sensitive and should be used as is in the asset resolution process.
The CASE_INSENSITIVE option is applied to Xreference Ids only. With this option, all Xreference Ids will be converted to uppercase regardless of the tagNochangecase element.

Files changed:
src/eagleinvsys/etl/core_etl/resolution_service.py
src/eagleinvsys/etl/core_etl/resolution_service_enum.py
metadata/core/ontology/templates/includes/defaultassetresolution.json
metadata/core/ontology/eagle-resolution.json

SDP-86331 Improve logic for mode alwayscommit=N

The logic for the alwayscommit = N mode has been enhanced to rollback committed records in case a validation issue is detected during the pre-load stage. Previously, these records were still loaded despite failing validation.

When alwayscommit = Y, the process attempts to load successful records in batches. Even if some fail within a batch, the remaining ones are processed.

When alwayscommit=N, successful records within a batch are rolled back, ensuring that no record from the batch appears in the position_detail (or any detail table).

Files Changed:
common_core/descriptor/dbdescriptor.py

SDP-86240 Metadata for entityshareclass resource was updated

The metadata synched with the Eagle Core database as the additional fields below are supported for load and extract:

Database Name Canonical Name

ENTITY_SHARE_CLASS.CLASS_CURRENCY classCurrency
ENTITY_SHARE_CLASS.MAJ_MINOR_NAV_QUOTE majorMinorNavQuote
ENTITY_SHARE_CLASS.MINOR_CURR_NAV_PRECISION minorCurrNavPrecision
ENTITY_SHARE_CLASS.NASDAQ_INST_TYPE nasdaqInstType
ENTITY_SHARE_CLASS.HEDGING_METHOD hedgingMethod

Files Changed:

metadata/core/ontology/definition/eagle-vocab-definition-entity-shared.json
metadata/core/ontology/eagle-vocab-entity-shared.json
metadata/core/ontology/entity/entityshareclass-models.json

SDP-85678 genericsmf resource metadata was updated to correct field type differences between source and target

The checksum functionality was raising an alert due to a field type mismatch between Eagle and Vault. The metadata for the genericsmf resource was corrected. The field type was change from Integer to Float for the following fields:

DERIVATIVES_EXT.CTD_COUPON
DERIVATIVES_EXT.SPEC_MARGIN_BALANCE_MIN
DERIVATIVES_EXT.SPEC_MARGIN_MIN

Files changed:
ontology/reference/genericsmf-models.json

SDP-86952 PACE object updated to support eHashLogin mode

New functionality is added for the Pace object: it now supports both the eLiteLogin mode and eHashLogin mode.

The variable for the login mode pace_login_mode can be set in the estar/tpe/dynamic/dataservices/configs/app_settings_target.json with one of two options:

eLiteLogin mode works as before: pace object will read username and password from vendor credentials config file (estar/tpe/dynamic/dataservices/configs/vendor_credentials.json)

eHashLogin mode needs only username, which is set in estar/tpe/dynamic/dataservices/configs/app_settings_target.json as pace_user_id variable

Files changed:
eagleinvsys/vendor/pace.py

SDP-85066 Data Rules support for warehousetradecharges resource

Data Rules warehousetradecharges resource was created to support data load into Oracle using core stored procedure - TRADESDBO.EGL_GEN_TRADE.TRADE_CHARGES_INSUPD
Resource also supports Eagle - Vault pipeline, extract data from Eagle and upload to Vault.

Supported modes: Batch, Temporal
Table: TRADESDBO.TRADE_CHARGES
PK: INSTANCE
Unique Key: TRADE_ID, CATEGORY
Batch Key: xrefAccountId + entityId^effectiveDate^sourceName^srcIntfcInst

Files changed:
metadata/core/ontology/vault/eagle-vault-model.json
metadata/core/ontology/warehouse/warehousetradecharges-models.json
metadata/core/ontology/warehouse/warehousetradecharges.json

Files added:
metadata/core/ontology/templates/db_oper/snowflake/warehousetradecharges.reinit_batch_many.sql
metadata/core/ontology/templates/eagle_wrhs_warehousetradecharges_insertupdate.txt
metadata/core/ontology/templates/eagle_wrhs_warehousetradecharges_reinitbatch.oracle.txt
metadata/core/ontology/templates/eagle_wrhs_warehousetradecharges_reinitbatch.snowflake.txt
metadata/core/ontology/warehouse/warehousetradecharges_in.json

SDP-86236 Added New Fields for Data Rules warehouseopenlot Resource

Additional fields have been added into the Data Rules Ontology to synchronize with the core Eagle Database.

New fields have been added for the HOLDINGDBO.LOT_LEVEL_POSITION.
coupon COUPON NUMBER(38,12)
custodianIncomeBalanceBase CUST_INCOME_BALANCE NUMBER(38,12)
custodianIncomeBalanceLocal CUST_INCOME_BALANCE_LOCAL NUMBER(38,12)
custodianPrincipalBalanceBase CUST_PRINCIPAL_BALANCE NUMBER(38,12)
custodianPrincipalBalanceLocal CUST_PRINCIPAL_BALANCE_LOCAL NUMBER(38,12)
srcSysC SRC_SYS_C VARCHAR2(25 BYTE)

New fields have been added for the HOLDINGDBO.POSITION_COST_LOT:
originalAccruedIncomeBase ORIG_ACCRUED_INCOME_BASE NUMBER(38,12)
commitmentAmountBase COMMITMENT_B NUMBER(38,12)
fundedCommitmentLocal FUNDED_CAPITAL_L NUMBER(38,12)
fundedCommitmentBase FUNDED_CAPITAL_B NUMBER(38,12)
recallableCapitalLocal RECALLABLE_CAPITAL_L NUMBER(38,12)
recallableCapitalBase RECALLABLE_CAPITAL_B NUMBER(38,12)
unfundedCommitmentLocal UNFUNDED_COMMITMENT_L NUMBER(38,12)
unfundedCommitmentBase UNFUNDED_COMMITMENT_B NUMBER(38,12)
capitalCallsBase CAPITAL_CALLS_B NUMBER(38,12)
capitalCallsLocal CAPITAL_CALLS_L NUMBER(38,12)
shareClass SHARE_CLASS VARCHAR2(10 BYTE)
overrideYield OVERRIDE_YLD VARCHAR2(5 BYTE)
openFactor OPEN_FACTOR NUMBER(38,12)

Files changed:
eagle-vocab-wh-shared.json
eagle-vocab-definition-wh-shared.json
warehouseopenlot_in.json
warehouseopenlot-models.json

Production Bugs

SDP-85314 Multileg SMF Support In Composite And History Mode For Additional Elements

New security elements typeOfBond - Tag 18565 and armInitFixedPeriodMonths - Tag 18566 are supported within the MultilegSMF interface for both load - GenericSMF and extract - SMFEXTRACT in composite and history mode.

EagleML Xpath:
EagleML/referenceTransaction/genericSMF/SMExtension/typeOfBond
EagleML/referenceTransaction/genericSMF/MBSExtension/armInitFixedPeriodMonths

Database fields:
SECURITYDBO.SECMASTER_DETAIL_EXT.TYPE_OF_BOND
SECURITYDBO.SECMASTER_DETAIL_EXT_HIST.TYPE_OF_BOND
SECURITYDBO.MBS_DETAIL_EXT.ARM_INIT_FIXED_PERIOD_MONTHS
SECURITYDBO.MBS_DETAIL_EXT_HIST.ARM_INIT_FIXED_PERIOD_MONTHS

Files changed:
eagle_default/in/xml/include/contract_mapping.inc
eagle_default/in/xml/include/contract_leg_alliases.inc
eagle_default/in/xml/include/pay_mapping.inc
eagle_default/in/xml/include/pay_leg_alliases.inc
eagle_default/in/xml/include/receive_leg_alliases.inc
eagle_default/in/xml/include/receive_mapping.inc
eagle_default/in/xml/xml-smf_multileg_swap.xml
eagle_default/in/xml/include/ins_upd_smf_req.inc

SDP-87379 Fixed SMF extract extension mapping issue

Added missing include references to correct extensions mapping in extract
Files changed:
estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/xml/t_smf_streaming.inc

SDP-87251 HTTP SOAP request Calls all MC2 instances if the localhost returns "Connection refused" Error

If MC2 SOAP request to localhost returns “Connection refused“ error, then on the next try the request will be sent to all MC2 instances based on the existing configuration from the estar/tpe/cfg/eaglemc2s/mc2-nodes.txt. All nodes will be used one by one until a success status is returned. The mc2 instances are only used if the localhost returns “Connection refused“ error.

If all mc2 nodes return a fail status, then the process will fail with an error description that contains all errors and failed requests.

If the mc2_nodes.txt does not exists or does not have any entries, then the process will fail.

Files changed:
eagle_ml-2-0_cm\in\xml\run_mc2_extract.xml
eagle_ml-2-0_cm\execute_on_mc2.inc