EagleML Release Notes - October 19, 2023

Owned by Eric Fardig (Deactivated)
Last updated: Nov 14, 2024 by Kinga Huszno
5 min read

Enhancements

SDP-78876 The Conversion workflow is enhanced to control the number of records extracted by the execution plan

A new RTR parameter was introduced to control the number of extracted records for a single resource during a conversion run. The parameter introduced is conversionlimit which has a default value of 20 million records per resource. This parameter will work with the service name buildandexecuteplan. If the calculated number of records for the date range exceed the default value, then the extract will not be triggered and an error will be raised: {{Attempt to run conversion for too many records at once(

{overall_records_count}

) please reduce date range.}}

The default value for conversionlimit can be set manually to overwrite the default and keep a client region stable, for example:

<taskParameter>
<name>conversionlimit</name>
<value>3000000</value>
</taskParameter>

Files changed:
src/eagleinvsys/etl/conversion_main.py

SDP-78708 Added new stream and IWS workflow for Optional Sink Schedule Processing

An optional process to manage Sink Schedules was introduced to help assist clients who are challenged to manage the sink schedule amount equaling 1.00. Introduced a new stream, eagle_ml-2-0_default_in_xml_sink_schd_manager, which triggers a sink_schd_manager workflow to delete future records from the Corporate Action and Sink Schedule tables when the Schedule Type - <scheduleType>SINK</scheduleType> and Sub Schedule Type - <scheduleSubType>SINK</scheduleSubType> are SINK and the Processing Option contains the value of SINK_MANAGER - <processingOptions>SCHEDULE_TRUE_REBOOK,SINK_MANAGER</processingOptions>. All records within the data file greater than the effective date for the provided source will be purged, prior records will be rebooked and all new and future records will be loaded as new SINK schedules.

<EagleML xmlns="Data & Analytics | BNY Mellon
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" eaglemlVersion="2-0"
eaglemlType="ReferenceTransactionMessage"
xsi:schemaLocation="Data & Analytics | BNY Mellon eagleml-main-2-0.xsd"
xsi:type="ReferenceTransactionMessage">
<referenceTransaction>
<header>
<objectType>Schedule</objectType>
<processingOptions>SCHEDULE_TRUE_REBOOK,SINK_MANAGER</processingOptions>
</header>
<schedule>
<objectType>Schedule</objectType>
<objectId>Schedule</objectId>
<sourceName>EAGLE PACE</sourceName>
<updateSource>MCADMIN</updateSource>
<accountingValidationFlag>Y</accountingValidationFlag>
<effectiveDate>2022-07-02</effectiveDate>
<primaryAssetId>SDP78282V001</primaryAssetId>
<primaryAssetType>INTERNAL</primaryAssetType>
<issueName>SAMPLE LTD</issueName>
<journalizingFlag>EXCLUDE_JOURNAL</journalizingFlag>
<cusip>SDP77789_01</cusip>
<primaryExchangeCode>OTC</primaryExchangeCode>
<bcusip>SDP77789_01</bcusip>
<assetCurrency>USD</assetCurrency>
<setEligibleFlag>Y</setEligibleFlag>
<securityMasterSource>1</securityMasterSource>
<scheduleId></scheduleId>
<scheduleType>SINK</scheduleType>
<scheduleSubType>SINK</scheduleSubType>
<scheduleStatus>RD</scheduleStatus>
<rate>2</rate>
<scheduleAmount>0.00538</scheduleAmount>
<scheduleFloat1>1.1</scheduleFloat1>
<endDate>2016-12-11</endDate>
<accrualDateCust>2015-02-18</accrualDateCust>
<determDate1>2015-02-18</determDate1>
<determDate2>2015-02-19</determDate2>
<nextAccrualDate>2015-02-18</nextAccrualDate>
<payDate>2015-02-18</payDate>
</schedule>
</referenceTransaction>
</EagleML>

This process will only execute If the <processingOptions> contains the value of SINK_MANAGER. This feature also supports the SCHEDULE_TRUE_REBOOK processing option which submits only those records that have changed. There is a dependency on a core table to be available in order to help facilitate the storage and orchestration of the records to retain and delete. This table is intended to be available in v17.R2.44. Implementation prior requires the table to be created.

Files changed:

eagle/estar/EAGLE_ML_2_0.pkg
eagle/estar/eagle_ml-2-0_default_in_xml_reference.pkg
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/xml/xml-sink_schd_manager.xml
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/xml/xml-sink_schd_manager_eofp.xml
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/xml/xml-ref_schedule.xml
eagle/estar/eagle_ml-2-0_default_in_xml_sink_schd_manager.pkg
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/xml/xml-sink_schd_manager.rsf
eagle/estar/tpe/dynamic/msgcenter/eis/profiles/eagle_wrf_generic_load.ejmspf
eagle/estar/tpe/dynamic/msgcenter/eis/ejmwf/eagle_wrf_generic_load_security_related_objects.ejmwf
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_workflows/eagle_wrf_generic_load_security_related_objects.wrf

SDP-78479 Added and fixed mapping for the ledgerGranularityRule and ledgerGranularityRuleName elements

Added and fixed mapping for the ledgerGranRule (ledgerGranularityRule previously) and ledgerGranularityRuleName elements in the MC/MC2 entity and accounting basis extracts.

Tag 8933 - ledgerGranRule - Xpath - EagleML/entityTransaction/genericEntity/acctBasis/ledgerGranRule
Tag 8939 - ledgerGranularityRuleName - Xpath - EagleML/entityTransaction/genericEntity/acctBasis/ledgerGranularityRuleName

Files changed:
eagle_ml-2-0_cm/out/xml/t_entity_streaming.inc
eagle_ml-2-0_cm/out/xml/t_entity_streaming_eml.inc
eagle_ml-2-0_cm/out/tagvalue/extract_entity_accounting_basis.inc
eagle_ml-2-0_cm/out/xml/t_entity_accounting_basis_streaming.inc

SDP-78302 Supported Message Attribute in failed_message_stats MC2 HealthCheck Alert

A Message attribute is now added to the body of the failed_message_stats MC2 HealthCheck alert. The max_message_length setting is introduced to limit the length of the value for the Message attribute. The message length default parameter value is set to 2000 characters and can be overridden in the mchealth.ini file. 

The code was updated to support this feature.

Files changed:
src/health/db/FailedMsgStatsDbApi.py
src/health/mc_health_env.py
rc/health/processors/StreamProcessAdaptor.py
src/health/processors/adaptors/msg_stat/FailedMsgStatsProcessor.py

SDP-77548 EDS Extension columns format processing fixes

Fixed an exception when trying to publish an EDS Extension column with an empty Format value.

Also added the validation and default value for number and integer columns.

SDP-77288 The BASE_CURRENCY field logic was enhanced as part of the warehouseposition metadata

Previously the field POSITION.BASE_CURRENCY was used to populate the baseCurrency on the EDS extract side. During ingestion to snowflake, the baseCurrency value was used to populated two fields:

  • POSITION.BASE_CURRENCY

  • POSITION_DETAIL.BASE_CURRENCY

A new field, assetLevelBaseCurrency, was introduced and corresponding mappings were adjusted to improve processing logic for below fields:

  • POSITION.BASE_CURRENCY - baseCurrency

  • POSITION_DETAIL.BASE_CURRENCY - assetLevelBaseCurrency

If the field assetLevelBaseCurrency is not included in the extract,, then the following is expected:

  • the field POSITION_DETAIL.BASE_CURRENCY is set to NULL if asiscopy is set to Y

  • the field POSITION_DETAIL.BASE_CURRENCY is equal to the baseCurrency if asiscopy is not used

Files changed:
metadata/core/ontology/definition/eagle-vocab-definition-wh.json
metadata/core/ontology/eagle-vocab-wh.json
metadata/core/ontology/warehouse/warehouseposition-models.json
metadata/core/ontology/warehouse/warehouseposition.json
metadata/core/ontology/warehouse/warehouseposition_in.json

SDP-76462 EDS support for entitybankrelationship Resource

The EDS entitybankrelationship resource was created to support the Eagle - Vault pipeline, to extract data from Eagle and upload to Vault.

Supported modes: Delta, Temporal, History
Table: RULESDBO.ENTITY_CUSTODY_RELATIONSHIP
History table: RULESDBO.ENTITY_CUSTODY_REL_HIST

Extract Filters:
entityfilter
fromdate, todate,
effectivedate, fromeffectivedate, toeffectivedate
entitybankalias

Files changed:
eagle/estar/tpe/dynamic/msgcenter/eis/ejmwf/eagle_wrf_cloud_extract.ejmwf
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_workflows/eagle_wrf_cloud_extract.wrf
metadata/core/ontology/accounting/entitybankrelationship.json
metadata/core/ontology/accounting/entitybankrelationship-models.json
metadata/core/ontology/accounting/entitybankrelationship_in.json

SDP-76455 EDS support for settledcashactivity Resource

The EDS settledcashactivity resource was created to support the Eagle - Vault pipeline, to extract data from Eagle and upload to Snowflake.

Supported modes: Batch, Temporal
Table: ESTAR.ESTAR_SETTLED_CASH_ACTIVITY
Batch Key: PORTFOLIO_ACCT, EFFECTIVE_DT, ACCT_BASIS, LNG_SHT_C, SHARE_CLASS

Extract Filters:
entityfilter
accountingbasis
processingsecuritytype, securitytype, investmenttype, releasestatus, primaryassetid, xrefid, ticker, securityalias, securitysubtype
fromdate, todate,
effectivedate, fromeffectivedate, toeffectivedate
postdate, frompostdate, topostdate
tradedate, fromtradedate, totradedate,
accountingdate, fromaccountingdate, toaccountingdate
monthendaccountingdate, frommonthendaccountingdate, tomonthendaccountingdate
settlementdate, fromsettlementdate, tosettlementdate
deltabatchfilter

Files added:
metadata/core/ontology/accounting/settledcashactivity.json
metadata/core/ontology/accounting/settledcashactivity-models.json
metadata/core/ontology/accounting/settledcashactivity.json

SDP-75650 EDS tradedcashactivity Resource

The EDS tradedcashactivity resource was created to support the Eagle - Vault pipeline, to extract data from Eagle and upload to Snowflake.

Supported upload modes: Batch, Temporal
Table: ESTAR.ESTAR_TRADED_CASH_ACTIVITY
Batch Key: PORTFOLIO_ACCT, EFFECTIVE_DT, ACCT_BASIS, LNG_SHT_C, SHARE_CLASS

Extract Filters:
entityfilter
accountingbasis
processingsecuritytype, securitytype, investmenttype, releasestatus, primaryassetid, xrefid, ticker, securityalias, securitysubtype
fromdate, todate,
effectivedate, fromeffectivedate, toeffectivedate
postdate, frompostdate, topostdate
accountingdate, fromaccountingdate, toaccountingdate
monthendaccountingdate, frommonthendaccountingdate, tomonthendaccountingdate
settlementdate, fromsettlementdate, tosettlementdate
deltabatchfilter

Files added:
metadata/core/ontology/accounting/tradedcashactivity.json
metadata/core/ontology/accounting/tradedcashactivity-models.json
metadata/core/ontology/accounting/tradedcashactivity.json

SDP-75321 Added support for Non Deliverable Swaps in Message Center Eagle ML

The Accounting Trades EagleML now supports the four below elements used in Non-Deliverable Swaps.

Tag 17989 - <comimissionSettlementAmount>
Tag 17988 - <otherFeeSettlementAmount>
Tag 18397 - <swapSettlementAmount>
Tag 66 - <settlementExhangeRate>

CONTRACT_17989 - Tag 17989- EagleML/accountingTransaction/multipleLegAccountingTrade/contractTradeLeg/comimissionSettlementAmount
PAY_17989 - Tag 17989- EagleML/accountingTransaction/multipleLegAccountingTrade/payTradeLeg/comimissionSettlementAmount
RECEIVE_17989 - Tag 17989- EagleML/accountingTransaction/multipleLegAccountingTrade/receiveTradeLeg/comimissionSettlementAmount
CONTRACT_17988 - Tag 17988- EagleML/accountingTransaction/multipleLegAccountingTrade/contractTradeLeg/otherFeeSettlementAmount
PAY_17988 - Tag 17988- EagleML/accountingTransaction/multipleLegAccountingTrade/payTradeLeg/otherFeeSettlementAmount
RECEIVE_17988 - Tag 17988- EagleML/accountingTransaction/multipleLegAccountingTrade/receiveTradeLeg/otherFeeSettlementAmount
CONTRACT_18397 - Tag 18397- EagleML/accountingTransaction/multipleLegAccountingTrade/contractTradeLeg/swapSettlementAmount
PAY_18397 - Tag 18397- EagleML/accountingTransaction/multipleLegAccountingTrade/payTradeLeg/swapSettlementAmount
RECEIVE_18397 - Tag 18397- EagleML/accountingTransaction/multipleLegAccountingTrade/receiveTradeLeg/swapSettlementAmount
CONTRACT_66 - Tag 66- EagleML/accountingTransaction/multipleLegAccountingTrade/contractTradeLeg/settlementExhangeRate
PAY_66 - Tag 66- EagleML/accountingTransaction/multipleLegAccountingTrade/payTradeLeg/settlementExhangeRate
RECEIVE_66 - Tag 66- EagleML/accountingTransaction/multipleLegAccountingTrade/receiveTradeLeg/settlementExhangeRate

Files changed:
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/xml/include/multiplelegtrade_aliases.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/xml/include/tradeleg_aliases.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/xml/include/tradeleg_columns.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/xml/xml-accttrade_singleleg.xml

SDP-74408 The reportdictionarydetail metadata was updated to correct several fields

The reportdictionarydetail metadata was corrected to keep the field type and length in sync between Oracle and Snowflake.

Files changed:
metadata/core/ontology/eagledb-vocab-common.json
metadata/core/ontology/reference/reportdictionarydetail-models.json

Production Bugs

SDP-78410 Updated Batch Cancel to apply the active Open Accounting Period

The Cancel Batch Resolution process was modified not to use the accounting date from the batch id lookup, but from another lookup to check the transaction status and return the next open accounting period as the accounting date on the cancel record. This prevents the cancel posting against a closed accounting period.

Files Changed:
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/include/csv-cancel_columns.inc

SDP-78343 Added Support for the Voluntary Corporate Actions Change Records to include the expiryDateFlag

Added the expiryDateFlag element - Tag 128777 - SECURITYDBO.VOL_CORP_ACTION_ELECTION.EXPIRY_DATE_PROC_FLAG to the Voluntary Corporate Actions update process. If the expiryDateFlag is included in the incoming file, it will be used in place of the lookup value returned.

Files changed:
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/xml/xml-ref_voluntarycaelections.xml

SDP-76947 Extract SQL query enhancement for SPDDELTATRIGGERLOGEXTRACT

Extract SQL query is enhanced for the case when the minAccountingDateFlag is used. A join with the ESTAR.TEMP_RESOLVE_COMPOSITE table is added to reduce the information retrieved when it resolves and retrieves the minimum of accounting date.

Files changed:
eagle_ml-2-0_cm\out\tagvalue\extract_spd_delta_trigger_log.inc

SDP-76530 AssetCash mappings was fixed to support security resolution with assetCash

During previous development, it was discovered that the security resolution for the AccountingAssetCash objects does not take into account the assetCurrency. This can cause issues if a client wants to have securities with the same identifiers, but different currencies. The security resolution allows such load, and such records are not processed as duplicates. In this case, if a client loads AccountingAssetCash, it can be bound to a security with a different currency. Therefore, mapping for Tag85 - assetCurency was added prior to resolution.

Note that this fix excludes the assetCashType = ‘BLINC’ which resolves the securityAlias by xreferences, which may cause different security aliases after resolution against once stored in the database.

Files changed:
eagle_default/in/include/trd_sec_columns.inc