Enhancements
SDP-78876 The Conversion workflow is enhanced to control the number of records extracted by the execution plan
A new RTR parameter was introduced to control the number of extracted records for a single resource during a conversion run. The parameter introduced is conversionlimit which has a default value of 20 million records per resource. This parameter will work with the service name buildandexecuteplan. If the calculated number of records for the date range exceed the default value, then the extract will not be triggered and an error will be raised: {{Attempt to run conversion for too many records at once(
{overall_records_count}
) please reduce date range.}}
The default value for conversionlimit can be set manually to overwrite the default and keep a client region stable, for example:
<taskParameter>
<name>conversionlimit</name>
<value>3000000</value>
</taskParameter>
Files changed:
src/eagleinvsys/etl/conversion_main.py
SDP-78708 Added new stream and IWS workflow for Optional Sink Schedule Processing
An optional process to manage Sink Schedules was introduced to help assist clients who are challenged to manage the sink schedule amount equaling 1.00. Introduced a new stream, eagle_ml-2-0_default_in_xml_sink_schd_manager, which triggers a sink_schd_manager workflow to delete future records from the Corporate Action and Sink Schedule tables when the Schedule Type - <scheduleType>SINK</scheduleType> and Sub Schedule Type - <scheduleSubType>SINK</scheduleSubType> are SINK and the Processing Option contains the value of SINK_MANAGER - <processingOptions>SCHEDULE_TRUE_REBOOK,SINK_MANAGER</processingOptions>. All records within the data file greater than the effective date for the provided source will be purged, prior records will be rebooked and all new and future records will be loaded as new SINK schedules.
<EagleML xmlns="
...
Data & Analytics | BNY Mellon
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" eaglemlVersion="2-0"
eaglemlType="ReferenceTransactionMessage"
xsi:schemaLocation="
...
Data & Analytics | BNY Mellon eagleml-main-2-0.xsd"
xsi:type="ReferenceTransactionMessage">
<referenceTransaction>
<header>
<objectType>Schedule</objectType>
<processingOptions>SCHEDULE_TRUE_REBOOK,SINK_MANAGER</processingOptions>
</header>
<schedule>
<objectType>Schedule</objectType>
<objectId>Schedule</objectId>
<sourceName>EAGLE PACE</sourceName>
<updateSource>MCADMIN</updateSource>
<accountingValidationFlag>Y</accountingValidationFlag>
<effectiveDate>2022-07-02</effectiveDate>
<primaryAssetId>SDP78282V001</primaryAssetId>
<primaryAssetType>INTERNAL</primaryAssetType>
<issueName>SAMPLE LTD</issueName>
<journalizingFlag>EXCLUDE_JOURNAL</journalizingFlag>
<cusip>SDP77789_01</cusip>
<primaryExchangeCode>OTC</primaryExchangeCode>
<bcusip>SDP77789_01</bcusip>
<assetCurrency>USD</assetCurrency>
<setEligibleFlag>Y</setEligibleFlag>
<securityMasterSource>1</securityMasterSource>
<scheduleId></scheduleId>
<scheduleType>SINK</scheduleType>
<scheduleSubType>SINK</scheduleSubType>
<scheduleStatus>RD</scheduleStatus>
<rate>2</rate>
<scheduleAmount>0.00538</scheduleAmount>
<scheduleFloat1>1.1</scheduleFloat1>
<endDate>2016-12-11</endDate>
<accrualDateCust>2015-02-18</accrualDateCust>
<determDate1>2015-02-18</determDate1>
<determDate2>2015-02-19</determDate2>
<nextAccrualDate>2015-02-18</nextAccrualDate>
<payDate>2015-02-18</payDate>
</schedule>
</referenceTransaction>
</EagleML>
This process will only execute If the <processingOptions> contains the value of SINK_MANAGER. This feature also supports the SCHEDULE_TRUE_REBOOK processing option which submits only those records that have changed. There is a dependency on a core table to be available in order to help facilitate the storage and orchestration of the records to retain and delete. This table is intended to be available in v17.R2.44. Implementation prior requires the table to be created.
Files changed:
eagle/estar/EAGLE_ML_2_0.pkg
eagle/estar/eagle_ml-2-0_default_in_xml_reference.pkg
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/xml/xml-sink_schd_manager.xml
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/xml/xml-sink_schd_manager_eofp.xml
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/xml/xml-ref_schedule.xml
eagle/estar/eagle_ml-2-0_default_in_xml_sink_schd_manager.pkg
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/xml/xml-sink_schd_manager.rsf
eagle/estar/tpe/dynamic/msgcenter/eis/profiles/eagle_wrf_generic_load.ejmspf
eagle/estar/tpe/dynamic/msgcenter/eis/ejmwf/eagle_wrf_generic_load_security_related_objects.ejmwf
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_workflows/eagle_wrf_generic_load_security_related_objects.wrf
SDP-78479 Added and fixed mapping for the ledgerGranularityRule and ledgerGranularityRuleName elements
Added and fixed mapping for the ledgerGranRule (ledgerGranularityRule previously) and ledgerGranularityRuleName elements in the MC/MC2 entity and accounting basis extracts.
Tag 8933 - ledgerGranRule - Xpath - EagleML/entityTransaction/genericEntity/acctBasis/ledgerGranRule
Tag 8939 - ledgerGranularityRuleName - Xpath - EagleML/entityTransaction/genericEntity/acctBasis/ledgerGranularityRuleName
Files changed:
eagle_ml-2-0_cm/out/xml/t_entity_streaming.inc
eagle_ml-2-0_cm/out/xml/t_entity_streaming_eml.inc
eagle_ml-2-0_cm/out/tagvalue/extract_entity_accounting_basis.inc
eagle_ml-2-0_cm/out/xml/t_entity_accounting_basis_streaming.inc
SDP-78302 Supported Message Attribute in failed_message_stats MC2 HealthCheck Alert
A Message attribute is now added to the body of the failed_message_stats MC2 HealthCheck alert. The max_message_length setting is introduced to limit the length of the value for the Message attribute. The message length default parameter value is set to 2000 characters and can be overridden in the mchealth.ini file.
The code was updated to support this feature.
Files changed:
src/health/db/FailedMsgStatsDbApi.py
src/health/mc_health_env.py
rc/health/processors/StreamProcessAdaptor.py
src/health/processors/adaptors/msg_stat/FailedMsgStatsProcessor.py
SDP-77548 EDS Extension columns format processing fixes
Fixed an exception when trying to publish an EDS Extension column with an empty Format value.
Also added the validation and default value for number and integer columns.
SDP-77288 The BASE_CURRENCY field logic was enhanced as part of the warehouseposition metadata
Previously the field POSITION.BASE_CURRENCY was used to populate the baseCurrency on the EDS extract side. During ingestion to snowflake, the baseCurrency value was used to populated two fields:
POSITION.BASE_CURRENCY
POSITION_DETAIL.BASE_CURRENCY
A new field, assetLevelBaseCurrency, was introduced and corresponding mappings were adjusted to improve processing logic for below fields:
POSITION.BASE_CURRENCY - baseCurrency
POSITION_DETAIL.BASE_CURRENCY - assetLevelBaseCurrency
If the field assetLevelBaseCurrency is not included in the extract,, then the following is expected:
the field POSITION_DETAIL.BASE_CURRENCY is set to NULL if asiscopy is set to Y
the field POSITION_DETAIL.BASE_CURRENCY is equal to the baseCurrency if asiscopy is not used
Files changed:
metadata/core/ontology/definition/eagle-vocab-definition-wh.json
metadata/core/ontology/eagle-vocab-wh.json
metadata/core/ontology/warehouse/warehouseposition-models.json
metadata/core/ontology/warehouse/warehouseposition.json
metadata/core/ontology/warehouse/warehouseposition_in.json
SDP-76462 EDS support for entitybankrelationship Resource
The EDS entitybankrelationship resource was created to support the Eagle - Vault pipeline, to extract data from Eagle and upload to Vault.
Supported modes: Delta, Temporal, History
Table: RULESDBO.ENTITY_CUSTODY_RELATIONSHIP
History table: RULESDBO.ENTITY_CUSTODY_REL_HIST
Extract Filters:
entityfilter
fromdate, todate,
effectivedate, fromeffectivedate, toeffectivedate
entitybankalias
Files changed:
eagle/estar/tpe/dynamic/msgcenter/eis/ejmwf/eagle_wrf_cloud_extract.ejmwf
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_workflows/eagle_wrf_cloud_extract.wrf
metadata/core/ontology/accounting/entitybankrelationship.json
metadata/core/ontology/accounting/entitybankrelationship-models.json
metadata/core/ontology/accounting/entitybankrelationship_in.json
SDP-76455 EDS support for settledcashactivity Resource
The EDS settledcashactivity resource was created to support the Eagle - Vault pipeline, to extract data from Eagle and upload to Snowflake.
Supported modes: Batch, Temporal
Table: ESTAR.ESTAR_SETTLED_CASH_ACTIVITY
Batch Key: PORTFOLIO_ACCT, EFFECTIVE_DT, ACCT_BASIS, LNG_SHT_C, SHARE_CLASS
Extract Filters:
entityfilter
accountingbasis
processingsecuritytype, securitytype, investmenttype, releasestatus, primaryassetid, xrefid, ticker, securityalias, securitysubtype
fromdate, todate,
effectivedate, fromeffectivedate, toeffectivedate
postdate, frompostdate, topostdate
tradedate, fromtradedate, totradedate,
accountingdate, fromaccountingdate, toaccountingdate
monthendaccountingdate, frommonthendaccountingdate, tomonthendaccountingdate
settlementdate, fromsettlementdate, tosettlementdate
deltabatchfilter
Files added:
metadata/core/ontology/accounting/settledcashactivity.json
metadata/core/ontology/accounting/settledcashactivity-models.json
metadata/core/ontology/accounting/settledcashactivity.json
SDP-75650 EDS tradedcashactivity Resource
The EDS tradedcashactivity resource was created to support the Eagle - Vault pipeline, to extract data from Eagle and upload to Snowflake.
Supported upload modes: Batch, Temporal
Table: ESTAR.ESTAR_TRADED_CASH_ACTIVITY
Batch Key: PORTFOLIO_ACCT, EFFECTIVE_DT, ACCT_BASIS, LNG_SHT_C, SHARE_CLASS
Extract Filters:
entityfilter
accountingbasis
processingsecuritytype, securitytype, investmenttype, releasestatus, primaryassetid, xrefid, ticker, securityalias, securitysubtype
fromdate, todate,
effectivedate, fromeffectivedate, toeffectivedate
postdate, frompostdate, topostdate
accountingdate, fromaccountingdate, toaccountingdate
monthendaccountingdate, frommonthendaccountingdate, tomonthendaccountingdate
settlementdate, fromsettlementdate, tosettlementdate
deltabatchfilter
Files added:
metadata/core/ontology/accounting/tradedcashactivity.json
metadata/core/ontology/accounting/tradedcashactivity-models.json
metadata/core/ontology/accounting/tradedcashactivity.json
SDP-75321 Added support for Non Deliverable Swaps in Message Center Eagle ML
The Accounting Trades EagleML now supports the four below elements used in Non-Deliverable Swaps.
Tag 17989 - <comimissionSettlementAmount>
Tag 17988 - <otherFeeSettlementAmount>
Tag 18397 - <swapSettlementAmount>
Tag 66 - <settlementExhangeRate>
CONTRACT_17989 - Tag 17989- EagleML/accountingTransaction/multipleLegAccountingTrade/contractTradeLeg/comimissionSettlementAmount
PAY_17989 - Tag 17989- EagleML/accountingTransaction/multipleLegAccountingTrade/payTradeLeg/comimissionSettlementAmount
RECEIVE_17989 - Tag 17989- EagleML/accountingTransaction/multipleLegAccountingTrade/receiveTradeLeg/comimissionSettlementAmount
CONTRACT_17988 - Tag 17988- EagleML/accountingTransaction/multipleLegAccountingTrade/contractTradeLeg/otherFeeSettlementAmount
PAY_17988 - Tag 17988- EagleML/accountingTransaction/multipleLegAccountingTrade/payTradeLeg/otherFeeSettlementAmount
RECEIVE_17988 - Tag 17988- EagleML/accountingTransaction/multipleLegAccountingTrade/receiveTradeLeg/otherFeeSettlementAmount
CONTRACT_18397 - Tag 18397- EagleML/accountingTransaction/multipleLegAccountingTrade/contractTradeLeg/swapSettlementAmount
PAY_18397 - Tag 18397- EagleML/accountingTransaction/multipleLegAccountingTrade/payTradeLeg/swapSettlementAmount
RECEIVE_18397 - Tag 18397- EagleML/accountingTransaction/multipleLegAccountingTrade/receiveTradeLeg/swapSettlementAmount
CONTRACT_66 - Tag 66- EagleML/accountingTransaction/multipleLegAccountingTrade/contractTradeLeg/settlementExhangeRate
PAY_66 - Tag 66- EagleML/accountingTransaction/multipleLegAccountingTrade/payTradeLeg/settlementExhangeRate
RECEIVE_66 - Tag 66- EagleML/accountingTransaction/multipleLegAccountingTrade/receiveTradeLeg/settlementExhangeRate
Files changed:
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/xml/include/multiplelegtrade_aliases.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/xml/include/tradeleg_aliases.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/xml/include/tradeleg_columns.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/xml/xml-accttrade_singleleg.xml
SDP-74408 The reportdictionarydetail metadata was updated to correct several fields
The reportdictionarydetail metadata was corrected to keep the field type and length in sync between Oracle and Snowflake.
Files changed:
metadata/core/ontology/eagledb-vocab-common.json
metadata/core/ontology/reference/reportdictionarydetail-models.json
Production Bugs
SDP-78410 Updated Batch Cancel to apply the active Open Accounting Period
The Cancel Batch Resolution process was modified not to use the accounting date from the batch id lookup, but from another lookup to check the transaction status and return the next open accounting period as the accounting date on the cancel record. This prevents the cancel posting against a closed accounting period.
Files Changed:
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/include/csv-cancel_columns.inc
SDP-78343 Added Support for the Voluntary Corporate Actions Change Records to include the expiryDateFlag
Added the expiryDateFlag element - Tag 128777 - SECURITYDBO.VOL_CORP_ACTION_ELECTION.EXPIRY_DATE_PROC_FLAG to the Voluntary Corporate Actions update process. If the expiryDateFlag is included in the incoming file, it will be used in place of the lookup value returned.
Files changed:
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/xml/xml-ref_voluntarycaelections.xml
SDP-76947 Extract SQL query enhancement for SPDDELTATRIGGERLOGEXTRACT
Extract SQL query is enhanced for the case when the minAccountingDateFlag is used. A join with the ESTAR.TEMP_RESOLVE_COMPOSITE table is added to reduce the information retrieved when it resolves and retrieves the minimum of accounting date.
Files changed:
eagle_ml-2-0_cm\out\tagvalue\extract_spd_delta_trigger_log.inc
SDP-76530 AssetCash mappings was fixed to support security resolution with assetCash
During previous development, it was discovered that the security resolution for the AccountingAssetCash objects does not take into account the assetCurrency. This can cause issues if a client wants to have securities with the same identifiers, but different currencies. The security resolution allows such load, and such records are not processed as duplicates. In this case, if a client loads AccountingAssetCash, it can be bound to a security with a different currency. Therefore, mapping for Tag85 - assetCurency was added prior to resolution.
Note that this fix excludes the assetCashType = ‘BLINC’ which resolves the securityAlias by xreferences, which may cause different security aliases after resolution against once stored in the database.
Files changed:
eagle_default/in/include/trd_sec_columns.inc