Enhancements

SDP-83345 Data Rules Rejection Reprocessing Logic for Batch load to the Oracle database

Implemented batch reprocessing logic. This logic works with the RTR parameter only:

<taskParameter>
<name>enablebatchexceptions</name>
<dataType>S</dataType>
<value>Y</value>
</taskParameter>

In case of an error on the batch upload, all records for the failed batch will be moved to the following location:

*/apps/eagle/data/msgcenter/cmw_staging_folder/reprocess/{CORR_ID}*

If data needs to be corrected, the user can download the file(s), modify the values, upload it back to the remote server and reprocess it again from User Interface.

If it is a reference data issue, for example the Source is not found the Reprocess can be executed after the reference data is corrected.

The dumped file in PSV pipe delimited format with the header no matter which format was initially loaded.

SDP-85570 Data Rules Rejection Reprocessing Logic for Batch load to the Oracle database

Improved Data Rules Rejection Reprocessing Logic for the Batch load to the Oracle database to handle row level errors. In addition, the eds_reprocess rule was also adjusted. Previously, when only single record reprocessing was implemented, eds_reprocess set the DO_NOT_DELETE variable to Y, in order to avoid deleting the entire batch when a single record for this batch is reprocessed. Now, when the entire batch is reprocessed, DO_NOT_DELETE has to be set to N, in order to delete the data and reload it.

Files changed:
metadata/core/ebs/eds_reprocess.json

SDP-83344 Data Rules Rejection Reprocessing functionality for single record

Below reprocessing improvements were implemented for single record:

• Fixed problem with reprocessing when original message in XML format

• Eliminated existing size limitation for original message

SDP-81724 Improved data structure for the SQL parameters, to store more information on validation

As part of this story, refactored new structure for SqlParam incorporated, rather than just list structure which had been in place.

The new structure allows to store more metadata information for each parameter (length, mandatory, dialect for date\datetime) and performs the validation based on this information.

The Validation is used only on Oracle and the next step is to extend processing rules information about (length, mandatory, dialect for date\datetime). Basic coverage of this data can be already gathered from ontology.

SDP-84271 Data Rules Validation Process as part of Rejection Reprocessing Functionality

Data Rules metadata was enhanced to have more information about the parameters for validation. Additional information was added to the ontology around stored procedure parameters in order to validate these parameters during the transformation and load process in Eagle/Oracle:

• Length

• Type

In addition, the process used for generating processing rules was improved to store more data about the parameters and use it during the transformation and load processes.

SDP-84987 The validation process was enhanced for data upload to the Oracle database via Data Rules

To improve end user experience, the errors returned have more details now.

For example:
for string parameter - parameter name and max length are added:
{{Param(in_investment_type) value too long DDDDDDDDDDDDDDDDDDDDDDDDD max length 2}}
for number or date - the parameter name is added.

SDP-79581 Data Rules smfidentifier Resource

The Data Rules smfidentifier resource was added to support the Eagle To Vault pipeline, to extract data from Eagle and upload to Snowflake.
The Oracle load is also supported via the PACE_MASTER.ESTAR_SEC.INSERT_UPDATE_XREF_INFO Stored Procedure.

Supported updatemode: Delta, Temporal, Batch

• Delta or Update - default load mode

• Temporal or Insert - works on record level only

• Batch - triggered by file with *_ASISCOPY_DELTABATCH_* file mask. The eagle_wrf_cloud_extract workflow will generate the mask if parameter usedeltabatch is set to Y in the profile for the cloud_extract.

It is recommended to use Batch mode only with ASISCOPY mode.

Tables: SECURITYDBO.XREFERENCE, SECURITYDBO.XREFERENCE_HIST

Batch Key: SECURITY_ALIAS in ASISCOPY mode

Extract Filters:
xrefid, xrefidtype
fromdate, todate,
sourcename
processingsecuritytype, securitytype, investmenttype, releasestatus, primaryassetid, ticker, securityalias, securitysubtype
heldonlyfilter
deltabatchfilter

*Please note:

• Full conversion is required for smfidentifier resource to correct the data in Vault, to resolve previously reported duplicates

• Disable delta jobs for executing smfidentifiers/smfidentifiershist and use the default profile for the smfidentifier resource where usedeltabatch is set to Y

Files added:
metadata/core/ontology/reference/smfidentifier-models.json
metadata/core/ontology/reference/smfidentifier.json
metadata/core/ontology/reference/smfidentifier_in.json
metadata/core/ontology/templates/db_oper/snowflake/smfidentifier.reinit_batch_many.sql
metadata/core/ontology/templates/eagle_ref_smfidentifier_insertupdate.txt
metadata/core/ontology/templates/eagle_ref_smfidentifier_reinitbatch.oracle.txt
metadata/core/ontology/templates/eagle_ref_smfidentifier_reinitbatch.snowflake.txt

Files changed:
rdf_metadata/rdf_meta_utils.py
rdf_metadata/rdf_physical_schema_generator.py
metadata/core/ontology/eagledb-inventory.json
metadata/core/ontology/eagle-editchecks.json
metadata/core/ontology/eagledb-vocab-common.json
metadata/core/ontology/reference/smfidentifiers-models.json
metadata/core/ontology/reference/smfidentifiers_in.json
metadata/core/ontology/reference/smfidentifiershist-models.json

SDP-85377 Accounting Periods Extract was enhanced to support accountingBasis filter

The ENTITYACCPERIODSEXTRACT has been enhanced to filter data based upon the ESTAR.ESTAR_ACCOUNTING_PERIODS.ACCT_BASIS using the accountingBasis parameter which was added as a paramter:
<taskParameter>
<name>accountingBasis</name>
<dataType>S</dataType>
<value>GAAP</value>
</taskParameter>

Also updated swagger to support accountingBasis filter for get_accounting_periods.

Files changed:
eagleml/swagger/swagger-extracts.json
eagle_ml-2-0_cm/out/tagvalue/extract_entity_accounting_periods.inc
eagle_ml-2-0_cm/out/xml/t_entity_accounting_periods.xml
eagle_ml-2-0_cm/out/xml/t_entity_accounting_periods_streaming.inc

SDP-85261 Python translator change to allow creates correct DBML request for close lots

There was a problem with the incorrect submessage type when the python translator was used. The Close Lot XSLT template was modified. The python translator now pulls the correct CLL_EGL_GEN_INTERFACE value for submessage type.

File changed:
eagle_default/in/xml/xslt/wrh_egl_gen_interface3.xsl

SDP-84859 Warehouse Open Lot outbound interface has been enhanced with new elements

The WRHSOPENLOTEXTRACT outbound interface was enhanced to support below elements:

warehouseTransaction/warehouseOpenLot/custodianPrincipalBalanceBase - HOLDINGDBO.LOT_LEVEL_POSITION.CUST_PRINCIPAL_BALANCE
warehouseTransaction/warehouseOpenLot/custodianPrincipalBalanceLocal - HOLDINGDBO.LOT_LEVEL_POSITION.CUST_PRINCIPAL_BALANCE_LOCAL
warehouseTransaction/warehouseOpenLot/custodianIncomeBalanceBase - HOLDINGDBO.LOT_LEVEL_POSITION.CUST_INCOME_BALANCE
warehouseTransaction/warehouseOpenLot/custodianIncomeBalanceLocal - HOLDINGDBO.LOT_LEVEL_POSITION.CUST_INCOME_BALANCE_LOCAL

Values of the elements are set to zero on the extract if the corresponding fields are NULL in the database.

Files changed:
eagle_ml-2-0_cm/out/xml/t_warehouse_openlot_streaming.inc

SDP-84771 Enhanced Data Rules extract logic to support override of group_key_expr using groupBy task parameter

Enhanced the Data Rules extract functionality by supporting the override of group_key_expr via groupBy task parameter.

Added a warning to the Data Rules resource processing status for duplicates during transformation.

for example:
{{["Warning", "Two or more 'editcheck' t_items with same name 'extract_editcheck'.", "eaglepy-i-1000", "", 0, "2024-05-21T10:13:47.462-04:00"]}}

List of t_items which require unique names due to status variable:

editcheck
creatertr
executeeds
match
daterule
datastorage

SDP-84693 Intrafund Cash Transfer Interface added to generic load workflow

The Intrafund Cash Transfers is supported by the eagle_wrf_generic_load workflow with _INTRAFUNDCASHTRANSFER_ file mask.

Files changed:
eagle_ml-2-0_workflows/eagle_wrf_generic_load.wrf
eagle_ml-2-0_workflows/eagle_wrf_generic_load_accounting_objects.wrf
eis/ejmwf/eagle_wrf_generic_load.ejmwf
eis/ejmwf/eagle_wrf_generic_load_accounting_objects.ejmwf
eis/profiles/eagle_wrf_generic_load.ejmspf

SDP-84490 genericsmf resource was updated to support a few new missing fields on load to Oracle database

The genericsmf resource was updated to support additional fields for the Oracle database.

Database Field Canonical Element
SECURITY_MASTER_DETAIL.PRIVATE_EQUITY_BENCHMARK - privateEquityBenchmark
DERIVATIVES.FWD_SPOT_INDICATOR - fwdSpotIndicator
MBS_DETAIL_EXT.ARM_INIT_FIXED_PERIOD_MONTHS - armInitFixedPeriodMonths
SECURITY_MASTER_EXT.UK_TAXABLE_FLAG - ukTaxableFlag
SECMASTER_DETAIL_EXT.TYPE_OF_BOND - typeOfBond

Files Changed:
eagle-vocab-sec-shared.json
eagle-vocab-definition-sec-shared.json
genericsmf-models.json

SDP-83786 EJM Processing Logic was enhanced for files with long file names

The EJM process logic was enhanced for file names longer than 200 characters.

A new variable, varReplacedFileNames, was added in workflow state to control which file names should be changed. This action enables the substitution replacement in last task_reporter step to avoid the error message: _Possibly the file was not processed._

Files changed:
eagle_ml-2-0_cm\get_w_state.inc
eagle_ml-2-0_cm\upd_w_state.inc
eagle_ml-2-0_cm\out\tagvalue\data_file_task.xml
eagle_ml-2-0_cm\out\xml\task_reporter.xml

SDP-83452 Added TBA Position netting via EagleML interface to the genericEntity

A new EagleML element was added for entityTransaction/genericEntity/netTBAEntityIndicator - Tag18577 to support TBA position netting via the EagleML interface as part of Entity election. The new XML element is supported for the load and the extract using the genericentity interface via the panel and direct SP load.

Files changed:
eagle_default/in/include/entity_aliases.inc
eagle_default/in/include/entity_panels.inc
eagle_default/in/include/csv-entity_columns.inc

eagleml/schema/eagleml-entity-2.0.xsd

SDP-82995 Warehouse Position Outbound Interface has been enhanced with new elements

The WRHSPOSITIONEXTRACT outbound interface was enhanced to support additional elements for the load and the extract:

warehouseTransaction/warehousePosition/custodianPrincipalBalanceBase - HOLDINGDBO.POSITION_DETAIL.CUST_PRINCIPAL_BALANCE
warehouseTransaction/warehousePosition/custodianPrincipalBalanceLocal- HOLDINGDBO.POSITION_DETAIL.CUST_PRINCIPAL_BALANCE_LOCAL
warehouseTransaction/warehousePosition/custodianIncomeBalanceBase- HOLDINGDBO.POSITION_DETAIL.CUST_INCOME_BALANCE
warehouseTransaction/warehousePosition/custodianIncomeBalanceLocal- HOLDINGDBO.POSITION_DETAIL.CUST_INCOME_BALANCE_LOCAL

Values of the elements are being zeroed if the corresponding fields are NULL in DB.

Files changed:
eagle_ml-2-0_cm/out/xml/t_warehouse_position_streaming.inc

SDP-82969 The WRHSOPENLOTEXTRACT logic was corrected for entities segregating derivatives

The issue was reported where the WRHSOPENLOTEXTRACT is appearing to duplicate lots on cash securities for entities segregating derivatives. The extract logic was updated as requested to correct the issue.

Files changed:
estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/tagvalue/extract_warehouse_openlot.inc

SDP-81860 warehouseperformance resource was updated to support data load to additional fields

Two database fields were missed on the data upload in the Eagle Oracle environment.

The metadata was corrected by providing description for below fields:

PERF_SEC_RETURNS.PSR_USER_DATE3 - pSRUserDate3
PERF_SEC_RETURNS.PSR_USER_INT1 - pSRUserInt1

Files changed:
warehouseperformance-models.json

SDP-81702 New EagleML Interface for Intrafund Cash Transfers Between Custodians

The EagleML Intrafund Cash Transfers Between Custodians interface is available for load. It enables the transfer of cash for a given entity between cash accounts related to different custodian banks. Entity should be established to use cash segregation for such cash transfers.
Object type name: CustodyCashTransfer
Stream for load: eagle_ml-2-0_default_in_xml_acct_cash
Generic load workflow file mask: CUSTODYCASHTRANSFER

Files added:
eagle_default/in/xml/xml-acct_custody_cash_transfer.xml

Files changed:
eagle_default/in/xml/xml-acct_cash.rsf

SDP-81701 New EagleML interface for Intrafund Cash Transfers

IntrafundCashTransfers interface is available for the EagleML load. It allows to transfer cash between cash accounts for an entity. If an entity uses cash segregation, then the client can use this option to move cash within the entity's custody basis. The interface includes entity resolution and validation checks for the entity, accounting basis, accounting periods, bank account and and cash accounts. It also can search for suitable cash accounts related to an entity in case of cash segregation.

Files added:
eagle_default/in/xml/xml-acct_intrafund_cash_transfer.xml

Files changed:
eagle_default/in/xml/xml-acct_cash.rsf
eagle_default/in/xml/xml-ent_cash_account.xml

SDP-80988 Added feature to resolve the resourceName from the file and pass for processing to EDS

The generic load workflow, eagle_wrf_generic_load, can route files for processing by Data Rules/EDS without configuration files using a global parameter.

New RTR parameters should be used at the same time : *GetResourceName* and *UseEDS*

<taskParameter>
<name>UseEDS</name>
<value>Y</value>
</taskParameter>
<taskParameter>
<name>GetResourceName</name>
<value>Y</value>
</taskParameter>

Global parameters are also supported in the w_config_custom.inc file:
*W_USE_EDS*
*W_GET_RESOURCE_NAME*

Files changed:
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/tagvalue/execute_wrkfl_event.xml
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/set_eds_params.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/w_config.inc

SDP-80208 Additional Security Master fields TYPE_OF_BOND and ARM_INITIAL_FIXED_PERIOD_MONTHS to support DIS BBPS SMF

New security elements typeOfBond and armInitFixedPeriodMonths are supported in the GenericSMF interface for the load and the extract for composite and history mode.

EagleML Xpath:
EagleML/referenceTransaction/genericSMF/SMExtension/typeOfBond
EagleML/referenceTransaction/genericSMF/MBSExtension/armInitFixedPeriodMonths

DB field:
SECURITYDBO.SECMASTER_DETAIL_EXT.TYPE_OF_BOND - Tag 18565
SECURITYDBO.SECMASTER_DETAIL_EXT_HIST.TYPE_OF_BOND - Tag 18565

SECURITYDBO.MBS_DETAIL_EXT.ARM_INIT_FIXED_PERIOD_MONTHS - Tag 18566
SECURITYDBO.MBS_DETAIL_EXT_HIST.ARM_INIT_FIXED_PERIOD_MONTHS - Tag 18566

Files changed:
eagleml/schema/eagleml-sec-shared-2-0.xsd
eagle_default\in\xml\xslt\smf_egl_gen_interface_template.xsl
eagle_default\in\xml\include\smf_aliases.inc
eagle_default\in\xml\include\sm_extension.inc
eagle_default/in/xml/include/mbs_extension.inc
eagle_ml-2-0_cm\out\xml\t_smf_streaming.inc
eagle_ml-2-0_cm\out\xml\t_smf_streaming_eml.inc

SDP-78140 Warehouse Position and Open Lot outbound interfaces have been enhanced with srcSysC element

The srcSysC element has been added to the Warehouse Position - WRHSPOSITIONEXTRACT and Open Lot - WRHSOPENLOTEXTRACT outbound interfaces.

warehouseTransaction/warehousePosition/srcSysC - HOLDINGDBO.POSITION_DETAIL.SRC_SYS_C
warehouseTransaction/warehouseOpenLot/srcSysC - HOLDINGDBO.LOT_LEVEL_POSITION.SRC_SYS_C

Files changed:
eagle_ml-2-0_cm/out/xml/t_warehouse_position_streaming.inc
eagle_ml-2-0_cm/out/xml/t_warehouse_openlot_streaming.inc

SDP-52544 The issueranalytic resource was enhanced to support data load to Oracle

The issueranalytic Data Rules resource has been enhanced to support the load to Oracle using the core stored procedure ESTAR.INS_UPD_ISSUER_ANALYTICS.

The following fields are not supported in SP and cannot be loaded into the Oracle database:
SECURITYDBO.ISSUER_ANALYTICS.RELEASE_DATE
SECURITYDBO.ISSUER_ANALYTICS.RELEASE_STATUS

Files changed:
metadata/core/ontology/reference/issueranalytic-models.json
metadata/core/ontology/reference/issueranalytic_in.json

Files added:
metadata/core/ontology/templates/eagle_ref_issueranalytic_insertupdate.txt

Production Bugs

SDP-85244 The mapping for the commitmentAmount element was corrected as part of Warehouse Position interface

The WarehousePosition load interface is fixed and now applies the value for the commitmentAmount element.

Files changed:
eagle_default/in/xml/xslt/wrh_egl_gen_interface3.xsl

SDP-85220 The Cancel Trade functionality was improved for IPT including Batch Cancels

IPT cancels used to fail in case of closed accounting period. The issue was corrected, the cancel transaction is applied to the first available accounting period now.

Level of Batchcancel - Tag1256 for IPT changed to ALL to follow the logic of the panel. Now IPT can be correctly canceled via Batch Cancel Trades panel.

Additional informational statuses were added to simplify troubleshooting. It is possible to check the list of trades available from {tpe/dynamic/dat/eagle/star/pan-canceltradequery.htm} lookup, the trade with {transactionId} to be canceled and variables after remapping the Batch Cancel Trades panel.

Files changed:
eagle_default/in/include/csv-cancel_columns.inc

SDP-84942 todate and fromdate filters were corrected for the reportdictionarylinkedasset resource

The todate and fromdate filters were referring to the RULESDBO.DICTIONARY.UPDATE_DATE field, but the RULESDBO.SECURITY_DICTIONARY_LINK table has the SECURITY_DATE field that works as update date field. In some cases, the SECURITY_DATE field can be empty and the DICTIONARY.UPDATE_DATE field should be used.

New logic for fromdate filter:
SECURITY_DICTIONARY_LINK.SECURITY_DATE >= :fromdate OR (SECURITY_DICTIONARY_LINK.SECURITY_DATE IS NULL AND DICTIONARIES.UPDATE_DATE >= :fromdate)

New logic for todate filter:
SECURITY_DICTIONARY_LINK.SECURITY_DATE <= :todate OR (SECURITY_DICTIONARY_LINK.SECURITY_DATE IS NULL AND DICTIONARIES.UPDATE_DATE <= :todate)

*Please note
Extract logic was corrected. cloud_conversion workflow should be executed for reportdictionarylinkedasset resource to correct data in Vault.

Files changed:
estar/tpe/dynamic/metadata/core/ontology/reference/reportdictionarylinkedasset.json

SDP-84915 Signage logic changes for CloseSwap and TRRESET events

The SETTLE_AMOUNT_BASE and SETTLEMENT_AMOUNT signage logic has been changed for the TRRESET / RECPT and TRRESET / DISB events in the WRHSTRADEEXTRACT. When signage logic is applied these fields will apply the revised logic for signage.

Files changed:
tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/tagvalue/extract_wrhs_trade_signage_logic.inc

SDP-84807 Forward Trades always routed to Cancel Trade panel

Forward Cancels are routed to the Cancel Trade panel only (eagle/star/trade/pan-cxltrade.htm) since the Batch Cancel panel does not support Forward Trades. Changes were done by interpreting the processingSecurityType outmapped by the eagle/star/pan-canceltradequery.htm lookup. If tag3931 = ‘FWXXXX’, then the batchId - Tag 701is removed and the data is passed to the Cancel Trade. Already canceled trades are removed from the trade lookup leaving only active trades eligible for cancelling.

Both legs of forward trade can be canceled by entityId and tradeTicketNumber.

Files changed:
eagle_default/in/include/csv-cancel_columns.inc

SDP-84481 MultilegSMF inbound interface enhanced with pulling data for existing security from DB for security update

The issue was that on initial security load some fields were calculated. On security updates, it was required to pass these fields in the incoming message.

The issue lookups were added to all three legs to pull data for existing security from database. It resolved the issue.

File added:
eagle_default/in/xml/include/smf_sm_upd.inc

File changed:
eagle_default/in/xml/xml-smf_multileg_swap.xml

SDP-83891 GenericSMF inbound interface was enhanced with skipping delisted securities functionality

Once a security becomes delisted, the Reuters Internal ID for it adds a caret symbol ^ to identify delisting of a security.
For example, the new ID will be ESRX.OQ^L18 as opposed to ESRX.OQ.
The problem was that even though the delisted Reuters ID sent against the same identifiers, a new security was getting created as the existing rule logic treated the combination of characters as a new security.

New functionality was implemented to identify delisting of a security. It’s not limited only to REUTERS id and caret ‘^' sign.
New global parameters were introduced:
W_SKIP_DELISTED_SECURITY - Y/N - Indicates if delisted security will be skipped or not. Default value is N.
W_DELISTED_SECURITY_IDENTIFIER_TYPES - indicates the list of Xreference identifiers types separated by comma or 'ALL’ used for validation. Default value is REUTERS.
W_DELISTED_SECURITY_TOKENS - tokens are used to indicate delisting of a security. List of tokens separated by comma. Default value is caret ^.

Example of the parameters usage in w_config_custom.inc:
<COL TAG="W_SKIP_DELISTED_SECURITY" EXPRESSION="'Y'"/>
<COL TAG="W_DELISTED_SECURITY_IDENTIFIER_TYPES" EXPRESSION="'REUTERS,CINS'"/>
<COL TAG="W_DELISTED_SECURITY_TOKENS" EXPRESSION="'^,#'"/>

Along with global parameters, the RTR parameter, SKIP_DELISTED_SECURITY, is also supported. It should be used with assetResolutionOption. If the parameter is present in the resolution options, it means that functionality for the delisted security is enabled:
<assetResolutionOption>SKIP_DELISTED_SECURITY,ALLOW_DUPL_XREF,REUTERS,SEDOL,ISIN,CUSIP,CINS</assetResolutionOption>.

If one of tokens is passed in one of listed security xreference identifiers - security resolution raises an editcheck.

File changed:
eagle_default/in/xml/include/security_resolution_generic.inc

SDP-83856 Warehouse Position and Taxlots extracts did not return data for secondary basis from portfolio valuation table

The logic for the warehouse position - WRHSPOSITIONEXTRACT and warehouse openlot - WRHSOPENLOTEXTRACT extracts was corrected to include the secondary basis in the query. RTR filters have a higher priority over default logic.

Files changed:
eagle_ml-2-0_cm/out/tagvalue/extract_holding_ext_port_valuation_fields.inc