Enhancements

SDP-75798 The EDS warehousecloselot resource was enhanced

Set of changes were implemented for the warehousecloselot resource:

• Temporal mode is supported now via corresponding Reinit Stored Procedure

• The main table was changed from TRADEDBO.TRADE to TRADEDBO.DISPOSAL_LOTS

• The logic for the DeltaBatch extract was enhanced to support the TRADEDBO.TRADE table for the batch select instead of the master table

Files changed:
metadata/core/ontology/eagle-interfaces.json
metadata/core/ontology/eagledb-crud-filters.json
metadata/core/ontology/warehouse/warehousecloselot_in.json
metadata/core/ontology/warehouse/warehousecloselot.json
metadata/core/ontology/templates/db_oper/snowflake/warehousecloselot.reinit_batch_many.sql
rdf_metadata/rdf_outflow.py
pyrules/eagleinvsys/common_core/dataset/ds_filters/delta_batch_filters/delta_batch_filter_cursor_handler.py
pyrules/eagleinvsys/common_core/dataset/ds_filters/filters.py
pyrules/eagleinvsys/common_core/dataset/ds_filters/filters_enum.py
pyrules/eagleinvsys/common_core/enums/cm_enum.py

SDP-75689 A new global option was introduced for the GenericSMF load interface

A new global option, W_USE_ALT_ASSET_ID_AS_PRIMARY, was implemented for the GenericSMF load interface. The option can be set in the w_config_custom.inc.

The logic activated by this option has two parts:

When the security is determined as a new security and the primaryAssetType from the file matches the global option then the instrumentId and instrumentIdType within the altAssetIdAndType model will replace the primaryAssetId/primaryAssetType in the data file prior to loading.
When security is exists, then the security will be updated. However, the primaryAssetId/primaryAssetType will remain as is in the database fields primaryAssetId/primaryAssetType.
Example for w_config_custom.inc:

<CODE>
:W_USE_ALT_ASSET_ID_AS_PRIMARY: := 'CLIENT_ID';
</CODE>
File added:
eagle_default/in/xml/include/primary_identifier_map.inc

Files changed:
eagle_default/in/xml/xml-smf.xml
eagle_default/in/xml/xml-smf_generic.xml
eagle_default/in/xml/xslt/eagleml_smf_gen_fields.xsl

SDP-75658 EDS delta cache refresh functionality was improved to recalculate header if indicated

The EDS delta cache refresh will rely on the last full reload of data cache to prevent old cached records from becoming outdated.
It is recommended to the ExecuteFullReload of data cache only if a new version of the ontology is installed.

Files changed:
src/common_core/dataset/idsdataframe.py

SDP-75369 The EDS warehouseperformance, warehousenavmulti, warehouseposition, and warehousecloselot resources were enhanced

Warehouse resources were validated and enhanced to pass the CheckSum validation
warehouseperformance
Fixed mapping for PERF_SUMMARY.PS_USER_FLOAT1 field

warehousecloselot
Fixed issues with UPDATE_DATE and UPDATE_SRC fields

warehousenavmulti
Added mapping for the following fields and fixed minor issues with mapping:
HOLDINGDBO.NAV_MULTI.FIXED_DISTRIBUTION_AMT - fixedDistributionAmount
HOLDINGDBO.NAV_MULTI.MILL_RATE - millRate
HOLDINGDBO.NAV_MULTI.PERIODIC_DISTRIBUTION_AMT - periodicDistributionAmount
HOLDINGDBO.NAV_MULTI.REPORTING_NAV_PRIOR - reportingNavPrior
HOLDINGDBO.NAV_MULTI.REPORTING_PUBLIC_OFFER_PRIOR - reportingPublicOfferPrior

warehouseposition
Excluded POSITION.POS_LOT_FLAG from CheckSum.
Added mapping for the following fields and fixed minor issues with mapping:
HOLDINGDBO.POSITION.PRICE_SOURCE - posPriceSource
HOLDINGDBO.POSITION.SHORT_PRICE_SOURCE - shortPriceSource
HOLDINGDBO.POSITION.XRATE_VAL_SOURCE - xrateValSource
HOLDINGDBO.POSITION.FWD_PRICE_SOURCE - fwdPriceSource
HOLDINGDBO.POSITION.PRIVATE_EQUITY_PRICE_SOURCE - privateEquityPriceSource
HOLDINGDBO.POSITION_DETAIL.CASH_RECEIVABLES_L - cashReceivablesLocal
HOLDINGDBO.POSITION_DETAIL.CASH_RECEIVABLES_B - cashReceivablesBase
HOLDINGDBO.POSITION_DETAIL.CASH_RECEIVABLES_S - cashReceivablesSystem
HOLDINGDBO.POSITION_DETAIL.UNEARN_SOLD_INT_ACC_L - unearnedSoldInterestAccrualLocal
HOLDINGDBO.POSITION_DETAIL.UNEARN_SOLD_INT_ACC_B - unearnedSoldInterestAccrualBase
HOLDINGDBO.POSITION_DETAIL.UNEARN_SOLD_INT_ACC_S - unearnedSoldInterestAccrualSystem
HOLDINGDBO.POSITION_DETAIL.PUSH_TYPE - pushType
HOLDINGDBO.POSITION_DETAIL.CASH_ACCT_ALIAS - cashAccountAlias
HOLDINGDBO.POSITION_DETAIL.SRC_SYS_C - srcSysC
HOLDINGDBO.POSITION_COST_DETAIL.COMMITMENT_B - commitmentBase
HOLDINGDBO.POSITION_COST_DETAIL.FUNDED_CAPITAL_L - fundedCapitalLocal
HOLDINGDBO.POSITION_COST_DETAIL.FUNDED_CAPITAL_B - fundedCapitalBase
HOLDINGDBO.POSITION_COST_DETAIL.RECALLABLE_CAPITAL_L - recallableCapitalLocal
HOLDINGDBO.POSITION_COST_DETAIL.RECALLABLE_CAPITAL_B - recallableCapitalBase
HOLDINGDBO.POSITION_COST_DETAIL.UNFUNDED_COMMITMENT_L - unfundedCommitmentLocal
HOLDINGDBO.POSITION_COST_DETAIL.UNFUNDED_COMMITMENT_B - unfundedCommitmentBase
HOLDINGDBO.POSITION_COST_DETAIL.CAPITAL_CALLS_L - capitalCallsLocal
HOLDINGDBO.POSITION_COST_DETAIL.CAPITAL_CALLS_B - capitalCallsBase

Files changed:
rdf_metadata/rdf_meta_utils.py
rdf_metadata/rdf_outflow.py
rdf_metadata/rdf_physical_schema_generator.py
metadata/core/ontology/eagledb-inventory.json
metadata/core/ontology/definition/eagle-vocab-definition-entity-shared.json
metadata/core/ontology/definition/eagle-vocab-definition-shared.json
metadata/core/ontology/definition/eagle-vocab-definition-wh-shared.json
metadata/core/ontology/eagle-vocab-entity-shared.json
metadata/core/ontology/eagle-vocab-shared.json
metadata/core/ontology/eagle-vocab-wh-shared.json
metadata/core/ontology/warehouse/warehousecloselot-models.json
metadata/core/ontology/warehouse/warehousenavmulti-models.json
metadata/core/ontology/warehouse/warehouseperformance_in.json
metadata/core/ontology/warehouse/warehouseposition-models.json
metadata/core/ontology/warehouse/warehouseposition_in.json
metadata/core/ontology/templates/db_oper/snowflake/warehouseposition.reinit_batch.sql
metadata/core/ontology/templates/db_oper/snowflake/warehouseposition.reinit_batch_many.sql
metadata/core/ontology/templates/eagle_wrhs_warehouseposition_reinitbatch.oracle.txt

SDP-75215 The EDS CSV Parser was Enhanced

The EDS CSV Parser was enhanced to read double quotes escaped lines correctly to process the carriage return - \r value on load and extract.

Files changed:

src/common_core/core_utils/string_utils.py
src/common_core/ds_parser/dsparser.py

SDP-75214 The EDS genericentity resource was updated to accurately populate the security.LEI_DETAILS.ID_LAST_UPDATE_DATE field

The EDS genericentity resource was updated to accurately populate the security.LEI_DETAILS.ID_LAST_UPDATE_DATE field on data upload to Oracle.

Files changed:
metadata/core/ontology/entity/genericentity-models.json

SDP-37474 The AccountingBasis field has been added to the accruedIncome extract - - ACCRUEDINCOMEEXTRACT

A new element, accountingBasis, has been created for the - ACCRUEDINCOMEEXTRACT. The value for this element is retrieved from the ESTAR.ESTAR_INCOME_POSITION.ACCT_BASIS field. By default, all basis's for a multi-basis entity return.

Files changed:
eagle_ml-2-0_cm/out/tagvalue/extract_accrued_income.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/xml/t_accrued_income.xml
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/xml/t_accrued_income_streaming.inc

SDP-75186 The Accrued Income extract - ACCRUEDINCOMEEXTRACT now supports filtering by accounting basis

A filter for the accountingBasis was added to the accruedIncome extract. It is now possible to extract records by the non-primary basis for a multi-basis entity for the ACCRUEDINCOMEEXTRACT.

Sample from RTR:

<taskParameter>
<name>accountingBasis</name>
<dataType>S</dataType>
<value>USTAX</value>
</taskParameter>

Files changed:
eagle_ml-2-0_cm/out/tagvalue/extract_accrued_income.inc

SDP-74666 WRHSTRADEEXTRACT - Updated signage logic for list of fields

The signage logic was removed for the below fields:

WarehouseTransaction/warehouseTrade/originalAcquisitionCostBase (ORIG_ACQ_COST_BASE)
WarehouseTransaction/warehouseTrade/originalAcquisitionCostLocal (ORIG_ACQ_COST_LOCAL)
New logic was added for OPEN SWAP and CLOSE SWAP transactions for the following fields:
WarehouseTransaction/warehouseTrade/accruedInterest (ACCRUED_INTEREST)
WarehouseTransaction/warehouseTrade/interestAdjBase (BASEINTERESTADJ)

Files changed:
eagle_ml-2-0_cm/out/tagvalue/extract_wrhs_trade_signage_logic.inc

SDP-74397 Several fields in the EDS issueprice and exchangeprice resources were updated

The mapping for exchangeprice and issueprice were corrected for fields in the SECURITYDBO.PRICE_EXCHANGE and the SECURITYDBO.PRICE tables

REC_LEG_PRICE_OR_MV
mapped to payPriceLeg/price if PRICE_TYPE_CODE = 5
mapped to payPriceLeg/marketValue if PRICE_TYPE_CODE = 4 or asiscopy
reversed behavior applied on load

PAY_LEG_PRICE_OR_MV
mapped to receivePriceLeg/price if PRICE_TYPE_CODE = 5
mapped to receivePriceLeg/marketValue if PRICE_TYPE_CODE = 4 or asiscopy
reversed behavior applied on load

With this particular logic EDS always load data from these fields in asiscopy mode.

Files changed:
ontology/reference/exchangeprice_in.json
ontology/reference/exchangeprice-models.json
ontology/reference/issueprice_in.json
ontology/reference/issueprice-models.json

SDP-74261 A new entitylist Data Rules Resource was introduced

The entitylist resource was implemented to support the extract from Oracle and the load to Snowflake as part of the Eagle to Vault pipeline.

Table: RULESDBO.ENTITY_LIST
LoadModes: Delta, Temporal

Extract Filters:
entityfilter, fromdate, todate,

Files changed:
metadata/core/ontology/eagle-resolution-processing.json
metadata/core/ontology/eagle-vocab-entity.json

Files added:
metadata/core/ontology/entity/entitylist-models.json
metadata/core/ontology/entity/entitylist.json
metadata/core/ontology/entity/entitylist_in.json
metadata/core/ontology/templates/eagle_ent_entitylist_insertupdate.txt

SDP-74137 MultilegSMF Swap Transaction Enhancement to Map Fields from Contract Leg to Pay and Receive Legs

The MultilegSMF load interface was enhanced that the below elements will now get their values from the contract leg and set on the contract. Prior to this change, it was possible to load different values in the initialExchangeOfPrincipal and finalExchangeOfPrincipal in each leg when the accountingValidationFlag = 'N'.

These elements get their value applied from contract initialExchangeOfPrincipal element.

Files changed:
eagle_default/in/xml/include/contract_mapping.inc
eagle_default/in/xml/include/pay_mapping.inc
eagle_default/in/xml/include/receive_mapping.inc

SDP-74133 Additional elements supported for the InterportfolioTransfer load

The InterportfolioTransfer inbound interface was enhanced to load the below elements.

accountingTransaction/interportfolioTransfer/quantityOrRatioIndicator - Tag 739
accountingTransaction/interportfolioTransfer/price - Tag 319
accountingTransaction/interportfolioTransfer/quantityRatio - Tag 61

Files changed:
eagle_default/in/xml/xml-acct_interporttransfer.xml

SDP-73882 The EDS warehousenavmulti resource was enhanced to support BundleBatch mode on the data upload to Snowflake

The warehousenavmulti resource had Batch mode only and processed batches one by one.
The Bundle Batch mode was enabled for the warehousenavmulti resource to process all batches at once.

A Filter called usedeltabatch was added for the extract.

Files changed:
metadata/core/ontology/templates/eagle_wrhs_warehousenavmulti_reinitbatch.oracle.txt
metadata/core/ontology/templates/eagle_wrhs_warehousenavmulti_reinitbatch.snowflake.txt
metadata/core/ontology/warehouse/warehousenavmulti.json
metadata/core/ontology/warehouse/warehousenavmulti_in.json

Files added:
metadata/core/ontology/templates/db_oper/snowflake/warehousenavmulti.reinit_batch_many.sql

SDP-73549 The metadata for vendorcashflow resource was updated

The mapping for SECURITYDBO.VENDOR_CASH_FLOW.UNIQUE_ID field was corrected on the load and extract.

Files changed:
vendorcashflow-models.json

SDP-71477 Updated the naicReviewDate element from ISTAR.XREFERENCE table for the InsuranceValuationLot extract

Moved the naicReviewDate from the eagle-vocab-definition-ref-shared.json, eagle-vocab-ref-shared.json to eagle-vocab-definition-shared.json, eagle-vocab-shared.json accordingly.
Added the xreference model for insurancevaluationlot extract for versions >= 2017.2.35
Added naicReviewDate to xreference model for insurancevaluationlot extract for versions >= 2017.2.35

Files changed:
ontology/definition/eagle-vocab-definition-ref-shared.json
ontology/definition/eagle-vocab-definition-shared.json
ontology/eagle-vocab-ref-shared.json
ontology/eagle-vocab-shared.json
ontology/warehouse/insurancevaluationlot.json
ontology/warehouse/insurancevaluationlot-models.json

SDP-71446 The EDS warehousenav resource was enhanced to improve performance on the data load to Snowflake

Previously the warehousenav resourse had only Batch mode implemented and loaded batches one by one.
The Bundle Batch mode was enabled for the warehousenav resource to process all batches at once to optimize the processing time.

Filter usedeltabatch was added to the extract.

Files changed:
metadata/core/ontology/templates/eagle_wrhs_warehousenav_reinitbatch.oracle.txt
metadata/core/ontology/templates/eagle_wrhs_warehousenav_reinitbatch.snowflake.txt
metadata/core/ontology/warehouse/warehousenav.json
metadata/core/ontology/warehouse/warehousenav_in.json
metadata/core/ontology/warehouse/warehousenav-models.json

Files added:
metadata/core/ontology/templates/db_oper/snowflake/warehousenav.reinit_batch_many.sql

SDP-67734 Introduced a new EDS genericcadetail resource

A new EDS resource called genericcadetail was implemented to improve the Eagle to Vault data pipeline for the genericca resource.

The genericcadetail resource provides support for below tables going forward: TRADESDBO.CORP_ACT_HIST_DETAIL
TRADESDBO.CORP_ACT_HIST_DETAIL_HIST

The above tables are marked as auxiliary for genericca resource and excluded from EDS extract in ASISCOPY mode.

Files changed:
ontology/reference/genericca.json
ontology/reference/genericca_in.json

Files added:
ontology/reference/genericcadetail.json
ontology/reference/genericcadetail_in.json
ontology/reference/genericcadetail-models.json

SDP-67733 Introduced a new EDS genericcaextension resource

A new EDS resource, genericcaextension, was implemented as part of the E2V pipeline to support below tables:
TRADESDBO.CORP_ACT_HIST_EXTENSION
TRADESDBO.CORP_ACT_HIST_EXTENSION_HIST

The above tables are marked as auxiliary for genericca resource and excluded from the EDS extract in ASISCOPY mode.

Files changed:
ontology/reference/genericca.json
ontology/reference/genericca_in.json
Files added:
ontology/reference/genericcaextension.json
ontology/reference/genericcaextension_in.json
ontology/reference/genericcaextension-models.json

Production Bugs

SDP-76056 The cm_exec_eds EDS interface was improved

The EDS stream eagle_ml-2-0_default_cm_exec_eds occasionally could process requests with the same correlation ID multiple times even during the duplicate correlation ID, where the EDS sub-process was executed anyway. Data Rules were corrected, and the issue was fixed. The EDS stream will now correctly reject RTR with duplicate correlation ID’s without EDS sub-process execution.

SDP-75416 Optimized performance of the PRICINGREPORTEXTRACT object

The PRICINGREPORTEXTRACT has been optimized to filter the data by Holdings during the intermediate processing. The extract has multiple steps and in previous releases of the PRICINGREPORTEXTRACT it was initially filtered by Holdings and then queried for Prices for the effective date requested without reapplying holdings. In a high-volume environment, the intermediate processing was taking significant time. This change streamlines the processing and improves throughput.

Files changed:
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/tagvalue/extract_pricing_report.inc

SDP-75008 The EDS reportdictionarylevel resource was updated to avoid duplicates in dictionary_format table

The SQL join for the reportdictionarylevel resource was corrected.

The RULESDBO.PACE_MASTER.GROUP_CODES table was added to excluded list of tables to resolve the issue with duplicates.

Files changed:
metadata/core/ontology/reference/reportdictionarylevel.json

SDP-74381 Added handling of special characters in Data Rules

The EDS extract and load process was improved to support special characters correctly. The following was improved:

• EDS extracts are encoded in utf-8 format

• Encoding for slash symbol

• Encoding for quote symbol

Files changed:
src/common_core/descriptor/dbdescriptor_sf.py
src/common_core/processing_env/instructions.py

SDP-73266 The EDS issueresg resource was corrected for history mode

The metadata for issueresg was corrected. The srcIntfcInst element was added.

Files changed:
metadata/core/ontology/reference/issueresg-models.json

SDP-73173 The issuerEntityRelationship object was updated to mirror Portfolio Data Center load logic

Fixed issuerEntityRelationship load logic according to Portfolio Data Center.
Adjusted DELETE issuerEntityRelationship functionality.
Fixed entity lookup to return exact value of entity id.

Files changed:
eagle_default/in/xml/xml-ref_issuerentityrelationship.xml
eagle_ml-2-0_cm/generate_sp_on_fly.inc
eagle_default/in/xml/include/ref_issuerentityrelationship_delete.inc
dynamic/metadata/bind/eagleml_ref_issuerentityrelationship.xml

SDP-69236 Added MS SQL Server support for the yieldCurve procedure during load

The SQL query for the yieldCurve load was added to the yield curve object for data load. The query will be generated after release install and can also be regenerated manually by sending the setup_version.xml to eagle_ml-2-0_default_cm_w_setup (restart of MC service is required).

Files changed:
eagle_ml-2-0_cm/generate_sp_on_fly.inc
eagle_default/in/xml/xml-ref_yieldcurve.xml

SDP-68416 Add and updated mapping for the following fields within the Position Model

HOLDINGDBO.POSITION_COST_DETAIL.LOCAL_TAX_COST_VALUE - EagleML/warehousePosition/localTaxCostValue
HOLDINGDBO.REAL_ESTATE_POSITION_DETAIL.SOURCE_NAME - EagleML/warehousePosition/realEstateUDF/reSourceName
HOLDINGDBO.POSITION_DETAIL_ANL.PRICE_DATE - EagleML/warehousePosition/analytics/pdaPriceDate
HOLDINGDBO.POSITION_DETAIL_ANL_FI.COUPON - EagleML/warehousePosition/analytics/pdafCoupon

Files changed:
metadata/core/ontology/eagle-vocab-wh-shared.json
metadata/core/ontology/warehouse/warehouseposition-models.json
metadata/core/ontology/warehouse/warehouseposition_in.json

SDP-64252 The EDS entityshareclass resource was updated to implement support for the RULESDBO.ENTITY_SHARE_CLASS_HIST table via the Eagle - Vault pipeline

The RULLESDBO.ENTITY_SHARE_CLASS_HIST database table is now supported for the load and the extract as part of the E2V pipeline.