EagleML Release Notes - October 18th 2022

Enhancements

SDP-66404 EDS load - Improved sniffing functionality for EagleML format

The Sniffing logic for the EagleML format was improved as part of EDS load to Oracle database. The SniffRowCount parameter is no longer required. The Data Rules will check all rows in the file. This change will help to resolve performance issue with EagleML format as part of EDS warehouseopenlot load.

Files changed:
pyrules/eagleinvsys/common_core/dataset/idsdataframe.py
pyrules/eagleinvsys/common_core/ds_parser/dsparser.py
pyrules/eagleinvsys/common_core/ds_parser/dsxmlparser.py
pyrules/eagleinvsys/common_core/enums/def_enums.py
pyrules/eagleinvsys/common_core/processing_env/env_db.py
pyrules/eagleinvsys/common_core/sink/isink.py
pyrules/eagleinvsys/etl/dynamic_template_processor.py

SDP-66330 Reportdictionary resources were updated to support additional filters

Changes were implemented to support filters required for delta extracts:
fromdate/todate filters for the entitypolicyrate resource are joined with the RULESDBO.POLICY_RATES.UPD_DATE
fromdate/todate filters for the entitypolicydetail resource are joined with the RULESDBO.POLICY.UPD_DATE
fromdate/todate filters for the entitypolicymap resource are joined with the RULESDBO.POLICY.UPD_DATE
fromdate/todate filters for the reportdictionarylinkedasset resource are joined with the RULESDBO.DICTIONARIES.UPDATE_DATE
fromdate/todate filters for the reportdictionarylevel resource are joined with the RULESDBO.DICTIONARIES.UPDATE_DATE

Files changed:
src/common_core/processing_env/processing_result.py
src/common_core/processing_env/proc_ctx_enum.py

SDP-66309 The MultilegSMF inbound interface was improved to resolve the Underlying Security on insert

As part of the Multileg SMF resolution, the Underlying security was being resolved on the panel lookup only using the xreference id and not including the xreference type. As a result, in some instances where multiple xreference id’s existed, the incorrect underlying security with the same xreference id, but a different xreference type could be resolved as the first one was returned. Additional resolution was added to the MultilegSMF inbound rule to resolve the Underlying Security by the xreference id and xreference type before the load process.

Files changed:
eagle_default/in/xml/xml-smf_multileg_swap.xml

SDP-66232 CalcTransactionId and calcCancelTransactionId enhancement for WRHSTRADEEXTRACT for PIKBONDCANCEL and COUPONCANCEL

The SQL query logic for the calcTransactionId and the calcCancelTransactionId was enhanced for the PIKBONDCANCEL/COUPONCANCEL trades. When extracting the WarehouseTrade with the EnhanceCalcTransactionId=Y and the IncludeExtendedDetails=Y, the calcTransactionId and calcCancelTransactionId values contain the transaction code of the original record like the PIKBOND and COUPON.

Example:

Transaction Ids from ENT66232trade001 PIKBOND record:

Transaction Ids from ENT66232trade003 PIKBONDCANCEL record:

Transaction Ids from ENT66232trade002 COUPON record:

Transaction Ids from ENT66232trade004 COUPONCANCEL record:

File changed:
eagle_ml-2-0_cm\out\tagvalue\extract_warehouse_trade.inc

SDP-65759 СalcMasterTransactionId enhancement for WRHSTRADEEXTRACT and 150/250 RGHTEXER/RGHTEXERCANCEL trade

The SQL query logic for the СalcMasterTransactionId was enhanced for 150 MTOPEN L N N RGHTEXER Y and 250 MTOPEN L N N RGHTEXERCANCEL Y trades. When extracting the WarehouseTrade with EnhanceCalcTransactionId=Y and IncludeExtendedDetails=Y, the calcMasterTransactionId value contains the LINK_EVENT_ID (EagleML - linkEventId) for the RGHTEXER and RGHTEXERCANCEL.

Example:

File changed:
eagle_ml-2-0_cm\out\tagvalue\extract_warehouse_trade.inc

SDP-65413 Rollback on Failure functionality was improved for EDS loads

The EDS WarehouseOpenLot did not rollback the whole batch on failure. While no data was loaded to the Position_Detail, the records still loaded to the holdingdbo.lot_level_position. EDS only supports Rollback functionality for In-Market/Oracle. If one record fails, the whole batch will be rejected. Upon correction, the whole batch needs to be reloaded.

In order to rollback the whole batch on failure, include the following RTR parameters:

<taskParameter>
<name>updatemode</name>
<value>batch</value>
</taskParameter>
<taskParameter>
<name>alwayscommit</name>
<value>N</value>
</taskParameter>

Files changed:
src/common_core/descriptor/dbdescriptor.py

SDP-64945 InactiveFlag Supported in Asset Resolution Options

The INACTIVE_FLAG was added as an asset resolution option for all modes.

1. If the INACTIVE_FLAG is not included, the resolution is processed as usual and the inactive records will be ignored during the resolution process

2. If the INACTIVE_FLAG is included and there are no active securities, a new security is created

3. If the INACTIVE_FLAG is included and the active security exists in the database, it is updated with the data provided.

It also resolves equally records with ‘N' and a null inactive_flag field in the database. For consistency, it is recommended to populate the <inactiveFlag>N</inactiveFlag> during creation and maintenance of active securities. When there is no inactive_flag value populated, there is no indicator to the client.

Files changed:
eagle_default/in/xml/include/security_resolution_generic.inc
eagle_default/in/xml/include/match_all_sec_resolution.inc
eagle_default/in/xml/include/match_all_new_record_check.inc

SDP-64696 EDS entitymanagerrelationship Resource

EDS entitymanagerrelationship Resource support load and extract to/from the RULESDBO.ENTITY_RANGE_DETAILS table for both Oracle and Snowflake database providers.

Files changed:
metadata/core/ontology/entity/entitymanagerrelationship-models.json
metadata/core/ontology/entity/entitymanagerrelationship_in.json
metadata/core/ontology/entity/entitymanagerrelationship.json
metadata/core/ontology/templates/ eagle_ent_entitymanagerrelationship_insertupdate.txt
metadata/core/ontology/eagle-resolution-processing.json

SDP-64158 New assetToBaseFxRate element for the TradedCashActivity and SettledCashActivity Extracts

The SETTLEDCAEXTRACT and TRADEDCAEXTRACT were enhanced to support new element assetToBaseFxRate.

This element is dynamically calculated during the extract and does not have a corresponding database field. The element calculates the asset currency to the base currency fx rate.

To include this element in extract set the parameter IncludeAssetToBaseFxRate to Y in extract RTR:

EagleML XPath:
EagleML/accountingTransaction/settledCashActivity/assetToBaseFxRate
EagleML/accountingTransaction/tradedCashActivity/assetToBaseFxRate

Files changed:
eagle_ml-2-0_cm\out\tagvalue\extract_acct_traded_cash_activity.inc
eagle_ml-2-0_cm\out\xml\t_acct_traded_cash_activity_streaming.inc
eagle_ml-2-0_cm\out\tagvalue\extract_acct_settled_cash_activity.inc
eagle_ml-2-0_cm\out\xml\t_acct_settled_cash_activity_streaming.inc

SDP-63482 Enabled Cloud Workflow to execute with all parameters supported by the generic extract

The Cloud workflow may be executed with any parameters supported by the generic extract without additional changes. The SQL queries and extracted data generated are similar for the generic extract and the cloud extract. All filters supported by the generic extract also may be passed in cloud workflow and work same way.
Files changed:
eagle_ml-2-0_cm/create_cm.inc

SDP-58257 EDS underlyingsmf Resource

A new EDS resource, UnderlyingSMF, was implemented to support the data upload to the SECURITYDBO.UNDERLYING_SECURITY for database Snowflake.
The extract is supported for both database providers: Snowflake and Oracle.

Files changed:
ontology/reference/underlyingsmf.json
ontology/reference/underlyingsmf-models.json
ontology/reference/underlyingsmf_in.json

SDP-57767 EDS genericIssueanalytic Resource was redesigned

The GenericIssueAnalytic resource was improved to support data upload in Delta or Update mode. Group key for batch was removed, to keep the resource it in sync with other reference resources.
The Batch Load mode was changed and it is no longer supports DELETE operations. Load via batch mode can be used for initial data load from Oracle to Snowflake.

Supported Load modes:
Oracle - Batch
Snowflake - Batch, Delta, Temporal.

Files changed:
genericissueanalytic.json
genericissueanalytic_in.json

SDP-52595 EDS issueridentifiers Resource

New EDS resource IssuerIdentifiers was implemented to support load to Snowflake database table SECURITYDBO.ISSUER_XREF.
Extract is supported for Snowflake and Oracle database providers.

Files changed:
ontology/reference/issueridentifiers.json
ontology/reference/issueridentifiers-models.json
ontology/reference/issueridentifiers_in.json

Production Bugs

SDP-66861 Warehouse position extract not returning data for all partitions with IncludePortfolioValuation parameter

The WRHSPOSITIONEXTRACT was not returning any rows for the non-primary partition when including the IncludePortfolioValuation parameter in extract.

The issue was corrected and the Warehouse position extract returns data for all partitions when requested.

Files changed:
eagle_ml-2-0_cm/extract_transform_sql.inc
eagle_ml-2-0_cm/out/tagvalue/extract_warehouse_position.inc

SDP-66968 Warehouse Position Inbound and Outbound interfaces were enhanced to support new fields

The Warehouse Position interface was extended for missing fields from POSITION_DETAIL_ANL, POSITION_DETAIL_ANL_EQ, POSITION_DETAIL_ANL_FI, POSITION_DETAIL_ANL_KRD and POSITION_DETAIL_ANL_USER tables for the load and the WRHSPOSITIONEXTRACT. The missed mapping was added for the following fields:

HOLDINGDBO.POSITION_DETAIL_ANL_FI.ADJ_DAYS_TO_CASH (tag 7967)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.ADJ_DAYS_TO_SELL (tag 7965)
HOLDINGDBO.POSITION_DETAIL_ANL.AVG_DAILY_VOLUME (tag 7963)
HOLDINGDBO.POSITION_DETAIL_ANL.DAYS_TO_CASH (tag 7966)
HOLDINGDBO.POSITION_DETAIL_ANL.DAYS_TO_SELL (tag 7962)
HOLDINGDBO.POSITION_DETAIL_ANL.LIQUIDITY_CLASSIFICATION (tag 5434)
HOLDINGDBO.POSITION_DETAIL_ANL.LIQUIDITY_SCORE (tag 7960)
HOLDINGDBO.POSITION_DETAIL_ANL.NOTIONAL_MV_ADJ (tag 8189)
HOLDINGDBO.POSITION_DETAIL_ANL.NOTIONAL_MV_LOCAL_ADJ (tag 8377)
HOLDINGDBO.POSITION_DETAIL_ANL.PARENT_PRICE (tag 2393)
HOLDINGDBO.POSITION_DETAIL_ANL.RAR_QUANTITY (tag 7961)
HOLDINGDBO.POSITION_DETAIL_ANL.REASONABLE_TRADE_SIZE (tag 7968)
HOLDINGDBO.POSITION_DETAIL_ANL.STAT_BOOK_YIELD (tag 7964)
HOLDINGDBO.POSITION_DETAIL_ANL.DEBT_TO_CAPITAL (tag 7211)
HOLDINGDBO.POSITION_DETAIL_ANL.DEBT_TO_EQUITY (tag 7212)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.ADJ_TRADING_COST (tag 15707)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.CAPITAL_STRUCTURE_SCORE (tag 15702)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.DV100_BASE (tag 7975)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.DV100_LOCAL (tag 7976)H
OLDINGDBO.POSITION_DETAIL_ANL_FI.E_SPREAD (tag 1589)
HOLDINGBO.POSITION_DETAIL_ANL_FI.FED_HOLDING_PERCENT_SCORE (tag 7989)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.FIXED_TRADING_COST (tag 15704)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.HIGH_LOW_PRICE_SCORE (tag 7988)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.INDEX_ELIGIBILITY_SCORE (tag 7986)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.INFLATION_FACTOR (tag 17566)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.ISSUE_SIZE_SCORE (tag 7983)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.ISSUE_SIZE_SCORE_TAX (tag 15703)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.I_SPREAD (tag 1590)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.MI_TRADING_COST (tag 15705)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.PARENT_OPTION_DELTA (tag 2392)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.PRICE_SCORE (tag 7982)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.RATING_SCORE (tag 15701)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.RUN_OFF_SCORE (tag 7987)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.SDV01_BASE (tag 7977)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.SDV01_LOCAL (tag 7978)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.SECTOR_SCORE (tag 7984)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.STATE_SCORE (tag 7981)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.SUB_SECTOR_SCORE (tag 7985)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.TOP_PARENT_ISSUER_SCORE (tag 15700)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.TOTAL_TRADING_COST (tag 15706)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.YTC (tag 8610)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.YTFC (tag 8611)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.ZV_MATURITY_YEARS (tag 16707)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.Z_SPREAD (tag 4942)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.LOGN_VOLATILITY_DURATION (tag 16082)
HOLDINGDBO.POSITION_DETAIL_ANL_FI.DAYS_TO_MATURITY_SCORE (tag 5800)
HOLDINGDBO.POSITION_DETAIL_ANL_KRD.KRBC_10YR_BY_30YR (tag 7192)

Xpath:
EagleML/warehouseTransaction/warehousePosition/analytics/adjustedDaysToCash
EagleML/warehouseTransaction/warehousePosition/analytics/adjustedDaysToSell
EagleML/warehouseTransaction/warehousePosition/analytics/averageDailyVolume
EagleML/warehouseTransaction/warehousePosition/analytics/daysToCash
EagleML/warehouseTransaction/warehousePosition/analytics/daysToSell
EagleML/warehouseTransaction/warehousePosition/analytics/liquidityClassification
EagleML/warehouseTransaction/warehousePosition/analytics/liquidityScore
EagleML/warehouseTransaction/warehousePosition/analytics/notionalMarketValueAdjustmentBase
EagleML/warehouseTransaction/warehousePosition/analytics/notionalMarketValueAdjustmentLocal
EagleML/warehouseTransaction/warehousePosition/analytics/parentPrice
EagleML/warehouseTransaction/warehousePosition/analytics/rarQuantity
EagleML/warehouseTransaction/warehousePosition/analytics/reasonableTradeSize
EagleML/warehouseTransaction/warehousePosition/analytics/statBookYield
EagleML/warehouseTransaction/warehousePosition/analytics/debtToCapital
EagleML/warehouseTransaction/warehousePosition/analytics/debtToEquity
EagleML/warehouseTransaction/warehousePosition/analytics/adjustedTradingCost
EagleML/warehouseTransaction/warehousePosition/analytics/capitalStructureScore
EagleML/warehouseTransaction/warehousePosition/analytics/dv100Base
EagleML/warehouseTransaction/warehousePosition/analytics/dv100Local
EagleML/warehouseTransaction/warehousePosition/analytics/excessSpread
EagleML/warehouseTransaction/warehousePosition/analytics/fedHoldingPercentScore
EagleML/warehouseTransaction/warehousePosition/analytics/fixedTradingCost
EagleML/warehouseTransaction/warehousePosition/analytics/highLowPriceScore
EagleML/warehouseTransaction/warehousePosition/analytics/indexEligibilityScore
EagleML/warehouseTransaction/warehousePosition/analytics/inflationFactor
EagleML/warehouseTransaction/warehousePosition/analytics/issueSizeScore
EagleML/warehouseTransaction/warehousePosition/analytics/issueSizeScoreTax
EagleML/warehouseTransaction/warehousePosition/analytics/interpolatedSpread
EagleML/warehouseTransaction/warehousePosition/analytics/miTradingCost
EagleML/warehouseTransaction/warehousePosition/analytics/parentOptionDelta
EagleML/warehouseTransaction/warehousePosition/analytics/priceScore
EagleML/warehouseTransaction/warehousePosition/analytics/ratingScore
EagleML/warehouseTransaction/warehousePosition/analytics/runOffScore
EagleML/warehouseTransaction/warehousePosition/analytics/sdv01Base
EagleML/warehouseTransaction/warehousePosition/analytics/sdv01Local
EagleML/warehouseTransaction/warehousePosition/analytics/sectorScore
EagleML/warehouseTransaction/warehousePosition/analytics/stateScore
EagleML/warehouseTransaction/warehousePosition/analytics/subSectorScore
EagleML/warehouseTransaction/warehousePosition/analytics/topParentIssuerScore
EagleML/warehouseTransaction/warehousePosition/analytics/totalTradingCost
EagleML/warehouseTransaction/warehousePosition/analytics/ytc
EagleML/warehouseTransaction/warehousePosition/analytics/ytfc
EagleML/warehouseTransaction/warehousePosition/analytics/zeroVolatilityMaturityYears
EagleML/warehouseTransaction/warehousePosition/analytics/zeroVolatilitySpread
EagleML/warehouseTransaction/warehousePosition/analytics/longnormalVolatilityDuration
EagleML/warehouseTransaction/warehousePosition/analytics/daysToMaturityScore
EagleML/warehouseTransaction/warehousePosition/analytics/krbc10yrBy30yr

Files changed:
eagle_default/in/xml/xslt/wrh_egl_gen_interface3.xsl
eagle_ml-2-0_cm/out/xml/t_warehouse_position_streaming.inc
eagle_ml-2-0_cm/out/xml/t_warehouse_position_xmldbapi_streaming.inc

SDP-66495 EntityAccountingBasis - fix for fwdPriceSource

The element fwdPriceSource (tag2920) on the EntityAccountingBasis was not a supported value from the incoming message. The logic of tag 2920 was fixed in the GenericEntity to support the value from the incoming message.

Related EagleML element:
EagleML/entityTransaction/entityAccountingBasis/fwdPriceSource

Files changed:
eagle_default\in\xml\xml-ent_entityaccountingbasis.xml

SDP-66461 Schedule interface was adjusted to support the journalizingFlag element

The Schedule inbound interface was corrected by mapping the journalizingFlag element into tag1293.

Files changed:
eagle_default/in/xml/xml-ref_schedule.xml

SDP-65825 Delta Based EJM process status in Automation Center

An EJM process with "GWDELTA_" in the schedule name was created in the Automation Center when performing a delta-based extraction. This process has always existed in the "In Progress" status.

The code of Delta Based extract was improved to set the “Completed” status and the “stop time” for EJM processes in AC.

Files changed:
eagle_ml-2-0_cm\out\tagvalue\extract_delta_based.inceagle_ml-2-0_cm\out\tagvalue\extract_delta_orch.inc

SDP-63453 EDS warehousecashactivity Resource was improved

Missing definitions for elements were added into the ontology:

estar_instance
holding_term
performance_eff_date
regulatory_intent
spd_perf_flow_rule
spd_perf_fx_rate
user_float7
user_float8
price_method
to_quantity
user_settl_inst1
user_settl_inst2
user_settl_inst3
user_settl_inst4
clearing_broker

These elements were mapped into the canonical elements:

estarInstance
holdingTerm
performanceEffectiveDate
regulatoryIntent
spdPerformanceFlowRule
spdPerformanceFXRate
priceMethod
toQuantity
userSettlementInstruction1
userSettlementInstruction2
userSettlementInstruction3
userSettlementIn

Files changed:
core/ontology/warehouse/warehousecashactivity-models.json
core/ontology/definition/eagle-vocab-definition-wh-shared.json

SDP-62816 EDS issuerrelationship Resource was improved to correct the issue with duplicated records

The records were duplicated due to an incorrect primary key. The primary key for the issuer relationship resource was adjusted.

Active Primary Key is set to: EFFECTIVE_DATE,SRC_INTFC_INST,FROM_ISSUER_ALIAS,TO_ISSUER_ALIAS,ROLE_TYPE

Files changed:
ssuerrelationship.json
issuerrelationship_in.json
issuerrelationship-models.json

SDP-62323 EDS genericca Resource was improved

The metadata for the genericca resource was updated to correct a few differences noted on injection into the Oracle and the Snowflake databases.

Files changed:
genericca-models.json
genericca_in.jsongenericca-models.json
genericca_in.json

SDP-62817 Incorrect columns were removed from issuer relationship resource

The metadata for ssuerrelationship resource was corrected. Fields added in errors were removed and the resource is in sync with corresponding Data Map Table.

Files changed:
issuerrelatonship-models.json