EagleML Release Notes - June 29, 2023

Owned by Eric Fardig (Deactivated)
Jun 29, 2023
6 min read

Enhancements

SDP-75037 Added a custom solution to provide the ability to change SMF XREF types

A solution for changing an xreference type is now supported within the smf object. This is optional feature which may be customized for each client's specific needs.

It is designed to retrieve conditions to execute using the w_config_custom.inc to identity if the xreference type should be changed. The two new parameters for configuration are below.

  1. :W_TYPE_CHANGE: defines if required to change type

  2. :W_TYPE_CHANGE_EXPR: is rules for change in form ‘oldType~ oldTypeValue~newType~newTypeValue~retypeExpr~retypeExprValue’
    a. oldType defines type of xId which to change
    b. newType is type of identifier after changing
    c. retypeExpr is expression to check
    The variables for one case can be separated by tildes ‘~’, and variables for a different case by a vertical dash ‘|’. If it is needed to check if value of xId equals to defined string, double quotes “ should be used. This is necessary for the parser worked correctly.

Use cases:

  1. Change type without condition: retypeExpr = '1'

  2. Change type based on string length of according xId: retypeExpr = 'StringLength(:xId:)>8'

  3. Change type based on defined substring of according xId: retypeExpr = 'SubString(GetVariable(:xId:),0,3)!="BBG"'
    Example: 'oldType~IAG_CLIENT_SEC_ID~newType~AIMID~retypeExpr~1|oldType~BBGID~newType~BBGID_MO~retypeExpr~SubString(GetVariable(:xId:),0,3)!="BBG"|'

For security type change (processingSecurityType, securityType, securitySubType) lookups were applied, but it requires to add values into PACE_MASTERDBO.CODE_VALUES. It is possible to change these values the same way changed within the xreferences.

Files changed:
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/xml/include/msgxchg_sec_core.inc
estar/tpe/dynamic/msgcenter/eagle_default/in/xml/include/multileg_swap_tag_remap.inc
estar/tpe/dynamic/msgcenter/eagle_default/in/xml/include/security_resolution_full.inc
estar/tpe/dynamic/msgcenter/eagle_default/in/xml/include/smf_generic_remap.inc
estar/tpe/dynamic/msgcenter/eagle_default/in/xml/xml-smf.xml
estar/tpe/dynamic/msgcenter/eagle_default/in/xml/xml-smf_generic.xml
estar/tpe/dynamic/msgcenter/eagle_default/in/xml/xml-smf_multileg_swap.xml

SDP-75004 EDS Extract in ASISCOPY mode excludes tables that ARE skipped on ASISCOPY Load

The Eagle to Vault pipeline was improved. Database tables that are not supported for data ingestion to Snowflake will be ignored by the EDS Extracts in ASISCOPY mode.

The ignored tables are declared via the eagledb-inventory:hasAsIsCopyExcludedTable predicate. The Data Rules resources and corresponding database tables below are where the change was implemented.

  • codevalue - INTERFACES, CODES, EGL_GENERIC_XLATION_ML

  • client - CLIENT_ADDRESS, CLIENT_RELATIONSHIP, CLIENT_RELATIONSHIP_DETAIL

  • entitypolicy - POLICY_DETAIL, POLICY_RATES, POLICY_MAP

  • entityaccountingbasis - ESTAR_ACCOUNTING_PERIODS

SDP-74677 New elements supported for IssuePcPrice for load

The below elements are now supported for the IssuePcPrice load.

  • accountingValidationFlag

  • priceConversionFxSource

  • priceConversionFxRate

  • payPriceLeg/notionalAmount

  • payPriceLeg/price

  • payPriceLeg/marketValue

  • receivePriceLeg/notionalAmount

  • receivePriceLeg/price

  • receivePriceLeg/marketValue

Also, in case of price conversion load failure, an error message will be shown.

Files changed:
eagle_default/in/xml/include/xml-ref_security_pricing_common.inc
eagle_default/in/xml/xml-ref_issuepcprice.xml

SDP-74541 Delta extracts for resources with composite and history mode

Delta extracts need to resolve the DeltaBasedUID to have two separate delta last_run_dates for the composite and history runs.

SDP-74530 Checksum functionality was improved for genericentity and entityanalytic

The below metadata and code changes were implemented for the genericentity and entityanalytic.

  • genericentity checksum supports all tables

  • entityanalytic checksum supports all tables

  • checksum performance was increased with calc_check_sum function

  • checksum on extract support all possible aliases

SDP-74258 Issuer Organization inbound interface extension enhanced to fill ISSUER_ALIAS field

The Issuer Organization extension uses ISSUER_ID as a business key. To support backwards compatibility, it will be maintained. As a result, added the ability to fill the ISSUER_ALIAS field if it exists in an extension table. It must be added to the extension table without a need to configure the field in XPanels.

File changed:
eagle_default/in/xml/include/custom_fields.inc

SDP-73432 EDS yieldcurve resource was enhanced to support load for Snowflake

The EDS yieldcurve resource was enhanced to support the load in Batch and Temporal modes for Snowflake. An additional yieldcurvecode filter was added for the extract.

Files changed:
metadata/core/ontology/reference/yieldcurve.json
metadata/core/ontology/reference/yieldcurve-models.json
metadata/core/ontology/reference/yieldcurve_in.json

Files added:
metadata/core/ontology/templates/db_oper/snowflake/yieldcurve.reinit_batch.sql
metadata/core/ontology/templates/db_oper/snowflake/yieldcurve.reinit_batch_many.sql
metadata/core/ontology/templates/eagle_ref_yieldcurve_reinitbatch.snowflake.txt
metadata/core/ontology/templates/eagle_ref_yieldcurve_reinitbatch_many.snowflake.txt

SDP-73402 Unpacker enhancement to skip timestamp restoration for zip files

The unpacker stream has been enhanced to support the ability to skip timestamp restoration for zip files. This can be set by using either of the two new variables below.

  • UnpackerUnzipSkipTimestampRestoration - A local variable that may be set in the IWS task unpacker or RTR and will have the highest priority.

  • W_UNPACKER_UNZIP_SKIP_TIMESTAMP_RESTORATION - A global variable that may be set in w_config_custom config and will have a priority lower than the UnpackerUnzipSkipTimestampRestoration variable.

Usage:

  • In the task_unpacker step in IWS or in RTR, set an UnpackerUnzipSkipTimestampRestoration variable in the Parameters of the task_unpacker.

  • <COL TAG="W_UNPACKER_UNZIP_SKIP_TIMESTAMP_RESTORATION" EXPRESSION="'Y'"/>
    or
    <CODE>:W_UNPACKER_UNZIP_SKIP_TIMESTAMP_RESTORATION: := 'Y';</CODE> in w_config_custom file.

Files changed:
eagle_ml-2-0_cm/unpacker/wrf_unzip.prf

SDP-73285 Profile Templates for Conversion and Cloud Workflows as part of Product

The Profile template eagle_wrf_cloud_conversion.profile was created for the eagle_wrf_cloud_conversion workflow and added to the package. Also, the profile template eagle_wrf_cloud_extract.profile was created for the eagle_wrf_cloud_extract workflow and added to the package.

Files added:
msgcenter/profile/workflows/eagle_wrf_cloud_conversion.profile
msgcenter/profile/workflows/eagle_wrf_cloud_extract.profile

Files changed:
eagle/estar/EAGLE_EXTRACTSERVICE.pkg
eagle/estar/eagle_default_workflows.pkg

SDP-72818 EDS genericissueanalytic resource was updated to fix mappings

The EDS genericissueanalytic resource was updated to fix the mapping for the following fields:
SECURITY_ANALYTICS_FI.BASIS_POINT_VALUE  - basisPointValue
SECURITY_ANALYTICS_FI.BASIS_POINT_VALUE_CURVE - basisPointValueCurve
SECURITY_ANALYTICS_FI.BASIS_POINT_VALUE_WRST - basisPointValueWrst
SECURITY_ANALYTICS_EXT.CONVEXITY_OAS_BID – analyticExtModel/convexityOasBid
SECURITY_ANALYTICS_EXT.CONVEXITY_OAS_ASK - analyticExtModel/convexityOasAsk
SECURITY_ANALYTICS_EXT.CONVEXITY_OAS_MID - analyticExtModel/convexityOasMid
SECURITY_ANALYTICS_EXT.MOD_DURATION_BID - analyticExtModel/modDurationBid
SECURITY_ANALYTICS_EXT.MOD_DURATION_ASK – analyticExtModel/modDurationAsk
SECURITY_ANALYTICS_EXT.MOD_DURATION_MID - analyticExtModel/modDurationMid
SECURITY_ANALYTICS_EXT.MACAULAY_DURATION_BID - analyticExtModel/macaulayDurationBid
SECURITY_ANALYTICS_EXT.MACAULAY_DURATION_ASK - analyticExtModel/macaulayDurationAsk
SECURITY_ANALYTICS_EXT.MACAULAY_DURATION_MID - analyticExtModel/macaulayDurationMid

The field SECURITYDBO.SECURITY_ANALYTICS_EQUITY.INDX_GENERAL_PX_RATIO was removed from genericissueanalytic and genericissueanalyticequity resources.

Files changed:
metadata/core/ontology/reference/genericissueanalytic-models.json
metadata/core/ontology/reference/genericissueanalyticequity-models.json
metadata/core/ontology/eagledb-vocab-common.json

SDP-71931 Added exceptions catch block for AccountingCash interface

An exception catch block exceptions_catch.inc was added for the eagle_ml-2-0_default_in_xml_acct_cash stream. This change exposes STATUSES, WARNINGS, ERRORS in the accounting cash interface.

Files changed:
estar/tpe/dynamic/msgcenter/eagle_default/in/xml/xml-acct_cash.xml

SDP-71683 Added support for several EntityAccountingBasis elements

The below elements were added to the EntityAccountingBasis object for loading.
performanceNavValuationFxSource Tag 18469
performanceNavForwardPriceSource Tag 18470

Fixed EntityAccountingBasis elements for loading:
vaultDate Tag 12005
postCutoffDt Tag 11848
settlementBeginDate Tag 18299

The below element was fixed for loading within the GenericEntity object when accountingValidationFlag = Y:
settlementBeginDate Tag 18299

Added elements to ENTITYEXTRACT legacy/streaming/MC2 extract:
performanceNavValuationFxSource Tag 18469
performanceNavForwardPriceSource Tag 18470

Fixed mapping for ledgerGranularityRuleName in MC2 ENTITYEXTRACT

The below elements were added to the ENTITYACCBASISEXTRACT streaming and MC2 extracts:
disableFlag
archiveDate
performanceNavValuationFxSource Tag 18469
performanceNavForwardPriceSource Tag 18470

SDP-70374 Enhanced the eagleml interface to support the Price Conversion Process to Support Unrealized Gain Loss - URGL for TRS on Single Name Fixed Income

Revised and expanded support for the IssuePrice, IssuePcPrice and ExchangePrice with Price Type Codes greater than 3 in the reference object to manage the routing and calculation to the Price Conversion processes. Please note, panel changes are only available post 2017 r2.41 Eagle core in market release.

Files changed:

eagle_default/in/xml/include/xml-ref_security_pricing_common.inc
eagle_default/in/xml/xml-ref_issueprice.xml

Production Bugs

SDP-74822 EagleML element couponRateFixingMethod is available for history only load in the multilegSMF object

The couponRateFixingMethod element can now be loaded in both history and composite tables. The correct XML path for generateSwapResetScheduleSwitch was added and now this element can be processed both from leg root node and from MultipleLegSwapModel:

<payLeg>
<MultipleLegSwapModel>
<generateSwapResetScheduleSwitch>Y</generateSwapResetScheduleSwitch>
</MultipleLegSwapModel>
</payLeg>
<payLeg>
<generateSwapResetScheduleSwitch>Y</generateSwapResetScheduleSwitch>
</payLeg>
Changes were applied for all three legs.

Files changed:

eagle_default/in/xml/include/contract_leg_alliases.inc
eagle_default/in/xml/include/contract_mapping.inc
eagle_default/in/xml/include/ins_upd_smf_req.inc
eagle_default/in/xml/include/pay_leg_alliases.inc
eagle_default/in/xml/include/pay_mapping.inc
eagle_default/in/xml/include/receive_leg_alliases.inc
eagle_default/in/xml/include/receive_mapping.inc

SDP-74719 EDS issuetenor Resource

The EDS issuetenor resource was created to support the Eagle Vault pipeline for extracting data from Eagle and uploading to Snowflake.

Table: SECURITYDBO.SECURITY_TENOR
Key: SECURITY_ALIAS + EFFECTIVE_DATE + SRC_INTFC_INST
LoadModes: Delta (by default), Temporal

Extract Filters:
sourcename
securityalias, processingsecuritytype, securitytype, investmenttype, releasestatus, primaryassetid, xrefid, ticker
fromdate, todate,
effectivedate, fromeffectivedate, toeffectivedate

SECURITYDBO.SECURITY_TENOR table was removed from genericsmf resource

Files changed:
metadata/core/ontology/eagle-vocab-enumerations.json
metadata/core/ontology/reference/genericsmf-models.json
metadata/core/ontology/reference/genericsmf.json
metadata/core/ontology/definition/eagle-vocab-definition-ref-shared.json
metadata/core/ontology/definition/eagle-vocab-definition-sec-shared.json
metadata/core/ontology/eagle-vocab-ref-shared.json
metadata/core/ontology/eagle-vocab-sec-shared.json

Files added:
metadata/core/ontology/reference/issuetenor-models.json
metadata/core/ontology/reference/issuetenor.json
metadata/core/ontology/reference/issuetenor_in.json

SDP-74686 Warehouseperformance resource was updated to correct the issue on EDS Extract side related to missing primary key

A SQL Select for the EDS extract is based on primary keys for tables defined in the metadata. The Warehouseperformance resource previously had the wrong primary keys: PERF_SUM_INST, SECURITY_ALIAS, PERF_ROLLUP_RETURNS_ID.

The field STRATEGY_DETAIL_ID was added starting from database version=2017.2.28.0 and is supported going forward.
This field was added to primary keys for below detail tables, warehouseperformance resource:
PERFORMDBO.PERF_SEC_RETURNS
PERFORMDBO.PERF_SEC_RETURNS_EXT.

The field STRATEGY_DETAIL_ID was not described as key in the Ontology which was the reason for incorrect data. The EDS Extract used to return extra records which is now resolved.

Currently list of keys used for EDS Extract:
PERF_SUM_INST, SECURITY_ALIAS, PERF_ROLLUP_RETURNS_ID, STRATEGY_DETAIL_ID.

Files changed:
warehouseperformance-models.json

SDP-74548 Fixed inactiveFlag loading in accounting validation mode

The inactiveFlag element for loading SMF records when using the accounting validation mode usage was updated. The value of the inactiveFlag - Tag 11043 from the incoming record is now stored before the security resolution and restores after the security resolution is finished. This provides the ability to update the inactive flag on the smf.

Files changed:
estar/tpe/dynamic/msgcenter/eagle_default/in/xml/include/security_resolution_generic.inc

SDP-74391 Checksum Validation Errors were corrected for genericissueanalytic, issuerrating, factor and timeseries

The Checksum validation errors were corrected for the genericissueanalytic, issuerrating, factor and timeseries resources. Most of those issues, except genericissueanalytic, were related to obsolete mapping of fields.
Replacement mapping in this case means that the field on the source side was replaced with the value from a different field on load by design repeating the EagleML logic for load.
This logic was enhanced as asiscopy mode was added. During an asiscopy extract and load, the value from the source database will be moved as is to the target database without replacement.
The errors for genericissueanalytic resource were related to the fields with the same column name between the tables. Only the value from the main table was loaded and extracted. The issue was fixed through the use of column prefixes.

Files changed:
Metadata Files for issuerrating, factor and timeseries resources were updated.

SDP-73528 EDS warehousependingtrade resource was enhanced to support new fields

The EDS warehousependingtrade resource was enhanced to support the below fields as part of Eagle Vault pipeline.

New supported elements:
TRADEDBO.PENDING_TRADES.RESEARCH_FEE_L – EagleML/warehousePendingTrade/researchFeeLocal
TRADEDBO.PENDING_TRADES.RESEARCH_FEE_B - EagleML/warehousePendingTrade/researchFeeBase
TRADEDBO.PENDING_TRADES.LEDGER_NOTES - EagleML/warehousePendingTrade/ledgerNotes
TRADEDBO.PENDING_TRADES.TO_SLEEVE_ID - EagleML/warehousePendingTrade/toSleeveId
TRADEDBO.PENDING_TRADES.TO_ORIGINAL_FACE_VALUE - EagleML/warehousePendingTrade/toOriginalFaceValue
TRADEDBO.PENDING_TRADES.RECALLABLE_CAPITAL_L - EagleML/warehousePendingTrade/recallableCapitalLocal
TRADEDBO.PENDING_TRADES.RECALLABLE_CAPITAL_B - EagleML/warehousePendingTrade/recallableCapitalBase
TRADEDBO.PENDING_TRADES.PE_INCOME_L - EagleML/warehousePendingTrade/privateEquityIncomeLocal
TRADEDBO.PENDING_TRADES.PE_INCOME_B - EagleML/warehousePendingTrade/privateEquityIncomeBase
TRADEDBO.PENDING_TRADES.PE_EXPENSE_L - EagleML/warehousePendingTrade/privateEquityExpenseLocal
TRADEDBO.PENDING_TRADES.PE_EXPENSE_B - EagleML/warehousePendingTrade/privateEquityExpenseBase
TRADEDBO.PENDING_TRADES.GAINLOSS_L - EagleML/warehousePendingTrade/gainLossLocal
TRADEDBO.PENDING_TRADES.GAINLOSS_B - EagleML/warehousePendingTrade/gainLossBase
TRADEDBO.PENDING_TRADES.COST_L - EagleML/warehousePendingTrade/costLocal
TRADEDBO.PENDING_TRADES.COST_B - EagleML/warehousePendingTrade/costBase
TRADEDBO.PENDING_TRADES.SHARE_CLASS - EagleML/warehousePendingTrade/shareClass
TRADEDBO.PENDING_TRADES.FUNDED_CAPITAL_L - EagleML/warehousePendingTrade/fundedCapitalLocal
TRADEDBO.PENDING_TRADES.FUNDED_CAPITAL_B - EagleML/warehousePendingTrade/fundedCapitalBase
TRADEDBO.PENDING_TRADES.NAV_PER_SHARE - EagleML/warehousePendingTrade/navPerShare
TRADEDBO.PENDING_TRADES.SUB_RED_INDICATOR - EagleML/warehousePendingTrade/subRedIndicator
TRADEDBO.PENDING_TRADES.LOWER_BOUND - EagleML/warehousePendingTrade/lowerBound
TRADEDBO.PENDING_TRADES.FEE_AMOUNT - EagleML/warehousePendingTrade/feeAmount
TRADEDBO.PENDING_TRADES.CAP_AMOUNT_BASE - EagleML/warehousePendingTrade/capAmountBase
TRADEDBO.PENDING_TRADES.BATCH_NUMBER - EagleML/warehousePendingTrade/batchNumber

The mapping for the UPDATE_DATE was also corrected.

Files changed:
metadata/core/ontology/reference/warehousependingtrade-models.json
metadata/core/ontology/reference/warehousependingtrade_in.json
metadata/core/ontology/definition/eagle-vocab-definition-acc-shared.json
metadata/core/ontology/definition/eagle-vocab-definition-shared.json
metadata/core/ontology/definition/eagle-vocab-definition-wh-shared.json
metadata/core/ontology/eagle-vocab-acc-shared.json
metadata/core/ontology/eagle-vocab-shared.json
metadata/core/ontology/eagle-vocab-wh-shared.json

SDP-72797 Added support for the issueFxRate processingSecurityType for both the from/to currencies security alias resolution

Support for the processingSecurityType element was added to the fromCurrency and toCurrency models. This change affects the security alias lookup only when the accountingValidationFlag = N

Files changed:
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/xml/xml-ref_fxrate.xml

Â