EagleML Release Notes - April 4, 2023

Enhancements

SDP-72427 EDS genericissueanalytic resource was updated

The Metadata for the genericissueanalytic resource was updated. The field type was updated to stay in sync with the corresponding EDM change:

The field SECURITY_ANALYTICS_EXT.ABBREV_TRADING_PERIOD type was changed from NUMBER to VARCHAR(255)

Files changed:
genericissueanalytic-models.json

SDP-71670 AccountingTrade inbound interface enhancement - maxEarnDaysOverride element

The AccountingTrade inbound interface is enhanced to support the new element maxEarnDaysOverride mapped to tag 17100. The number of days for desired earning days should be passed in this new element.

This element allows to bypass the Max Earn Days Limit (365 days) by an override value from the maxEarnDaysOverride. It can be used to pass through the error “Max earn days limit exceeded…“.

This enhancement is targeted for a specific business need and is not advised as normal BAU.

EagleML Xpath: EagleML/accountingTransaction/accountingTrade/maxEarnDaysOverride

element:

Files changed:
eagle_default/in/include/trade_aliases.inc
eagle_default/in/include/csv-sell_columns.inc
eagle_default/in/include/csv-sell_panels.inc
eagle_default/in/include/csv-buy_columns.inc
eagle_default/in/include/csv-buy_panels.inc
eagle_default/in/include/csv-deliver.inc
eagle_default/in/include/csv-receive.inc
eagle_default/in/xml/xml-accttrade_deliver.xml
eagle_default/in/xml/xml-accttrade_receive.xml

SDP-71444 GenericIssueAnalytic load and extract enhancement to support uncovered AnalyticExtModel fields.

The below GenericIssueAnalytic fields of the AnalyticExtModel model which were not previously supported have been added to the load and extract interfaces as well as Ontology.

EagleML Xpath:
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/roundLotSize
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/priceToCashFlow
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/tradeLotSize
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/sortinoRatio
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/abbrevTradingPeriod
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/averageVolume5d
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/averageVolume20d
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/interpolYieldSpread
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/interpolYieldSpreadGovt
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/zBasisConstMaturity
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/assetSwapSpread
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/ytmBid
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/ytmAsk
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/ytmMid
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/pricePositionMultiplier
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/maxDrawdownLength
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/maxDrawdownPercent
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/dvdYield12
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/peRatioEst
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/epsLtg
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/downsideRisk
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/totalVotingShares
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/totalNonVotingShares
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/convexityOasBid
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/convexityOasAsk
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/convexityOasMid
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/modDurationBid
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/modDurationAsk
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/modDurationMid
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/macaulayDurationBid
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/macaulayDurationAsk
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/macaulayDurationMid
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/sharpeRatio
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/underlyingContValue
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/volatility30d

Files changed:
metadata/bind/eagleml_ref_genericissueanalytic.xml
eagle_default/in/xml/xml-ref_genericissueanalytic.xml
eagle_ml-2-0_cm/out/xml/t_generic_issue_analytic.xml
eagle_ml-2-0_cm/out/xml/t_generic_issue_analytic_streaming.inc

SDP-71358 Updated the signage logic for PAR_OR_SHARES in the WRHSTRADEEXTRACT

Conditions were adjusted for the original quantity field to support and display a negative value for the signage key.
PAR_OR_SHARES - EagleML/WarehouseTransaction/warehouseTrade/originalQuantity

Signage Key - 350MTACTIONLNYREVSPLITN

Files changed:
eagle_ml-2-0_cm/out/tagvalue/extract_wrhs_trade_signage_logic.inc

SDP-71297 Checksum functionality was implemented at record level

EDS Extracts on the Eagle side were enhanced to support a new rowCheckSum element logic. The hash is calculated using an industry-standard algorithm for all columns in every row. The goal is to make sure that the data ingested into Snowflake matches the data extracted from Oracle at the record level.

In order to enable check sum logic, two parameters should be specified in the RTR:

calculatechecksum set to Y - to Include additional fields in the result file rowCheckSum
excludezero set to N - in order to checksum of each row calculated correctly, ALL values should be extracted even zeros

Usage:
<taskParameter>
<name>calculatechecksum</name>
<value>Y</value>
</taskParameter>
<taskParameter>
<name>excludezero</name>
<value>N</value>
</taskParameter>

Example of value in rowCheckSum field:

{"CLIENT": "7aadd8e19b83e0c6f9f264c75bf373d5",
"INSTITUTION": "d41d8cd98f00b204e9800998ecf8427e",
"INDIVIDUAL": "d41d8cd98f00b204e9800998ecf8427e",
"CLIENT_ADDRESS": "d41d8cd98f00b204e9800998ecf8427e",
"CLIENT_RELATIONSHIPS": "d41d8cd98f00b204e9800998ecf8427e",
"CLIENT_RELATIONSHIP_DETAIL": "d41d8cd98f00b204e9800998ecf8427e"}

After data upload, there is a new hash value calculated for every row in the destination tables and stored in the zrow_target_check_sum field. This is compared with the hash value from the incoming file stored in the zrow_source_check_sum field.
Hash values are compared and in case of a mismatch, the record is inserted into the ESTAR.RECONCILATION_LOG in Snowflake.

SDP-71166 EDS codevalue resource was enhanced to support extract and load in history mode

The EDS Codevalue resource was enhanced to support the extract from PACE_MASTERDBO.CODE_VALUES_HIST and upload data to the corresponding Snowflake table EAGLE_DEV_EDS.PACE_MASTERDBO.CODE_VALUES_HIST in History mode.

Please use the historyOnlyFlag to extract data in History mode.

<taskParameter>
<name>historyOnlyFlag</name>
<value>Y</value>
</taskParameter>

Files changed:
codevalue.json
codevalue_in.json.
codevalue-models.json

SDP-70747 New ExtractLevelDeltaMode parameter was introduced to switch delta control between workflow and single extract

A new flag, ExtractLevelDeltaMode, was added which allows to switch off delta logic on the workflow level and switch on delta logic on the single extract level. The delta information will be stored in the Redis Cache for each extract instead of the Oracle Data Base for the whole workflow.

The new logic is supported only for EDS extracts for the eagle_wrf_generic_extract and eagle_wrf_cloud_extract workflows (profiles with EBS_ResourceName, ResourceName parameters). The FeedType parameter is not supported.

Example of usage in profile:
EBS_ResourceName – ebs:genericsmf
ebs:default:DeltaBased - Y
ebs:default:DeltaBasedUID – [unique_uid]
ebs:default:increaselastrundatetime - NEVER
ebs:default:ExtractLevelDeltaMode - Y

Files changed:
eagle_ml-2-0_cm/out/tagvalue/extract_calculate_date.inc
eagle_ml-2-0_cm/mc2/ebs/mc2_ebs_apply_deltabased_pars.inc
eagle_ml-2-0_cm/in/xml/exec_eds.xml

Workflows changed:
eagle_wrf_generic_extract
eagle_wrf_cloud_extract

SDP-70746 Additional logic was implemented for EDS Delta Based extracts and parameter increaselastrundatetime

Previously the increaselastrundatetime parameter did not support the option ONLY_DATA_EXTRACTED and the last_run_date was updated each time despite the number of extracted records.
The ONLY_DATA_EXTRACTED option for increaselastrundatetime parameter is now supported. If the extract returns NO_DATA, then the last_run_date will not be updated. If the extract returns data, then the last_run_date will be updated.

Usage:
<taskParameter>
<name>increaselastrundatetime</name>
<dataType>S</dataType>
<value>ONLY_DATA_EXTRACTED</value>
</taskParameter>

Files changed:
common_core/enums/def_enums.py
common_core/descriptor/dbdescriptor_oci.py
common_core/descriptor/dbdescriptor_sf.py
common_core/dataset/idsdataframe_delta.py

SDP-68815 EagleML Inbound - Added Support for AccrualFactor

The AccrualFactor inbound interface support has been added to EagleML. The below element can be sent in to add accrual factors.
<objectType>AccrualFactor</objectType>

Supported table: SECURITYDBO.ACCRUAL_FACTOR
In addition support for deleting AccrualFactor records from database was added. To delete a record, the <action>DELETE</action> should be passed in the header:

<header>
<objectType>AccrualFactor</objectType>
<action>DELETE</action>
</header>

File added:
eagle_default/in/xml/xml-ref_smf_accrual_factor.xml

Files changed:
eagle_default/in/xml/xml-ref_objects.rsf

SDP-68138 SMF Custom Extension enhancement - fieldCompareFlag

SMF Custom Extension logic has been enhanced to now support fieldCompareFlag with historyOnlyFlag.

Files changed:
eagle_default/in/xml/include/custom_fields.inc
eagle_default/in/xml/xml-smf_generic.xml

SDP-67722 Data Rules were updated to support the timestamp format for the fields RELEASE_DATE and SOURCE_DATE

Corresponding Data Rules were updated to support the timestamp format for the fields RELEASE_DATE and SOURCE_DATE. The change was required to support Eagle data load to Snowflake AS IS . The metadata was enhanced with additional elements sourceTimeStamp and releaseTimeStamp where it was required. Below resources were updated:

genericissueanalytic
issueresg
issuerrating
issuerorganization
issuevariableraterating
rating
schedule
vendorcashflow
timeseries

SDP-61241 EDS warehousecloselot resource was updated to support additional elements for load and extract

Data rules for warehousecloselot were updated to support mapping and additional XSD elements in Ontology:
bookCostBase
bookCostLocal
distributedInvestmentIncomeBase
distributedInvestmentIncomeLocal
expectedCreditLossAmountBase
expectedCreditLossAmountLocal
expectedCreditLossPreviousAmountBase
expectedCreditLossPreviousAmountLocal
expectedCreditLossPreviousStageAmount
expectedCreditLossStageAmount
equityMethodAcctSecurityGainBase
equityMethodAcctSecurityGainLocal
holdingTerm
incRepayFeeBase
incRepayFeeLocal
equityMethodAcctSecurityLossBase
equityMethodAcctSecurityLossLocal
srcSysC
undistributedInvestmentIncomeBase
undistributedInvestmentIncomeLocal
unrealizedUndistributedInvestmentIncomeBase
unrealizedUndistributedInvestmentIncomeLocal

Files changed:
warehousecloselot-models.json
eagle-vocab-wh-shared.json
eagle-vocab-definition-wh-shared.json

Production Bugs

SDP-71849 Added support for fixed_income_ext.firstCallDateIssuance in Accounting mode

Added support for the element firstCallDateIssuance in Accounting mode.

If the accountingValidationFlag set to Y, then the additional validation is done by default.
If the accountingValidationFlag set to N or null, then the panel checks do not occur.

Files changed:
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/xml/include/fixedincome_extension.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/xml/include/smf_aliases.inc

SDP-71308 Added partition support for eagle_ml-2-0_default_in_xml_impairments stream

Added partition support for eagle_ml-2-0_default_in_xml_impairments stream. The interface will perform a lookup to find the entities partition and pass it to the impairments event. Previously, a default value of 1 was being used.
Files changed:
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/xml/xml-impairments.xml

SDP-69611 EDS clientcommunication resource was updated to support the clientId filter

The clientcommunication extract did not return any data if the filter clientId was used. The clientId field type was corrected the clientcommunication extract is working as expected in case when the clientId filter is used.

File changed:
metadata/core/ontology/reference/clientcommunication.json

SDP-69608 EDS clientcommunication resource was updated to support new fields on extract

The RULESDBO.CLIENT table was joined in order to extract the clientId on the Oracle and Snowflake extract. Mapping for the sourceName element was added to the Oracle and Snowflake extracts.

Files changed:
metadata/core/ontology/reference/clientcommunication.jsontest- test-metadata/core/ontology/reference/clientcommunication-models.json
metadata/core/ontology/reference/clientcommunication_in.json
metadata/core/ontology/eagledb-vocab-common.json