EagleML Release Notes - September 2020
- Olga Galis (Deactivated)
Enhancements:
SDP-41113: WarehousePerformance Inbound Interface was updated to support new STRATEGY_DETAIL_ID field in case it exists
The data load for the WarehousePerformance Interface was failing with a database Error Executing 'ESTAR'=>ORA-01400: cannot insert NULL into ("PERFORMDBO"."PERF_SEC_RETURNS_EXT"."STRATEGY_DETAIL_ID"). The STRATEGY_ID field was added to Primary Keys, and the STRATEGY_DETAIL_ID column was added to the PERF_SEC_RETURNS and PERF_SEC_RETURNS_EXT tables as part of recent EDM work.
When the field STRATEGY_DETAIL_ID was part of the following tables PERFORMDBO.PERF_SEC_RETURNS and PERFORMDBO.PERF_SEC_RETURNS_EXT, the Warehouse Performance load via ESTAR.PRF_EGL_GEN_INTERFACE_EXT was failing. To fix the load issue, the field STRATEGY_DETAIL_ID was set as 0. This field was added for future development.
Files changed:
eagle_ml-2-0_cm/generate_sp_on_fly.inc
SDP-41078: SP PRF_EGL_GEN_INTERFACE_EXT is regenerated as a wrapper in case of error
In case of the SP ESTAR.PRF_EGL_GEN_INTERFACE_EXT being invalid or not existing, it will be regenerated for a second time as a wrapper to call the core stored procedure PERFORMDBO.EGL_GEN_PERFORM.PRF_EGL_GEN_INTERFACE.
In this case, the load logic into the Performance Analytics tables is not supported.
Files changed:
eagle_ml-2-0_cm/generate_sp_on_fly.inc
SDP-40996: Pricing Report Extract supports additional filters
Introduced new filters to manage the data requested and returned within the PRICINGREPORTEXRACT.
Implemented the Zero Balances Filter
Introduced a new switch, IncludeZeroBalances, that will control the behavior of the ZeroBalance filter for the Pricing Report extracts. When the IncludeZeroBalances is included, it will trigger action only when the value is set to N. Any other value except N, will be ignored. The ShareParValue and BookValue fields will be controlled separately.
If IncludeZeroBalances set to Y or omitted - there is no filter.
If IncludeZeroBalances set to N - the filter is applied.
<taskParameter>
<name>IncludeZeroBalances</name>
<dataType>S</dataType>
<value>N</value>
</taskParameter>
When the IncludeZeroBalances is set to N, the ZeroBalanceSwitch becomes active.
The ZeroBalanceSwitch has possible values of:
ParBookValue
NOT (HOLDINGDBO.POSITION_DETAIL.SHARE_PAR_VALUE = 0
AND HOLDINGDBO.POSITION_DETAIL.BOOK_VALUE = 0)ShareParValue
NOT (HOLDINGDBO.POSITION_DETAIL.SHARE_PAR_VALUE = 0)
<taskParameter>
<name>ZeroBalanceSwitch</name>
<dataType>S</dataType>
<value>ParBookValue </value>
</taskParameter>
By default, when the IncludeZeroBalances is set to N and if the ZeroBalanceSwitch is not ShareParValue, it will act as ParBookValue and filter out the par / shares and book values that have a value of 0 on the HOLDINGDBO.POSITION_DETAIL table and have an entry in the PORTFOLIO_VALUATION table.
Also, introduced the ability to retrieve the Market Date from a secondary table and calculate the Market Days Stale. The new RTR parameter is: UsePriceExchangeMarketDateSource
Possible values: Y/N. Default Value: N
<taskParameter>
<name>UsePriceExchangeMarketDateSource</name>
<dataType>S</dataType>
<value>Y</value>
</taskParameter>
When the switch is set to Y, it will pull the Market Date from the PRICE_EXCHANGE table and calculate the Market Days Stale using the Pricing_Exchange table. The calculation will subtract Effective Date less the Market Date from the PRICE_EXCHANGE table accounting for the fact that the Market Date may not always be populated.
File changed:
eagle_ml-2-0_cm/out/tagvalue/extract_pricing_report.inc
SDP-39911: Pricing Report Outbound Interface was updated to extract the additional element daysPriceUnchanged
All records returned from the Pricing Extract include a new element daysPriceUnchanged that populates with the value available from the corresponding database fields without an additional switch required:
RULESDBO.PRICE_RULE_SECURITIES.DAYS_UNCHANGED
RULESDBO.PRICE_RULE_SECURITIES_ARCH.DAYS_UNCHANGED
Files changed:
eagle_ml-2-0_cm/out/tagvalue/extract_pricing_report.inc
eagle_ml-2-0_cm/out/xml/t_pricing_report_streaming.inc
SDP-40523: To improve performance of SQL statement for SMF Extract bind parameters were added for CREATE_DATE filter
To improve performance of the SMFEXTRACT, bind parameters that were added for the field CREATE_DATE. This change improves performance of the SQL statement and prevents the possibility of an SQL injection.
If the RTR parameter onlyNewSecurities is set to Y, then bind parameters are added to the filter:
AND SECURITY_MASTER.CREATE_DATE between
TO_DATE(:1, 'YYYY-MM-DD HH24-MI-SS') AND
TO_DATE(:2, 'YYYY-MM-DD HH24-MI-SS')
Files changed:
eagle_ml-2-0_cm/extract_filter_bind_pars_sql.inc
eagle_ml-2-0_cm/extract_filter_date_one.inc
eagle_ml-2-0_cm/out/tagvalue/extract_smf.inc
SDP-40124: Stream Category and Description were updated and repackaged for a few Eagleml Default Interfaces
The Stream Category and Description were updated for a few streams. There were no Installer changes at this time. Therefore, updates are available only in the case of a Fresh Install for the streams listed below:
eagle_ml-2-0_default_cm_exec_eds
eagle_default_eds_warehouse
Files changed:
eagle_ml-2-0_default_cm_execextract
eagle_ml-2-0_default_cm_extract_validation
eagle_ml-2-0_default_cm_profiled_load_validation
SDP-40122: Mandatory CA Exclusion load process was updated to improve CA resolution and allow Delete action
The Mandatory CA Exclusion load process no longer requires the corpActInstance and is correctly resolved based on incoming data. The ability to delete the Mandatory CA Exclusion was also introduced. In order to Delete, it is necessary to pass the referenceTransaction/header/action as DELETE or DEL in the incoming mandatoryCAExclusion file.
Files changed:
eagle_default/in/xml/xml-ref_mandatorycaexclusions.xml
SDP-40117: GenericIssueAnalytic Inbound interface was updated to support the data load into specific database tables via RTR parameter analyticModelsSwitch
Generic Issue Analytic Inbound Interface was updated to support new functionality that allows loading data into specific database tables.
The switch analyticModelsSwitch /tag1420 supports multiple values which indicates the database tables for data load and update.
Supported values for the analyticModelsSwitch are:
ALL – insert/update data into all interface tables supported by default
EQ - insert/update data into securitydbo.security_analytics_equity and main tables
FI – insert/update data into securitydbo.security_analytics_fi and main tables
MTG – insert/update data into securitydbo.security_analytics_mtg and main tables
DERV – insert/update data into securitydbo.security_analytics_derv and main tables
SAE - insert/update data into securitydbo.security_analytics_ext and main tables
PE - insert/update data into securitydbo.security_price_earn_equity and main tables
EPS - insert/update data into securitydbo.security_eps_equity and main tables.
Please note that main tables securitydbo.SECURITY_ANALYTICS and securitydbo.SECURITY_ANALYTICS_USER are always updated.
The list of main tables may vary based upon the Eagle core version.
Xpath:
EagleML/referenceTransaction/genericIssueAnalytic/analyticModelsSwitch
Files changed:
eagle_default/in/xml/xml-ref_genericissueanalytic.xml
SDP-39763: Additional logic implemented to check for an existing Eagle Accounting and Warehouse Positions as part of GenericSMF Inbound Interface
The GenericSMF Inbound Interface was updated to align with the Front End business logic to not allow the assetCurrency to be changed for a security held in a portfolio regardless if the accountingValidationFlag is set to Y or N. The currency update of SMF held in a portfolio is not allowed.
If the Accounting Trade was processed and Eagle STAR Accounting Position exists for the Security record, then the fields: Asset Currency/tag85, Processing Security Type/tag3931, Quantity Type/tag12, Price Multiplier/tag18 and Quantity Scale/tag19 may not be updated for this Security.
If the Warehouse Position exists for the Security record, then the fields: Asset Currency/tag85, Quantity Type/tag12, Price Multiplier/tag18 and Quantity Scale/tag19 may not be updated for this Security.
If the accountingValidationFlag set to Y, then the additional validation is done by default.
If the accountingValidationFlag set to N, then new logic is turned on when the heldSecurityValidationFlag as part of incoming message:
<heldSecurityValidationFlag>Y</heldSecurityValidationFlag>
Files changed:
eagle_default/in/xml/include/generic_smf_validations.inc
eagle_default/in/xml/include/msgxchg_sec_core.inc
eagle_default/in/xml/xslt/eagleml_smf_gen_fields.xsl
eagle_default/in/xml/xslt/smf_egl_gen_interface_template.xsl
eagle_default/in/xml/xslt/smf_egl_gen_interface_template_mssql.xsl
SDP-39473: New inbound interface Position Default was implemented to add debt default period by Entity / Security. Security Default Inbound interface was enhanced to support load by Security Type
A new inbound interface, positionDefault, was built to feed the securitydbo.position_default table which enables the load by entity / security.
Implemented the logic below:
Mandatory fields description:
Security, Entity, beginDate – mandatory
accountingBasis – if missed – pulled Primary Accounting Basis from entity
CostingRule – calculated from the accountingBasis for specified entity
addPeriodsForAllBasesSwitch – mandatory for the ADD action (ignored for CHANGE and DELETE)
Periods may be added for a single basis (addPeriodsForAllBasesSwitch = N) or for all bases of Entity (addPeriodsForAllBasesSwitch = Y).
An error will be raised when any key element is missed, unresolved or calculated.
Periods are added only for Accounting Bases configured for a specified Entity. In cases where the accountingBasis is not setup for Entity, an error will be raised.
Enhanced the securityDefault inbound interface with the load mode by Security Type.
In order to switch modes, added new parameter: addPeriodsBasedOnSwitch
Possible values: Security, Security Type
Default value: Security
Based upon the addPeriodsBasedOnSwitch selected, the Security or Security Type value is required.
Corrected elements below for the securityDefault interface:
Updated the element processingSecurityType to securityType
Updated securityDefaultComments to comments
Both interfaces were added to eagle_wrf_generic_load workflow
Files added:
metadata/bind/eagleml_ref_position_default.xml
msgcenter/eagle_default/in/xml/xml-ref_position_default.xml
Files changed:
metadata/bind/eagleml_ref_security_default.xml
metadata/bind/eagleml_reference.xml
msgcenter/eagle_default/in/xml/xml-ref_security_default.xml
msgcenter/eagle_default/in/xml/xml-ref_objects.rsf
msgcenter/eagle_ml-2-0_cm/out/xml/t_ref_security_default_streaming.inc
eagle_wrf_generic_load worflow files
SDP-38929: EntityToEntity inbound interface was improved to support many to one relation, to allow load many Entity Details to One Entity
EntityToEntity was improved to support the load of multiple Account Xreferences into the RULESDBO.ENTITY_DETAIL for 1 Entity of any type, like PORT or INDX.
The Entity Resolution and Entity Batting Order options are supported, for example:
<entityResolutionOption>MATCH_ALL</entityResolutionOption>
<entityBattingOrder>XREF_TYPE2,XREF_TYPE3,XREF_TYPE1</entityBattingOrder>
Find 1st Match is supported by default if Entity Resolution Option is not set. If the EntityId is not resolved, the load of entity details will be failed.
File changed:
eagle_default/in/xml/xml-ent_entitytoentity.xml
SDP-38535: Accounting Trade Interface was updated to support additional fields for Expected Credit Losses
The mapping was added for several new fields to support Expected Credit Loss functionality for the Accounting Trade Interface:
EagleML/accountingTrade/eclStage
EagleML/accountingTrade/eclLocal
EagleML/accountingTrade/eclBase
EagleML/accountingTrade/purchasedImpaired
EagleML/accountingTrade/regulatoryIntent
Files changed:
eagle_default/in/include/csv-buy_columns.inc
eagle_default/in/include/csv-conversion_columns.inc
eagle_default/in/include/csv-receive.inc
eagle_default/in/include/trade_aliases.inc
SDP-35454: Generic Load workflow was enhanced to support the load and delete of additional feed types - securityDefault and positionDefault
Default workflow eagle_wrf_generic_load was updated to support the load and delete of additional object types - securityDefault and positionDefault.
securityDefault and positionDefault records may be deleted via eagle_wrf_generic_load workflow if DELETE action is defined in a message header.
<referenceTransaction>
<header>
<objectType>positionDefault</objectType>
<action>DELETE</action>
</header>
</referenceTransaction>
Files changed:
eagle_ml-2-0_workflows/eagle_wrf_generic_load.wrf
eagle_ml-2-0_workflows/eagle_wrf_generic_load_accounting_objects.wrf
eagle_ml-2-0_workflows/eagle_wrf_generic_load_security_related_objects.wrf
eis/ejmwf/eagle_wrf_generic_load.ejmwf
eis/ejmwf/eagle_wrf_generic_load_accounting_objects.ejmwf
eis/ejmwf/eagle_wrf_generic_load_security_related_objects.ejmwf
Production Bugs:
SDP-40925: Security Resolution Update logic was improved for assetResolutionOption set to USE_FIRST_AS_MAIN + MATCH_ALL
The following issue occurred in Security Resolution Logic for the Security Resolution Option USE_FIRST_AS_MAIN + MATCH_ALL: If the Security Alias was not resolved by the first pair of Security Xreference Identifiers and was resolved on step 2 by 2 – 10 pairs, but the first pair of identifiers contained a mismatch with the Xrefence Type, then the Xref Id for the first pair was updated in error. The corrected behavior is to fail the load with error of mismatch in the first pair of identifiers.
The Security Resolution logic was updated: If the First Xref contains a mismatch, in case assetResolutionOption set to USE_FIRST_AS_MAIN + MATCH_ALL OR USE_EXIST_XREF_ONLY + USE_FIRST_AS_MAIN + MATCH_ALL or USE_EXIST_XREF_ONLY + MATCH_ALL, then the security update will be rejected.
File changed:
eagle_default/in/xml/include/security_resolution_generic.inc
SDP-40702: Classification Matrix Outbound Interface was updated
CLASSIFICATIONMATRIXEXTRACT was updated to correct the mapping for the RULESDBO.EGL_CM_CODES.CM_CODE_INST field.
The related XSD model for the ClassificationMatrix feed type was also updated to support a new element classificationMatrixCodeInstance.
XMLPath: EagleML/referenceTransaction/classificationMatrix/classificationMatrixCodeInstance
Files changed:
eagle_ml-2-0_cm/out/xml/t_ref_classification_matrix_streaming.inc
SDP-40646: DEMANDLIST Extract was updated to support mapping for Portfolios and Portfolios Lists
The DEMANDLIST Extract was returning incorrect content as more records were being returned than were expected. The Demand List extract logic was missing the mapping for tag5/Portfolios and tag1081/Portfolios Lists as part of DemandListProfileName which caused content discrepancies between the front-end report and the extract file. The extract rule was updated to correct the issue.
File changed:
eagle_ml-2-0_cm/out/tagvalue/extract_demandlist.inc
SDP-40140: SMF History Extract in DeltaBased Mode was corrected
SMF History Extract executed in DeltaBased mode was failing with the database error: ORA-00904: "SECURITYDBO"."SECMASTER_HISTORY"."INSTANCE": invalid identifier.
The issue occurred if the following three parameters were specified in the RTR , then SQL query was generated incorrectly:
<taskParameter>
<name>DeltaBased</name>
<dataType>S</dataType>
<value>Y</value>
</taskParameter>
<taskParameter>
<name>DeltaBasedUID</name>
<dataType>S</dataType>
<value>SMF_DELTA_HISTORY</value>
</taskParameter>
<taskParameter>
<name>UseInstance</name>
<dataType>S</dataType>
<value>Y</value>
</taskParameter>
The instance field for SMF History Extract was redefined as SECURITY_ALIAS, and the issue was fixed.
Files changed:
eagle_ml-2-0_cm/out/tagvalue/extract_delta_based.inc
eagle_ml-2-0_cm/out/tagvalue/extract_delta_based_param.inc
eagle_ml-2-0_cm/out/tagvalue/extract_delta_based_send.inc
eagle_ml-2-0_cm/out/tagvalue/extract_delta_based_send_dt.inc
SDP-39961: Corrected the issue with Asynchronous SOAP requests
Asynchronous SOAP requests stopped working after post-processing logic for the T-Stream to deliver archives to the FTP Server. ASYNC extracts were throwing unexpected errors during HTTP distribution and callback mode execution. This issue was fixed by altering the logic to save an extract file from a staging directory to a permanent location.
Files changed:
eagle_ml-2-0_cm/reporter_dist.inc
eagle_ml-2-0_cm/reporter_dist_methods.inc
SDP-37481: MultipleLegAccountingTrade feed type was updated to correct issues when the calculateLocalAmountsFlag was set to Y
MultipleLegAccountingTrade Inbound interface was updated to correct issues occurring when the calculateLocalAmountsFlag was set to Y with the MC_MODE panel logic allowed to override incoming values for the following tags:
4483 – ilbIndexRatio
45 – price
47 – commission
11579 – commissionSettlementDate
9357 – otherFeeSettlementDate
3752 – otherFee
50 - localNetAmount
165 – principal
In order to avoid load issues in case calculateLocalAmountsFlag set to Y, tag4404/totalSettlementAmountLocal is required now.
XMLPath:
accountingTransaction/multipleLegAccountingTrade/contractTradeLeg/totalSettlementAmountLocal
accountingTransaction/multipleLegAccountingTrade/payTradeLeg/totalSettlementAmountLocal
accountingTransaction/multipleLegAccountingTrade/receiveTradeLeg/totalSettlementAmountLocal
The field value directly affects the localNetAmount and the principal calculation if the calculateLocalAmountsFlag is set to Y. The mapping and rule logic was updated to support the tags necessary for inner-panel calculations.
File changed:
eagle_default/in/xml/xml-accttrade_multipleleg.xml