Enhancements
SDP-61931 The Generic Issue Analytic Inbound and Outbound interfaces were enhanced to support new fields
The XSD and Generic Issue Analytic interface were updated to support additional Bloomberg fields listed below.
SECURITYDBO.SECURITY_ANLYTICS_EXT.WORKOUT_DATE (tag 18394)
SECURITYDBO.SECURITY_ANLYTICS_EXT.WORKOUT_PRICE (tag 18395)
Xpath:
EagleML/referenceTransaction/genericIssueAnalytic/analyticExtModel/workoutDate
EagleML/referenceTransaction/genericIssueAnalytic/analytics/analyticExtModel/workoutPrice
Files changed:
eagle_default/in/xml/xml-ref_genericissueanalytic.xml
metadata/bind/eagleml_ref_genericissueanalytic.xml
eagle_ml-2-0_cm/out/xml/t_generic_issue_analytic_streaming.inc
SDP-59347 The Warehouse Position Inbound and Outbound interfaces were enhanced to support new fields
The XSD and Warehouse Position interface was updated to support additional fields listed below.
HOLDINGDBO.POSITION_COST_DETAIL.ORIG_ACCRUED_INCOME_BASE (18342)
HOLDINGDBO.POSITION_COST_DETAIL.TAX_PAYABLE_LOCAL (tag 18343)
HOLDINGDBO.POSITION_COST_DETAIL.TAX_PAYABLE_BASE (tag 18344)
HOLDINGDBO.POSITION_COST_DETAIL.TAX_PAYABLE_SYS (tag 18345)
HOLDINGDBO.POSITION_COST_DETAIL.ORIG_TAX_PAYABLE (tag 18346)
Xpath:
EagleML/warehouseTransaction/warehousePosition/originalAccruedIncomeBase
EagleML/warehouseTransaction/warehousePosition/taxPayableLocal
EagleML/warehouseTransaction/warehousePosition/taxPayableBase
EagleML/warehouseTransaction/warehousePosition/taxPayableSys
EagleML/warehouseTransaction/warehousePosition/originalTaxPayable
Files changed:
eagle_default/in/xml/xslt/wrh_egl_gen_interface3.xsl
eagle_ml-2-0_cm/out/xml/t_warehouse_position_streaming.inc
eagle_ml-2-0_cm/out/xml/t_warehouse_position_xmldbapi_streaming.inc
SDP-63525 GenericSMF inbound interface was enhanced to support History mode for initial security load
During initial load of a security, tag 1257 (HISTORY ONLY) is not sent to the Stored Procedure even when it is populated in incoming file. As a result, the entire record is loaded into the security master table.
A new flag was introduced to load the initial security in History mode. As a result, only a specific list of tags will be inserted into the master table.
Usage:
<header>
<objectType>GenericSMF</objectType>
<processingOptions>SEC_SHELL_RECORD_ONLY</processingOptions>
</header>
or the following parameter in w_config_custom.inc
<COL TAG="W_SEC_SHELL_RECORD_ONLY" EXPRESSION= "'Y'"/>
Files changed:
eagle_default/in/xml/xslt/eagleml_smf_gen_fields.xsl
eagle_default/in/xml/include/security_resolution_generic.inc
SDP-63538 Roll Up in Warehouse Position and Warehouse Open Lot interfaces was enhanced to support a new parameter
A new parameter called in_update_source was added into the EGL_GEN_HOLDNG.POSFINAL_EGL_GEN and EGL_GEN_HOLDNG.LOTFINAL_EGL_GEN Stored Procedures.
In EagleML rules we call these Stored Procedures with specific list of parameters and an incorrect number of parameters can lead to an error during the POSFINAL_EGL_GEN or LOTFINAL_EGL_GEN call. Warehouse Position and Warehouse Open Lot interfaces were therefore enhanced to support the new parameter in_update_source in the POSFINAL_EGL_GEN and LOTFINAL_EGL_GEN Stored Procedure calls.
Files changed:
eagle_default/in/xml/xml-dbdirect_invar_eofp.xml
eagle_default/in/xml/xml-dbdirect_invar_rollup.xml
eagle_default/in/xml/include/xml-warehouse_common.inc
SDP-64139 WRHSTRADEEXTRACT - Updated signage logic for list of fields
Signage Key and Conditions were adjusted for the following fields:
SETTLEMENT_AMOUNT - EagleML/WarehouseTransaction/warehouseTrade/settlementAmount
SETTLE_AMOUNT_BASE - EagleML/WarehouseTransaction/warehouseTrade/settleAmountBase
TRADE_AMOUNT - EagleML/WarehouseTransaction/warehouseTrade/tradeAmount
NETSETTLEAMT - EagleML/WarehouseTransaction/warehouseTrade/netsettleamt
Affected signage key:
180MTOPENLNNBUYY
Files changed:
eagle_ml-2-0_cm/out/tagvalue/extract_wrhs_trade_signage_logic.inc
SDP-58789 Added filters for Warehouse Performance DataSet Extract
DictionaryId and PerfFreqCode filters were previously added for EDS ontology based extract but the same filters still did not work for the Warehouse Performance EDS dataset based extract. The filters were added and the issue was resolved.
Additional supported filters:
Parameter | Type | Filter SQL statement
dictionaryid default "MASTERTABLE.DICTIONARY_ID = :dictionaryid"
endeffectivedate DATE "MASTERTABLE.END_EFFECTIVE_DATE = :endeffectivedate"
perfsummarytype CHAR(2) "MASTERTABLE.PERF_SUMMARY_TYPE = :perfsummarytype"
perffreqcode CHAR(8) "MASTERTABLE.PERF_FREQ_CODE = :perffreqcode"
perfrollupreturnsid default "PERF_SEC_RETURNS.PERF_ROLLUP_RETURNS_ID = :perfrollupreturnsid"
Files changed:
core/datasets/warehouseperformance.json
SDP-56042 EDS Schedule Resource
EDS Schedule Resource was implemented to support load and extract to SECURITYDBO.SCHEDULE for both Oracle and Snowflake database providers.
Files changed:
metadata/core/ontology/reference/schedule_in.json
metadata/core/ontology/templates/eagle_ref_schedule_insertupdate.txt
metadata/core/ontology/reference/schedule.json
metadata/core/ontology/reference/schedule-models.json
SDP-60366 Implemented DeltaBatch extract for EDS warehouse objects
New parameter supported for the following EDS extracts:
warehousecashactivity
warehousependingtrade
warehousetrade
New filter requires 3 parameters:
usedeltabatch
fromdate
todate
<taskParameter>
<name>usedeltabatch</name>
<dataType>S</dataType>
<value>Y</value>
</taskParameter>
<taskParameter>
<name>fromdate</name>
<value>2017-04-16 11:00:00</value>
</taskParameter>
<taskParameter>
<name>todate</name>
<value>2022-04-17 13:00:00</value>
</taskParameter>
As a result, the extract for EDS warehouse objects will contain only batches (Entire batch Entity + Effective Date + Source) where at least one record has update_date in the specified range.
A new extract mode 'Delta-Batch' was added for Warehouse Resources to exclude duplicates. The 'Delta-Batch' mode will extract all records for a defined batch if one or more records in a batch is updated in Eagle.
To resolve issue with duplicates, the following steps should be executed:
EDS Extract in Delta-Batch mode should be executed on in-market region.
Batch load mode should be used to upload data to Snowflake/Vault
Files changed:
common_core/dataset/ds_filters/delta_batch_filters/delta_batch_filter_cursor_handler.py
SDP-63514 EDS code was updated to support new parameters for the ROLLUP procedures
New parameters were added to the core Rollup procedures for Warehouse Position and Openlot for Eagle DB version = 2017.2.35.0. The new parameters are supported for both in-market and Vault.
Core procedures:
EGL_GEN_HOLDNG.POSFINAL_EGL_GEN
EGL_GEN_HOLDNG.LOTFINAL_EGL_GEN
Files changed:
ontology
warehouseposition_in.json
warehouseopenlot_in.json
ontology
templates
eagle_wrhs_warehouseposition_rollup.oracle.txt
eagle_wrhs_warehouseposition_rollup.snowflake.txt
eagle_wrhs_warehouseopenlot_rollup.oracle.txt
eagle_wrhs_warehouseopenlot_rollup.snowflake.txt
ontology/templates/db_oper
snowflake
warehouseposition.reinit_batch.sql
warehouseopenlot.reinit_batch.sql
SDP-63581 Data Rules metadata was updated to remove BKUP_UPDATE_SOURCE field
Ontology models were corrected for the following EDS resources - issuerrating, smfidentifiers, smfidentifiershist.
The database fields listed below were removed:
XREFERENCE.BKUP_UPDATE_SOURCE
XREFERENCE_HIST.BKUP_UPDATE_SOURCE
ISSUER_RATINGS.BKUP_UPDATE_SOURCE
DDL should be regenerated in Create mode for above resources.
Files changed:
ontology\\reference\\issuerrating-models.json
ontology\\reference\\smfidentifiers-models.json
ontology\\reference\\smfidentifiershist-models.json
SDP-64059 EDS genericsmf resource was enhanced to support History mode on load and update
The historyOnlyFlag flag was previously supported for extract only. As a result, history only mode was not available for EDS genericsmf resource.
The history only mode is now available for the initial security load and update.
Files changed:
metadata/core/ontology/reference/genericsmf-models.json
metadata/core/ontology/reference/genericsmf-in.json
SDP-62741 EDS issuerrole resource was improved to load data in history mode
The issuerrole model was updated with additional mapping for ASOF or history mode. Now processing of elements in rdf_meta_utils checks if ASOF mode is supported by resource and includes such elements in method definition.
Files changed:
rdf_metadata/rdf_meta_utils.py
ontology/reference/issuerrole-models.json
expected-results/issuerrole_resource_output.json
SDP-63113 EDS issuerrating extracts was enhanced to support a new filter
A new filter called “reason” was added to exclude Canceled Ratings or any other unwanted records from issuerrating extract. It appeared there was no such filter neither as part of resource nor in eagledb-crud-filters, It is supported now.
Usage:
<taskParameter>
<name>reason</name>
<dataType>S</dataType>
<value>E(LzlsEnHa)</value>
</taskParameter>
Files changed:
ontology/eagledb-crud-filters.json
ontology/reference/issuerrating.json
expected-results/issuerrating_resource_output.json
SDP-62120 Alter DDL REST API was enhanced to support data type changes
New functionality allows to change the data type of the columns in Alter mode.
For column data type changes supported by Snowflake provider only column type will be changed and column name won’t be changed.
Details on supported data type changes can be found here https://docs.snowflake.com/en/sql-reference/sql/alter-table-column.html
For column data type changes which are not supported by Snowflake provider column with old data type will be renamed to <COLUMN_NAME>_OLD and after that column with new data type will be created with <COLUMN_NAME>
Files changed:
rdf_metadata/rdf_physical_schema_generator.py
SDP-62798 Cache update service was refactored to support new components structure
Cache Update service was refactored to support new data rules components structure.
Cache Update service contributes to microservices approach. Cache Update method was added in Cache Update service and will be deleted in EDS service.
Production Bugs
SDP-63892 The RelativeDelta Function was restored
The RelativeDelta function and import were added to exprbase.py
The Function was imported from external library and our own function wrapper was added to prevent code smell: “unused import”
Files changed:
common-core/lang_expr/exprbase.py
SDP-63877 EDS genericsmf was updated to improve update_date logic
The UPDATE_DATE logic is in sync with EagleML now.
SYSDATE is used to populate UPDATE_DATE field on insert and update.
Files changed:
metadata/core/ontology/reference/genericsmf-models.json
metadata/core/ontology/reference/genericsmf.json
metadata/core/ontology/reference/genericsmf_in.json
SDP-63728 The Issuer Rating Inbound interface was fixed to allow empty submittedDate field
The empty submittedDate field in the incoming file caused an error in the Stored Procedure due to an incorrect pasring of the field. The logic for submittedDate field was fixed to allow an empty value.
Files changed:
eagle_default/in/xml/xml-ref_issuerrating.xml
SDP-62892 Error is not being raised when extension data is not passed
There was an issue when an empty extension model was passed in incoming message like this as shown below.
<customExtensions>
<custom_wrhscashactExt>
</custom_wrhscashactExt>
</customExtensions>
This issue is now fixed and the error is not being raised.
File changed:
eagle_default/in/xml/include/wrhs_ext_custom_fields.inc
SDP-64226 The Accounting Cash interface was updated to improve performance for Cancel Trades
The changes in SDP-61374 resulted in a performance degradation due to an incorrect query which is used to check if a record exists in the database. This query was fixed and bind parameters were added which resulted in equal and improved performance.
Files changed:
eagle_default/in/xml/xml-acct_cash_cancel.xml
SDP-63714 The spdDeltaTriggerLog extract sql query was fixed for data from different partitions
The extract with entityId filter was not working as expected for entities which are from partitions that are not the primary partitions. As a result, multiple objects could have been extracted from the PRICE_FX_EXCEPTION table. This issue was caused by query generation that works incorrectly for data from nonprimary partitions.
Related file:
eagle_ml-2-0_cm/out/tagvalue/extract_spd_delta_trigger_log.inc
SDP-64275 Warehouse Trade outbound interface supports extensions extract
Database fields were being incorrectly collected by the extension table due to issues with the building of those queries.
File changed:
eagle_ml-2-0_cm/extract_extension_table_fields.inc