Enhancements

SDP-77472 Improved JSON DB Descriptor

The JSON parser that handles the list of objects has only one dictionary populated including complex elements values. The JSON DB Descriptor was improved to correctly retrieve a complex element value.

Files changed:
src/common_core/ds_parser/dsjsonparser.py

SDP-77448 transactionType element mapped to TRANSACTION_TYPE db field in the warehouseCashActivity inbound interface

The transactionCode element was mapped only to the TRANSACTION_TYPE field in the warehouseCashActivity inbound interface. When the transactionType element is passed in the incoming message, it has a higher priority than the transactionCode and is used for mapping to the TRANSACTION_TYPE field. If the transactionType is not populated and the transactionCode is populated on the incoming message, then this will be mapped too the TRANSACTION_TYPE field.

File changed:
eagle_default/in/xml/xslt/wrh_egl_gen_interface3.xsl

SDP-77412 The warehousenavmonthend resource was updated to correct mapping for missed elements

Mapping for the below elements was added in the MONTHLY_ESTAR_NAV table.

bidNav - ESTAR.MONTHLY_ESTAR_NAV.BID_NAV
offerNav - ESTAR.MONTHLY_ESTAR_NAV.OFFER_NAV
estimateVersion - ESTAR.MONTHLY_ESTAR_NAV.ESTIMATE_VERSION

The field type was updated according to EagleML XSD Schema:
assets - ESTAR.MONTHLY_ESTAR_NAVASSETS from Integer to Decimal
navChangePercent - ESTAR.MONTHLY_ESTAR_NAVNAV_CHANGE_PERCENT from Integer to Decimal

Files changed:
eagle-vocab-shared.json
eagle-vocab-acc-shared.json
eagle-vocab-definition-shared.json
eagle-vocab-definition-acc-shared.json
warehousenavmonthend-models.json
warehousenavmonthend.reinit_batch_many.sql

SDP-77304 Improvements made to the languagetranslation multilingual extract EDS resource

Oracle descriptor implementation was changed to cx_Oracle for EDS extract languagetranslation.

Files changed:
src/eagleinvsys/rdf_metadata/rdf_meta_utils.py
src/eagleinvsys/rdf_metadata/rdf_outflow.py
src/eagleinvsys/rdf_metadata/rdf_vocab.py
ontology/eagle-interfaces.json
ontology/reference/codevalue.json
ontology/reference/languagetranslation.json

SDP-75116 Conversion Workflow Optimizations

The conversion workflow logic which is responsible for the python pypy process spawning was improved to limit the number of pypy workers allocated during data conversion based upon the execution plan. The below changes were implemented to optimize this conversion process and maintain the Eagle environment.

• Allocate no more than half of the available CPU for conversion

• Allocate pypy workers no more than number of RTR’s for processing

• New OCI_PAGE_SIZE variable limit

Files changed:
src/common_core/descriptor/dbdescriptor_oci.py
src/eagleinvsys/etl/conversion_executor.py

SDP-74235 Added Partition support for the eagle_ml-2-0_default_in_xml_impairments stream

Partition support for the eagle_ml-2-0_default_in_xml_impairments stream has been added and resolution for Tag 95 - entityPartition has been included. The value from the incoming file value will be used before the partition is resolved if it’s present on the record.
Files changed:
eagle_default/in/xml/xml-impairments.xml

SDP-70816 The Conversion Workflow was enhanced to allow skip delete on data upload to Snowflake

In order to skip delete  prior to data ingestion, a new RTR parameter was added when running the conversion workflow.

<taskParameter>
<name>deletedata</name>
<value>N</value>
</taskParameter>

In this case, the file mask of the result file will contain extra substring NODELETE

K26C9A17D6FE1891_ASISCOPY_CONVERSION_NODELETE_20220517_20220616_GENERICSMF_20230125.xml

The DELETE is skipped in case additional substring is added to the file name after CONVERSION_NODELETE and only load process will be executed.

It is recommended to use additional parameters like sourcename and deletedata set to N to move additional data without deleting the existing data from Snowflake within the given date range.

SDP-53501 EDS warehousenavmonthend resource available

Added a resource to support the database table, ESTAR.MONTHLY_ESTAR_NAV, via the Eagle to Vault pipeline using the new warehousenavmonthend resource. The ReinitMany procedure is implemented for better performance.

Files changed:
rdf_metadata/rdf_physical_schema_generator.py
warehouse/warehousenavmonthend.json
warehouse/warehousenavmonthend_in.json
warehouse/warehousenavmonthend-models.json
templates/eagle_wrhs_warehousenavmonthend_reinitbatch.oracle.txt
templates/eagle_wrhs_warehousenavmonthend_reinitbatch.snowflake.txt
templates/db_oper/snowflake/warehousenavmonthend.reinit_batch_many.sql

Production Bugs

SDP-78109 Pricing report extract was corrected to produce consistent results

The pricing report extract - PRICINGREPORTEXTRACT depends on a nested SQL query from the RULESDBO.PRICE_VALIDATION_STATUS_DETAIL, where a field is derived by using the ‘xmlagg’ function. When returning multiple rows of data, the order of the data returned was generating a different result order. To fix this, the ORDER BY statement was added in nested query when running the xmlagg function.

Files changed:
eagle_ml-2-0_cm/out/tagvalue/extract_pricing_report.inc

SDP-77641 EDS Extensions for Oracle from EDS UI

Code changes were applied to support EDS extensions for any database schema from the EDS UI.

Files changed:
src/eagleinvsys/rdf_metadata/rdf_meta_templates_processor.py
src/eagleinvsys/rdf_metadata/rdf_meta_utils.py
src/eagleinvsys/rdf_metadata/rdf_ontology_generator.py
src/eagleinvsys/rdf_metadata/rdf_ontology_generator_utils.py
src/eagleinvsys/rdf_metadata/rdf_physical_schema_generator.py

SDP-76987 AccountingTrade load interface modifications for impliedCommissionIndicator when applying net amount calculations

The mapping for Tag 78 - impliedCommissionIndicator was fixed to change ‘N' to “NO’ and ‘Y' to 'YES’ according to the panels incoming values so the net amount was calculated correctly.

Files changed:
eagle_default/in/include/csv-buy_columns.inc
eagle_default/in/include/csv-capfundinvestments_columns.inc
eagle_default/in/include/csv-deliver.inc
eagle_default/in/include/csv-fxfwd_columns.inc
eagle_default/in/include/csv-fxspot.inc
eagle_default/in/include/csv-receive.inc
eagle_default/in/include/csv-sell_columns.inc
eagle_default/in/xml/include/tradeleg_columns.inc
eagle_default/in/xml/xml-impairments.xml

SDP-76703 The Mapping for the Schedule resource was corrected

The source name was not resolved on data processing due to missing mapping and was raising error. The mapping was corrected for the sourceName element which resolved this issue.

Files changed:
ontology/reference/schedule-models.json
ontology/reference/schedule_in.json

SDP-76694 The CheckSum issues were corrected for the warehousetrade and the warehouseperformance resources

The metadata for warehousetrade and warehouseperformance resources was improved to calculate consistent Checksums.

Files changed:
ontology/warehouse/warehouseperformance-models.json
ontology/warehouse/warehousetrade-models.json

SDP-76656 Enhanced Entity Extract with custom extension tables to use EFFECTIVE_DATE

There was an issue with SQL generated for the Entity Extract when interacting with custom extension tables. Custom extension tables were joined without the EFFECTIVE_DATE which caused multiplied records to appear in the SQL result. The issue was fixed by modifying the join to use the EFFECTIVE_DATE.

Files changed:
eagle_ml-2-0_cm/dbkeysmap.inc

SDP-76597 Enhanced the RemoveEntityDuplicates flag in the entity extract to work with extensions having a large number of columns

EagleML already includes functionality to remove duplicates on the entity extract. For larger extension tables nearing 1000 rows, the query for the extract will violate Oracle edit checks. The SQL query for the generic entity extract - ENTITYEXTRACT was refactored and now the RemoveEntityDuplicates flag may be used for entities with larger extension tables without generating an error.

Files changed:
eagle_ml-2-0_cm/out/tagvalue/extract_entity.inc

SDP-76519 Added support to lookup accruedInterestType during the load process

EagleML now can lookup Tag 3715 - accruedInterestType for the following processingSecurityTypes:

• DBIBFD (Interest Bearing Debt)

• DBIBPK (Interest Bearing Debt PIK)

• DBIBMU (Interest Bearing Municipal)

• DBIBMA (Interest Bearing at Maturity)

• DBIBST (Interest Bearing Short Term)

• DBIVIV (Inverse Floating Debt Instrument)

• DBSTST (Short Term Index Fund)

Files changed:
eagle_default/in/include/csv-buy_panels.inc
eagle_default/in/include/csv-sell_columns.inc