EagleML Release Notes - August 23rd 2022

Owned by Jeffrey Santora (Deactivated)
Last updated: Aug 26, 2022 by Olga Galis (Deactivated)
4 min read

Enhancements

SDP-64635 Lastruntime is updated in MC while using EDS extract

The eds_exec stream was modified to use the lastruntime field stored in the database instead of using redis cache. This provides the ability to update the lastruntime and monitor the process in Message Center for inMarket clients using IWS.

Files changed:
eagle_ml-2-0_cm/rtr_parse_params.inc
eagle_ml-2-0_cm/in/xml/exec_eds.xml

SDP-64726 WRHSTRADEEXTRACT - Updated signage logic for list of fields

Signage Keys and Conditions were adjusted for the following fields.

COST_LOCAL – EagleML/WarehouseTransaction/warehouseTrade/costLocal
COST_BASE - EagleML/WarehouseTransaction/warehouseTrade/costBase
ORIG_ACQ_COST_BASE - EagleML/WarehouseTransaction/warehouseTrade/originalAcquisitionCostBase
ORIG_ACQ_COST_LOCAL - EagleML/WarehouseTransaction/warehouseTrade/originalAcquisitionCostLocal

Affected transactionCodes: OPENSWAP, OPENFUTURE, NONTAXMERGP, NONTAXSPINP, WRITE.
Affected recordType = 150, mttype = MTOPEN, cancelFlag = N, longShortIndicator = S.

Files changed:
eagle_ml-2-0_cm/out/tagvalue/extract_wrhs_trade_signage_logic.inc

SDP-63111 IssuerESG Inbound interface was updated to support new elements

The IssuerESG inbound interface was updated to support several new elements.

Data Base fields:
SECURITYDBO.ISSUER_ESG[_HIST].ENTERPRISE_VALUE (tag 18426)
SECURITYDBO.ISSUER_ESG[_HIST].ENTERPRISE_VALUATION_YEAR (tag 18427)
SECURITYDBO.ISSUER_ESG[_HIST].CARBON_EMISSIONS_SCOPE_1 (tag 18428)
SECURITYDBO.ISSUER_ESG[_HIST].CARBON_EMISSIONS_SCOPE_2 (tag 18429)
SECURITYDBO.ISSUER_ESG[_HIST].CARBON_EMISSIONS_SCOPE_1_2 (tag 18430)
SECURITYDBO.ISSUER_ESG[_HIST].CARBON_EMISSIONS_YEAR (tag 18431)
SECURITYDBO.ISSUER_ESG[_HIST].CARBON_EMISSIONS_KEY (tag 18432)
SECURITYDBO.ISSUER_ESG[_HIST].CARBON_EMISSIONS_SCOPE_3 (tag 18433)

Xpath:
EagleML/referenceTransaction/referenceTransaction/issuerESG/enterpriseValue
EagleML/referenceTransaction/referenceTransaction/issuerESG/enterpriseValuationYear
EagleML/referenceTransaction/referenceTransaction/issuerESG/carbonEmissionsScope1
EagleML/referenceTransaction/referenceTransaction/issuerESG/carbonEmissionsScope2
EagleML/referenceTransaction/referenceTransaction/issuerESG/carbonEmissionsScope1and2
EagleML/referenceTransaction/referenceTransaction/issuerESG/carbonEmissionsReviewYear
EagleML/referenceTransaction/referenceTransaction/issuerESG/carbonEmissionsKey
EagleML/referenceTransaction/referenceTransaction/issuerESG/carbonEmissionsScope3

File changed:
eagle_default/in/xml/xml-ref_issuer_esg.xml
metadata/bind/eagleml_ref_issueresg.xml

SDP-64032 IssuerESG Outbound interface was updated to support new elements

The IssuerESG outbound interface was updated to support several new elements.

Data Base fields:
SECURITYDBO.ISSUER_ESG[_HIST].ENTERPRISE_VALUE (tag 18426)
SECURITYDBO.ISSUER_ESG[_HIST].ENTERPRISE_VALUATION_YEAR (tag 18427)
SECURITYDBO.ISSUER_ESG[_HIST].CARBON_EMISSIONS_SCOPE_1 (tag 18428)
SECURITYDBO.ISSUER_ESG[_HIST].CARBON_EMISSIONS_SCOPE_2 (tag 18429)
SECURITYDBO.ISSUER_ESG[_HIST].CARBON_EMISSIONS_SCOPE_1_2 (tag 18430)
SECURITYDBO.ISSUER_ESG[_HIST].CARBON_EMISSIONS_YEAR (tag 18431)
SECURITYDBO.ISSUER_ESG[_HIST].CARBON_EMISSIONS_KEY (tag 18432)
SECURITYDBO.ISSUER_ESG[_HIST].CARBON_EMISSIONS_SCOPE_3 (tag 18433)

Xpath:
EagleML/referenceTransaction/referenceTransaction/issuerESG/enterpriseValue
EagleML/referenceTransaction/referenceTransaction/issuerESG/enterpriseValuationYear
EagleML/referenceTransaction/referenceTransaction/issuerESG/carbonEmissionsScope1
EagleML/referenceTransaction/referenceTransaction/issuerESG/carbonEmissionsScope2
EagleML/referenceTransaction/referenceTransaction/issuerESG/carbonEmissionsScope1and2
EagleML/referenceTransaction/referenceTransaction/issuerESG/carbonEmissionsReviewYear
EagleML/referenceTransaction/referenceTransaction/issuerESG/carbonEmissionsKey
EagleML/referenceTransaction/referenceTransaction/issuerESG/carbonEmissionsScope3

File changed:
eagle_ml-2-0_cm/out/xml/t_issuer_esg_streaming.inc

SDP-18636 Added new elements to support the ability to load them in IssuePrice Interface

New fields were added in issuerPrice Object to support the ability to load for the following fields.

Data Base fields:
SECURITYDBO.PRICE.PERCENT_CHANGE_1M (tag 18434)
SECURITYDBO.PRICE.PERCENT_CHANGE_3M (tag 18435)
SECURITYDBO.PRICE.PERCENT_CHANGE_6M (tag 18436)
SECURITYDBO.PRICE.PERCENT_CHANGE_1YR (tag 18437)
SECURITYDBO.PRICE.PERCENT_CHANGE_WTD (tag 18438)
SECURITYDBO.PRICE.PERCENT_CHANGE_MTD (tag 18439)
SECURITYDBO.PRICE.PERCENT_CHANGE_QTD (tag 18440)
SECURITYDBO.PRICE.PERCENT_CHANGE_YTD (tag 18441)
SECURITYDBO.PRICE.PX_DIRTY_CLEAN (tag 18442)
SECURITYDBO.PRICE.PX_SETTLE (tag 18443)
SECURITYDBO.PRICE.PX_SETTLE_LAST_DATE (tag 18444)
SECURITYDBO.PRICE.QUOTED_SPEC (tag 18445)
SECURITYDBO.PRICE.SETTLEMENT_TYPE (tag 18446)

Xpath:
EagleML/referenceTransaction/referenceTransaction/issuePrice/percentChange1m
EagleML/referenceTransaction/referenceTransaction/issuePrice/percentChange3m
EagleML/referenceTransaction/referenceTransaction/issuePrice/percentChange6m
EagleML/referenceTransaction/referenceTransaction/issuePrice/percentChange1yr
EagleML/referenceTransaction/referenceTransaction/issuePrice/percentChangeWtd
EagleML/referenceTransaction/referenceTransaction/issuePrice/percentChangeMtd
EagleML/referenceTransaction/referenceTransaction/issuePrice/percentChangeQtd
EagleML/referenceTransaction/referenceTransaction/issuePrice/percentChangeYtd
EagleML/referenceTransaction/referenceTransaction/issuePrice/pxDirtyClean
EagleML/referenceTransaction/referenceTransaction/issuePrice/pxSettle
EagleML/referenceTransaction/referenceTransaction/issuePrice/pxSettleLastDate
EagleML/referenceTransaction/referenceTransaction/issuePrice/quotedSpec
EagleML/referenceTransaction/referenceTransaction/issuePrice/settlementType

Files changed:
tpe/dynamic/metadata/bind/eagleml_ref_issueprice.xml
tpe/dynamic/msgcenter/eagle_default/in/xml/xml-ref_issueprice.xml
tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/xml/t_issueprice_streaming.inc

SDP-64345 Added new elements to support the ability to extract them in the IssuePrice Interface

New fields were added in issuerPrice Object to support the ability to extract for the following fields.

Data Base fields:
SECURITYDBO.PRICE.PERCENT_CHANGE_1M (tag 18434)
SECURITYDBO.PRICE.PERCENT_CHANGE_3M (tag 18435)
SECURITYDBO.PRICE.PERCENT_CHANGE_6M (tag 18436)
SECURITYDBO.PRICE.PERCENT_CHANGE_1YR (tag 18437)
SECURITYDBO.PRICE.PERCENT_CHANGE_WTD (tag 18438)
SECURITYDBO.PRICE.PERCENT_CHANGE_MTD (tag 18439)
SECURITYDBO.PRICE.PERCENT_CHANGE_QTD (tag 18440)
SECURITYDBO.PRICE.PERCENT_CHANGE_YTD (tag 18441)
SECURITYDBO.PRICE.PX_DIRTY_CLEAN (tag 18442)
SECURITYDBO.PRICE.PX_SETTLE (tag 18443)
SECURITYDBO.PRICE.PX_SETTLE_LAST_DATE (tag 18444)
SECURITYDBO.PRICE.QUOTED_SPEC (tag 18445)
SECURITYDBO.PRICE.SETTLEMENT_TYPE (tag 18446)

Xpath:
EagleML/referenceTransaction/referenceTransaction/issuePrice/percentChange1m
EagleML/referenceTransaction/referenceTransaction/issuePrice/percentChange3m
EagleML/referenceTransaction/referenceTransaction/issuePrice/percentChange6m
EagleML/referenceTransaction/referenceTransaction/issuePrice/percentChange1yr
EagleML/referenceTransaction/referenceTransaction/issuePrice/percentChangeWtd
EagleML/referenceTransaction/referenceTransaction/issuePrice/percentChangeMtd
EagleML/referenceTransaction/referenceTransaction/issuePrice/percentChangeQtd
EagleML/referenceTransaction/referenceTransaction/issuePrice/percentChangeYtd
EagleML/referenceTransaction/referenceTransaction/issuePrice/pxDirtyClean
EagleML/referenceTransaction/referenceTransaction/issuePrice/pxSettle
EagleML/referenceTransaction/referenceTransaction/issuePrice/pxSettleLastDate
EagleML/referenceTransaction/referenceTransaction/issuePrice/quotedSpec
EagleML/referenceTransaction/referenceTransaction/issuePrice/settlementType

Files changed:
tpe/dynamic/metadata/bind/eagleml_ref_issueprice.xml
tpe/dynamic/msgcenter/eagle_default/in/xml/xml-ref_issueprice.xml
tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/xml/t_issueprice_streaming.inc

SDP-62855 naicDesignation and insuranceValuationLot Outbound Interfaces were updated to support a new element

The naicDesignation and insuranceValuationLot Outbound interfaces were updated to support a new element called naicReviewDate.

Data Base fields:
ISTAR.ISTAR_XREFERENCE.NAIC_REVIEW_DT

naicDesignation Xpath:
EagleML/referenceTransaction/referenceTransaction/naicDesignation/xreference/naicReviewDate

insuranceValuationLot Xpath:
EagleML/warehouseTransaction/insuranceValuationLot/naicDesignation/xreference/naicReviewDate

Files changed:
eagle_ml-2-0_cm/out/tagvalue/extract_naic_designation.xml
eagle_ml-2-0_cm/out/tagvalue/extract_warehouse_istarlot.xml
eagle_ml-2-0_cm/out/xml/t_ref_naic_designation.xml
eagle_ml-2-0_cm/out/xml/t_warehouse_istarlot_streaming.inc

SDP-62727 Add Default Effective Date Value for Cloud Extract

The Cloud workflow has changed as follows.

  1. ebs:default:effectivedate now resolved by expression:

    iif(StringLength(:param_effectivedate:) = 0 AND (StringLength(:param_fromeffectivedate:) = 0 AND StringLength(:param_toeffectivedate:) = 0), convert(today(), "yyyy-mm-dd"), :param_effectivedate:)

  2. ‘fromeffectivedate’ and ‘toeffectivedate’ added as a parameters

  3. ebs:default:fromeffectivedate and ebs:default:toeffectivedate resolves from input parameters from above.

When we have no effectivedate parameters in the history mode extract, the workflow automatically sets effective date to current date to prevent region overload and extracting the entire table.

Files changed:
eagle_ml-2-0_workflows/eagle_wrf_cloud_extract.wrf
eagle_ml-2-0_workflows/eagle_wrf_cloud_extract_schd.wrf
eis/ejmwf/eagle_wrf_cloud_extract.ejmwf

SDP-64381 Kafka Confluent Extracts Support

Support of Kafka Confluent as a target sink for extracts was added. With new feature Data Rules has ability to use Kafka Confluent infrastructure as an extract sink to send data between platforms. Kafka can be used as an optimal integration instrument, as Kafka is a persistent system and offers strong durability and fault tolerance guarantees. On top of it, Data Rules implementation has scalable micro-batching system which allows to split data to micro-batches with ~5000 records. To use new Kafka Confluent producer for extracts you need to add TargetSinkType - ssKafkaType parameter to your RTR or MT profile and specify name of the Distribution profile via StreamingDistributionProfileName - your_kafka_distr_profile_name parameter. It’s also recommended to specify max_batch_size RTR parameter to avoid issues with data load if you extract big chunk of records (recommended size - 4000-5000). Before usage, Kafka Confluent Topic should be created and configuration should be set for producer config In-Market via streaming distribution profile and consumer config in Cloud using the DNA Kafka Consumer Service configuration API. The API is used to store the non-secure portion of the configuration https://apps.sd.dev.az.eagleinvsys.com:8443/api/vault/dnakafkaclient/api/doc/

RTR Example:

<EagleML xmlns="http://www.eagleinvsys.com/2011/EagleML-2-0 " xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="RunTaskRequest" eaglemlVersion="2-0" xsi:schemaLocation="http://www.eagleinvsys.com/2011/EagleML-2-0 eagleml-main-2-0.xsd" eaglemlType="RunTaskRequest">

<header>

<messageId>E30X5FTN4N317JFF</messageId>

<sentBy>MC2EJM</sentBy>

<sendTo>http://www.eagleinvsys.com/eagle_default_ruleservice </sendTo>

<creationTimestamp>2018-04-13T09:52:39.73-05:00</creationTimestamp>

<expiryTimestamp/>

<expiryLoadFlag/>

</header>

<taskIdentifier>

<processingSequenceNumber/>

<correlationId>{GUID}</correlationId>

<businessTaskId>EBS_warehouseposition</businessTaskId>

<sequenceNumber/>

</taskIdentifier>

<taskTypeEnum>NEW</taskTypeEnum>

<synchronousExecution>yes</synchronousExecution>

<taskParameters>

<taskParameter>

<name>ServiceName</name>

<value>EBS</value>

</taskParameter>

<taskParameter>

<name>ServiceMethodName</name>

<value>extract</value>

</taskParameter>

<taskParameter>

<name>ResourceName</name>

<value>warehouseposition</value>

</taskParameter>

<taskParameter>

<name>OutputFormat</name>

<value>csv</value>

</taskParameter>

<taskParameter>

<name>maxrows</name>

<value>1000000</value>

</taskParameter>

<taskParameter>

<name>max_batch_size</name>

<value>4000</value>

</taskParameter>

<taskParameter>

<name>effectivedate</name>

<value>2016-09-24</value>

</taskParameter>

<taskParameter>

<name>StreamingDistributionProfileName</name>

<value>kafka_producer</value>

</taskParameter>

<taskParameter>

<name>TargetSinkType</name>

<value>ssKafkaType</value>

</taskParameter>

</taskParameters>

</EagleML>

 

Files changed:
common_core\kafka_core\config\kafka_ssl_config.py
common_core\kafka_core\config\kafka_config_enum.py
common_core\kafka_core\config\abstract_client_config.py
eagleinvsys\mc2py\eds_kafkaendpoint\kafka_bootstrap.py

SDP-64771 Added new type of distribution profile known as streaming distribution profile

To be able to work with new messaging approach with Kafka Confluent, a new distribution profile type was added which is known as streaming distribution profile.

To use this new type of distribution profile, a new parameter was added to the instructions {{StreamingDistributionProfileName}}.
This parameter can be used in RTRs directly or added to MT profiles, workflows, schedules in Automation Center or services, streams or workflows which trigger EDS extracts.
The value of this parameter is the name of the profile (without _profile postfix and .json).

The new profile has the same naming convention as other distribution profiles - {name}_profile.json - and the same location: (estar/tpe/dynamic/metadata/distributions by default)

Distribution profiles used for Kafka messaging have different fields and parameters in the profile itself, comparing to default profiles.

Files changed:
eagleinvsys/eds_transform.py
eagleinvsys/etl/core_etl/genetl.py
eagleinvsys/etl/etl_main.py
eagleinvsys/etl/transform.py
common_core/sink/kafkasink.py
common_core/processing_env/instructions.py
common_core/enums/distrib_enums.py
common_core/enums/cm_enum.py
distribution/distribution_executor.py

SDP-62915 Ontology workflow definition was extended to support multiple feed types

Ontology workflow definition was extended to support the execution of Ontology based Resources and FeedTypes tasks, dataset based interfaces. Currently archived and single text files are supported via workflow execution.

SDP-64264 Z-fields logic was implemented for warehouseposition summary extension table

The additional logic was implemented to update Z-fields in the warehouseposition summary extension table on data load. To populate Z-fields, the sql block was enhanced with construction for warehouseposition.reinitbatch.sql

Files changed:
ontology/templates/db_oper/snowflake/warehouseposition.reinit_batch.sql

SDP-52536 Date filter was improved for Ontology and Dataset based extracts

We support multiple parameters if filter defined differently, so for all date filters:

{{"MASTERTABLE.SETTLEMENT_DATE @(:settlementdate)"}}

All date filters were changed to the same definition for the case equal(=).

Cases for >= and <= are still support only single date value.

For single parameter - no changes in the query.

AND MASTERTABLE.EFFECTIVE_DATE = TO_DATE(:effectivedate,'YYYY-MM-DD')

For multiple values:

WHERE (1=1)
AND MASTERTABLE.EFFECTIVE_DATE IN (TO_DATE('2007-08-20', 'YYYY-MM-DD'), TO_DATE('2007-08-31','YYYY-MM-DD'))

This logic work on Oracle and Snowflake.

For Ontology based extracts and below date filters:

·         accountingdate

·         effectivedate

·         releasedate

·         postdate

·         valuationdate

·         tradedate

·         settlementdate

·         startdate

·         enddate

·         ledgereffdate

·         monthendaccountingdate

For EDS Datasets:

Only effectivedate filter is supported.

Usage example:

<taskParameter>
<name>effectivedate</name>
<value>2007-03-17,2007-05-19</value>
</taskParameter>

Production Bugs

SDP-63865 Delta based ENTITYEXTRACT failed with sourcename or activestatus filters

Due to the fact that the sourcename and activestatus filters use binds that ran after the date filters that also leverage binds in the SQL query, a "not all variables bound" error was received. The order of the filters in the WHERE part of the SQL has been changed so that the SQL query does not raise errors anymore.

Files changed:
eagle_ml-2-0_cm/out/tagvalue/extract_entity.inc

SDP-65040 ContraSecAlias was not updating on the first load

An adhoc call was added to update the contra_sec_alias field during the initial load of a forward security and resolve it with the new security alias value. Previously, the logic was running after the security was resolved and therefore the initial load did not resolve the contra_sec_alias field.

Files changed:
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/xml/include/msgxchg_sec_core.inc

Â