EagleML Release Notes - July 2019

Enhancements:

SDP-24599: Batch Mode for Loading Underlying Securities

Batch Mode for loading underlying securities was implemented. Batch Underlying load process expects incoming data to be grouped by security + underlying type,

where records sorted by time periods and contain element referenceTransaction/header/batchKey as

SECURITY IDENTIFIER^IDENTIFIER TYPE^UNDERLYING TYPE.

Batch Underlying Load process deletes all existing Underlying records with the following key: security + underlying type and after that loads incoming records.

Processing option UNDERLYING_UPSERT_MODE allows to avoid delete step for Batch Underlying Load process.

New file:

xml-smf_underling_batch.xml

SDP-23971: highlyLiqInvMin Element was added to the EntityLiquidity Interface Data Map

A new field was added to the database RULESDBO.ENTITY_LIQUIDITY.HIGHLY_LIQ_INV_MIN and was enabled in the inbound mapping and extract.

The EntityLiquidity element is part of EntityLiquidity Interface for load and extract.

SDP-23798: Added support for the CLIENTEXTRACT via MC2 engine

The Swagger file was updated to support the definition of the CLIENTEXTRACT. The CLIENTEXTRACT is fully supported via EagleML and MC2.

SDP-23796: Swagger file was updated with missing feed types

The Swagger file was updated to support the definition of the following feed types in the swagger-extracts.json file:

• PORTFOLIOVALUATIONEXTRACT

• ENTITYLISTLISTEXTRACT

• CALBUSINESSDATESEXTRACT

• REFGIPSCOMPOSITEEXTRACT

• RPTTIMESTAMPS

• CODETRANEXTRACT

• CODEVALEXTRACT

SDP-23717: Delta Mode was fixed for the Warehouse Trade Charges extract

The Warehouse Trade Charges extract was adjusted to correct support for delta mode operations.

File changed:

eagle_ml-2-0_cm/out/tagvalue/extract_warehouse_trade_charges.inc

SDP-23684: Code Translation for Primary Exchange Code is currently supported for Security-based Reference and SMF inbound interfaces

The functionality Code Translation is currently supported in the following inbound interfaces:

• AccountingCA

• GenericCA

• CancelCA

• MandatoryCAExclusion

• VoluntaryCAElection

• AnalyticEquity

• AnalyticFI

• GenericIssueAnalytic

• SMFIdentifiers

• SMFIdentifier

• ReferenceElementOverride

• IssuePrice

• ExchangePrice

• IssuePcPrice

• IssueFxRateDemand

• Rating

• RecoveryRate

• IssuerRole

• IssuerRole – delete rule

• IssueVariableRate

• ModelPrice

• NaicDesignation

• Schedule

• SecurityNotes

• SecurityOverrides

• TimeSeries

• VendorCashFlow

• AccountingSMF

• MultilegSMF

• UnderlyingSMF

• DeleteSMF

• SMF MI

The Code Translation of the Exchange Code may be triggered by adding the node referenceTransaction/header/codeTranslations with the value 'primaryExchangeCode' in the incoming data message.
Primary Exchange Codes in an incoming message will be sent to the lookup Code Translation with the following parameters:
• From Code Short Name = Primary Exchange Code from an incoming message (tag 1981, <primaryExchangeCode> node)
• From Source = Source Name from an incoming message (<sourceName> node)
• To Source = 'EAGLE PACE'
• From Code Name = 'IEXCHANGE CODE'
• To Code Name = 'IEXCHANGE CODE'

The output value To Code Short Name will be mapped to tag 1981 if the code translation is successful, else it will be mapped to tag 1981 from the incoming message.

SDP-23480: Added <sequenceNumber> element node in MC2 extracts

Added changes to set the <sequenceNumber> element with a value of ‘1’ by default for the MC2 extract to be in consistent with the MC extract.

File changed:
extract_process_params_on_start.inc

SDP-22966: New element cancelAccountingDateSwitch was added for Cash Interfaces

A new element cancelAccountingDateSwitch was added to the following Cash Interfaces:

• AccountingCash

• AccountingAssetCash

• AccountingReclaimCash

If the file has the cancelAccountingDateSwitch flag set to Y, it is possible to cancel a record in a closed accounting period.

File changed:
xml-acct_cash_cancel.xml

SDP-22890: Warehouse Position and OpenLot extracts were enhanced to work correctly with extractLevel and DeltaBased filters

The Warehouse Position and OpenLot extracts were enhanced to use the update_date from the position table when the DeltaBased option is enabled and the extractLevel parameter is used at the same time.

Files changed:

eagle_ml-2-0_cm/extract_sorting_sql.inc
eagle_ml-2-0_cm/out/tagvalue/extract_delta_based_param.inc

SDP-22303: Provided ability to Cancel Multiple EJM Workflows by Correlation ID

Added ability to cancel several processes by defining a list of correlation Id's in the RTR. This logic is realized in eagle_ml-2-0_default_cm_control_process stream.

To use this option define a list of correlation Id values separated by commas in the CancelCorrelationId task parameter:

<taskParameter>
<name>CancelCorrelationId</name>
<dataType>S</dataType>
<value>value1,value2,value3</value>
</taskParameter>

Files changed:
control_process.xml
control_cancel_process.inc

SDP-22292:  Cancel Workflow via RTR Parameters CancelWorkflowName or CancelCorrelationId

Wrong error was returned on Cancel Workflow in case CancelCorrelationId task parameter was populated without CancelWorkflowName in RTR .
Updated condition for handling required RTR Parameters to Cancel Workflow. Cancel Workflow can be done via RTR Parameters CancelWorkflowName or CancelCorrelationId.

Files changed:
w_config.inc
execute_wrkfl_event.xml

SDP-22222: EJM Performance Optimization on Task Reporter Step

The StreamID parameter was added to the task files query to improve the query performance.
The parameter was added dynamically only to non-PRF file data streams.

Files changed:
eagle_ml-2-0_cm/out/tagvalue/data_file_task.xml
eagle_ml-2-0_cm/out/xml/task_reporter_generate_task_unit.inc

SDP-21756: Warehouse Trade was enhanced to extract additional data from the TRADESDBO.TRADE_DETAIL_EXT table

The Warehouse Trade Extract was enhanced to support inclusion of the TradeDetailExt table.

Files changed:
extract_warehouse_trade.inc
t_warehouse_trade_streaming.inc

SDP-21584: Grouping issue resolved for the Issuer Organization Extract

The Issuer Organization transformation rule was modified. Two additional keys, SRC_INTFC_INST and EFFECTIVE_DATE, were added to correct the grouping of the extracted dtata from history table .

Files changed:
eagle_ml-2-0_cm\\out\\xml
t_issuer_org_streaming.inc
eagle_ml-2-0_cm\\out\\xml
t_issuer_org.xml

SDP-20423: Warehouse XSL Transformation Rule was enhanced to process and pass Correlation ID Node in XMLDBAPI output

The Warehouse XSL Transformation rule was enhanced to process and pass the correlation id node in XMLDBAPI output. This node is used for correct rollback functionality, when the XSL-translated data is loaded in sub-batchmode.

Sub-matchmode is when the load consists of two steps. The first step is load and second step is roll up.

SDP-20291: Reinsert/Upsert mode was improved for Center Roles and PACE Roles

Modified the behavior for Center and PACE Roles.  The problem for Center Roles was that in case of adding a new role to a user, all existing assignments were removed.
The current assignment of the PACE and Center Roles work the same way. The default mode for adding Center and Pace Roles to a User is UPSERT.

The following improvements were made:

• If a Center role is already assigned to user, then the Role is reassigned only when the isPrimaryRole flag is changed.
  This change eliminates reduntant DB and Panels calls in case of nothing is changed in the Center role for User assignment.

• Implemented a REINSERT mode. In this mode, all Center and PACE roles are removed from the user and assigned only roles from the incoming User file.
  To run the User update in REINSERT mode it is required to use the <processingOptions>REINSERT</processingOptions> in the userAdministractionTransaction Header.
  
  Here is the example:
  <userAdministrationTransaction>
  <header>
  <objectType>User</objectType>
  <processingOptions>REINSERT</processingOptions>
  </header>

Files changed:
msgcenter\eagle_default\in\xml\include\user_common.inc

SDP-18601: Improvements in EJM to allow using colon(:) symbol in BusinessTaskId

The EJM code has been enhanced to allow user to use BusinessTaskId with colon(:) symbol.

Sample:

<businessTaskId>WRHS:POSITION:EXTRACT</businessTaskId>

Files changed:
dynamic/msgcenter/eagle_ml-2-0_cm/collect_task_parameters.inc
dynamic/msgcenter/eagle_ml-2-0_cm/create_cm.inc
dynamic/msgcenter/eagle_ml-2-0_cm/create_cm_load.inc
dynamic/msgcenter/eagle_ml-2-0_cm/create_tsr.inc
dynamic/msgcenter/eagle_ml-2-0_cm/execute_task.inc
dynamic/msgcenter/eagle_ml-2-0_cm/out/xml/t_eagle_header_fields.inc
dynamic/msgcenter/eagle_ml-2-0_cm/out/xml/t_eagle_header_streaming.inc
dynamic/msgcenter/eagle_ml-2-0_cm/out/xml/task_reporter.xml
dynamic/msgcenter/eagle_ml-2-0_cm/out/xml/taskacknowledgement.xml
dynamic/msgcenter/eagle_ml-2-0_cm/in/xml/realtime_rtr.xml
dynamic/msgcenter/eagle_ml-2-0_cm/rtr_map_params.inc

SDP-18372: Code Translation for Primary Exchange Code supported for Generic SMF Inbound Interface

The functionality Code Translation is currently supported in the inbound interface GenericSMF.

The Exchange Code Translation may be triggered by adding the node referenceTransaction/header/codeTranslations with the value 'primaryExchangeCode' in the incoming SMF data message.
The Primary Exchange Codes in an incoming message will be sent to the lookup Code Translation with the following parameters:
• From Code Short Name = Primary Exchange Code from an incoming message (tag 1981, <primaryExchangeCode> node)
• From Source = Source Name from an incoming message (<sourceName> node)
• To Source = 'EAGLE PACE'
• From Code Name = 'IEXCHANGE CODE'
• To Code Name = 'IEXCHANGE CODE'

The output value To Code Short Name will be mapped to tag 17 if the code translation is successful, else it will be mapped to tag 1981 from the incoming message.

SDP-10938: EffectiveDate and Source filters added to the EntitySecurityLending Extract

The EffectiveDate and Source filters were added to the EntitySecurityLending Extract (ENTITYSECLENDINGEXTRACT).

Files changed:
extract_entity_sec_lending.inc

SDP-10923: Automatic Detection of Encrypted Files based on the File Content is On by Default

Automatic detection of encrypted files is enabled by default and can be OPT-OUT in eagle_ml-2-0_custom_cm/w_config_custom.inc via new flag

 <COL TAG="W_AUTO_DETECT_ENCRYPTION" EXPRESSION="'N'"/>

The automatic detection of encrypted files requires Python Rule Service to be installed on the region. Automatic detection of encrypted files via the unpacker stream is only supported for Eagle V2015 version and higher.

Files changed:

eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/ndfa/decrypt_incoming.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/unpacker/wrf_unzip.prf

New files:

eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/check_encrypted.inc
eagle/estar/tpe/dynamic/msgcenter/pyrules/eagleinvsys/mc2py/check_encrypted.py (Python Rule Service)

SDP-9271: New step load_underlying_generic_smf_mi with mask UNDERLYINGGENERICSMFMI added to the eagle_wrf_generic_load workflow

Added support for a new step in the load_underlying_generic_smf_mi with mask UNDERLYINGGENERICSMFMI in the eagle_wrf_generic_load workflow when loading Generic SMF for Underlying security before loading data via the eagle_ml-2-0_default_in_xml_smf_mi in load_generic_smf_mi step.

Files changed:

eagle_wrf_generic_load.wrf
eagle_wrf_generic_load_smf_objects.wrf

Production Bugs
SDP-23886: The query for the Client Extract was redesigned

The SQL query for the Client extract was updated. The problem identified was in the FROM clause where there was an RTRIM operator used for join tables as it had negative impact on SQL performance. As part of solution the RTRIM operator was deleted. Also, streaming mode was added which provides the ability to delete two joins to the entity_xreference table without changing the content of the result extract.  These modifications and enabling streaming mode results in a faster transformation providing improved throughput.

SDP-23330: Rule for Schedule Streams was updated to show the correct Status in Automation Center

The rule for schedule stream was updated to show the correct status in Automation Center when the ConsolidateTSR is used and there are errors on the sub processes. Previously if the workflow failed, the failed status was properly displayed in the Monitoring Tool, but not in Automation Center. The corresponding status was marked as SUCCESS in Automation Center. This issue was fixed.

File changed:
workflow_restart.xml

SDP-22135: Issuer Role was improved to avoid unnecessary ADD panel calls

Fixed issues with the a variable override between data processing loops. Added tags remap before the panel call to avoid issues with panel execution.

Files changed:
xml-ref_issuerrole_common.inc

SDP-21588: Grouping issue was corrected for the Issuer Relationship Extract

The Issuer Relationship extract rule was adjusted to enable correct grouping of the extracted records.

Files changed:

eagle_ml-2-0_cm\out\xml\t_issuer_relationship_streaming.inc
eagle_ml-2-0_cm\out\xml\t_issuer_rel.xml