EagleML Release Notes - September 2019

Owned by Olga Galis (Deactivated)
Last updated: Nov 13, 2024 by Kinga Huszno
8 min read

Enhancements

SDP-27317: Added support for the Date Rule as part of the File Mask in Profile Loader

Introduced the ability to use the date rule in any place of the file mask.  The date rule is resolved before matching the mask to the actual file name.
The name of the date rule should start with “at” (@) symbol. If there are more symbols in the file name after the daterule name, then the daterule name should be enclosed into curly braces.
Examples:
SDP27317@Today
SDP27317_@{Previous Business Day}_DATA

SDP-27314: Wildcard Support was added in the File Masks in the Profile Loader

Enabled wildcard support in the file mask.  An unlimited number of wildcards may now be created in the file mask.  This provides greater flexibility for clients to manage files and packages.
In addition, wildcards may be used in both file and package names. Also, wildcard symbols are now automatically added to the beginning and end of the specified file mask.  For example, abc*def will automatically be treated as abc*def)

Files changed:

eagle_ml-2-0_cm/ndfa/launch_load_by_profile.inc
eagle_ml-2-0_cm/unpacker/custom_routing.inc
eagle_ml-2-0_cm/unpacker/send_to_wrf.xml

New files:

eagle_ml-2-0_cm/ndfa/check_mask_match.inc
eagle_ml-2-0_cm/ndfa/parse_in_mask.inc

SDP-27312:  EaglePY Service Version set in the w_config_eagle_components_version.inc

A number of enhancements to EagleML functionality requires installation of the EaglePY Service and pyruleservice components.
To activate Python script during the Multileg SMF load, an additional logic was introduced on the setup step.
The Python script will be used for data transformation if the parameter, <CODE> :g_eaglepy_version: :='2019.9.x. '; </CODE> is set in the w_config_eagle_components_version.inc

Files changed:
setup_component_version.inc

SDP-26233: MultilegSMF Floater Model - Load and Extract

The logic for loading Multileg Securities via the workflow eagle_wrf_genric_load was enhanced.
A new file mask MULTILEGSMFMI is supported.
The following installation steps should be performed starting from the September EagleML Release and going forward for EagleML clients in order for the MultilegSMF Floater Model to work:

Please Install AllServices starting from September Release. Additional details can be found on Installation Components and Repositories page.

The Python script will be used for data transformation if the parameter, <CODE> :g_eaglepy_version: :='2019.9.x. '; </CODE>, is set in the w_config_eagle_components_version.inc

The Incoming EagleML Multileg SMF message will be transformed to sequence the EagleML messages (Multileg SMF, Underlying SMFs for every leg - Contract, Pay, Receive; then Issue Variable Rate and other Security-related objects for every leg) by Python script.  After transformation, the sequence of EagleML messages will be sent to the Reference No Split stream eagle_ml-2-0_default_in_xml_ref_no_split to load every part of the Multileg.

After transformation, the sequence of EagleML messages will be sent to the Reference No Split stream eagle_ml-2-0_default_in_xml_ref_no_split to load every part of Multileg. 
Python script is supported for Linux/Oracle regions.

The following feed types should be used to Extract Multileg SMF:

Generic SMF – feedtype = SMFEXTRACT.
Issue Variable Rate – feedtype = ISSUEVARRATEEXTRACT.
Earnings Time Periods – feedtype = EARNINGSTIMEPERIODEXTRACT.
Schedule - feedtype = SCHEDULEEXTRACT.
Security Notes - feedtype = SECURITYNOTESEXTRACT.
Issuer Role - feedtype = ISSUERROLEEXTRACT.

Files changed:

xml-smf_generic_batch.prf 
run_python.inc 
eagle_wrf_generic_load.wrf 
eagle_wrf_generic_load_smf_objects.wrf 
eagle_wrf_generic_load.ejmwf 
eagle_wrf_generic_load_smf_objects.ejmwf

SDP-27112: Enhanced FileNamePattern generation as part of Date Rule

The filename pattern was improved, by reading a date from the incoming file, generating an unique GUID() identifier and integrating this into the file name. Generated file names are distinct now.

Upon processing, the temporary, working files are deleted.

Files changed:

eagle_ml-2-0_cm\calc_date_from_rule.inc
eagle_ml-2-0_cm\http_calls.inc

SDP-26756: The Output for Warehouse Extracts was synchronized between EagleML, MC2 and EBS when the excludezero filter is applied

Warehouse Extracts executed via Extract Service worked incorrectly in case 'excludezero' parameter was set to 'N'. 
The issues was corrected and the output was synchronized between EagleML/MC0, Extract Service/MC2 and EBS.

  • If excludezero is set to Y or is Null, then the zero values will be excluded.

  • if excludezero is set to N, then the zero values will be included into extract

File changed:

extract_fields_query.inc

SDP-26705: Field size mismatch between EagleML and Database on LEI_DETAILS.LEGAL_FORM

The LEI_DETAILS.LEGAL_FORM Field Size was Changed in V17. The length of the LEI_DETAILS.LEGAL_FORM field is dependent on the environment version and it is set to 255 characters for V17 and 30 characters for prior versions.

Files changed:

eagle_default/in/xml/xml-ref_issuerorganization.xml

SDP-26614: Filter DeltaBased was Corrected for the Generic Extract via MC2 Engine

Calculated dates were not correctly updated in the database as part of the packer step in the eagleml_wrf_generic_extract workflow impacting the delta based processing. 
To address, added the eagle_default_calculate_date_final step to update the calculated date in the database for a workflow executed via MC2.

Files changed:
eagle_ml-2-0_cm/out/tagvalue/extract_calculate_date.inc
eagleml_wrf_generic_extract workflow

SDP-26489: Improved the date rule calculation for inbound files.

Improved the date rule calculation for inbound files.
The date rule calculation always had used Today() as the incoming parameter setting everything to the current date.
After the enhancement, the date from the incoming message may be used as an incoming parameter too.
For enabling this option attribute the fromCurrentDate should be set = N

For example:
If set the following options in data message
< calcDates>
< dateRule dateElementId="startDate" ruleName = "Previous Business Day" fromCurrentDate = "N"/>
< /calcDates>

If the startDate in the incoming message is <startDate>2019-09-06</startDate>, then the calculated value of the startDate will be 2019-09-05 because the previous business day is 2019-09-05 for 2019-09-06.

Files changed:
date_rule_calculation.inc

SDP-26189: DeltaBased Mode Support for the Settled Cash Activity Extract

The Delta Based mode was added for the Settled Cash Activity extract.

Usage:
< taskParameter>
< name>DeltaBased</name> 
< dataType>S</dataType> 
< value>Y</value> 
< /taskParameter>

It is highly recommended to use DeltaBased = Y option with DeltaBasedUID parameter, in order to have a unique last run date value.

Files changed:

eagle_ml-2-0_cm/out/tagvalue/extract_acct_settled_cash_activity.inc

eagle_ml-2-0_cm/out/tagvalue/extract_acct_settled_cash_activity.xml

SDP-26065: EagleML rules for the Generic SMF Load were updated

The FIRST_UNDER_SEC_ALIAS variable did not work correctly on the Generic SMF Load as the value from the GLOBAL variable was propagated to the next records in the file during processing.
The workaround for Message Center issue was provided on the EagleML side by removing the global property  SCOPE=”GLOBAL”  in the corresponding inbound rules.

Files changed:
xml-smf_underling_batch.xml
xml-smf_underling.xml
xml-smf_generic.xml
smf_core_mapping_tmp.inc
underlying_link_ins_upd.inc
underlying_smf_ins_upd.inc
xml-ref_yieldcurve.xml

SDP-25949: Datetime Format Supported for Issue Rating (Load and Extract)

The load interface for the Issue Rating was enhanced to support the datetime format via the submittedDate field.  Support is enabled in coordination with EagleML as of the v17R2.16 version forward.  Prior versions will continue to only support the date.

Files changed:
eagle_default/in/xml-ref_issuerating_all. xml
eagle_default/in/xslt/eagleml_rating_data_model.xsl

SDP-25525: Eagle Products Versions are printed in w_config_eagle_components_version.inc

Introduced a new include file, w_config_eagle_components_version.inc, which is auto-generated during the installation process as part of the eagle_ml-2-0_default_cm_w_setup stream.
The file contains the latest version of every Eagle Core Product installed on the Server and the last Update date, including the EagleML and MC2 components.

For example:
< !-EAGLEML-WITHOUT-OVERLAYS-2019.08.12.2-SNAPSHOT Update date: 2019-08-13 07:52:45->
< !-pypy-2019.7.1-20190722.113614 Update date: 2019-08-12 09:05:31->
< !-pyruleservice-2019.8.1.080914 Update date: 2019-08-12 09:06:44->

Files changed:
setup_component_version.inc
w_setup_eagle_version.xml
eagle_default_deploy.xml
eagle_default_deploy_submit_version.inc

SDP-24969: Generic Load Workflow was Enhanced to work with two Engines MC or MC2

The Generic Load workflow, eagle_wrf_generic_load, was modified to run under two engines: MC and MC2.  The parameters from the MC2 load workflow , eagle_wrf_generic_load_mc2, were added to the MC workflow.
The MC2 workflow is no longer supported and all additional enhancements will be done within the eagle_wrf_generic_load workflow.

SDP-24305: DateRule logic triggered only once in the Generic Load Workflow

Enhanced the process to have each date rule called and calculated only once per workflow run. All calculated values are stored in a code value lookup and are always available. The task parameter DateRules was added. 
For example:
< taskParameter>
<name>DateRules</name><dataType>S</dataType>
<value>Today,Previous Business Day</value>
< /taskParameter>

The new data tag dateRule was added in XSD schema for transaction header in the data message.

For example:

< calcDates>
< dateRule dateElementId="startDate" ruleName = "Previous Bus Day" fromCurrentDate = "Y"/>
< dateRule dateElementId="endDate " ruleName = "Today" fromCurrentDate = "N"/>
< /calcDates>

The new logic was implemented only for the warehouse inbound rule (EagleML format) only. The Date rule will be calculated with input date of Today. Logic for using the date from the incoming data file as an input date will be added in a future release.

Files changed:
eagle_default/in/xml/include/date_rule_calculation.inc 
eagle_default/in/xml/xml-warehouse_v2.xml 
eagle_ml-2-0_cm/in/xml/runtaskrequest.xml 
eagle_ml-2-0_cm/in/xml/w_setup_eagle_version.xml 
eagle_ml-2-0_cm/w_config.inc
XSD changes

SDP-14959: Updated Logic for the 'IncludeCurveAnalytics' Parameter for the Warehouse Position Extract

The logic for the 'IncludeCurveAnalytics' parameter was changed in the Warehouse Position extract. Previously, if the 'IncludeCurveAnalytics' was set to Y, then the POSITION_DETAIL_ANL_USER table was excluded from the list of requested tables and the PD_CURVE_ANALYTICS table was included instead. If the 'IncludeCurveAnalytics' is set to Y, then the POSITION_DETAIL_ANL_USER and PD_CURVE_ANALYTICS tables will be always included in the SQL query.

File changed:
extract_warehouse_position.inc

Production Bugs:

SDP-27658: Taxlots and Holdings Extracts - Corrected Mapping for fields BOOK_PRICE and BOOK_PRICE_LOCAL in XSD schema

The annotations of the elements bookPriceLocal and bookPriceBase were updated in the BookPrice.model group to point in the proper database location.

Files changed:
eagleml/schema/eagleml-wh-shared-2-0.xsd

SDP-27608: DeltaBased Warehouse Position Extract was Updated

The DeltaBased Warehouse Position Extract was failing when the extractLevel and datasources filters were used together as the wrong SQL query was being used to insert the position id into the temporary table. -The issue was corrected.

File changed:
eagle_default\in\xml\extract_warehouse_position.inc

SDP-27556, SDP-27512, SDP-27420: GenericSMF Load replaces some Xref data with Primary Asset or Alt Asset values

When the GenericSMF is loaded via panels (accountingValidationFlag=Y) and the Primary Asset Type and/or Alt Asset Type has any values for the 'ISIN', 'SEDOL', 'RIC', 'BBID', 'SICOVM', 'VALOREN', 'CMM', 'INTERNAL', 'CINS', some xrefs were replaced by the Primary Asset Type or Alt Asset Type values. Added logic to reorder the xrefs before sending to the panel based on the panel replacement logic. Changes were made for the following SMF types:

  • Mortgage Backed Security (DBFBFB, DBFBIO, DBFBPO, DBTATA, DBFBAL, DBFBCC)

  • Option Contract (OPOPCR, OPOPSW, OPOPDB, OPOPEQ, OPOPFB, OPOPCM, OPOPIX)

  • Right (EQWRXX)

  • Warrant (EQRTXX)

  • Short Term Debt (DBDCST, DBIBMA, DBIBRP, DBIBST, DBSTST, DBIBRR)

  • Future Contract (FTXXXX).

File changed:
accounting_smf_xref_order.inc

SDP-27473: datasources_SourcesFilter parameter correction in Swagger

The datasources_SourcesFilter description in the Swagger was not using the correct name. The API used the "sourcesFilter" filter name instead of "datasources". Added changes to make the API use the same source filter name ("datasources") as in the EagleML rules.

Files changed:
swagger-extracts.json

SDP-27223: Parameter WaitEventSPGenerated is set to ON by default during Install

The Parameter WaitEventSPGenerated is set by default in the w_config_version.inc during the Install of EagleML and AllServcies.
It is no longer dependent on any other variables.

Files changed:
eagle_ml-2-0_cm\in\xml\w_setup_eagle_version.xml

SDP-27153: New Fields originalAcquisitionCostBase and originalAcquisitionCostLocal supported on the Warehouse Position Extract

The fields originalAcquisitionCostBase and originalAcquisitionCostLocal were added to the Warehouse Position Extract. 
The new fields are extracted by default.

Files changed:
t_warehouse_position_streaming.inc

SDP-27132: Updated the Logic of the Workflow State Variable for the Payload Message in Memory Only Near-Real Time Processing

Incorrect severity codes were assigned to the task status sporadically. The issue occurred due to the workflow state not being stored properly.
The workflow state variable was missing the correlation id at the moment the Stored Procedure was updating the workflow state. The logic for collecting the workflow state was updated for the payload message to properly store and supply the initial CorrelationId.

Files changed:
eagle_ml-2-0_cm/in/xml/realtime_rtr.xml

SDP-27129: Adjusted Issuer Role logic to populate the end date correctly

The Issuer Role logic was adjusted to populate the end date when the record with the Issuer Alias + Role type + Sec alias.  As a result, the existing record will be updated or closed and a new record inserted.

Files changed:
xml-ref_issuerrole_common.inc

SDP-26750: Fixed the DateRule Variances in the Generic Extract if executed via MC2

The Calc Date step did not execute correctly in the eagle_wrf_generic_extract workflow running via MC2.  The feed type CALCULATEDATE was added on MC2 to synchronize with MC0. Currently the same logic is applied to the DateRule filter via MC0 and MC2.

Files changed:
eagle_ml-2-0_cm/out/tagvalue/extract_calculate_date.xml
eagle_ml-2-0_cm/out/xml/extract_service_start.inc
eagle_ml-2-0_cm/out/xml/t_eagleml-2-0_out_extract_service.rsf

New files:
eagle_ml-2-0_cm/out/tagvalue/extract_calculate_date.inc
eagle_ml-2-0_cm/out/xml/extract_service_calculate_date.xml

SDP-26600: Issue with Accounting Basis field corrected for the Warehouse Open Lot

The accountingBasis was ignored if part of an incoming Open Lot file.  The Accounting Basis for the corresponding Entity was retrieved from the Stored Procedure instead.
Fixed the style-sheet transformed EagleML Open Lot inbound files to DBML requests. 
The IN_ACCOUNT and IN_ACCOUNT_BASIS are now passed to the stored procedure bind for the load.

SDP-26278: Resolved an exception during SMF Processing at Midnight

When the update time for the generic SMF's current time processing spanned across midnight, the time value rolled to the next day, but the day value remained the previous day.
The logic was revised to increase the day value for cases when the current time value is approaching midnight.

Files changed:
eagle_default/in/xml/include/msgxchg_sec_core.inc
eagle_default/in/xml/include/secmastersynthetic.inc
eagle_default/in/xml/xml-smf.xml
eagle_default/in/xml/xml-smf_generic.xml
eagle_default/in/xml/xml-smf_multileg_swap.xml

SDP-25320: Added the Ticker field to the Warehouse Position extract in V17

The ticker field was missing on the list of fields for the sub-query for the SECURITY_MASTER table. This sub-query is used to reduce the number of fields for the
Warehouse Position extract on V17. The ticker field was added to the sub-query.

Files changed:
dynamic/msgcenter/eagle_ml-2-0_cm/out/tagvalue/extract_asset_properties_fields.inc

SDP-22810: Schedule Execution failed with error message "Schedule timed out in 0 seconds"

The schedule execution was failing with the error message “Schedule timed out in 0 seconds” when the :PendingId: task parameter is not empty.
The parameter is set after the Message Center Service is restarted and the event execution timeout was not defined and had a ‘0’ value. 
The code was changed to have the event execution timeout defined in case the :PendingId: task parameter is not empty.

File changed:
eagle_ml-2-0_cm/exec_star_schedule.inc