EagleML Release Notes - August 2018

Owned by Ilie Tomac Jnr (Deactivated)
Last updated: Aug 29, 2018
7 min read

ESP-340 EagleML - Generic Delete Processes does not allow for Delete of Master Fund

The generic delete process was not able to delete a master fund.

Functionality for executing the panel to delete a master fund was added to the generic delete workflow and is conditionally controlled.

Files changed:

t_eagle_ml-2-0_default_in_xml_delete_entity_generic.xml 


ESP-347 Regenerate binds based on environment core SP - securityOverrides
The inbound securityOverrides interface received new logic that allows the rule to auto generate an include with an XMLDBAPI request. Meta information for this change was created and NG support was also added.

Files changed:
xml-ref_security_overrides.xml
eagleml_ref_securityoverrides.xml


ESP-375: EJM: Enhance EAS Alert functionality to support time zones, business calendars and date offsets / For Oracle regions
ESP-1764: EJM: Enhance EAS Alert functionality to support time zones, business calendars and date offsets/ For MSSQL regions

EAS alert functionality was improved to be able to use time zones, business calendars and date offsets on both Oracle and MSSQL platforms.

The timezonename and businesscalendar parameters may be set in the w_config_custom.inc file or the spf file. The corresponding config variables are W_EAS_TIME_ZONE_NAME and W_EAS_BUSINESS_CALENDAR. The spf file equivalents are timezonename and businesscalendar.

Files changed:

eagle_ml-2-0_cm/out/xml/read_task_info.inc
eagle_ml-2-0_cm/out/xml/check_business_date.inc


ESP-376: EagleML: Add FUND_MGMT_STYLE/FUND_GEO_FOCUS from the Secmaster_Detail_Ext table to the XSD and EagleML inbound interface

The SECMASTER_DETAIL_EXT.FUND_MGMT_STYLE and SECMASTER_DETAIL_EXT.FUND_GEO_FOCUS fields were added to the XSD schema, inbound and outbound EagleML rules. 

The fields have been added to the XSD (as <fundManagementStyle> and <fundGeoFocus>). These fields were also added to the inbound and outbound GenericSMF rules.

Files changed:

eagle_default/in/xml/xslt/smf_egl_gen_interface_template.xsl
eagle_default/in/xml/xslt/smf_egl_gen_interface_template_mssql.xsl
eagle_ml-2-0_cm/out/xml/t_smf_streaming.inc
eagle_ml-2-0_cm/out/xml/t_smf_streaming_eml.inc


ESP-425: EJM: Adjust possibility to load multiple files with same file mask in eagle_wrf_generic_load in parallel mode or sequentially by parameter

Enabled the ability to choose parallel vs sequential execution for files with the same mask in the generic load workflow.

These options are controlled by the RTR parameter ParallelExecution and the global w_config.inc variable W_USE_PARALLEL_LOAD both of which have possible values of Y or N.  By default execution of files with the same mask is performed sequentially.  A value of Y will load in parallel mode.  A value of N will load in sequential mode.

Files changed:

eagle_ml-2-0_cm/out/tagvalue/cmw_datafile_control.prf

eagle_ml-2-0_cm/out/tagvalue/data_file_task.xml

eagle_ml-2-0_cm/out/tagvalue/execute_wrkfl_event.xml

RTR example to load data in sequential mode:

<EagleML xmlns="http://www.eagleinvsys.com/2011/EagleML-2-0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="RunTaskRequest" eaglemlVersion="2-0" xsi:schemaLocation="http://www.eagleinvsys.com/2011/EagleML-2-0 eagleml-main-2-0.xsd" eaglemlType="RunTaskRequest">
<header>
<messageId>TST_TSR_ACK_RENAME</messageId>
<sentBy>http://www.eagleinvsys.com</sentBy>
<sendTo>http://www.eagleinvsys.com/eagle_ml-2-0_default_cm_control_message</sendTo>
<creationTimestamp/>
</header>
<taskIdentifier>
<correlationId>cm_wrf_200718_8</correlationId>
<businessTaskId>cm_wrf_200718_8</businessTaskId>
</taskIdentifier>
<taskTypeEnum>LOAD</taskTypeEnum>
<taskParameters>
<taskParameter>
<name>StreamName</name>
<dataType>S</dataType>
<value>eagle_ml-2-0_default_cm_execwkfl</value>
</taskParameter>
<taskParameter>
<name>ActionType</name>
<dataType>S</dataType>
<value>EXECUTE</value>
</taskParameter>
<taskParameter>
<name>WorkflowName</name>
<dataType>S</dataType>
<value>eagle_wrf_generic_load</value>
</taskParameter>
<taskParameter>
<name>WorkflowData</name>
<dataType>S</dataType>
<value>EAGLE_ML_ESP425.zip</value>
</taskParameter>
<taskParameter>
<name>ParallelExecution</name>
<dataType>S</dataType>
<value>N</value>
</taskParameter>
</taskParameters>
</EagleML>


ESP-434 ESP-2058 EagleML - ADD/Change Sector Setup/Change with Accounting Validation Flag = Y

Support for loading MASTER/SECTOR entities was added. The mutualEntityType field now supports both the MASTER and SECTOR values.

Examples of values in the inbound file:

<mutualEntityType>SECTOR</mutualEntityType>

<mutualEntityType>MASTER</mutualEntityType>

Files changed:

csv-entity_columns.inc

entity_aliases.inc

entity_panels.inc

extract_entity.inc

t_entity_streaming.inc

t_entity_streaming_eml.inc


ESP-990 WH pending trade fails in 17 LX

An error for the ‘wrong number or types of arguments in call to INS_PENDING_TRADE’ was occurring when attempting to load Warehouse Pending Trade records in any environment on the 2017 Linux version. This was occurring because the INS_PENDING_TRADE procedure was changed to include some new fields.

The bind that was being generated by the XSLT included predefined stored procedure parameters.   The change now auto generates the bind which would retrieve the active stored procedure parameters dynamically.

The new fields in the stored procedure were also added to the WarehousePendingTrade XSD type, included in the inbound and outbound interfaces for the Warehouse Pending Trades object.

File added:
/dynamic/metadata/bind/eagleml_wrhs_warehousependingtrade.xml

Files changed:
/dynamic/metadata/bind/eagleml_warehouse.xml
/dynamic/msgcenter/eagle_default/in/xml/xslt/wrh_egl_gen_interface3.xsl
/dynamic/msgcenter/eagle_ml-2-0_cm/out/xml/t_warehouse_pending_trades_streaming_eml.inc
/dynamic/msgcenter/eagle_ml-2-0_cm/setup_bind.inc


ESP-1173 RealTime processing: need to develop method to display info in EJM Tasks

The eagle_ml-2-0_default_cm_realtime_ejm_stat stream was developed to generate records in the MSGCENTER_DBO.MSG_MESSAGE_STAT table and link them to records in the MSGCENTER_DBO.MSG_DETAIL table.

This functionality will allow information to be displayed correctly in the EJM task tab and also check whether a received RTR was processed in Real-Time Processing.

In case of an error from the python script the trigger will be sent directly to the eagle_ml-2-0_default_cm_task_reporter stream which will then generate a TSR with the appropriate status for the error.

Files changed:

eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/tagvalue/realtime_ejm_stat.rsf
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/tagvalue/realtime_ejm_stat.xml
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/in/xml/realtime_rtr.xml
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/xml/task_reporter.xml
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/xml/taskacknowledgement.xml 
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/w_config.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/xml/include/smf_aliases.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_default/in/xml/xml-create_tsr.xml


ESP-1219: EagleML: Add logic to not override the partitionCount derived from transformation tasks in eagle_wrf_generic_load_profile_executor 

There was logic in the uncompress_transform which would override the partitionCount parameter set in the transformation tasks.
The send_to_wrf task is able to handle multiple partition outputs and therefore the partition parameter in the transformation task should not be overwritten in the uncompress_transfromtask. This allows for maximum transform + load performance.

The '!partitioncount;1' parameter was removed to enable the send_to_wrf task to maintain the partitionCount set in the transformation task.

Files changed: 

eagle\estar\tpe\dynamic\msgcenter\eagle_ml-2-0_cm\unpacker\custom_routing.inc


ESP-1225 Fetching the Current, Prior and Next Business day using EagleML RTR

The new ProcessCenter EagleML object was created. The object is extract only and supports the following filters:

  • ProcessCenterName(required)
    • CalendarBusinessName(required)
    • Common filter(maxrow)

Data mappings can be found on the confluence page for this new object in the extracts section.

Files changed:
extract_process_center.inc
extract_process_center.xml
t_ref_process_center_streaming.xml
t_ref_process_center_streaming.inc


ESP-1429 Additional fields for analytics XSD
These new elements have been added to the GenericIssueAnalytic XSD object they were added in the <analyticExtModel> group:
convexityOasBid
convexityOasAsk
convexityOasMid
modDurationBid
modDurationAsk
modDurationMid
macaulayDurationBid
macaulayDurationAsk
macaulayDurationMid
sharpeRatio
underlyingContValue
volatility30d
These fields were also added to the <genericIssueAnalyticModel> node in the Generic SMF object in the XSD.

Files changed: 
\dynamic\eagleml\schema\eagleml-ref-shared-2-0.xsd
\dynamic\eagleml\schema\eagleml-shared-2-0.xsd


ESP-1560 InsuranceValuationLot extract fields missing from EML xsd & json schema

InsuranceValuationLot extract were not passing validation against the XSD and JSON schema.

To make these extract valid against the XSD and the JSON schema the following changes were made:

  1. The following fields in the InsuranceValuationLot (XSD group InsuranceValuationLot.model) have been renamed:
    totalLocalMarketValue to marketValueBase
    naicMarketCode to scheduleDCode
    baseEarnedIncome to incomeEarned
    totalAveragePrice to valPrice
  2. The following fields in the streaming t rule for InsuranceValuationLot have been renamed:
    marketValueB to marketValueBase
    naicDesignation to designation
    schdDCode to scheduleDCode
  3. The sourceNumber field has been deleted from the streaming t rule for InsuranceValuationLot.

The following files have been changed:
\dynamic\msgcenter\eagle_ml-2-0_cm\out\xml\t_warehouse_istar_lot_streaming.xml
\dynamic\msgcenter\eagle_ml-2-0_cm\out\xml\t_warehouse_istar_lot_streaming.inc
\dynamic\eagleml\schema\eagleml-insurance-2-0.xsd


ESP-1575 - Add parentCompositeName to openlot, position and wh trade extracts

The new paranetComposoiteName field was added to the Common Entity Model for these Warehouse Extracts:

- Warehouse Position
- Warehouse OpenLot
- Warehouse Trade

The parantCompositeId field already had a mapping in those objects, but the parentCompositeName was also required so the field is now being resolved as follows:
(select RTRIM(e.ENTITY_NAME) 
from RULESDBO.ENTITY e 
where e.ENTITY_ID = eext.USER_CHAR1 
and e.ENTITY_TYPE = 'COMP') EM_EX_PARENT_COMP_NAME

Files changed: 
- extract_entity_fields.inc
- t_entity_common_ids_fields_streaming.inc
- t_entity_common_ids_fields.inc


ESP-1585 Improve the delete functionality for the Factor interface

Added support within the Factor interface to be able to delete multiple records at a time.

New files:
parse_common_objects.inc 
resolve_source_instance.inc 
ref_factor_delete.inc

Files changed:
xml-ref_objects.rsf 
xml-ref_delete.xml
referenceobject.inc 
parse_issuer_xref.inc 
parse_security_xrefs.inc


ESP-1610 Add wash sale to the generic extract and load workflows 
The Accounting WashSale object was not supported in the generic extract and load workflows.

The inbound and outbound rules were added to eagle_wrf_generic_load and eagle_wrf_generic_extract respectively.

The inbound interface can now be loaded through the generic load workflow if the file includes the _WASHSALE_ file mask.

The WASHSALEEXTRACT feed type was added as one of the accounting extract in the generic extract workflow.

The following files have been changed:
eagle_ml-2-0_workflows/eagle_wrf_generic_extract.wrf 
eagle_ml-2-0_workflows/eagle_wrf_generic_extract_accounting_objects.wrf 
eagle_ml-2-0_workflows/eagle_wrf_generic_extract_security_related_objects.wrf 
eagle_ml-2-0_workflows/eagle_wrf_generic_load.wrf 
eagle_ml-2-0_workflows/eagle_wrf_generic_load_accounting_objects.wrf 
eagle_ml-2-0_workflows/eagle_wrf_generic_load_security_related_objects.wrf


ESP-1678 XSD Adds - Unigestion - Bloomberg - Issuer, Issuer Analytics, Price, CPA

New EagleML objects and fields had to be added to the XSD.

  1. These fields have been added to the XSD schema:
    The element settlementType has been added to IssuePrice and ExchangePrice
    The element cpStockOption has been added to GenericCA 
    The element issuerShortName has been added to IssuerOrganization
  2. The EagleML object issuerAnalytic (complex type IssuerAnalytic) has been added to the schema with the following elements:
    releaseStatus
    releaseDate
    commonShares
    cashFlowFromOper
    userChar1 - userChar5
    userDate1 - userDate5
    userFloat1 - userFloat27

    Files changed:
    \dynamic\eagleml\schema\eagleml-issuer-2-0.xsd
    \dynamic\eagleml\schema\eagleml-enum-2-0.xsd


ESP-1769: Add bind file for raw service

 Support was added for the installation of the bind_raw_data.xml bind file for raw service.

Files changed:

eagle_ml-2-0_cm/setup_bind.inc


ESP-1775: Eagle - Accounting Trade Conversion Message failing

A new rule for loading CONVERSION trades was created. The booking of these trades is executed by the following panels: eagle/star/trade/pan-conversion.htm, eagle/star/trade/pan-irswapopen.htm, eagle/star/trade/pan-trswapopen.htm, eagle/star/trade/pan-crswapopen.htm. These panels require the rule to have mappings for several tags that weren’t in the Accounting Trade XSD.

The new rule makes it possible to load CONVERSION trades through the Accounting Trade object. The following fields were added to the Accounting Trade XSD:

<accountingDate> - tag 36 
<accountingBasis> - tag 21 
<factorDate> - tag 105 
<swapBeginAccrualDate> - tag 2857
<originatingOpenEffectiveDate> - tag 2860
<pendingTradeStatus> - tag 3720
<overridePostingCutOffDateFlag> - tag 9154
<overrideEditChecksFlag> - tag 2285
<calculateNetAmountsFlag> - tag 2286
<methodOfConvertingLTDAmortization> - tag 7001 
<chooseMethodOfConvertingInterest> - tag 7002
<dataEntryMethod> - tag 10485 was present in XSD

Files changed:
eagle_default\in\include\trade_aliases.inc
eagle_default\in\xml\xml-accttrade_conversion.xml
eagle_default\in\include\csv-conversion_columns.inc
eagle_default\in\xml\xml-acct_trades.rsf


ESP-1797 XSD Adds - Unigestion Issuer Analytics 2

These new fields have been added to the IssuerAnalytic XSD type:
sharesOutstanding
corpMeetingLastDate
corpMeetingNextDate
pretaxIncome
salesRevTurn
bsNumberOfTsySh

Files changed:
\dynamic\eagleml\schema\eagleml-issuer-2-0.xsd


ESP-1799 TAR.GZ compression doesn't work for multiple files in Windows Environments

The TAR.GZ compression option was not working properly in Windows environments when the eagle_wrf_generic_extract workflow was producing more than one file.

New logic was added to the generic extract workflow which compresses the extracted files into the TAR format and then performs a second compression into the GZ format. The extension for files of this type is filename.tar.gz.

Files changed:

compress_extract.inc


ESP-1802 WarehouseNavMulti load fails in BK13 region

The WarehouseNavMulti was failing due to the number of input parameters.  The error was related to a mismatch between the number of input parameters in the bind_wrhs_nav_multi.xml and the database procedure on the region.

Binds for the WarehouseNavMulti, WarehouseNav, WarehousePerformance, and WarehouseCashFlowProjection that previously were part of the EagleML package were removed from the EagleML package and the binds will now be automatically generated during the setup.

New files:
eagleml_wrhs_warehousenavmulti.xml
eagleml_wrhs_warehousenav.xml
eagleml_wrhs_performance.xml
eagleml_wrhs_cashflowproj.xml

The following files have been changed:
eagleml_warehouse.xml

Removed files:
bind_wrhs_nav_multi.xml
bind_wrhs_nav.xml
bind_wrhs_performance.xml
bind_wrhs_cashflowproj.xml


ESP-1828 Add support of extended details for warehouse cash activity feed type

The outbound cash activity interface did not have support for the extended details elements.

The cash activity feed was added to the list of interfaces that support the extended details model.

Files changed:
extract_warehouse_cash_activity.inc


ESP-1842: EagleML: Add meaningful error descriptions to the GL balance inbound interface

The GL balance inbound interface was returning the database field name upon failure in place of EagleML elements.  This created some confusion for users and created a debugging challenge. 

Editchecks were added for the fields that make up the unique insert key for the ledgerdbo.gl_detail_posting table. These allow the errors to be displayed in a more meaningful way.

<EDITCHECK EXPR="StringLength(:tag1288:) &gt; 0" TAG="1288" ERRORMESSAGEEXPR="'ledgerAccount is null.'" />
<EDITCHECK EXPR="StringLength(:tag1109:) &gt; 0" TAG="1109" ERRORMESSAGEEXPR="'effectiveDate is null.'" />
<EDITCHECK EXPR="StringLength(:tag86:) &gt; 0" TAG="86" ERRORMESSAGEEXPR="'baseCurrency is null.'" />
<EDITCHECK EXPR="StringLength(:tag34:) &gt; 0" TAG="34" ERRORMESSAGEEXPR="'shareClass is null.'" />
<EDITCHECK EXPR="StringLength(:tag182:) &gt; 0" TAG="182" ERRORMESSAGEEXPR="'contraClass is null.'" />
<EDITCHECK EXPR="StringLength(:tag85:) &gt; 0" TAG="85" ERRORMESSAGEEXPR="'localCurrency is null.'" />

Files changed: 

eagle_default/in/xml/xml-wh_gl_detail_posting.xml


ESP-1852 Real-Time processing: fix condition for resending RTR to core control_message stream

The default condition for resending the RTR to the eagle_ml-2-0_default_cm_control_message stream was changed.

The new default logic is to process RTRs containing a parameter with a value of eagle_wrf_generic_load using RealTime Processing and use the core EJM processes otherwise.

Files changed:

eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/in/xml/realtime_rtr.xml


ESP-1853 Real-Time processing: fix issue with processing multiple records in one file

A python script was changed to be able to correctly process multiple reference transactions in the same incoming file.

Files changed:

eagle\estar\tpe\dynamic\pyrules\eagleinvsys\mc2py\bundle_utils.py


ESP:1871 in dbkeysmap.inc strings are over 2k charachters

The var_dblogicalkeys_config1 and var_dblogicalkeys_config2 which are in the dbkeysmap include were both over 2000 characters. This causes issues with extracting objects with extensions.

Literals were added to each object type row.

The w_config.inc file logic for the inclusion of extensions in extracts was changed. The default logic is now to extract the extensions with the core data. This is controlled by setting the EscapeExpressionExtract variable to “N”.

Files changed: eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/dbkeysmap.inc


ESP-1997 - EAGLEML - CASHJOURNALEXTRACT not returning data from a secondary partition

The Accounting Cash Extract was not returning secondary partition data in the case that the entity filter was applied.

The outbound interface logic was changed to return data from all partitions when using the entity filter with the Accounting Cash Extract.

Files changed:
- extract_acct_cash_journal.inc


ESP-2003 Removed Case Sensitivity in Logic fo Generating Stored Procedures

When a stored procedure was being generated based on a core stored procedure it would get created incorrectly due to the generator being case sensitive.

The logic was changed so that the case sensitivity would be handled. Additionally regular expressions were added for right parsing.

Files changed:

estar\tpe\dynamic\msgcenter\ eagle_ml-2-0_cm\ generate_sp_on_fly.inc