EagleML Release Notes - June 2019

Enhancements:

SDP-23032: Updated index hint to address throughput in execution of message streams

A hint in the task reporter query was updated which gets the summary information about loaded files. It should improve the throughput time of the query execution.

Changed files:
task_reporter_generate_task_unit.inc

SDP-23005: The FTP/SFTP file transfer was timing out via the Message Center curl command if files are larger than 150 MB

For larger files greater than 150MB, a timeout was occurring.  The issue was resolved by adding an additional call to the curl command with a timeout set to 1800 seconds. The new, optional parameter is defined as W_FTP_CURL_TIMEOUT_30MIN and should be set with a value of Y in the w_config_custom.inc.

File changed:
reporter_dist_methods.inc

SDP-22742: Created new Inbound interface for EntityLiquidity

• Created a new inbound interface to load Entity Liquidity object records to the database
• Created functionality to delete the Entity Liquidity object records from the database via Inbound Interface
• The Inbound interface for EntityLiquidity was added to the eagle_wrf_generic_load workflow

Changed files:
eagleml_entity.xml
ent_entityliquidity_delete.inc
generate_sp_on_fly.inc
parse_ent_common_fields.inc
xml-ent_delete.xml
xml-ent_objects.rsf

Added files:
eagleml_ent_entityliquidity.xml
ent_entityregulatory_delete.inc
xml-ent_entityliquidity.xml

SDP-22672: GENISSUEANALYTICEXTRACT was updated to support streaming

The rules for the GENISSUEANALYTICEXTRACT were updated to support streaming.

Files changed:
t_generic_issue_analytic_streaming.xml
t_eagleml-2-0_streaming.rsf
extract_generic_issue_analytic.inc
w_config.inc

SDP-22107: TIBCOMode supported for streaming extracts (T-transformation)

The parameter TIBCOMode has an impact on the mapping of attributes for extract data.
The EagleML and transactionMessage nodes did not contain important attributes for TIBCO mode when using streaming mode.
Support was provided to the group of extracts below:

• WORKINGTRIALBALANCEEXTRACT
• PREFERREDSHAREEXTRACT
• UNAPPROVEDVARMARGINEXTRACT
• WEIGHTEDAVERAGEEXTRACT
• MDCLENEAGEGRAPHEXTRACT
• WASHSALEEXTRACT

Extra grouping in the out_t_streaming includes was deleted (includes were optimized) for the listed extracts:

• WRHSBROKTRDQUOTEEXTRACT
• WRHSCASHACTIVITYEXTRACT
• WRHSCASHFLOWPROJEXTRACT
• WRHSCLOSELOTEXTRACT
• WRHSCSTDACTIVITYEXTRACT
• WRHSCSTDBALANCEEXTRACT
• WRHSNAVEXTRACT
• WRHSPENDINGTRADESEXTRACT
• WRHSPERFORMANCEEXTRACT
• WRHSPOSITIONEXTRACT
• WRHSTRADEEXTRACT
• WRHSNAVMONTHENDEXTRACT

Files changed:

t_eml_header_streaming.inc
t_rule_start_obj_header_streaming.inc
t_rule_start_trx_header_streaming.inc
t_acct_preferred_share_streaming.inc
t_acct_unapproved_var_margin_streaming.inc
t_acct_weighted_average_streaming.inc
t_ref_mdc_leneage_graph_streaming.inc
t_ref_mdc_leneage_graph_streaming.xml
t_warehouse_brokertradequote_streaming.inc
t_warehouse_brokertradequote_streaming.xml
t_warehouse_brokertradequote_streaming_eml.inc
t_warehouse_cash_activity_streaming.inc
t_warehouse_cash_activity_streaming.xml
t_warehouse_cash_activity_streaming_eml.inc
t_warehouse_cash_flow_proj_streaming.inc
t_warehouse_cash_flow_proj_streaming.xml
t_warehouse_cash_flow_proj_streaming_eml.inc
t_warehouse_close_lot_streaming_eml.inc
t_warehouse_closelot_streaming.inc
t_warehouse_custody_activity_streaming.inc
t_warehouse_custody_activity_streaming.xml
t_warehouse_custody_activity_streaming_eml.inc
t_warehouse_custody_balance_streaming.inc
t_warehouse_custody_balance_streaming.xml
t_warehouse_custody_balance_streaming_eml.inc
t_warehouse_nav_streaming.inc
t_warehouse_nav_streaming.xml
t_warehouse_nav_streaming_eml.inc
t_warehouse_pending_trades_streaming.inc
t_warehouse_pending_trades_streaming.xml
t_warehouse_pending_trades_streaming_eml.inc
t_warehouse_performance_streaming.inc
t_warehouse_performance_streaming.xml
t_warehouse_performance_streaming_eml.inc
t_warehouse_position_streaming.inc
t_warehouse_trade_streaming.inc
t_warehouse_trade_streaming.xml
t_warehouse_trade_streaming_eml.inc
t_wash_sale_streaming_eml.inc
t_warehouse_nav_month_streaming.xml
t_acct_working_trial_balance_streaming.inc
t_acct_working_trial_balance_streaming.xml

SDP-21983: Added support for BatchMode for the Warehouse DeltaBased extracts

Logic was enhanced to allow BatchMode to be disabled for DeltaBased Warehouse Extracts via a RTR parameter:

<taskParameter>
<name>BatchMode</name>
<dataType>S</dataType>
<value>N</value>
</taskParameter>

Files changed:

eagle_ml-2-0_cm/extract_start.inc

SDP-21389: Swagger file was updated to support additional feed types

The definition of the following feed types have been added to the swagger-extracts.json file:

• ENTITYCOLLATERALSUMMEXTRACT
• ENTITYANALYTICEXTRACT
• ENTITYPEERGROUPRANKINGEXTRACT
• ENTITYRATINGEXTRACT
• ENTITYSECLENDINGEXTRACT
• ENTITYSTATISTICEXTRACT
• GENISSUEANALYTICEXTRACT
• PRICINGREPORTEXTRACT
• ISSUERRELEXTRACT
• ISSUEEXCHANGEPRICEEXTRACT
• ISSUERENTRELEXTRACT
• ISSUERESGEXTRACT
• ISSUERRATINGEXTRACT
• ISSUERREGULATORYEXTRACT
• CONTROLCENTEREVENTLOGEXTRACT
• EAGLEADMINISTRATIONSETTINGSEXTRACT

Support of the following Feeds have been added in Extract Services (MC2).

• ENTITYCOLLATERALSUMMEXTRACT
• ENTITYANALYTICEXTRACT
• ENTITYPEERGROUPRANKINGEXTRACT
• ENTITYRATINGEXTRACT
• ENTITYSECLENDINGEXTRACT
• ENTITYSTATISTICEXTRACT
• PRICINGREPORTEXTRACT
• ISSUERENTRELEXTRACT
• ISSUERESGEXTRACT
• ISSUERRATINGEXTRACT
• ISSUERREGULATORYEXTRACT
• CONTROLCENTEREVENTLOGEXTRACT
• EAGLEADMINISTRATIONSETTINGSEXTRACT

SDP-20827: Corrected the mapping for the AccountingTrade Python rule

The python CSV-to-EagleML transform for the AccountingTrade did not recognize security tags 1233 and 1234. Mapping for these tags was added in the python rule for the AccountingTrade, which is required for in-memory only processing.

Files changed:
t_eml_acct.py

SDP-20549: Allow the Profile Loader to work without any translation required

It is now possible to configure the profile generic loader to pass an incoming file without translation. To configure, the user should enter "no translation" in the field where the name of translator is expected.
The Incoming file will be passed to load by profile generic loader instance.

Files changed:
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/unpacker/custom_routing.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/unpacker/send_to_wrf.xml

SDP-20513: Updated the Warehouse Performance rules for loading and extracting the new Liquidity data model

The EagleML Warehouse Performance incoming transformation rule and corresponding extract were enhanced to process data for the PERFORMDBO.PSR_LIQUIDITY table.
The PSR_LIQUIDITY is a new table and is available only for certain Eagle versions. A corresponding core stored procedure changes outside of EagleML are required to support loading and updates to this table.

Files changed:

eagle_default/in/xml/xslt/wrhs_performance.xsl
eagle_default/in/xml/include/xml-dbdirect_wrhs_delete_statement.inc
eagle_ml-2-0_cm/out/tagvalue/extract_warehouse_performance.inc
eagle_ml-2-0_cm/out/xml/t_warehouse_liquidity_fields_streaming.inc
eagle_ml-2-0_cm/out/xml/t_warehouse_performance_streaming.inc

SDP-20512: Created Outbound interface for Entity Liquidity

The outbound interface for EntityLiquidity object was created.
Feed type: ENTITYLIQUIDITYEXTRACT.

Supported EagleML filters:

• Date Filter
• Sources Filter
• Entity Filter
• User Group Filter
• Common Filter

Changed files:
extracts.rsf
extract_service.rsf
t_eagleml-2-0_streaming.rsf
w_config.inc

Added files:
extract_entity_liquidity.xml
extract_entity_liquidity.inc
t_entity_liquidity_streaming.xml
t_entity_liquidity_streaming.inc
extract_service_entity_liquidity.xml

SDP-20149: Added XSD Enumeration for the User and the UserGroup in SCIM format

Added XSD Enumeration for the User and the UserGroup objects in SCIM format

The Extract rules were updated to provide support for Object Types ScimUser and ScimGroup.

File changed:
eagle_default/in/xml/xml_user_objects.rsf
eagle_ml-2-0_cm/out/xml/t_user_group_scim_streaming.inc
eagle_ml-2-0_cm/out/xml/t_user_user_scim_streaming.inc

SDP-19866: The information about incoming files is stored in the database by default

The following parameters have been set with a default value in the eagle_ml-2-0_cm/w_config.inc:
:W_WAITEVENTS_DATABASE_MODE: := 'Y';
:W_ENABLE_EXTENDED_FILE_STATISTICS: := 'Y';

These options enable logic for storing extended information about the incoming files in the database instead of a file on the app server.  

SDP-19850: Updated the Warehouse Position rules for loading and extracting the new Liquidity data model

The EagleML Warehouse Position incoming transformation rule and corresponding extract were enhanced to process data for the HOLDINGDBO.POSITION_DETAIL_LIQUIDITY table.
The POSITION_DETAIL_LIQUIDITY is a new table and available only for certain Eagle versions. A corresponding core stored procedure changes outside of EagleML are required to support loading and updates to this table.

Files changed:

eagle_default/in/xml/xslt/wrh_egl_gen_interface3.xsl
eagle_default/in/xml/xslt/wrhs_performance.xsl
eagle_default/in/xml/include/xml-dbdirect_wrhs_delete_statement.inc
eagle_ml-2-0_cm/out/tagvalue/extract_warehouse_position.inc
eagle_ml-2-0_cm/out/xml/t_warehouse_liquidity_fields_streaming.inc
eagle_ml-2-0_cm/out/xml/t_warehouse_position_streaming.inc

SDP-19625: Added support of JMSExtractDeliveryStream and CollectDistributionErrors RTR parameters in generic extract workflow

Added support for two new RTR parameters JMSExtractDeliveryStream and CollectDistributionErrors in the eagle_wrf_generic_extract workflow when the ExtractDeliveryMethod='JMS'.
The corresponding global parameter are also supported with the W_COLLECT_DISTRIBUTION_ERRORS global parameter.

RTR Sample Parameters:

<taskParameter>
<name>CollectDistributionErrors</name>
<dataType>S</dataType>
<value>JMS</value>
</taskParameter>
<taskParameter>
<name>JMSExtractDeliveryStream</name>
<dataType>S</dataType>
<value>JMS_stream_sample_extract</value>
</taskParameter>

w_config_custom Sample Parameters:

<COL TAG="W_COLLECT_DISTRIBUTION_ERRORS" EXPRESSION= "'JMS'"/>
<COL TAG="W_JMS_DEF_EXTRACT_STREAM" EXPRESSION="'JMS_EAGLE_extract'" USERDESCRIPTION="Default JMS EXTRACT Delivery Stream"/>

Files changed:
eagle_default_cm_file_distribution.xml
reporter_dist_methods.inc
eagle_wrf_generic_extract.wrf
eagle_wrf_generic_extract.ejmwf

SDP-18029: The PRICINGREPORTEXTRACT was updated to include additional elements

The following new elements were added to Pricing Report:

priceLevel
levelComments
reason
comments
marketDate
marketDaysStale

Added MC2 support for the PRICINGREPORTEXTRACT

Changed files:
extract_filter_sources.inc
extract_pricing_report.inc
t_pricing_report_streaming.inc

SDP-16994: The XSD version was added to the w_config_version.inc file

The extract always gets the XSD version from the main xsd file. To simplify debugging, made changes to store the XSD version as the global parameter g_EagleML20_XSDRevision in the w_config_version.inc. The parameter is set by the SetupEagleVersion process via the eagle_ml-2-0_default_cm_w_setup stream. Extracts now read this parameter once, instead of from the eagleml-main-2-0.xsd file each execution.

Files changed:
w_setup_eagle_version.xml
get_revision_num.inc

SDP-15946: Optimized execution of sql queries for most used EJM includes

The execution of sql requests was optimized. Bind variables were added when executing sql queries for the most common shared include files.

Files changed:

execute_task.inc
control_workflow_paused.inc
get_stream_info.inc
check_env_name.inc
calc_date_from_rule.inc
check_stream_status.inc
include\get_sp_params_list.inc

SDP-15247: Rating extract supports date and datetime format

The Rating Extract was enhanced to support both date and datetime format for submittedDate field.

Files changed:
/eagle_ml-2-0_cm/out/tagvalue/extract_rating.inc
/eagle_ml-2-0_cm/out/xml/t_rating_streaming.inc

SDP-14722: Error message was improved in case of failure of accounting trade

Previously the 'UNKNOW' error was returned in case no panel was selected for the accounting trade load. This is generally because the message is missing a valid event type, processing security type, etc.
The Error description was improved:
'No panel found to load. Please check values for tag55, tag3931 and tag4533.'

File changed:
accttrade_routing_check.inc
csv-buy_panels.inc
csv-sell_panels.inc
xml-accttrade_buy.xml
xml-accttrade_sell.xml

SDP-10431: Enhanced the Profile Loader to support processing of multiple archived files 

Support for the processing of archived data has been added to the profile loader workflow. In the past, it could only receive plain uncompressed files. It is now possible to configure the loader for the condition when data comes in from one or multiple archives.

The following changes in the profiles syntax have been added:

• If the file is part of the archive, the archive mask should be specified before the file mask as follows: [package mask]/[sourсe file mask]
• Only one file per profile should be a bundle trigger. To mark a file as a bundle trigger, prefix it with the tilde symbol: ~bundle_archive/source_file
• Archived and plain files may appear together in one profile.

Files changed:
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/ndfa/launch_load_by_profile.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/ndfa/ndfa_exec.xml
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/unpacker/custom_routing.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/unpacker/send_to_wrf.xml
eagle/estar/tpe/dynamic/pyrules/eagleinvsys/wrf_generator/_main_.py
Files added:
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/run_wrf_instance.inc

SDP-9466: Added the ability to extract a Static or Dynamic Entity List

Production Bugs

SDP-23322: Corrected logic for CreateConsolidatedTSR query

The CreateConsolidatedTSR query was working incorrectly when the varchar columns in the subselect contained non-alphanumeric values.   As a result, the ConsolidatedTSR was not created. This issue was identified and corrected.

Changed files:
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/xml/task_reporter.xml

SDP-22713: Fixed lastTradeDate mapping for Exchange Price Object type

The value for the lastTradeDate is loading correctly now via the RDC Price Event and panel.

Changed files:
xml-ref_security_pricing_common.inc

SDP-22593: Resolved issue with malformed CASHJOURNALEXTRACT file related to Records grouping

The records grouping was removed from transformation. Every extracted record is now enveloped under a separate <accountingTransaction> node. 

Changed file:
t_acct_cash_journal_streaming.inc

SDP-22083: Logic for the Distribution Rate Summary model was improved within the WRHSNAVMULTIEXTRACT

The total summary value in the Distribution Rate model was equal to sum of the rates of all processed records for a given Portfolio Account, Valuation Date and Share Class.
The subtotals were corrected to account for canceled records.  The logic was changed to reflect the correct total sum reversing cancels when they are present in the dataset.

Changed rules:
extract_nav_multi_me_details_tables.inc

SDP-20418: GenericSMF load replaces some xref data with Primary Asset or Alt Asset values

When the GenericSMF was loaded via panels (accountingValidationFlag=Y) and the Primary Asset Type, and/or Alt Asset Type had any values in the Xref ('ISIN', 'SEDOL', 'RIC', 'BBID', 'SICOVM', 'VALOREN', 'CMM', 'INTERNAL', 'CINS') some xrefs were replaced by the Primary Asset Type or Alt Asset Type values.

Added logic to reorder the xref before sending to the panel based on the panel replacement logic. Changes were made for the following SMF types: Equity ('EQCSCS', 'EQCSPF', 'EQCSPK'); Mutual Fund ('EQEQMF'); and Long Term Debt ('DBFLTP', 'DBIBFD', 'DBIBMU', 'DBIBPK', 'DBIVIV', 'DBMFDD', 'DBAMTL', 'DBIBTL').

SDP-20235: Entitlement filter for SP-Based Extracts was improved

Fixed issues related to the Entitlement filter with the attribute "all entities are included" in SP-Based Extracts.

The following changes were implemented:
1) If the Entity filter and the Entitlement filter (RTR parameter ‘usergroups’) are used at the same time, the Entitlement filter works as an additional restriction for Entity filter (default logic).
2) If the Entitlement filter is only used (RTR parameter ‘usergroups’), the extract performs according to the Business group assigned:

• When the Business group with a list of specific entities is used, it works as the source for allowed portfolios for next processing in extract (default logic).
• When the Business group with the attribute "all entities are included" is used, the extract will be cancelled with the error message: "User Group with access to all entities cannot be used without Entity filter".

This logic works only for following extracts:
CASHJOURNALEXTRACT Accounting Cash Journal
COMPLIANCE2A7PRICINGEXTRACT Accounting Compilance 2a7 Picing Report
CONSEARNEDINCOMEEXTRACT Accounting Cons Earned Income
EXPENSEACTIVITYEXTRACT Accounting Expense Activity
FAIRVALUECOMPARISONEXTRACT Accounting Fair Value Comparison
POSITIONASOFEXTRACT Accounting Position AsOf
REALIZEDGAINLOSSEXTRACT Accounting Realized Gain Loss
TRIALBALANCEEXTRACT Accounting Trial Balance
UNAPPROVEDVARMARGINEXTRACT Accounting Unapproved Var Margin
UNSETTLEDTRANSACTIONSEXTRACT Accounting Unsettled Transactions
WEIGHTEDAVERAGEEXTRACT Accounting Weighted Average
WORKINGTRIALBALANCEEXTRACT Accounting Working Trial Balance

Changed files:
extract_filter_entity.inc
extract_filter_entity_list_resolution.inc
extract_filter_entity_resolution_v1.inc
extract_filter_entity_resolution_v2.inc