EagleML Release Notes - May 2019

Enhancements:

SDP-18751: Added history mode for the Entity Regulatory load

The Entity Regulatory object type did not support the History mode during the load process and the Effective Date was missing in XSD.

Added support of the Entity Regulatory load into the history table
Added support of the Entity Regulatory deletion
Added the Effective Date mapping for the inbound and outbound rules.

Files changed:
metadata/bind/eagleml_ent_entityregulatory.xml
eagle_default/in/xml/xml-ent_entityregulatory.xml
eagle_ml-2-0_cm/generate_sp_on_fly.inc
eagle_default/in/xml/xml-ent_objects.rsf
eagle_default/in/xml/include/parse_ent_common_fields.inc
eagle_default/in/xml/include/xml-ent_delete.xml

Files added:
eagle_default/in/xml/include/ent_entityregulatory_delete.inc

SDP-20872: Added history mode for the Entity Regulatory extract

The Entity Regulatory extract was enhanced to allow data extraction from the history table.

The extract in history mode now supports the effective date filter for the ENTITYREGULATORYEXTRACT

Extract filter example:
<taskParameter>
<name>historyonlyflag</name>
<dataType>S</dataType>
<value>Y</value>
</taskParameter>
<taskParameter>
<name>effectivedate</name>
<dataType>S</dataType>
<value>2018-10-18</value>
</taskParameter>

Also, the effectiveDate node was added into the extract and the XSD schema.

Files changed:
eagle_ml-2-0_cm\out\tagvalue\extract_entity_regulatory.inc
eagle_ml-2-0_cm\out\xml\t_entity_regulatory_streaming.inc

SDP-20442: Allowed using the Batting Order with the USE_FIRST_AS_MAIN Resolution Option

The ability to use the USE_FIRST_AS_MAIN Asset Resolution option in 1st Match mode was added to the Security Resolution logic.

Example of Asset Resolution option element for USE_FIRST_AS_MAIN + 1st Match mode:
<assetResolutionOption>ALL,USE_FIRST_AS_MAIN</assetResolutionOption>

The following logic was implemented for the USE_FIRST_AS_MAIN Asset Resolution Option in 1st Match mode:

Step 1. Attempt to resolve the Security Alias by the first pair of Security Xreference identifiers specified in the Batting Order without additional options (Asset Currency, Xref Exchange, Maturity Date, Nra Tax Country).
Step 2. If the Security Alias was not resolved on step 1, the common security resolution logic is executed. The Security Alias is resolved by all pairs of the Security Xreference identifiers specified in the Batting Order and specified additional options in the Asset Resolution Option node are taken into account (Asset Currency, Xref Exchange, Maturity Date, Nra Tax Country).

Files changed:
eagle_default\in\xml\include\security_resolution_generic.inc

SDP-20412: Improved error handling in the TSR when ConsolidatedTSR=Y

When the eagle_wrf_generic_extract workflow was executed with the CreateConsolidatedTSR=Y, errors from the eagle_ml-2-0_default_cm_file_distribution stream were ignored and were missing in the final TSR.

The rule was changed to collect the status for the eagle_ml-2-0_default_cm_file_distribution stream and report errors in the Consolidated TSR.

Files changed:
task_reporter.xml
check_stream_status.inc

SDP-20265: Included SECURITY_MASTER_DETAIL.ALT_INVESTMENT_TYPE field in extract when IncludeExtendedDetails is Enabled

The field SECURITY_MASTER_DETAIL.ALT_INVESTMENT_TYPE (alias SMD_ALT_INVESTMENT_TYPE) was added to the Warehouse Trade Extract - WRHSTRADEEXTRACT, Warehouse Position Extract - WRHSPOSITIONEXTRACT and Warehouse Open Lot Extract - WRHSOPENLOTEXTRACT.

This field is part of the tagvalue extract of eagle_ml-2-0_default_out_q stream only and is not a part of the EagleML Extract.

Files changed:
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/tagvalue/extract_asset_properties_fields.inc

SDP-19587: Enhanced the Warehouse Open Lot to include Book_Cost_L (tag 17039) and Book_Cost_B (tag 17038) elements

New elements bookCostBase (tag 17038) and bookCostLocal (tag 17039) were added in both the Inbound and Outbound interfaces of the Warehouse Open Lot - WRHSOPENLOTEXTRACT object type. The elements bookPriceLocal and bookPriceBase were also added to the outbound interface.

Files changed:
wrh_egl_gen_interface3.xsl
t_warehouse_openlot_streaming.inc
t_warehouse_openlot_xmldbapi_streaming.inc
t_warehouse_openlot_xmldbapi.xml

SDP-18817: TIBCOMode supported for streaming extracts (T-transformation)

The parameter TIBCOMode has an impact on the mapping of attributes for data extracts.

The EagleML and transactionMessage nodes did not contain important attributes in TIBCO mode when streaming was enabled.

Support was added for all streaming extracts except for the list below:

1. MDCLENEAGEGRAPHEXTRACT
2. WRHSSPDDELTALOGEXTRACT
3. UNAPPROVEDVARMARGINEXTRACT
4. WORKINGTRIALBALANCEEXTRACT
5. PREFERREDSHAREEXTRACT
6. WEIGHTEDAVERAGEEXTRACT
7. WASHSALEEXTRACT
8. METADATAEXTRACT

Files changed:

t_eagle_header_streaming.inc
t_rule_header_streaming.inc
t_rule_start_header_streaming.inc
t_rule_end_header_streaming.inc
t_eagle_header_streaming.inc
t_eagle_header_fields.inc
t_rule_end_obj_header_streaming.inc
t_rule_end_trx_header_streaming.inc
t_rule_start_obj_header_streaming.inc
t_rule_start_trx_header_streaming.inc
t_ref_dictionary_streaming.inc
t_smf_streaming_eml.inc
t_xref_streaming.inc
t_ref_dictionary_streaming.inc
t_warehouse_brokertradequote_streaming_eml.inc
t_warehouse_cash_activity_streaming_eml.inc
t_warehouse_cash_flow_proj_streaming_eml.inc
t_warehouse_closelot_streaming.inc
t_warehouse_custody_activity_streaming_eml.inc
t_warehouse_custody_balance_streaming_eml.inc
t_warehouse_disclosure_status_streaming.inc
t_warehouse_gl_detail_posting_streaming.inc
t_warehouse_istarlot_streaming.inc
t_warehouse_nav_month_streaming.inc
t_warehouse_nav_multi_streaming.inc
t_warehouse_nav_streaming_eml.inc
t_warehouse_openlot_streaming.inc
t_warehouse_pending_trades_streaming_eml.inc
t_warehouse_performance_streaming_eml.inc
t_warehouse_position_streaming.inc
t_warehouse_trade_streaming_eml.inc
t_warehouse_washsale_trade_streaming.inc
t_forward_points_streaming.xml
t_forwardpoints_streaming.inc
t_generic_issuer_streaming.inc
t_generic_issuer_streaming.xml
t_issue_fx_pc_rate_streaming.inc
t_issue_fx_pc_rate_streaming.xml
t_issueprice_streaming.inc
t_issueprice_streaming.xml
t_issuer_rel_streaming.xml
t_issuer_relationship_streaming.inc
t_issuer_role_streaming.inc
t_issuer_role_streaming.xml
t_user_entitlement_streaming.inc
t_user_group_streaming.inc
t_user_user_streaming.inc
t_codeval_streaming.inc
t_codeval_streaming.xml
t_rating_streaming.inc
t_ref_pace_calendar_streaming.inc
t_ref_pace_calendar_streaming.xml
t_rule_start_trx_header_streaming.inc
t_voluntary_ca_election_streaming.inc
t_voluntary_ca_election_streaming.xml
t_generic_corporate_actions_streaming.inc
t_generic_corporate_actions_streaming.xml
t_ref_dictionary_streaming.inc
t_rule_start_trx_header_streaming.inc
t_entity_accounting_basis_streaming.inc
t_entity_accounting_periods_streaming.inc
t_entity_streaming_eml.inc
t_acct_gl_activity_streaming.inc
t_acct_monthly_portfolio_valuation_streaming.inc
t_acct_open_rec_pay_multi_streaming.inc
t_acct_trial_balance_streaming.inc
t_acct_unsettled_transactions_streaming.inc
t_acct_fair_value_comparison_streaming.inc
t_acct_cons_earned_inc_streaming.inc
t_acct_cash_journal_streaming.inc

SDP-18312: Added support for the DICTIONARYEXTRACT in the Generic Extractor workflow

The Generic Extract workflow was enhanced to support the Dictionary object.

SDP-18242: Updated the index hint to improve throughput in message streams execution

The hint in the task reporter query for retrieving summary information about loaded files was updated to improve the throughput of the query execution.

Changed files:
task_reporter_generate_task_unit.inc

SDP-18124: Changed the eagle_ml-2-0_default_in_xml_entity message stream to memory based

Added support to load the Entity objects as single messages within in-memory only processing.

The following objects are supported:

GenericEntity, EntityManagerRelationship, EntityBankRelationship, EntityAccountingBasis, EntityToComposite, EntityComposite, EntityManager, EntityToList, EntityXref, EntityList, AccountingEntity, EntityAccountingPeriods, EntityAccountingPeriods CLOSE, BenchmarkAssignment, EntityPeerGroupRanking, EntityStatistic, EntityRegulatory, EntityCollateralSumm, EntitySecurityLending, CashAccount, CashAccountRelationship, EntityAnalytic, CustomBenchmark, EntityRating

Files changed:
realtime_rtr.xml
xml-ent_entitymanager.xml
xml-ent_entityxref.xml
xml-ent_entityaccounting.xml
xml-ent_entityaccountingperiods_close.xml
xml-ent_entityaccountingperiods.xml
xml-ent_benchmarkassignment.xml
xml-ent_entity_peer_group_ranking.xml
xml-ent_entitystatistics.xml
xml-ent_entity_collateral_summ.xml
xml-ent_entityseclending.xml
xml-ent_cash_account.xml
xml-ent_cash_account_relationship.xml
xml-ent_entity_analytics.xml
xml-ent_custombenchmark.xml
xml-ent_entityregulatory.xml
xml-ent_entity_rating.xml

SDP-18120: Added the source description field to the RATINGSEXTRACT Interface

Added the source description element to the Ratings outbound Interface. It will now appear under the ratingDataModel as EagleML/ReferenceTransaction/rating/ratingDataModel/sourceDescription.

The mapping for the ratingSrc node in the Rating Extract was also corrected to return the source name instead of source instance.

Files changed:

extract_rating.inc
t_rating.xml
t_rating_streaming.inc
t_rating_streaming.xml

SDP-14296: Enhanced the Warehouse Position to include the Book_Cost_L (tag 17039) and Book_Cost_B (tag 17038) elements

New elements bookCostBase (tag 17038) and bookCostLocal (tag 17039) were added in both the inbound and outbound interfaces of the Warehouse Position - WRHSPOSITIONEXTRACT object type. The elements bookPriceLocal and bookPriceBase were also added to the outbound interface.

Files changed:
wrh_egl_gen_interface3.xsl
t_warehouse_position_streaming.inc
t_warehouse_position_xmldbapi_streaming.inc
t_warehouse_position_xmldbapi.xml

SDP-10041: Enhanced logic for the GenericSMF Forward extract to default the W_FORWARD_SINGLE_LEG_MODE flag to Y when the spokecheckflag is set

The Generic SMF extract process was modified to add the singleLegLoadFlag in the extracted SMF if the RTR for load has the spokecheckflag set.

The RTR parameter

<taskParameter>
<name>SetSingleLegLoadFlagValue</name>
<dataType>S</dataType>
<value>Y</value>
</taskParameter>

Files changed:
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/tagvalue/extract_smf.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/xml/t_smf_streaming.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/xml/t_smf_streaming_eml.inc

SDP-15208: Added support for SecurityNotes, Issuer Relationship, and Schedule Module Extracts

Added extract support for SecurityNotes for Oracle and MSSQL.

Supported filters: common, security, date. Added SecurityNotes extract in eagle_wrf_generic_extract workflow.

Added new fields to load: noteCharKey, shareProperty, privateNote.

Fils changed:
eagle_default\in\xml\xml-ref_secnotes.xml
eagle_default\in\xml\xml-ref_secnotes_mi.xml
eagle_default\in\xml\t_eagle_ml-2-0_default_in_xml_delete_security_notes.xml
eagle_default\in\xml\xslt\eagleml_smf_notes.xsl
eagle_default\in\xml\xslt\eagleml_smf_notes_batch.xsl
eagle_default\in\xml\rt_eagle_ml-2-0_default_in_xml_delete_objects.rsf
eagle_ml-2-0_cm\out\tagvalue\extract_ref_security_notes.xml
eagle_ml-2-0_cm\out\tagvalue\extract_ref_security_notes.inc
eagle_ml-2-0_cm\out\tagvalue\extracts.rsf
eagle_ml-2-0_cm\out\xml\t_ref_security_notes_streaming.xml
eagle_ml-2-0_cm\out\xml\t_ref_security_notes_streaming.inc
eagle_ml-2-0_cm\out\xml\t_eagleml-2-0_streaming.rsf

Production Bugs

SDP-22084: Rule changes to address issues with Extracts related to the NUGGETIZER when the RecordCountPerNugget is defined

An issue was found when the W_RECORD_COUNT_PER_NUGGET was set to Y, then the resulting extract had incorrect data with one record containing data from another record. The nuggetizer logic was improved to fix this issue as this was related to Message Center initialization after cache clean up.

SDP-22015: Fixed schedule wrapper workflow handling for WAITEVENTS DB mode

Schedule based workflows were enhanced to receive data if they were launched in the schedule wrapper workflow with the parameter W_WAITEVENTS_DATABASE_MODE set to Y.

Files changed:
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/cancel_global_event_wait.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/get_global_event_state.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/set_global_event.inc

SDP-21840: Resolved regression issue introduced by the ExchangePrice rule redesign

The source resolution logic has been updated to resolve the source instance based on the exact source name instead of a wild card search. This will prevent incorrect instance resolution when there are similar source names like 'TEST' and 'TEST RESOLUTION.'

Files changed:
/eagle_default/in/xml/include/xml-ref_security_pricing_common.inc
/eagle_default/in/xml/include/resolve_source_instance.inc

SDP-21828: Changed :tag55: options on the xml-acct_cash_xml.xml rule file to add CASHDIV for Conversion Dividends

The Accounting Asset Cash object was modified to support the cash type of CASHDIV for the assetCashType parameter. This allows a user to pass a conversion dividend via EagleML. By using the assetCashType of CASHDIV, the transaction will map as a dividend receivable to the position_detail table. In addition, the transaction will have the ability to be settled via the multiple settlements panel.

Files changed:
/eagle_default/in/xml/xml-acct_cash_asset.xml

SDP-20292: Addressed Fresh installation issue when the W_WAITEVENTS_DATABASE_MODE=Y is set in w_config_custom.inc

The fresh installation of EagleML components failed when the W_WAITEVENTS_DATABASE_MODE=Y was set in the w_config_custom.inc and the stored procedure for the W_WAITEVENTS_DATABASE_MODE did not exist on the region.
A check was added to prevent the W_WAITEVENTS_DATABASE_MODE stored procedure call when the stored procedure does not exist on the region.

Files changed:
estar\tpe\dynamic\msgcenter\eagle_ml-2-0_cm\in\xml\w_setup_eagle_version.xml
estar\tpe\dynamic\msgcenter\eagle_ml-2-0_cm\ndfa\ndfa_exec.xml
estar\tpe\dynamic\msgcenter\eagle_ml-2-0_cm\out\tagvalue\execute_wrkfl_event.xml
estar\tpe\dynamic\msgcenter\eagle_ml-2-0_cm\add_info_files_workflow.inc
estar\tpe\dynamic\msgcenter\eagle_ml-2-0_cm\generate_sp_on_fly.inc
estar\tpe\dynamic\msgcenter\eagle_ml-2-0_cm\get_upd_orch_wait_state.inc
estar\tpe\dynamic\msgcenter\eagle_ml-2-0_cm\waitevent_sp_generate.inc (new)

SDP-20184: Replaced the direct SP call for the REALIZEDGAINLOSSEXTRACT with a panel call

The FeedType RealizedGainLossExtract failed for some entities with error: ORA-01406 fetched column value was truncated.

Added logic to support RTR parameter AccountingValidationFlag to run the extract via panel.

<taskParameter>
<name>AccountingValidationFlag</name>
<dataType>S</dataType>
<value>Y</value>
</taskParameter>

Added support for for the displayUDF RTR parameter. The following fields are retrieved when displayUDF is set in the RTR:

userField17
userField18
smSecurityType
securityType2
granularityCategory
userGroup19
naicMarketCode
groupCode

Files changed:

eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/xml/t_acct_realized_gain_loss_streaming.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/tagvalue/extract_acct_realized_gain_loss.inc

SDP-19537: Resolved Corporate Actions Sink amount issue in the SMF Schedule object type

The load of the Sink Schedule data failed as the value of the total sink amount exceeded 1.0. This issue was caused as the sink amount was rounded to 10 digits.
The corresponding DB field allows to store numeric values with 12 digits (DB type - NUMBER(28,12) ). Rounding was increased to 12 digit.

SDP-19087: Updated the Date Format in Trade Extract to DateTime instead of Date

The format for the checkedDate, instructedDate, authorizedDate and preparedDate fields was changed from date to datetime in the WarehouseTrade Extract.

Files changed:
/eagle_ml-2-0_cm/out/tagvalue/extract_warehouse_trade.inc

SDP-17822: Added the source in the /eagle_default/in/xml/xml-wh_brokertradequote_v2.xml and the in /eagle_default/in/xml/xslt/wrh_egl_gen_interface3.xsl

The deletion query for the warehouseBrokerTradeQuote interface was updated to include the sourceName field in the query. The mapping for source was added in the XSLT transformation and the inbound rule.

Files changed:
eagle_default/in/xml/xml-wh_brokertradequote_v2.xml
eagle_default/in/xml/xslt/wrh_egl_gen_interface3.xsl