Enhancements:
SDP-23032 Review index hint to address slowness in execution of message streams
Updated additional index hint to address slowness in execution of message streams
Hint in the task reporter query for getting summary information about loaded files was updated. It should improve time of the query execution.
Changed files:
task_reporter_generate_task_unit.inc
SDP-23005 EagleML - EJM The FTP Transmission of Files larger than 150 MB timeout when Message Center is sending them using curl
FTP/SFTP file transfer was timing out via Message Center curl command if files are larger than 150 MB
The issue was resolved by adding an additional call to curl command with timeout set to 1800 seconds. New optional parameter W_FTP_CURL_TIMEOUT_30MIN is available and should be set to ‘Y’ in w_config_custom.inc
File changed:
reporter_dist_methods.inc
SDP-22742 EagleML: Create Entity rules for loading new Liquidity data model
Created new inbound interface for EntityLiquidity
- Created new inbound interface to load Entity Liquidity object records to database.
- Created functionality to delete Entity Liquidity object records from database via Inbound Interface.
- Inbound interface for EntityLiquidity added to eagle_wrf_generic_load workflow.
Changed files:
eagleml_entity.xml
ent_entityliquidity_delete.inc
generate_sp_on_fly.inc
parse_ent_common_fields.inc
xml-ent_delete.xml
xml-ent_objects.rsf
Added files:
eagleml_ent_entityliquidity.xml
ent_entityregulatory_delete.inc
xml-ent_entityliquidity.xml
SDP-22672 GENISSUEANALYTICEXTRACT has to support streaming
GENISSUEANALYTICEXTRACT was updated to support streaming
Rules for GENISSUEANALYTICEXTRACT were updated to support streaming.
Files changed:
t_generic_issue_analytic_streaming.xml
t_eagleml-2-0_streaming.rsf
extract_generic_issue_analytic.inc
w_config.inc
SDP-22107 Allstate: TIBCOMode doesn't work correctly in case of extraction with streaming T transformation - part 6
TIBCOMode supported for streaming extracts (T-transformation)
The parameter TIBCOMode has an impact on the mapping of attributes for extract data.
EagleML and transactionMessage nodes do not contain important attributes for TIBCO mode when using streaming mode.
Support was provided to the last group of extracts below:
- WORKINGTRIALBALANCEEXTRACT
2. PREFERREDSHAREEXTRACT
3. UNAPPROVEDVARMARGINEXTRACT
4. WEIGHTEDAVERAGEEXTRACT
5. MDCLENEAGEGRAPHEXTRACT
6. WASHSALEEXTRACT
Extra grouping in out_t_streaming includes was deleted (includes were optimized) for the listed extracts:
- WRHSBROKTRDQUOTEEXTRACT
8. WRHSCASHACTIVITYEXTRACT
9. WRHSCASHFLOWPROJEXTRACT
10. WRHSCLOSELOTEXTRACT
11. WRHSCSTDACTIVITYEXTRACT
12. WRHSCSTDBALANCEEXTRACT
13. WRHSNAVEXTRACT
14. WRHSPENDINGTRADESEXTRACT
15. WRHSPERFORMANCEEXTRACT
16. WRHSPOSITIONEXTRACT
17. WRHSTRADEEXTRACT
18. WRHSNAVMONTHENDEXTRACT
Files changed:
1. t_eml_header_streaming.inc
2. t_rule_start_obj_header_streaming.inc
3. t_rule_start_trx_header_streaming.inc
4. t_acct_preferred_share_streaming.inc
5. t_acct_unapproved_var_margin_streaming.inc
6. t_acct_weighted_average_streaming.inc
7. t_ref_mdc_leneage_graph_streaming.inc
8. t_ref_mdc_leneage_graph_streaming.xml
9. t_warehouse_brokertradequote_streaming.inc
10. t_warehouse_brokertradequote_streaming.xml
11. t_warehouse_brokertradequote_streaming_eml.inc
12. t_warehouse_cash_activity_streaming.inc
13. t_warehouse_cash_activity_streaming.xml
14. t_warehouse_cash_activity_streaming_eml.inc
15. t_warehouse_cash_flow_proj_streaming.inc
16. t_warehouse_cash_flow_proj_streaming.xml
17. t_warehouse_cash_flow_proj_streaming_eml.inc
18. t_warehouse_close_lot_streaming_eml.inc
19. t_warehouse_closelot_streaming.inc
20. t_warehouse_custody_activity_streaming.inc
21. t_warehouse_custody_activity_streaming.xml
22. t_warehouse_custody_activity_streaming_eml.inc
23. t_warehouse_custody_balance_streaming.inc
24. t_warehouse_custody_balance_streaming.xml
25. t_warehouse_custody_balance_streaming_eml.inc
26. t_warehouse_nav_streaming.inc
27. t_warehouse_nav_streaming.xml
28. t_warehouse_nav_streaming_eml.inc
29. t_warehouse_pending_trades_streaming.inc
30. t_warehouse_pending_trades_streaming.xml
31. t_warehouse_pending_trades_streaming_eml.inc
32. t_warehouse_performance_streaming.inc
33. t_warehouse_performance_streaming.xml
34. t_warehouse_performance_streaming_eml.inc
35. t_warehouse_position_streaming.inc
36.t_warehouse_trade_streaming.inc
37.t_warehouse_trade_streaming.xml
38. t_warehouse_trade_streaming_eml.inc
39. t_wash_sale_streaming_eml.inc
40. t_warehouse_nav_month_streaming.xml
41. t_acct_working_trial_balance_streaming.inc
42. t_acct_working_trial_balance_streaming.xml
SDP-21983 'BatchMode' support for WH DeltaBased Extracts
Added support for 'BatchMode' for Warehouse DeltaBased extracts
Logic was enhanced to allow BatchMode to be disabled for DeltaBased Warehouse Extracts via RTR parameter:
<taskParameter>
<name>BatchMode</name>
<dataType>S</dataType>
<value>N</value>
</taskParameter>
Files changed:
- /eagle_ml-2-0_cm/extract_start.inc
SDP-21389 EagleML: Swagger add missing feedtypes for MC
Add missing description for extracts to Swagger
Definition of the following feed types have been added to swagger-extracts.json file:
- ENTITYCOLLATERALSUMMEXTRACT
- ENTITYANALYTICEXTRACT
- ENTITYPEERGROUPRANKINGEXTRACT
- ENTITYRATINGEXTRACT
- ENTITYSECLENDINGEXTRACT
- ENTITYSTATISTICEXTRACT
- GENISSUEANALYTICEXTRACT
- PRICINGREPORTEXTRACT
- ISSUERRELEXTRACT
- ISSUEEXCHANGEPRICEEXTRACT
- ISSUERENTRELEXTRACT
- ISSUERESGEXTRACT
- ISSUERRATINGEXTRACT
- ISSUERREGULATORYEXTRACT
- CONTROLCENTEREVENTLOGEXTRACT
- EAGLEADMINISTRATIONSETTINGSEXTRACT
Support of the following Feeds have been added in Extract Service (MC2).
ENTITYCOLLATERALSUMMEXTRACT
ENTITYANALYTICEXTRACT
ENTITYPEERGROUPRANKINGEXTRACT
ENTITYRATINGEXTRACT
ENTITYSECLENDINGEXTRACT
ENTITYSTATISTICEXTRACT
PRICINGREPORTEXTRACT
ISSUERENTRELEXTRACT
ISSUERESGEXTRACT
ISSUERRATINGEXTRACT
ISSUERREGULATORYEXTRACT
CONTROLCENTEREVENTLOGEXTRACT
EAGLEADMINISTRATIONSETTINGSEXTRACT
SDP-20827 Eagle ML - The in-memory trade processing does not use tags 1233 and 1234
Corrected mapping for AccountingTrade Python rule
The python CSV-to-EagleML transform for AccountingTrade did not recognize security tags 1233 and 1234. Added mapping for this tags in Python rule for AccountingTrade required for in-memory only processing.
Files changed:
t_eml_acct.py
SDP-20549 Allow the Profiled Loader to work without any translation required
Allow the Profiled Loader to work without any translation required
It is now possible to configure profiled generic load to pass incoming file without translation. In order to do that, user should enter "no translation" in the field where name of translator supposed to be.
Incoming file will be passed to load by profiled generic load instance.
Files changed:
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/unpacker/custom_routing.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/unpacker/send_to_wrf.xml
SDP-20513 EagleML: Update Performance rules for loading and extracting new Liquidity data model additions
Updated Performance rules for loading and extracting new Liquidity data model
EagleML Warehouse Performance incoming transformation rule and corresponding extract were enhanced to process data for the PERFORMDBO.PSR_LIQUIDITY table.
Note: PSR_LIQUIDITY is a new table and exists only on certain Eagle versions. Core stored procedure also should be upgraded to support data loading to this table.
Files changed:
- /eagle_default/in/xml/xslt/wrhs_performance.xsl
- /eagle_default/in/xml/include/xml-dbdirect_wrhs_delete_statement.inc
- /eagle_ml-2-0_cm/out/tagvalue/extract_warehouse_performance.inc
- /eagle_ml-2-0_cm/out/xml/t_warehouse_liquidity_fields_streaming.inc
- /eagle_ml-2-0_cm/out/xml/t_warehouse_performance_streaming.inc
SDP-20512 EagleML: Create Entity rules for extracting new Liquidity data model
Outbound interface for Entity Liquidity was created
The outbound interface for EntityLiquidity object was created.
Feed type: ENTITYLIQUIDITYEXTRACT.
Supported EagleML filters:
- Date Filter
- Sources Filter
- Entity Filter
- User Group Filter
- Common Filter
Changed files:
extracts.rsf
extract_service.rsf
t_eagleml-2-0_streaming.rsf
w_config.inc
Added files:
extract_entity_liquidity.xml
extract_entity_liquidity.inc
t_entity_liquidity_streaming.xml
t_entity_liquidity_streaming.inc
extract_service_entity_liquidity.xml
SDP-20149 Added XSD Enumeration for User and UserGroup in SCIM format
Added XSD Enumeration for User and UserGroup in SCIM format
Added XSD Enumeration for User and UserGroup objects in SCIM format
Extract rules were updated to provide support for Object Types – ScimUser and ScimGroup.
File changed:
• /eagle_default/in/xml/xml_user_objects.rsf
• /eagle_ml-2-0_cm/out/xml/t_user_group_scim_streaming.inc
• /eagle_ml-2-0_cm/out/xml/t_user_user_scim_streaming.inc
SDP-19866 EagleML: make W_WAITEVENTS_DATABASE_MODE=Y and W_ENABLE_EXTENDED_FILE_STATISTICS=Y is our default configuration
Made information about incoming file to be stored in data base by default
Following parameters have been set with default value:
:W_WAITEVENTS_DATABASE_MODE: := 'Y';
:W_ENABLE_EXTENDED_FILE_STATISTICS: := 'Y';
These options enable logic for storing extended information about incoming files in Data Base instead of file on the app server.
SDP-19850 EagleML: Update Position rules for loading and extracting new Liquidity data model additions
Updated Position rules for loading and extracting new Liquidity data model
EagleML Warehouse Position incoming transformation rule and corresponding extract were enhanced to process data for the HOLDINGDBO.POSITION_DETAIL_LIQUIDITY table.
Note: POSITION_DETAIL_LIQUIDITY is a new table and exists only on certain Eagle versions. Core stored procedure also should be upgraded to support data loading to this table.
Files changed:
/eagle_default/in/xml/xslt/wrh_egl_gen_interface3.xsl
/eagle_default/in/xml/xslt/wrhs_performance.xsl
/eagle_default/in/xml/include/xml-dbdirect_wrhs_delete_statement.inc
/eagle_ml-2-0_cm/out/tagvalue/extract_warehouse_position.inc
/eagle_ml-2-0_cm/out/xml/t_warehouse_liquidity_fields_streaming.inc
/eagle_ml-2-0_cm/out/xml/t_warehouse_position_streaming.inc
SDP-19625 JMSExtractDeliveryStream task paramter doesn't work for generic extract workflow
Added support of JMSExtractDeliveryStream and CollectDistributionErrors RTR parameters for generic extract workflow
Support of for two new RTR parameters JMSExtractDeliveryStream and CollectDistributionErrors was provided in eagle_wrf_generic_extract workflow if ExtractDeliveryMethod='JMS'.
The corresponding global parameter is also supported W_COLLECT_DISTRIBUTION_ERRORS
Files changed:
eagle_default_cm_file_distribution.xml
reporter_dist_methods.inc
eagle_wrf_generic_extract.wrf
eagle_wrf_generic_extract.ejmwf
SDP-18029 Update PRICINGREPORTEXTRACT to include additional elements
PRICINGREPORTEXTRACT was updated to include additional elements
Following new elements were added to Pricing Report:
priceLevel
levelComments
reason
comments
marketDate
marketDaysStale
Added MC2 support for PRICINGREPORTEXTRACT
Modified common source filter to support MC2
Changed files:
extract_filter_sources.inc
extract_pricing_report.inc
t_pricing_report_streaming.inc
SDP-16994 EJM optimization - Add logic to store XSD revision in config_version file
XSD version was added to w_config_version.inc file
Extract always gets XSD version from the main xsd file. Added changes allow to store XSD version as global parameter g_EagleML20_XSDRevision in “w_config_version.incâ€. The parameter is set by SetupEagleVersion process via eagle_ml-2-0_default_cm_w_setup stream. Currently extracts read this parameter g_EagleML20_XSDRevision instead of eagleml-main-2-0.xsd file each time.
Files changed:
w_setup_eagle_version.xml
get_revision_num.inc
SDP-15946 Query optimization - Dev - Part 1 (Add bind variables for frequently used queries)
Optimized execution of sql queries for most used EJM includes
The execution of sql requests was optimized.
The bind variables were added when executing sql queries for most used includes.
Files changed:
execute_task.inc
control_workflow_paused.inc
get_stream_info.inc
check_env_name.inc
calc_date_from_rule.inc
check_stream_status.inc
include\get_sp_params_list.inc
SDP-15247 Change an incoming EagleML data field to allow dateordatetime instead of just date
Rating exract supports date and datetime format
Rating Extract was enhanced to support both date and datetime format for field submittedDate.
Files changed:
/eagle_ml-2-0_cm/out/tagvalue/extract_rating.inc
/eagle_ml-2-0_cm/out/xml/t_rating_streaming.inc
SDP-14722 Client wants more granular details on the error on the EagleML trade loads
Error message was improved in case of failure of accounting trade
Previously 'UNKNOW' error was returend in case no panel was selected for accounting trade load.
Error description was imporved:
'No panel found to load. Please check values for tag55, tag3931 and tag4533.'
File changed:
accttrade_routing_check.inc
csv-buy_panels.inc
csv-sell_panels.inc
xml-accttrade_buy.xml
xml-accttrade_sell.xml
SDP-10431 add support for profiled generic load to process archived multiple files by different translators
Enhanced profiled generic load to support processing of multiple archived files
User is now able to use packages that contain one or several files as source files for profiled generic load and configure translation and outgoing feed types for each file from package separately.
To specify packaged files user should use prefix of package mask separated from processed file by slash symbol, as follows: [package mask]/[sourÑe file mask]
It is now also possible to specify incoming package or source file as bundle trigger. On bundle trigger arrival instance of profiled generic load will be launched. Only one bundle trigger file should be used per profile. To specify bundle trigger use ~ as prefix of file mask.
Files changed:
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/ndfa/launch_load_by_profile.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/ndfa/ndfa_exec.xml
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/unpacker/custom_routing.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/unpacker/send_to_wrf.xml
eagle/estar/tpe/dynamic/pyrules/eagleinvsys/wrf_generator/_main_.py
Files added:
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/run_wrf_instance.inc
Production Bugs
SDP-23322 task_reporter failes to assemble ConsolidatedTSR for workflow that waits for external data if waitevents DB mode or extended statistics is enabled
Correced logic for CreateConsolidatedTSR query
CreateConsolidatedTSR query was working incorrectly in case of varchar columns in sub-select contained non-alpha values, and as a result ConsolidatedTSR was not created. This issue was fixed.
Changed files:
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/xml/task_reporter.xml
SDP-22713 Regression - Missing 7724 tag in exchangePrice
Fixed lastTradeDate mapping for Exchange Price Object type
Value for lastTradeDate is loading correctly now via RDC Price Evnet and panel.
Changed files:
xml-ref_security_pricing_common.inc
SDP-22593 malformed cashjournal extracts
Resolved issue with malformed CASHJOURNALEXTRACT file related to Records grouping
The records grouping was removed from transformation. Every extracted record is now enveloped under a separate <accountingTransaction> node.
Changed file:
t_acct_cash_journal_streaming.inc
SDP-22083 WHRSNAVMULTIEXTRACT needs additional logic to interpret / offset Cancel Records in the Distribution Rate Summary element
Logic for Distribution Rate Summary model was improved for WRHSNAVMULTIEXTRACT
The total summary value in the Distribution Rate model was equal to sum of rate of all processed records for given Portfolio Account, Valuation Date and Share Class.
Such approach had a problem in case of canceled records. Calculations were corrected to properly account for canceled records.
The logic was changed to reflect correct total sum.
Changed rules:
• extract_nav_multi_me_details_tables.inc
SDP-20418 EagleML SMF Load workflow updating CUSIP populating into TICKER field
GenericSMF load replaces some xref data with Primary Asset or Alt Asset values
When GenericSMF is loaded via panels (accountingValidationFlag=Y) and Primary Asset Type and/or Alt Asset Type has any values in 'ISIN', 'SEDOL', 'RIC', 'BBID', 'SICOVM', 'VALOREN', 'CMM', 'INTERNAL', 'CINS' some xrefs are replaced by Primary Asset Type or Alt Asset Type values. Added logic to reorder of xref before sending to panel based on panel replacement logic. Changes made for following SMF types: Equity ('EQCSCS', 'EQCSPF', 'EQCSPK'), Mutual Fund ('EQEQMF'), Long Term Debt ('DBFLTP', 'DBIBFD', 'DBIBMU', 'DBIBPK', 'DBIVIV', 'DBMFDD', 'DBAMTL', 'DBIBTL').
SDP-20235 usergroups are not properly supported for SP-Based extracts
Entitlement filter for SP-Based Extracts was improved
Fixed issues related to Entitlement filter with attribute "all entities are included" in SP-Based Extracts.
The following changes were implemented:
1) If Entity filter and Entitlement filter (RTR parameter is ‘usergroups’) are used at the same time, the Entitlement filter works as additional restriction for Entity filter (default logic).
2) If Entitlement filter is only used (RTR parameter is ‘usergroups’), extract performed according to Business group assigned:
- When Business group with list of specific entities is used, then it works as source for allowed portfolios for next processing in extract (default logic).
- When Business group with attribute "all entities are included" is used, extract will be cancelled with the error message: "User Group with access to all entities cannot be used without Entity filter".
This logic works only for following extracts:
CASHJOURNALEXTRACT Accounting Cash Journal
COMPLIANCE2A7PRICINGEXTRACT Accounting Compilance 2a7 Picing Report
CONSEARNEDINCOMEEXTRACT Accounting Cons Earned Income
EXPENSEACTIVITYEXTRACT Accounting Expense Activity
FAIRVALUECOMPARISONEXTRACT Accounting Fair Value Comparison
POSITIONASOFEXTRACT Accounting Position AsOf
REALIZEDGAINLOSSEXTRACT Accounting Realized Gain Loss
TRIALBALANCEEXTRACT Accounting Trial Balance
UNAPPROVEDVARMARGINEXTRACT Accounting Unapproved Var Margin
UNSETTLEDTRANSACTIONSEXTRACT Accounting Unsettled Transactions
WEIGHTEDAVERAGEEXTRACT Accounting Weighted Average
WORKINGTRIALBALANCEEXTRACT Accounting Working Trial Balance
Changed files:
extract_filter_entity.inc
extract_filter_entity_list_resolution.inc
extract_filter_entity_resolution_v1.inc
extract_filter_entity_resolution_v2.inc