Enhancements:
SDP-23032: Updated index hint to address throughput in execution of message streams
A hint in the task reporter query was updated which gets the summary information about loaded files. It should improve the throughput time of the query execution.
Changed files:
task_reporter_generate_task_unit.inc
SDP-23005: The FTP/SFTP file transfer was timing out via the Message Center curl command if files are larger than 150 MB
For larger files greater than 150MB, a timeout was occurring. The issue was resolved by adding an additional call to the curl command with a timeout set to 1800 seconds. The new, optional parameter is defined as W_FTP_CURL_TIMEOUT_30MIN and should be set with a value of Y in the w_config_custom.inc.
File changed:
reporter_dist_methods.inc
SDP-22742: Created new Inbound interface for EntityLiquidity
- Created a new inbound interface to load Entity Liquidity object records to the database
- Created functionality to delete the Entity Liquidity object records from the database via Inbound Interface
- The Inbound interface for EntityLiquidity was added to the eagle_wrf_generic_load workflow
Changed files:
eagleml_entity.xml
ent_entityliquidity_delete.inc
generate_sp_on_fly.inc
parse_ent_common_fields.inc
xml-ent_delete.xml
xml-ent_objects.rsf
Added files:
eagleml_ent_entityliquidity.xml
ent_entityregulatory_delete.inc
xml-ent_entityliquidity.xml
SDP-22672: GENISSUEANALYTICEXTRACT was updated to support streaming
The rules for the GENISSUEANALYTICEXTRACT were updated to support streaming.
Files changed:
t_generic_issue_analytic_streaming.xml
t_eagleml-2-0_streaming.rsf
extract_generic_issue_analytic.inc
w_config.inc
SDP-22107: TIBCOMode supported for streaming extracts (T-transformation)
The parameter TIBCOMode has an impact on the mapping of attributes for extract data.
The EagleML and transactionMessage nodes did not contain important attributes for TIBCO mode when using streaming mode.
Support was provided to the group of extracts below:
- WORKINGTRIALBALANCEEXTRACT
- PREFERREDSHAREEXTRACT
- UNAPPROVEDVARMARGINEXTRACT
- WEIGHTEDAVERAGEEXTRACT
- MDCLENEAGEGRAPHEXTRACT
- WASHSALEEXTRACT
Extra grouping in the out_t_streaming includes was deleted (includes were optimized) for the listed extracts:
- WRHSBROKTRDQUOTEEXTRACT
- WRHSCASHACTIVITYEXTRACT
- WRHSCASHFLOWPROJEXTRACT
- WRHSCLOSELOTEXTRACT
- WRHSCSTDACTIVITYEXTRACT
- WRHSCSTDBALANCEEXTRACT
- WRHSNAVEXTRACT
- WRHSPENDINGTRADESEXTRACT
- WRHSPERFORMANCEEXTRACT
- WRHSPOSITIONEXTRACT
- WRHSTRADEEXTRACT
- WRHSNAVMONTHENDEXTRACT
Files changed:
t_eml_header_streaming.inc
t_rule_start_obj_header_streaming.inc
t_rule_start_trx_header_streaming.inc
t_acct_preferred_share_streaming.inc
t_acct_unapproved_var_margin_streaming.inc
t_acct_weighted_average_streaming.inc
t_ref_mdc_leneage_graph_streaming.inc
t_ref_mdc_leneage_graph_streaming.xml
t_warehouse_brokertradequote_streaming.inc
t_warehouse_brokertradequote_streaming.xml
t_warehouse_brokertradequote_streaming_eml.inc
t_warehouse_cash_activity_streaming.inc
t_warehouse_cash_activity_streaming.xml
t_warehouse_cash_activity_streaming_eml.inc
t_warehouse_cash_flow_proj_streaming.inc
t_warehouse_cash_flow_proj_streaming.xml
t_warehouse_cash_flow_proj_streaming_eml.inc
t_warehouse_close_lot_streaming_eml.inc
t_warehouse_closelot_streaming.inc
t_warehouse_custody_activity_streaming.inc
t_warehouse_custody_activity_streaming.xml
t_warehouse_custody_activity_streaming_eml.inc
t_warehouse_custody_balance_streaming.inc
t_warehouse_custody_balance_streaming.xml
t_warehouse_custody_balance_streaming_eml.inc
t_warehouse_nav_streaming.inc
t_warehouse_nav_streaming.xml
t_warehouse_nav_streaming_eml.inc
t_warehouse_pending_trades_streaming.inc
t_warehouse_pending_trades_streaming.xml
t_warehouse_pending_trades_streaming_eml.inc
t_warehouse_performance_streaming.inc
t_warehouse_performance_streaming.xml
t_warehouse_performance_streaming_eml.inc
t_warehouse_position_streaming.inc
t_warehouse_trade_streaming.inc
t_warehouse_trade_streaming.xml
t_warehouse_trade_streaming_eml.inc
t_wash_sale_streaming_eml.inc
t_warehouse_nav_month_streaming.xml
t_acct_working_trial_balance_streaming.inc
t_acct_working_trial_balance_streaming.xml
SDP-21983: Added support for BatchMode for the Warehouse DeltaBased extracts
Logic was enhanced to allow BatchMode to be disabled for DeltaBased Warehouse Extracts via a RTR parameter:
<taskParameter>
<name>BatchMode</name>
<dataType>S</dataType>
<value>N</value>
</taskParameter>
Files changed:
eagle_ml-2-0_cm/extract_start.inc
SDP-21389: Swagger file was updated to support additional feed types
The definition of the following feed types have been added to the swagger-extracts.json file:
- ENTITYCOLLATERALSUMMEXTRACT
- ENTITYANALYTICEXTRACT
- ENTITYPEERGROUPRANKINGEXTRACT
- ENTITYRATINGEXTRACT
- ENTITYSECLENDINGEXTRACT
- ENTITYSTATISTICEXTRACT
- GENISSUEANALYTICEXTRACT
- PRICINGREPORTEXTRACT
- ISSUERRELEXTRACT
- ISSUEEXCHANGEPRICEEXTRACT
- ISSUERENTRELEXTRACT
- ISSUERESGEXTRACT
- ISSUERRATINGEXTRACT
- ISSUERREGULATORYEXTRACT
- CONTROLCENTEREVENTLOGEXTRACT
- EAGLEADMINISTRATIONSETTINGSEXTRACT
Support of the following Feeds have been added in Extract Services (MC2).
- ENTITYCOLLATERALSUMMEXTRACT
- ENTITYANALYTICEXTRACT
- ENTITYPEERGROUPRANKINGEXTRACT
- ENTITYRATINGEXTRACT
- ENTITYSECLENDINGEXTRACT
- ENTITYSTATISTICEXTRACT
- PRICINGREPORTEXTRACT
- ISSUERENTRELEXTRACT
- ISSUERESGEXTRACT
- ISSUERRATINGEXTRACT
- ISSUERREGULATORYEXTRACT
- CONTROLCENTEREVENTLOGEXTRACT
- EAGLEADMINISTRATIONSETTINGSEXTRACT
SDP-20827: Corrected the mapping for the AccountingTrade Python rule
The python CSV-to-EagleML transform for the AccountingTrade did not recognize security tags 1233 and 1234. Mapping for these tags was added in the python rule for the AccountingTrade, which is required for in-memory only processing.
Files changed:
t_eml_acct.py
SDP-20549: Allow the Profile Loader to work without any translation required
It is now possible to configure the profile generic loader to pass an incoming file without translation. To configure, the user should enter "no translation" in the field where the name of translator is expected.
The Incoming file will be passed to load by profile generic loader instance.
Files changed:
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/unpacker/custom_routing.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/unpacker/send_to_wrf.xml
SDP-20513: Updated the Warehouse Performance rules for loading and extracting the new Liquidity data model
The EagleML Warehouse Performance incoming transformation rule and corresponding extract were enhanced to process data for the PERFORMDBO.PSR_LIQUIDITY table.
The PSR_LIQUIDITY is a new table and is available only for certain Eagle versions. Core stored procedure also should be upgraded to support data loading to this table.
Files changed:
eagle_default/in/xml/xslt/wrhs_performance.xsl
eagle_default/in/xml/include/xml-dbdirect_wrhs_delete_statement.inc
eagle_ml-2-0_cm/out/tagvalue/extract_warehouse_performance.inc
eagle_ml-2-0_cm/out/xml/t_warehouse_liquidity_fields_streaming.inc
eagle_ml-2-0_cm/out/xml/t_warehouse_performance_streaming.inc
SDP-20512: Created Outbound interface for Entity Liquidity
The outbound interface for EntityLiquidity object was created.
Feed type: ENTITYLIQUIDITYEXTRACT.
Supported EagleML filters:
- Date Filter
- Sources Filter
- Entity Filter
- User Group Filter
- Common Filter
Changed files:
extracts.rsf
extract_service.rsf
t_eagleml-2-0_streaming.rsf
w_config.inc
Added files:
extract_entity_liquidity.xml
extract_entity_liquidity.inc
t_entity_liquidity_streaming.xml
t_entity_liquidity_streaming.inc
extract_service_entity_liquidity.xml
SDP-20149: Added XSD Enumeration for the User and the UserGroup in SCIM format
Added XSD Enumeration for the User and the UserGroup objects in SCIM format
The Extract rules were updated to provide support for Object Types ScimUser and ScimGroup.
File changed:
eagle_default/in/xml/xml_user_objects.rsf
eagle_ml-2-0_cm/out/xml/t_user_group_scim_streaming.inc
eagle_ml-2-0_cm/out/xml/t_user_user_scim_streaming.inc
SDP-19866: The information about incoming files is stored in the database by default
The following parameters have been set with a default value in the eagle_ml-2-0_cm/w_config.inc:
:W_WAITEVENTS_DATABASE_MODE: := 'Y';
:W_ENABLE_EXTENDED_FILE_STATISTICS: := 'Y';
These options enable logic for storing extended information about the incoming files in the database instead of file on the app server.
SDP-19850: Updated Warehouse Position rules for loading and extracting new Liquidity data model
EagleML Warehouse Position incoming transformation rule and corresponding extract were enhanced to process data for the HOLDINGDBO.POSITION_DETAIL_LIQUIDITY table.
Note: POSITION_DETAIL_LIQUIDITY is a new table and available only for certain Eagle versions. Core stored procedure also should be upgraded to support data loading to this table.
Files changed:
/eagle_default/in/xml/xslt/wrh_egl_gen_interface3.xsl
/eagle_default/in/xml/xslt/wrhs_performance.xsl
/eagle_default/in/xml/include/xml-dbdirect_wrhs_delete_statement.inc
/eagle_ml-2-0_cm/out/tagvalue/extract_warehouse_position.inc
/eagle_ml-2-0_cm/out/xml/t_warehouse_liquidity_fields_streaming.inc
/eagle_ml-2-0_cm/out/xml/t_warehouse_position_streaming.inc
SDP-19625: Added support of JMSExtractDeliveryStream and CollectDistributionErrors RTR parameters in generic extract workflow
Support of for two new RTR parameters JMSExtractDeliveryStream and CollectDistributionErrors was provided in eagle_wrf_generic_extract workflow if ExtractDeliveryMethod='JMS'.
The corresponding global parameter is also supported W_COLLECT_DISTRIBUTION_ERRORS
Files changed:
eagle_default_cm_file_distribution.xml
reporter_dist_methods.inc
eagle_wrf_generic_extract.wrf
eagle_wrf_generic_extract.ejmwf
SDP-18029: PRICINGREPORTEXTRACT was updated to include additional elements
The following new elements were added to Pricing Report:
priceLevel
levelComments
reason
comments
marketDate
marketDaysStale
Added MC2 support for PRICINGREPORTEXTRACT
Modified common source filter to support MC2
Changed files:
extract_filter_sources.inc
extract_pricing_report.inc
t_pricing_report_streaming.inc
SDP-16994: XSD version was added to w_config_version.inc file
Extract always gets XSD version from the main xsd file. Added changes to allow store XSD version as global parameter g_EagleML20_XSDRevision in w_config_version.inc. The parameter is set by SetupEagleVersion process via eagle_ml-2-0_default_cm_w_setup stream. Now extracts will read this parameter g_EagleML20_XSDRevision once, instead of eagleml-main-2-0.xsd file each time.
Files changed:
w_setup_eagle_version.xml
get_revision_num.inc
SDP-15946: Optimized execution of sql queries for most used EJM includes
The execution of sql requests was optimized.
The bind variables were added when executing sql queries for most used includes.
Files changed:
execute_task.inc
control_workflow_paused.inc
get_stream_info.inc
check_env_name.inc
calc_date_from_rule.inc
check_stream_status.inc
include\get_sp_params_list.inc
SDP-15247: Rating extract supports date and datetime format
Rating Extract was enhanced to support both date and datetime format for submittedDate field.
Files changed:
/eagle_ml-2-0_cm/out/tagvalue/extract_rating.inc
/eagle_ml-2-0_cm/out/xml/t_rating_streaming.inc
SDP-14722: Error message was improved in case of failure of accounting trade
Previously 'UNKNOW' error was returend in case no panel was selected for accounting trade load.
Error description was imporved:
'No panel found to load. Please check values for tag55, tag3931 and tag4533.'
File changed:
accttrade_routing_check.inc
csv-buy_panels.inc
csv-sell_panels.inc
xml-accttrade_buy.xml
xml-accttrade_sell.xml
SDP-10431: Enhanced Profiled Loader to support processing of multiple archived files
Support for processing of archived data has been added to profiled loader workflow. In the past it could only receive plain uncompressed files. Now it is possible to configure the loader for the condition when data comes in one or multiple archives.
The following changes in the profiles syntax have been added:
- If the file is part of the archive, then archive mask should be specified before the file mask, as follows: [package mask]/[sourсe file mask]
- As usual, one of files per profile should be a bundle trigger. To mark a file as bundle trigger, you should prefix it with tilde symbol: ~bundle_archive/source_file
- Archived and plain files can appear together in one profile.
Files changed:
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/ndfa/launch_load_by_profile.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/ndfa/ndfa_exec.xml
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/unpacker/custom_routing.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/unpacker/send_to_wrf.xml
eagle/estar/tpe/dynamic/pyrules/eagleinvsys/wrf_generator/_main_.py
Files added:
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/run_wrf_instance.inc
SDP-9466: Added ability to extract Dynamic Entity List
Production Bugs
SDP-23322: Corrected logic for CreateConsolidatedTSR query
CreateConsolidatedTSR query was working incorrectly in case of varchar columns in sub-select contained non-alpha values, and as a result ConsolidatedTSR was not created. This issue was fixed.
Changed files:
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/xml/task_reporter.xml
SDP-22713: Fixed lastTradeDate mapping for Exchange Price Object type
Value for lastTradeDate is loading correctly now via RDC Price Evnet and panel.
Changed files:
xml-ref_security_pricing_common.inc
SDP-22593: Resolved issue with malformed CASHJOURNALEXTRACT file related to Records grouping
The records grouping was removed from transformation. Every extracted record is now enveloped under a separate <accountingTransaction> node.
Changed file:
t_acct_cash_journal_streaming.inc
SDP-22083: Logic for Distribution Rate Summary model was improved for WRHSNAVMULTIEXTRACT
The total summary value in the Distribution Rate model was equal to sum of rate of all processed records for given Portfolio Account, Valuation Date and Share Class.
Such approach had a problem in case of canceled records. Calculations were corrected to properly account for canceled records.
The logic was changed to reflect correct total sum.
Changed rules:
• extract_nav_multi_me_details_tables.inc
SDP-20418: GenericSMF load replaces some xref data with Primary Asset or Alt Asset values
When GenericSMF was loaded via panels (accountingValidationFlag=Y) and Primary Asset Type, and/or Alt Asset Type had any values in 'ISIN', 'SEDOL', 'RIC', 'BBID', 'SICOVM', 'VALOREN', 'CMM', 'INTERNAL', 'CINS' some xrefs were replaced by Primary Asset Type or Alt Asset Type values.
Added logic to reorder of xref before sending to panel based on panel replacement logic. Changes made for following SMF types: Equity ('EQCSCS', 'EQCSPF', 'EQCSPK'), Mutual Fund ('EQEQMF'), Long Term Debt ('DBFLTP', 'DBIBFD', 'DBIBMU', 'DBIBPK', 'DBIVIV', 'DBMFDD', 'DBAMTL', 'DBIBTL').
SDP-20235: Entitlement filter for SP-Based Extracts was improved
Fixed issues related to Entitlement filter with attribute "all entities are included" in SP-Based Extracts.
The following changes were implemented:
1) If Entity filter and Entitlement filter (RTR parameter is ‘usergroups’) are used at the same time, the Entitlement filter works as additional restriction for Entity filter (default logic).
2) If Entitlement filter is only used (RTR parameter is ‘usergroups’), extract performed according to Business group assigned:
- When Business group with list of specific entities is used, then it works as source for allowed portfolios for next processing in extract (default logic).
- When Business group with attribute "all entities are included" is used, extract will be cancelled with the error message: "User Group with access to all entities cannot be used without Entity filter".
This logic works only for following extracts:
CASHJOURNALEXTRACT Accounting Cash Journal
COMPLIANCE2A7PRICINGEXTRACT Accounting Compilance 2a7 Picing Report
CONSEARNEDINCOMEEXTRACT Accounting Cons Earned Income
EXPENSEACTIVITYEXTRACT Accounting Expense Activity
FAIRVALUECOMPARISONEXTRACT Accounting Fair Value Comparison
POSITIONASOFEXTRACT Accounting Position AsOf
REALIZEDGAINLOSSEXTRACT Accounting Realized Gain Loss
TRIALBALANCEEXTRACT Accounting Trial Balance
UNAPPROVEDVARMARGINEXTRACT Accounting Unapproved Var Margin
UNSETTLEDTRANSACTIONSEXTRACT Accounting Unsettled Transactions
WEIGHTEDAVERAGEEXTRACT Accounting Weighted Average
WORKINGTRIALBALANCEEXTRACT Accounting Working Trial Balance
Changed files:
extract_filter_entity.inc
extract_filter_entity_list_resolution.inc
extract_filter_entity_resolution_v1.inc
extract_filter_entity_resolution_v2.inc