Enhancements:
SDP-23032: Updated index hint to address throughput in execution of message streams
A hint in the task reporter query was updated which gets the summary information about loaded files. It should improve the time the throughput time of the query execution.
Changed files:
task_reporter_generate_task_unit.inc
SDP-23005: The FTP/SFTP file transfer was timing out via the Message Center curl command if files are larger than 150 MB
...
File changed:
reporter_dist_methods.inc
SDP-22742: Created new Inbound interface for EntityLiquidity was created
Created a new inbound interface to load Entity Liquidity object records to the database
Created functionality to delete the Entity Liquidity object records from the database via Inbound Interface
The Inbound interface for EntityLiquidity was added to the eagle_wrf_generic_load workflow
Changed files:
eagleml_entity.xml
ent_entityliquidity_delete.inc
generate_sp_on_fly.inc
parse_ent_common_fields.inc
xml-ent_delete.xml
xml-ent_objects.rsf
...
The parameter TIBCOMode has an impact on the mapping of attributes for extract data.
The EagleML and transactionMessage nodes did not contain important attributes for TIBCO mode when using streaming mode.
Support was provided to the group of extracts below:
WORKINGTRIALBALANCEEXTRACT
PREFERREDSHAREEXTRACT
UNAPPROVEDVARMARGINEXTRACT
WEIGHTEDAVERAGEEXTRACT
MDCLENEAGEGRAPHEXTRACT
WASHSALEEXTRACT
Extra grouping in the out_t_streaming includes was deleted (includes were optimized) for the listed extracts:
WRHSBROKTRDQUOTEEXTRACT
WRHSCASHACTIVITYEXTRACT
WRHSCASHFLOWPROJEXTRACT
WRHSCLOSELOTEXTRACT
WRHSCSTDACTIVITYEXTRACT
WRHSCSTDBALANCEEXTRACT
WRHSNAVEXTRACT
WRHSPENDINGTRADESEXTRACT
WRHSPERFORMANCEEXTRACT
WRHSPOSITIONEXTRACT
WRHSTRADEEXTRACT
WRHSNAVMONTHENDEXTRACT
Files changed:
t_eml_header_streaming.inc
t_rule_start_obj_header_streaming.inc
t_rule_start_trx_header_streaming.inc
t_acct_preferred_share_streaming.inc
t_acct_unapproved_var_margin_streaming.inc
t_acct_weighted_average_streaming.inc
t_ref_mdc_leneage_graph_streaming.inc
t_ref_mdc_leneage_graph_streaming.xml
t_warehouse_brokertradequote_streaming.inc
t_warehouse_brokertradequote_streaming.xml
t_warehouse_brokertradequote_streaming_eml.inc
t_warehouse_cash_activity_streaming.inc
t_warehouse_cash_activity_streaming.xml
t_warehouse_cash_activity_streaming_eml.inc
t_warehouse_cash_flow_proj_streaming.inc
t_warehouse_cash_flow_proj_streaming.xml
t_warehouse_cash_flow_proj_streaming_eml.inc
t_warehouse_close_lot_streaming_eml.inc
t_warehouse_closelot_streaming.inc
t_warehouse_custody_activity_streaming.inc
t_warehouse_custody_activity_streaming.xml
t_warehouse_custody_activity_streaming_eml.inc
t_warehouse_custody_balance_streaming.inc
t_warehouse_custody_balance_streaming.xml
t_warehouse_custody_balance_streaming_eml.inc
t_warehouse_nav_streaming.inc
t_warehouse_nav_streaming.xml
t_warehouse_nav_streaming_eml.inc
t_warehouse_pending_trades_streaming.inc
t_warehouse_pending_trades_streaming.xml
t_warehouse_pending_trades_streaming_eml.inc
t_warehouse_performance_streaming.inc
t_warehouse_performance_streaming.xml
t_warehouse_performance_streaming_eml.inc
t_warehouse_position_streaming.inc
t_warehouse_trade_streaming.inc
t_warehouse_trade_streaming.xml
t_warehouse_trade_streaming_eml.inc
t_wash_sale_streaming_eml.inc
t_warehouse_nav_month_streaming.xml
t_acct_working_trial_balance_streaming.inc
t_acct_working_trial_balance_streaming.xml
SDP-21983: Added support for BatchMode for the Warehouse DeltaBased extracts
Logic was enhanced to allow BatchMode to be disabled for DeltaBased Warehouse Extracts via a RTR parameter:
<taskParameter>
<name>BatchMode</name>
<dataType>S</dataType>
<value>N</value>
</taskParameter>
Files changed:
...
eagle_ml-2-0_cm/extract_start.inc
SDP-21389: Swagger file was updated Definition updated to support additional feed types
The definition of the following feed types have been added to the swagger-extracts.json file:
ENTITYCOLLATERALSUMMEXTRACT
ENTITYANALYTICEXTRACT
ENTITYPEERGROUPRANKINGEXTRACT
ENTITYRATINGEXTRACT
ENTITYSECLENDINGEXTRACT
ENTITYSTATISTICEXTRACT
GENISSUEANALYTICEXTRACT
PRICINGREPORTEXTRACT
ISSUERRELEXTRACT
ISSUEEXCHANGEPRICEEXTRACT
ISSUERENTRELEXTRACT
ISSUERESGEXTRACT
ISSUERRATINGEXTRACT
ISSUERREGULATORYEXTRACT
CONTROLCENTEREVENTLOGEXTRACT
EAGLEADMINISTRATIONSETTINGSEXTRACT
Support of the following Feeds have been added in Extract Service Services (MC2).
ENTITYCOLLATERALSUMMEXTRACT
ENTITYANALYTICEXTRACT
ENTITYPEERGROUPRANKINGEXTRACT
ENTITYRATINGEXTRACT
ENTITYSECLENDINGEXTRACT
ENTITYSTATISTICEXTRACT
PRICINGREPORTEXTRACT
ISSUERENTRELEXTRACT
ISSUERESGEXTRACT
ISSUERRATINGEXTRACT
ISSUERREGULATORYEXTRACT
CONTROLCENTEREVENTLOGEXTRACT
EAGLEADMINISTRATIONSETTINGSEXTRACT
SDP-20827: Corrected the mapping for the AccountingTrade Python rule
The python CSV-to-EagleML transform for the AccountingTrade did not recognize security tags 1233 and 1234. Mapping for these tags was added in the python rule for the AccountingTrade, which is required for in-memory only processing.
Files changed:
t_eml_acct.py
SDP-20549: Allow the Profiled Profile Loader to work without any translation required
It is now possible to configure profiled the profile generic load loader to pass an incoming file without translation. In order to do that, To configure, the user should enter "no translation" in the field where the name of translator supposed to beis expected.
The Incoming file will be passed to load by profiled profile generic load loader instance.
Files changed:
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/unpacker/custom_routing.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/unpacker/send_to_wrf.xml
SDP-20513: Updated Warehouse Updated the Warehouse Performance rules for loading and extracting the new Liquidity data model
The EagleML Warehouse Performance incoming transformation rule and corresponding extract were enhanced to process data for the PERFORMDBO.PSR_LIQUIDITY table.
Note: The PSR_LIQUIDITY is a new table and is available only for certain Eagle versions. Core A corresponding core stored procedure also should be upgraded changes outside of EagleML are required to support data loading and updates to this table.
Files changed:
...
eagle_default/in/xml/xslt/wrhs_performance.xsl
...
eagle_default/in/xml/include/xml-dbdirect_wrhs_delete_statement.inc
...
eagle_ml-2-0_cm/out/tagvalue/extract_warehouse_performance.inc
...
eagle_ml-2-0_cm/out/xml/t_warehouse_liquidity_fields_streaming.inc
...
eagle_ml-2-0_cm/out/xml/t_warehouse_performance_streaming.inc
SDP-20512: Created Outbound interface for Entity Liquidity was created
The outbound interface for EntityLiquidity object was created.
Feed type: ENTITYLIQUIDITYEXTRACT.
Supported EagleML filters:
Date Filter
Sources Filter
Entity Filter
User Group Filter
Common Filter
Changed files:
extracts.rsf
extract_service.rsf
t_eagleml-2-0_streaming.rsf
w_config.inc
...
SDP-20149: Added XSD Enumeration for the User and the UserGroup in SCIM format
Added XSD Enumeration for the User and the UserGroup objects in SCIM format
The Extract rules were updated to provide support for Object Types – ScimUser and ScimGroup.
File changed:
• /eagle_default/in/xml/xml_user_objects.rsf
• /eagle_ml-2-0_cm/out/xml/t_user_group_scim_streaming.inc
• /eagle_ml-2-0_cm/out/xml/t_user_user_scim_streaming.inc
...
SDP-19866: The information about incoming files is stored in the data base database by default
Following The following parameters have been set with a default value :
:W_WAITEVENTSin the eagle_ml-2-0_cm/w_config.inc:
:W_WAITEVENTS_DATABASE_MODE: := 'Y';
:W_ENABLE_EXTENDED_FILE_STATISTICS: := 'Y';
These options enable logic for storing extended information about the incoming files in Data Base the database instead of a file on the app server.
SDP-19850: Updated the Warehouse Position rules for loading and extracting the new Liquidity data model
The EagleML Warehouse Position incoming transformation rule and corresponding extract were enhanced to process data for the HOLDINGDBO.POSITION_DETAIL_LIQUIDITY table.
Note: The POSITION_DETAIL_LIQUIDITY is a new table and available only for certain Eagle versions. Core A corresponding core stored procedure also should be upgraded changes outside of EagleML are required to support data loading and updates to this table.
Files changed:
/eagle_default/in/xml/xslt/wrh_egl_gen_interface3.xsl
/eagle_default/in/xml/xslt/wrhs_performance.xsl
/eagle_default/in/xml/include/xml-dbdirect_wrhs_delete_statement.inc
/eagle_ml-2-0_cm/out/tagvalue/extract_warehouse_position.inc
/eagle_ml-2-0_cm/out/xml/t_warehouse_liquidity_fields_streaming.inc
/eagle_ml-2-0_cm/out/xml/t_warehouse_position_streaming.inc
...
SDP-19625: Added support of JMSExtractDeliveryStream and CollectDistributionErrors RTR parameters in generic extract workflow
Support of Added support for two new RTR parameters JMSExtractDeliveryStream and CollectDistributionErrors was provided in the eagle_wrf_generic_extract workflow if when the ExtractDeliveryMethod='JMS'.
The corresponding global parameter is are also supported with the W_COLLECT_DISTRIBUTION_ERRORS
Files changed:
eagle_default_cm_file_distribution.xml
reporter_dist_methods.inc
eagle_wrf_generic_extract.wrf
eagle_wrf_generic_extract.ejmwf
SDP-18029: global parameter.
RTR Sample Parameters:
<taskParameter>
<name>CollectDistributionErrors</name>
<dataType>S</dataType>
<value>JMS</value>
</taskParameter>
<taskParameter>
<name>JMSExtractDeliveryStream</name>
<dataType>S</dataType>
<value>JMS_stream_sample_extract</value>
</taskParameter>
w_config_custom Sample Parameters:
<COL TAG="W_COLLECT_DISTRIBUTION_ERRORS" EXPRESSION= "'JMS'"/>
<COLTAG="W_JMS_DEF_EXTRACT_STREAM"EXPRESSION="'JMS_EAGLE_extract'"USERDESCRIPTION="Default JMS EXTRACT Delivery Stream"/>
Files changed:
eagle_default_cm_file_distribution.xml
reporter_dist_methods.inc
eagle_wrf_generic_extract.wrf
eagle_wrf_generic_extract.ejmwf
SDP-18029: The PRICINGREPORTEXTRACT was updated to include additional elements
...
priceLevel
levelComments
reason
comments
marketDate
marketDaysStale
Added MC2 support for the PRICINGREPORTEXTRACT
Modified common source filter to support MC2
Changed files:Changed files:
extract_filter_sources.inc
extract_pricing_report.inc
t_pricing_report_streaming.inc
SDP-16994: The XSD version was added to the w_config_version.inc file
Extract The extract always gets the XSD version from the main xsd file. Added To simplify debugging, made changes to allow store the XSD version as the global parameter g_EagleML20_XSDRevision in the w_config_version.inc. The parameter is set by the SetupEagleVersion process via the eagle_ml-2-0_default_cm_w_setup stream. Now extracts will Extracts now read this parameter g_EagleML20_XSDRevision once, instead of from the eagleml-main-2-0.xsd file each timeexecution.
Files changed:
w_setup_eagle_version.xml
get_revision_num.inc
...
The execution of sql requests was optimized.
The bind Bind variables were added when executing sql queries for most used includesthe most common shared include files.
Files changed:
execute_task.inc
control_workflow_paused.inc
get_stream_info.inc
check_env_name.inc
calc_date_from_rule.inc
check_stream_status.inc
include\get_sp_params_list.inc
...
SDP-15247: Rating extract supports date and datetime format
The Rating Extract was enhanced to support both date and datetime format for submittedDate field.
...
SDP-14722: Error message was improved in case of failure of accounting trade
Previously the 'UNKNOW' error was returend returned in case no panel was selected for the accounting trade load.
Error description was imporvedThis is generally because the message is missing a valid event type, processing security type, etc.
The Error description was improved:
'No panel found to load. Please check values for tag55, tag3931 and tag4533.'
File changed:
accttrade_routing_check.inc
csv-buy_panels.inc
csv-sell_panels.inc
xml-accttrade_buy.xml
xml-accttrade_sell.xml
SDP-10431: Enhanced Profiled the Profile Loader to support processing of multiple archived files
User is now able to use packages that contain one or several files as source files for profiled loader and configure translation and outgoing feed types for each file from package separately. To specify packaged files user should use prefix of package mask separated from processed file by slash symbol, Support for the processing of archived data has been added to the profile loader workflow. In the past, it could only receive plain uncompressed files. It is now possible to configure the loader for the condition when data comes in from one or multiple archives.
The following changes in the profiles syntax have been added:
If the file is part of the archive, the archive mask should be specified before the file mask as follows: [package mask]/[sourсe file mask]
...
Only one
...
file per profile should be
...
a bundle trigger. To
...
mark a file as a bundle trigger
...
, prefix it with the tilde symbol: ~bundle_archive/source_file
Archived and plain files may appear together in one profile.
Files changed:
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/ndfa/launch_load_by_profile.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/ndfa/ndfa_exec.xml
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/unpacker/custom_routing.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/unpacker/send_to_wrf.xml
eagle/estar/tpe/dynamic/pyrules/eagleinvsys/wrf_generator/_main_.py
Files added:
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/run_wrf_instance.inc
SDP-9466: Added the ability to extract a Static or Dynamic Entity List
Added two new FeedTypes were added:
STATICENTITYLISTEXTRACT
DYNAMICENTITYLISTEXTRACT
Both feeds are based on ENTITYLISTLISTEXTRACT rules and have the same output structure and are supported by Generic Extract workflow.
STATICENTITYLISTEXTRACT – extracts only static Entity with type LIST, dynamic lists will be ignored.
EntityFilter(Optional) – type can be only EntityList or ProcessCenter.DYNAMICENTITYLISTEXTRACT– extracts only dynamic Entity with type LIST, static lists will be ignored.
EntityFilter(required) with type EntityList
Changed files:
eagle_ml-2-0_cm/extract_filter_entity.inc
eagle_ml-2-0_cm/out/tagvalue/extract_entity_list_list.inc
eagle_ml-2-0_cm/out/tagvalue/extracts.rsf
eagle_ml-2-0_cm/out/xml/extract_service.rsf
eagle_ml-2-0_cm/out/ndfaxml/launchextract_loadservice_by_profilestart.inc
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/ndfaout/ndfa_exec.xml
eagle/estar/tpe/dynamic/msgcenter/eagle_ml/t_eagleml-2-0_cm/unpacker/custom_routing.inc
eagle/estar/tpe/dynamic/msgcenter/_out_extract_service.rsf
eagle_ml-2-0_cm/out/unpackerxml/send_to_wrf.xml
eagle/estar/tpe/dynamic/pyrules/eagleinvsys/wrf_generator/_main_.py
Files added:
eagle/estar/tpe/dynamic/msgcenter/t_eagleml-2-0_streaming.rsf
eagle_ml-2-0_cm/out/xml/t_entity_list_list_streaming.inc
eagle_ml-2-0_cm/runw_wrf_instanceconfig.inc
Production Bugs
SDP-23322: Corrected logic for CreateConsolidatedTSR query
The CreateConsolidatedTSR query was working incorrectly in case of when the varchar columns in sub-select the subselect contained non-alpha values, and as alphanumeric values. As a result, the ConsolidatedTSR was not created. This issue was fixedidentified and corrected.
Changed files:
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/xml/task_reporter.xml
...
SDP-22713: Fixed lastTradeDate mapping for Exchange Price Object type
Value The value for the lastTradeDate is loading correctly now via the RDC Price Evnet Event and panel.
Changed files:
xml-ref_security_pricing_common.inc
...
The records grouping was removed from transformation. Every extracted record is now enveloped under a separate <accountingTransaction> node.
Changed file:
t_acct_cash_journal_streaming.inc
SDP-22083: Logic for the Distribution Rate Summary model was improved for within the WRHSNAVMULTIEXTRACT
The total summary value in the Distribution Rate model was equal to sum of rate the rates of all processed records for a given Portfolio Account, Valuation Date and Share Class.
Such approach had a problem in case of canceled records. Calculations Date and Share Class.
The subtotals were corrected to properly account for canceled records.
The logic was changed to reflect the correct total sum reversing cancels when they are present in the dataset.
Changed rules:
• extract_nav_multi_me_details_tables.inc
...
SDP-20418: GenericSMF load replaces some xref data with Primary Asset or Alt Asset values
When the GenericSMF was loaded via panels (accountingValidationFlag=Y) and the Primary Asset Type, and/or Alt Asset Type had any values in the Xref ('ISIN', 'SEDOL', 'RIC', 'BBID', 'SICOVM', 'VALOREN', 'CMM', 'INTERNAL', 'CINS') some xrefs were replaced by the Primary Asset Type or Alt Asset Type values.
Added logic to reorder of the xref before sending to the panel based on the panel replacement logic. Changes were made for the following SMF types: Equity ('EQCSCS', 'EQCSPF', 'EQCSPK'), ; Mutual Fund ('EQEQMF'), ; and Long Term Debt ('DBFLTP', 'DBIBFD', 'DBIBMU', 'DBIBPK', 'DBIVIV', 'DBMFDD', 'DBAMTL', 'DBIBTL').
...
SDP-20235: Entitlement filter for SP-Based Extracts was improved
Fixed issues related to the Entitlement filter with the attribute "all entities are included" in SP-Based Extracts.
The following changes were implemented:
1) If the Entity filter and the Entitlement filter (RTR parameter is ‘usergroups’) are used at the same time, the Entitlement filter works as an additional restriction for Entity filter (default logic).
2) If the Entitlement filter is only used (RTR parameter is ‘usergroups’), the extract performed performs according to the Business group assigned:
When the Business group with a list of specific entities is used,
thenit works as the source for allowed portfolios for next processing in extract (default logic).
When the Business group with the attribute "all entities are included" is used, the extract will be cancelled with the error message: "User Group with access to all entities cannot be used without Entity filter".
This logic works only for following extracts:
CASHJOURNALEXTRACT Accounting Cash Journal
COMPLIANCE2A7PRICINGEXTRACT Accounting Compilance 2a7 Picing Report
CONSEARNEDINCOMEEXTRACT Accounting Cons Earned Income
EXPENSEACTIVITYEXTRACT Accounting Expense Activity
FAIRVALUECOMPARISONEXTRACT Accounting Fair Value Comparison
POSITIONASOFEXTRACT Accounting Position AsOf
REALIZEDGAINLOSSEXTRACT Accounting Realized Gain Loss
TRIALBALANCEEXTRACT Accounting Trial Balance
UNAPPROVEDVARMARGINEXTRACT Accounting Unapproved Var Margin
UNSETTLEDTRANSACTIONSEXTRACT Accounting Unsettled Transactions
WEIGHTEDAVERAGEEXTRACT Accounting Weighted Average
WORKINGTRIALBALANCEEXTRACT Accounting Working Trial Balance
...