EagleML Release Notes - March 2019

Owned by Sujatha Calangi (Unlicensed)
Last updated: Nov 13, 2024 by Kinga Huszno
7 min read

SDP-17628:  User was loaded with Disabled Status in EML User Stream

Object type User was loaded with disabled state when elements <accountState> or <active> (in case of SCIM load) were absent in incoming data file.

Logic was updated to load User with enabled state by default.

Files changed:
user_common.inc

 

SDP-10666:  Critical errors in "eagle_ml-2-0_default_in_xml_smf_mi" stream, but final status is SUCCESS

Streams eagle_ml-2-0_default_in_xml_smf_mi and eagle_ml-2-0_default_in_xml_issuer_mi had SUCCESS status although XSL-transformations inside them failed.

Added XSL-transformation errors handling to report correct status.

Files changed:
xml-ref_issuer.prf
xml-smf_generic_batch.prf

 

SDP-17749:  EntityList filter including Dynamic and Static filters creates incorrect sub query when EntityId contains: symbol

There was a problem with extracts when EntityList filter contained entities with special characters. As a result extract returned incorrect "no data" status.

Entity filter logic was enhanced to handle EntityId with special characters properly.

File changed:
eagle_ml-2-0_cm/extract_filter_entity.inc

 

SDP-14430:  IssuerOrg Delta extract can be run with historyonlyflag enabled

Incorrect query was generated for Issuer Organization extract when combination of parameters historyonlyflag = ‘Y’ and deltabased = ‘Y’ was used.

The logic was updated to use correct table (ISSUER_ORGANIZATION_HIST) in delta mode filters.

File changed:
\eagle_ml-2-0_cm\out\tagvalue\extract_delta_based_param.inc

 

SDP-10589 EJM:  Prevent multiple STAR Schedule executions during reprocess

Multiple star schedule executions were triggered when Message Center Processing resumed after interruptions in its execution.

To avoid multiple STAR/Pace executions, the event instance was saved into task params after the event was started.

After restarting the MC services, the status is checked for completion.

Files changed:
exec_pace_event.inc
exec_star_schedule.inc

 

SDP-9627:  EntityID filter is not working for REFTRADEQUOTESEXTRACT

Entity filter was ignored in Trade Quotes extract.

Fixed Entity and Held filters sequence in Trade Quotes outbound interface.
Added User groups filter to Trade Quotes outbound interface.

File changed:
extract_ref_trade_quotes.inc

 

SDP-18268:  Add ignore rule in the ruleset xml-ent_objects.rsf for condition |objectType|='Ignore'

Added ignore rule in the ruleset xml-ent_objects.rsf for condition |objectType|='Ignore'

Files changed:
ignored_object_type.xml
xml-ent_objects.rsf

 

SDP-14549:  WEIGHTEDAVERAGEEXTRACT not partition aware, missing two elements in output

Weighted Average Extract did not return data for funds that were not on the 1st partition. 

Extract rule was improved to retrieve data from different partitions. 

File changed:
extract_acct_weighted_average.inc

 

SDP-17928:  Correct spelling of error messages and file status on file processing cancellation

Error messages that were thrown for file processing cancellation failures were incorrect.

When the processing of a file is cancelled, its status should be set as FAILED and appropriate error message should be added to file statistics.

Fixed incorrect status for cancelled files. Error messages were updated to provide more descriptive information to the user.

Files changed:
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/tagvalue/execute_wrkfl_event.xml
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/get_file_stat.inc

 

SDP-17929:  Add error message to external file statistic when decryption failed

Processing of the file aborts if file decryption attempt was unsuccessful and no information is available for user at a high level which leads to user confusion.

Added error description to file statistic if file processing failed due to decryption failure.

Example of error:

gpg encrypted with RSA key, ID B3F93AB6.gpg decryption failed No secret key.

Files changed:
eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/ndfa/decrypt_incoming.inc

 

SDP-18084:  Grouping by SRC_INTFC_INST doesn't work for Issue Variable Rate extract

Grouping for Issue Variable Rate Extract with keys Security Alias + Effective Date + Source Instance (DB fields: SECURITY_ALIAS + EFFECTIVE_DATE + SRC_INTFC_INST) worked incorrectly. The reason was the Source Instance field was not extracted and was set an empty value.

To resolve the issue SRC_INTFC_INST field was added to the list of fields in the request.

File changed:
extract_issue_variable_rate.inc

 

SDP-17899:  WRHSOPENLOTEXTRACT and WRHSPOSITIONEXTRACT -- Add a sequence value to the extendedPortValuationModel

New field 'priceRuleSecurityInstance' is added in node pvAssetPrice in extendedPortValuationModel. This field has a unique value for each price record and can be used to differentiate between the returned records.

Files changed:
extract_holding_ext_port_valuation_fields.inc
t_holding_portfolio_valuation_fields_streaming.inc

 

SDP-17866:  ClassificationMatrix - add new fields and filters

New fields, parameter and filters were added for ClassificationMatrix extract.

New fields: 
- classificationMatrix/classificationMatrixLevel 
- classificationMatrix/classificationMatrixCategory/levelCount
- classificationMatrix/classificationMatrixLevel/levelCount
- classificationMatrix/classificationMatrixLevel/valueLevel10


New parameter and Usage example:
ExcludeClassMatrixId - parameter to include or exclude node classificationMatrixId in the extract.
Accepted values - Y or N, default value - N.

 Usage:
<taskParameter>
<name>ExcludeClassMatrixId</name>
<value>Y</value>
</taskParameter>

New filter and Usage example:
pendingOnly - filters records based on their SAVE_FINAL value in database.
Accepted values - Y or N

Usage:
<taskParameter>
<name>pendingOnly</name>
<value>Y</value>
</taskParameter>

pendingOnly = Y, extracts only pending records (SAVE_FINAL = N)
pendingOnly = N, extracts only finalized records (SAVE_FINAL = Y);
Important: if no value for parameter is entered in RTR, extract returns records with SAVE_FINAL = Y, N and NULL.

Files changed:
\eagle_ml-2-0_cm\out\xml\t_ref_classification_matrix_streaming.inc
\eagle_ml-2-0_cm\out\tagvalue\extract_ref_classification_matrix.inc

 

SDP-10868:  Update realtime_ejm_stat to report stream as failure when underlying records have failed

Rule for realtime_ejm_stat was enhanced to show correct statistics for No. Of Failed, Start Processing Time, End Processing Time and Status on the File Statistics tab.

Files changed:
eagle_ml-2-0_cm\out\tagvalue\realtime_ejm_stat.xml


SDP-9259:  Query optimization during message processing

DB calls which are used during in-memory processing were reviewed and updated.
1. Added bind variables for system DB calls.
2. Remove unnecessary DB calls on reporting step and execkwkfl step.
3. Decrease number of calls on realtime_ejm_stat step, replace some SP calls by anonymous block with update statement.
4. Improve queries on reporting step for getting summary and detail information.

Files changed: 
eagle_ml-2-0_cm/out/tagvalue/realtime_ejm_stat.xml
eagle_ml-2-0_cm/out/xml/task_reporter_gen_task_unit_realtime.inc
eagle_ml-2-0_cm/out/xml/taskacknowledgement.xml
eagle_ml-2-0_cm/get_stream_folder.inc
eagle_ml-2-0_cm/get_stream_name.inc
eagle_ml-2-0_cm/out/tagvalue/execute_wrkfl_event.xml
eagle_ml-2-0_cm/out/xml/task_reporter.xml
eagle_ml-2-0_cm/out/xml/task_reporter_generate_task_unit.inc
eagle_ml-2-0_cm/reporter_dist.inc
eagle_ml-2-0_cm/reporter_dist_methods.inc

 

SDP-18582:  TRDX_EGL_GEN_INTERFACE and TRDC_EGL_GEN_INTERFACE feeds fail to load via the xml-warehouse_v2.xml

Warehouse Trade XREF or Warehouse Trade Charges EagleML types were not processed by EagleML rules.

Warehouse common include was updated to support load for Warehouse Trade XREF and Warehouse Trade Charges EagleML objects.

Files changed:
eagle_default\in\xml\include\xml-warehouse_common.inc

 

 

SDP-10945: CONSEARNEDINCOMEEXTRACT missing mappings for TotalFund and Sector parameters

Grouping was incorrect in the Consolidated Earned Income outbound interface if a requested Entity is a Master Entity that contains Sector Entities.

The issue with grouping was fixed: Sector Name field was added to the group key.

Also the Consolidated Earned Income outbound interface was enhanced to support ReportDetailLevelSwitch filter. Possible Values are POS, LOT. By default, when filter is omitted, POS value is used. When ReportDetailLevelSwitch = LOT, Report is run on Transaction Lot Level. Otherwise, the report is run on Position Level.

The interface was enhanced to support new elements for Transaction Lot Level:

openEventId
closeEventId
monthEndDate
origAcctDate
tradeDate
masterTicketNumber

Files changed:
eagle_ml-2-0_cm\out\tagvalue\extract_acct_cons_earned_inc.inc
eagle_ml-2-0_cm\out\xml\t_acct_cons_earned_inc.xml
eagle_ml-2-0_cm\out\xml\t_acct_cons_earned_inc_streaming.inc

 

SDP-9497:  Add rtr parameter to disable the execution of update_tasks_statistics.inc logic for workflows

The DisableTaskStatisticsUpdate parameter was added to allow the option of disabling the execution of the update_task_statistics.inc for workflows.

The default value for this parameter is “N” and must be set to “Y” to disable the logic.

Files changed:
update_task_statistics.inc
execute_wrkfl_event.xml

 

SDP-18529: Issue with DateRule parameter in control message for Date Rule logic

Updated date rules functionality for task parameters with the following structure:

Date parameter name + DateRule (ex: effectivedateDateRule).

Added ability to exclude such Date parameters from the logic and process them as simple task parameters without date rules calculation.

To enable this feature, parameter W_SKIP_DATERULE_PARAM_LIST in w_config_custom.inc is set with a list of excluded parameters separated by commas.
Example of usage:
<CODE>:W_SKIP_DATERULE_PARAM_LIST:=: W_SKIP_DATERULE_PARAM_LIST :+ ','+'UseDateRule';</CODE>

Files changed:
collect_task_parameters.inc
w_config.inc

 

SDP-16300:  UserChar1 (tag 1887) is not passed to panel on  AccountingTrade Load

When AccountingTrade data is loaded as single message via EJM, field UserChar1 with tag id 120 was overwritten as empty value. DB request that got workflow state was returned in format '120::field1:value1:field2:value2'. After parsing the returned result, empty value is assigned to tag 120.

The issue was fixed by replacing substring '120::' from the request result before parsing. This change helps in avoiding the same issue in the following interfaces:

GenericEntity, ExchangePrice, AnalyticEquity, AnalyticFI, IssueAnalytic, GenericIssueAnalytics, IssuerOrganization, IssuerAnalytics, IssuePCPrice, IssuerRelationship, GenericCA, GenericSMF, MultipleLegAccountingTrade, UnderlyingSMF.

Files changed:
orch_query.inc

 

SDP-9933:  Add issuerShortName into the Issuer Organization EagleMl rule

Support for issuerShortName field was added to Issuer Organization load and extract rules.
Note: currently issuerShortName exists in a limited number of database versions.

Files changed:
/metadata/bind/eagleml_ref_issuerorganization.xml
/eagle_default/in/xml/xml-ref_issuerorganization.xml
/eagle_ml-2-0_cm/out/xml/t_issuer_org_streaming.inc

 

SDP-18766:  Allow users to specify workflow schedule launch parameters by their names, not tags

Automation Center allows to set up schedule launch parameters with names but this was not supported by workflow_restart.sch rule that is used for workflow_restart.xml rule.

EJM Logic was enhanced to let Users to set up parameters with names to make workflow schedules more user friendly.

Added handling for incoming parameters from automation center based on parameter name

Tags related to parameters:
:tagName: := '1166' – :paramName: := ProfileName
:tagName: := '1174' – :paramName: := WORKFLOWENGINE
:tagName: := '1176' – :paramName: :=  WorkflowName / PROFILEDWORKFLOW
:tagName: := '1177' –  :paramName: := corrIdPrefix
:tagName: := '1178' –  :paramName: := DataTaskEmailDistributionList
:tagName: := '1179' –  :paramName: := EnableTimeoutCancel (Y/N)
:tagName: := '1180' –  :paramName: := EJMRunTimeout

Files changed:
estar/tpe/dynamic/msgcenter/eagle_ml-2-0_cm/schedule/workflow_restart.sch

 

SDP-15818:  New Feedtype to extract Warehouse Trade Charges data

A new outbound interface was added to the warehouseTradeCharges EagleML object.

Feed type: WRHSTRADECHARGESEXTRACT.

Supported filters:

  • Entity filters

  • User Groups filter

  • Date filters: updatedate, effectivedate, accountingdate, postdate, ledgereffdate, monthendaccountingdate, tradedate, settlementdate

  • Source filter

  • SMF filters

  • maxrows filter

  • CustomWhere filter

Files added:
extract_warehouse_trade_charges.inc
extract_warehouse_trade_charges.xml
t_warehouse_trade_charges_streaming.inc
t_warehouse_trade_charges_streaming.xml

Files changed:
extracts.rsf
t_eagleml-2-0_streaming.rsf
w_config.inc

 

SDP-15250:  Add concatenated values to calcMasterTransactionId on WRHSOPENLOTEXTRACT and WRHSCLOSELOTEXTRACT

New field 'calcMasterTransactionId' was added to Warehouse OpenLot and CloseLot Extracts. 
The field contains a concatenated value from 'acctBasis', 'entityId', 'securityAlias', 'longShortIndicator' and some others fields.

Files changed:
extract_warehouse_openlot.inc
t_warehouse_openlot_streaming.inc
extract_warehouse_closelot.inc
t_warehouse_close_lot_streaming.xml
t_warehouse_closelot_streaming.inc

 

SDP-16498:  Unable to load EagleML ‘accountingTrade’ feedtype with buySellIndicator=’REC’ or ‘DEL’

When Receive or Deliver Accounting Trade was loaded, there was an error about values of Cross Reference Identifiers were missing (tags 1233 and 1234).

Obsolete mapping was removed for all Accounting Trade types which was overwriting these tags with empty values.

Files changed:
eagle_default\in\xml\xml-accttrade_buy, xml-accttrade_cancel, xml-accttrade_capfundinvestments, xml-accttrade_conversion, xml-accttrade_deliver, xml-accttrade_fxfwd, xml-accttrade_fxspot, xml-accttrade_receive, xml-accttrade_sell, trd_sec_columns,
eagle_default\in\include\ csv-buy_columns, csv-capfundinvestments_columns, csv-conversion_columns, csv-deliver, csv-receive, csv-sell_columns.

 

SDP-10499: Redesign the ExchangePrice rule to include the logic from the RDC_PRICE_UPLOAD 

The Exchange Price loading rule has been updated to support the accountingValidationFlag with a value of Y and RDC loading process. The Issue PC Price load and Exchange Price load now use common logic for mapping and panel processing.

Files Changed:
eagle_default/in/xml/xml-ref_issuepcprice.xml 
eagle_default/in/xml/xml-ref_exchangeprice.xml
eagle_default/in/xml/xml-ref_objects.rsf

New Files:
eagle_default/in/xml/include/xml-ref_security_pricing_common.inc

 

SDP-18497: Change the workflow_restart logic similar to EJM Monitoring Tool to redirect the workflow execution to the correct engine
The workflow_restart logic was enhanced to redirect the execution of a workflow to either MC or MC2  based on the parameter WORKFLOWENGINE .

If the workflow profile has a parameter WORKFLOWENGINE with a value of Y then the workflow execution will be launched on the MC2 engine.

Files Changed:
eagle_ml-2-0_cm/out/tagvalue/workflow_restart.xml

 

SDP-14021:  The xml-ref_issuepcprice.xml does not have mapping for the EXassetResolutionOption

A series of mapping for the following elements that were previously included were missing in the xml-ref_issuepcprice.xml: assetResolutionOption, battingOrder, securityAlias. The tagNochangecase was missing for the exchangePrice.

The issue was fixed in SDP-10499. The Issue PC Price load and Exchange Price load now use the common logic for mapping and the panel processing.  The missing elements have been restored. 

 

SDP-18942:  Improve performance of reference data extracts by applying a Bind Variable for the Max Effective Date query


Max Effective Date sub query didn't have bind variable. This affected the  query execution time.
Support for bind variables was added for the Max Effective Date sub-query for all extracts which supported this option.

Affected extract list:
- ENTITYEXTRACT – history mode
- ISSUEPRICEEXTRACT
- ISSUEEXCHANGEPRICEEXTRACT
- ISSUERORGEXTRACT– history mode
- ISSUERRATINGEXTRACT
- ISSUEFXPCRATEEXTRACT
- ISSUEFXRATEEXTRACT
- GENCORPACTIONSEXTRACT
- SCHEDULEEXTRACT
- FACTOREXTRACT
- SMFEXTRACT– history mode
- ISSUEVARRATEEXTRACT

Changed file:
extract_maxeffdate_tables.inc

 

SDP-18684:  Issuer Ratings Extract issue: missing include file in EagleML
EagleML package didn't contain a link to the  file - t_issuer_identifiers_fields_streaming_json.inc, whic h caused an issue when only EaglML package was installed without EXTRACTSERVICE package.
The missing link was added to the include file.

 

SDP-18615:  Profiled Generic Load should distinguish interfaces with similar names correctly

When the names of interfaces were similar as in the case of USER and USERGROUP, the send_to_wrf step of eagle_wrf_generic_load_profile_executor workflow was not distinguishing it correctly.

The logic was changed to fix this issue.

Changed files:

eagle_ml-2-0_cm\unpacker\send_to_wrf.xml