EagleML Release Notes - August 2017

This release uses XSD Revision 265.

Enhancements

EagleML

Added Extract Filter to Warehouse Broker Trade Quote Extract

A logical filter was unavailable to filter the Broker Trade Quote extract, which restricted extracting an individual record upon loading in multiple records at the same time.

A new task parameter, masterTicketNumber, was added in RTR for adding the new filter. The code was enhanced in the Warehouse Broker Trade Quote objects to reflect the addition of the masterTicketNumber parameter.
        
This issue was tracked as Item #143412.
        
Enhanced Mapping for Tag 9153 for Payment In Kind Preferred Element

The tag 9153, defined in eagle_default/in/xml/include/system_flags.inc, was hardcoded to "MC_MODE" in the system_flags.inc include, resulting in an error in some accounting events where :9153: != "MC_MODE".

The mapping for tag 9153 was enhanced for the Payment In Kind preferred element in the xml-ref_ca.xml rule.
        
This issue was tracked as Item #152777.
        
Enhanced WRHSNAVMONTHENDEXTRACT with Additional Field Elements

Certain field elements were added to the Prior NAV Month End information:
The fields include:                                        Desired EagleML Element Name:
Prior Net Asset Amount                                  priorNetAssetsNAV
Prior Public Offering                                       reportingPublicOfferPrior
Prior Raw NAV                                               priorRawNAV
Prior Reporting NAV                                      reportingNavPrior
Prior Shares Outstanding                              priorSharesOutstanding
Prior Valuation Date                                      priorValuationDate
Raw NAV Month over Month Change           rawNAVMonthOverMonthChange
Reporting NAV Month over Month Change   reportingNAVMonthOverMonthChange

The reportingNavChange field was added to the Nav Report Details, along with reportingPublicOfferPrior and reportingNavPrior fields which were added to the Nav Report Details/Prior NAV. The WarehouseNAVMonthEnd interface was enhanced with the fields from the Nav Report Details/Prior NAV and reportingNavChange.

The XSD was enhanced to reflect these additions.
        
This issue was tracked as Item #152872.
        
Added Support for Entitlements for Additional Extracts

Support was added to entitlement/usergroups in Warehouse Nav Multi (WRHSNAVMULTIEXTRACT) and Warehouse GL Balances (WRHSGLDETAILPOSTINGEXTRACT) extracts.

The following flag was added to support the new filter:

before the call of include for entity filters (extract_filter_entity.inc):

Support was also added for deletion of the Warehouse Multi Nav and Warehouse GL Balances records in other partitions in the 'eagle_ml-2-0_default_in_xml_delete_objects' stream.
        
This issue was tracked as Item #152902.
        
Added BusinessTaskID to Header Node of Extracts

The ability to extract security changes periodically as they occur and push them downstream was unavailable with the CorrelationID output in the header node of extracts. A periodical trigger of the extract was required by Automation Center in order to facilitate the publishing of a comprehensible extract output by adding the BusinessTaskID with the CorrelationID.

The core EagleML code was enhanced to include BusinessTaskID node for addition into extacted data. BusinessTaskID will not be enabled by default.

The following parameter needs to be included in the custom config include:

This issue was tracked as Item #152971.
        
Added Configuration Parameter to startDate from Issuer Role

The startDate column was being overwritten from the existing record from the incoming message. The eagle_default/in/xml/include/xml-ref_issuerrole_common.inc include was enhanced. The W_ISSUER_ROLE_USE_DB_START_DATE config parameter was added, with the default value set to Y.

The following code was enhanced:

This issue was tracked as Item #153080.
        
Added Ability to Lookup Issuers Using Multiple Issuer Data Elements

The ability to look up an Issuer using the Issuer_ID, Issuer Xref fields, LEI, INDUSTRY_CODE, COUNTRY_INCORPORATION, and CURRENCY data elements was previously unavailable for the Issuer Org, Issuer Ratings, LEI, Issuer Analytics, Issuer Roles, and Issuer Relationships data categories. EagleML was enhanced to include these now.

The Issuer resolution was added for the Issuer Organization, Issuer Rating, Issuer Role, and Issuer Relationship categories. The Issuer was resolved by Currency, LEI, Industry Code, Country incorporation via the listing of the required fields in the issuerResolutionOption element.

For example:

There is one more option supported - MATCH_ALL:

If MATCH_ALL is used, the system will try to find an Issuer which possesses all Xrefs listed in both issuerBattingOrder, where Xref priority is defined, as well as the incoming message.

Note that if more than one Issuer is returning based on the results of the resolution, an error is returned specifying "too many records".

The following files were enhanced to reflect these changes:

• xml-ref_issuerorganization.xml

• xml-ref_issuerrating.xml

• xml-ref_issuerrelationship.xml

• xml-ref_issuerrole_main.xml

• issuer_xref_resolution.inc

This issue was tracked as Item #153336.
        
Enhanced the Loading of Issuer Data in History Mode

The ability to only load Issuer data in history mode into the issuer_organization history table was added.

The xml-issuerorganization.xml rule was updated with added changes to support history mode.
        
This issue was tracked as Item #153337.
        
Enhanced EagleML Common Model to join ESTAR.ESTAR_PORTFOLIO_VALUATION to Warehouse Position and Open Lot Extract Objects

The XSD was updated with a new model for Open Lot and Position extracts. The extract_holding_ext_port_valuation_fields.inc include was created and added references in the Position and Open Lot extracts.

The common model for Position and Open Lot reflects the data from the ESTAR.ESTAR_PORTFOLOIO_VALUATION using when the IncludePortfolioValuation parameter is set to Y. The Prior Effective Date, Prior Price Local, Last Prince Change Date, Days Price Unchained, and Prior Exchange Rate elements were added.

The default for IncludePortfolioValuation = Y for Open Lot and Position.

For example:

New mapping was added to the following: t_warehouse_openlot_streaming.xml, t_warehouse_position_streaming.xml, t_warehouse_position.xml, t_warehouse_openlot.xml rules, and the t_holding_portfolio_valuation_fields.inc and t_holding_portfolio_valuation_fields_streaming.inc includes.
        
This issue was tracked as Item #153530.
        
Enhanced EagleML Common Model to Resolve Price Type and Price Source for ESTAR.ESTAR_PORTFOLIO_VALUATION

The XSD was enhanced with a new model to include Price Type and Price Source in resolution for the ESTAR.ESTAR_PORTFOLIO_VALUATION table.

The code was enhanced to reflect these changes:

New mapping was added to the following: t_warehouse_openlot_streaming.xml, t_warehouse_position_streaming.xml, t_warehouse_position.xml, t_warehouse_openlot.xml rules, and the t_holding_portfolio_valuation_fields.inc, t_holding_portfolio_valuation_fields_streaming.inc includes.
        
This issue was tracked as Item #153579.
        
Enhanced EagleML Common Model to Join to ESTAR.MONTHLY_PORTFOLIO_VALUATION to Retrieve Prior Effective Date Data Elements

The common model was expanded to include four new fields populated by the ESTAR.MONTHLY_PORTFOLIO_VALUATION table:

• ESTAR_PORTFOLIO_VALUATION2.TOT_URGL_B

• ESTAR_PORTFOLIO_VALUATION2.SECURITY_GL_LOCAL

• ESTAR_PORTFOLIO_VALUATION2.SECURITY_GL_BASE

• ESTAR_PORTFOLIO_VALUATION2.URGL_CURRENCY_B

These elements will be on the WRHSPOSITIONEXTRACT object only.

The following fields on the WRHSPOSITIONEXTRACT object are populated when IncludePortfolioValuation = ALL:

• priceSource

• priceType

• priorEffectiveDate

• priorPrice

• lastChangeDate

• daysPriceUnchanged

• priorExchangeRate

• priorUrglSecurityBase

• priorUrglSecurityLocal

• priorUrglCurrencyBase

• priorTotalUrglBase

This issue was tracked as Item #153790.

Enhanced EagleML Common Model to Resolve Price Type and Price Source for ESTAR.ESTAR_MONTHLY_VALUATION

The XSD was enhanced with a new model to include Price Type and Price Source in resolution for the ESTAR.ESTAR_MONTHLY_VALUATION table.

The code was enhanced to reflect these changes:

New mapping was added to the following: t_warehouse_openlot_streaming.xml, t_warehouse_position_streaming.xml, t_warehouse_position.xml, t_warehouse_openlot.xml rules, and the t_holding_portfolio_valuation_fields.inc and t_holding_portfolio_valuation_fields_streaming.inc includes.
        
This issue was tracked as Item #153791.
        
Enabled Signage Flag Support for TRADED_SHARES Element in SUBSCRIBEREEDEMEXTRACT

The calculated field CALC_TRADED_SHARES was added into the SUBSCRIBEREEDEMEXTRACT when using the IncludeSubscribeReedemCalcFields parameter:

A comparison of the data available for Mutual Fund Implementation revealed a requirement that the database value be multiplied by -1 when the SUB_RED_INDICATOR value element was populated with 'R'.

The following  logic was implemented to reflect these changes:

This issue was tracked as Item #153989.

EJM

Enhanced Validation for Addressing RUNPREPROCESSOR Failure in the eagle_ml-2-0_default_cm_task_reporter Stream

The RUNPREPROCESSOR command failure resulted in nothing being triggered due to a missing filemask transmission. A validation for RUNPREPROCESSORs was unavailable to check for failure in specific processes and stop subsequent steps from starting.

The preprocessor_check.inc include was developed alongside enhancements to the task_reporter.xml rules to address this failure:

This issue was tracked as Item #152631.
        
Implemented the Management and Cancellation of Workflows and EJM Processes

Functionality was introduced to cancel single streams by deleting the processing file from the processing folder of a stream directory, updating the status of the processing Message Center task and message in the MSGCENTER_DBO.MSG_MESSAGE_STAT and MSGCENTER_DBO.TASKS in order to mark them as completed, and restarting the Message Center processor.

The functionality to cancel an EJM process is identical to the cancellation for single streams, with the addition of sending a trigger to the task reporter in order to have the EJM process finish correctly.

An example of trigger data:
CorrelationId:CORRELATION_ID_FROM_RTR:BusinessTaskId:CORRELATION_ID_FROM_RTR:ProcessCorrelationId:CORRELATION_ID_FROM_RTR:TaskParams:FileStream;TARGET_STREAM!DataFileName;PROCESSING_DATA_FILE_NAME!:varTaskStatus:FAILED:varErrorDescr:Task cancelled:IsEOFW:1:

Canceling an entire workflow was implemented by collecting a list of correlation IDs for all executed subtasks of a workflow, and updating the necessary PROC_STATUS and ORCH_STATE in ORCH table for the task, as well as sending a trigger file to the eagle_ml-2-0_default_cm_task_reporter stream. The removal of the procwaitstate_, procstate_, proctasks_ files for current task is then followed by the cancellation of global events and log records for the workflows.

Note: For some workflows the file should be only deleted from the stream to cancel, for example, the inbound stream from loading data, as well as streams such as eagle_ml-2-0_default_cm_datafile_control or eagle_ml-2-0_default_cm_parallel_exec prior to cancelling the file from the processing folder of the stream.

The ability to cancel a workflow substep are identical to the steps for cancelling an EJM process, by updating the workflow specific features in ORCH tables, triggering processing, and canceling global events.

The cancellation process uses the eagle_ml-2-0_default_cm_control_process stream to in order to set the pause value. The eagle_ml-2-0_default_cm_execwkfl stream controls all steps or substeps of the workflows. The ability to pause and resume the workflow is faciliated by the use of the CorrelationID selection.

Restarting Message Center is completed by pausing all events, restarting Message Center, and resuming all events.

The following parameters were added:

• W_RETRY_CNT_WORKFLOW_PAUSED_CHECK is a w_config parameter to set count of cycle. The default value is 30.

• W_WAIT_TIME_WORKFLOW_PAUSED_CHECK is a w_config parameter to set time of waiting of changing Pause Status. The default value is 10 seconds.

This issue was tracked as Item #154171.

I2I

Added Ability to Trigger Processes from Hub

A new I2I workflow was created, eagle_wrf_i2i, with default profiles in the following cases:

• eagle_wrf_i2i_full.profile

• eagle_wrf_i2i_delta.profile

• eagle_wrf_i2i_sec_alias_sync_fwd.profile

• eagle_wrf_i2i_sec_alias_sync_non_fwd.profile

These profiles replaced the curent default workflows, which will be excluded from the Default Package, to make way for support for only the eagle_wrf_i2i:

• eagle_wrf_i2i_sec_alias_sync.wrf

• eagle_wrf_delta_extract.wrf

• eagle_wrf_full_extract.wrf

The new workflow depends on flags which determine whether Extract or Load will be launched remotely.

For example:

• If W_I2I_IS_HUB = YES, then the GenericExtract will be launched locally and the Generic Load remotely.

• If W_I2I_IS_SPOKE = YES, then GenericExtract will be launched remotely and the Generic Load locally.

A new I2I common include was created and populated with logic for PACKER, ExecWkfl, and Message Distribution. The i2i_common_include.inc include also resolved the credentials for LinkId and SpokeSID. The i2i_update_wkfl_params_by_spoke_sid.inc and i2i_hub_process.inc includes were excluded from Packages.
        
This issue was tracked as Item #151941.
        
Implemented Web Service Transmission of Control Messages in I2I Workflow

The ability to change the destination source and navigate among destinations was previously unavailable in the hub model. The addition of the LinkId parameter below addresses this issue and will be unique per each SpokeSID.

The parameter wil be added in the workflow and will be set in a profile or in the RTR for the I2I process:

The configuration file for the I2I was enhanced to allow quick switching between destination credentials.

A new I2I common include was created and populated with logic for PACKER, ExecWkfl, and Message Distribution. The i2i_common_include.inc include also resolved the credentials for LinkId and SpokeSID. The i2i_update_wkfl_params_by_spoke_sid.inc and i2i_hub_process.inc includes were excluded from Packages.

This issue was tracked as Item #154061.

Software Fixes

EagleML

Issue with Warehouse Cash Projection Incorrectly Passing Amounts to EJM Extract

The maturityDate, accountingBasis, prinAmount, and cfUserFloat11 amounts were not being extracted despite being in the incoming EagleML data file.

The following rules were enhanced to address this issue:

• t_warehouse_cash_flow_proj_streaming.xml

• t_warehouse_cash_flow_proj.xml

• extract_warehouse_cash_flow_proj.xml

This issue was tracked as Item #137031.
        
Added Restriction to CorrelationID Character Handling

The EagleML definition for CorrelationID was xsd:normalizedString, which allowed slashes and backslashes. Restrictions for these characters are required since CorrelationID is used as part of the file names.

The XSD was enhanced to reflect a check for CorrelationID in the runtaskrequest.xml rule.
        
This issue was tracked as Item #151846.
        
Resolved XSD Conflict for IncludeExtendedDetails Duplicate Element

The sharesOutstanding element, contained in the common XSD group ExtendedAssetProperties.model and in the extract_asset_properties_fields.inc include, had a mapping in the SECURITYDBO.EQUITY database table. The sharesOutstanding element was also mapped to the HOLDINGDBO.MONTHLY_ESTAR_NAV database table and contained in the WarehouseNAVMonthEnd.model and in the T rules of the extract.

The sharesOutstanding element was renamed to navSharesOutstanding in the WarehouseNAVMonthEnd.model.

The navSharesOutstanding element was moved to the NavSharesOutstanding.model in the XSD:

The XSD schema, inbound, and t rules have been enhanced to reflect these changes.
        
This issue was tracked as Item #152614.
        
Fixed Entity Filters While Running INSURANCEVALUATIONLOTEXTRACT in Held Mode

An issue arose where entity filters were ignored in situations where the Insurance Valuation Lot Extract was running in Held Mode.

The extract_filter_entity.inc include was enhanced in order to support the held filter usage in entities.

The following parameter was added to address the change in the extract_warehouse_istarlot.xml rule:

This issue was tracked as Item #153105.
        
Added Parameter to Filter INSURANCEVALUATIONLOTEXTRACT by MONTH_END_DATE

The Update_Date parameter was the only active date parameter for instances where a client was using a CustomWhere clause while running the Insurance Valuation Lot Extract. The entry for the Month End Date was missing in the eagle_ml-2-0_cm/out/tagvalue/extract_warehouse_istarlot.xml file.

The code was updated to include a Month End Date filter for extracts.

This issue was tracked as Item #153108.
        
Issue with Issuer Role Details For Existing IDs and Small Aliases

Existing Issuer IDs coupled with small Issuer ID aliases have resulted in an error showing a discrepancy between the request result as an existing Issuer alias.

The code was updated to reflect changes made to the query result:

was changed to

This issue was tracked as Item #153175.
        
Resolved Final TSR Information Missing TransactionID for VoluntaryCAElection Object

Sending a TransactionID causes an RTR failure which does not return the TransactionID when using the VoluntaryCAElections object.

The xml-ref_voluntarycaelections.xml rule was enhanced to reflect the addition of TransactionID content in the final TSR information:

This issue was tracked as Item #153512.
        
Resolved Issue Passing in SourceName Parameter with WRHSGLDETAILPOSTINGEXTRACT

The parameter passed in on the RTR, such as accountingdate, does not persist through processing.

For example, MSTARDIRECT sourcename is being changed to STARDIRECT by the execution of the Sources Filter.

The logic in the sourcename/datasources filter is modified to reflect cases in which:

• No datasources/sourcename filter set in the extract CM old logic which would be used for datasources override.

• Datasources/sourcename filters are set in the incoming CM (datasources have higher priority and would always override sourcename from the incoming settings), the source filter would be set according to the incoming settings.

This issue was tracked as Item #153723.
        
Resolved Syntactical Error in Issuer Organization Rule

The varNGParamValues_Add variable was missing a colon in the first line:

The rule was updated to include proper synatx.
 
This issue was tracked as Item #153884.
        
Issue with Pledge Information on  Warehouse Custody Extract

The SECURITY_MASTER table was being used to get pledgeName and pledgeType, connected by the CUSTODY_BALANCE.SECURITY_ALIAS field instead of CUSTODY_BALANCE.PL_SECURITY_ALIAS.

To resolve this issue, the CUSTODY_BALANCE.PL_SECURITY_ALIAS replaced the CUSTODY_BALANCE.SECURITY_ALIAS for connection to the SECURITY_MASTER table.
        
This issue was tracked as Item #153949.
        
Issue withl Date Fields in Extracts

Key date fields were returning a minimal value of 1900-01-01 on the Warehouse Position and Open Lot Extracts.

The logic was updated to address the population of Position and Open Lot fields with the 1900-01-01 value. Date fields with 1900-01-01 value will be returned as NULL if the ExcludeDefaultDate = 'Y'.

The following parameter was added:

This issue was tracked as Item #153968.
        
Resolved PACE_MASTER_DBO.OW.POST_ORCH_REQ Stored Procedure Expecting Missing Arguments

The PACE_MASTER.DBO.OW_POST_ORCH_REQ stored procedure expected 13 arguments and receiving only 12, produced an error on the discrepancy of expected and received parameters. The issue was only found in versions 12.x.

The stored procedure was enhanced to address the case in 12.x environments in which 13 arguments were expected to be passed, when only 12 were available. The nugg_upd_params.inc was updated to include this case.
        
This issue was tracked as Item #154336.
        
Applied GenericCA Local Fix Resolution for Missing Xref Identifier Error to EagleML

Additional logic that was applied in a previous release affected tag 302 (toAssetID), which is used in almost all panels in xml-ref_ca.xml for load data. The calculation resulted by Asset Identifiers data from <toAsset> node was fixed to reflect the ability to directly take calculation from /toAsset/securityAlias node.

Additional logic was added to check for <toAsset> node existence for resolving to_security_alias, and to accomodate empty <toAsset> node processing.
        
This issue was tracked as Item #154345.
        
Added Support For Additional Core SP Parameters for GenericSMF

A discrepancy was found between the bind arguments from the database for INSERT_UPDATE_SECURITY_INFO stored procedure and the stored procedure arguments in certain regions. The following arguments did not have bind definitions:

• IN_FI_FIXING_DT_BUSINESS_CENTER

• IN_ASM_NAIC_NAIC_CAP_STRUCTURE_CODE

• IN_XREF_END_DATE

The XSD schema and rules were updated to reflect this change. The rules and XSD files are contained in the field for mapping to stored procedure parameter IN_FIXING_DT_BUSINESS_CENTER.

This issue was tracked as Item #154540.
        
Fixed Discrepancy in XSD Output from Share Classes Within the Same Fund

An issue occured when executing the WRHSNAVMULTIEXTRACT with the IncludeAverageNetAssets set to Y where the correct SQL was executed, generating the correct DAT file, but an incorrect XSD output. In the case where two share classes are within the same fund, the expectated result was to have distinct outputs. Changes were made to the t_warehouse_nav_multi_streaming.inc include to address this issue.

This issue was tracked as Item #154827.

Resolved Invalid Identifier Error During PL/SQL Developer Execution

During execution via PL/SQL Developer, an ISNULL: invalid identifier error is thrown on the portion of the Select statement in the SQL that is generated and stored in the /data/msgcenter/cmw_staging_folder/ when running the SUBSCRIBEREDEEMEXTRACT object.

The error reads:
AND (ISNULL(ESTAR_SUBSCRIBE_REDEEM.CANCEL_FLAG, 'N') != 'Y')

The code was changed to NVL in order to address this issue.
        
This issue was tracked as Item #154899.

EJM

Issue with TSR File Naming

The TSR file naming convention overlapped with the RTR file naming, causing decreased stability in infrastructure file watchers.

Changes were made to keep the TSR file naming scheme kept independent of the RTR file. The task_report_sender.xml, task_reporter.xml, taskacknowledgement.xml, wrkflw_subprocess_report.inc files were updated to address this issue.
        
This issue was tracked as Item #151987.
        
Issue with Corr_ID Duplication in EAGLE_WRF_GENERIC_LOAD Workflow

A final TSR message was not being sent from the generic_load workflow. Due to the processing error, the reporting step was not executed. The final TSR was then re-created, causing duplicate Corr_ID in some runs. The issue persisted only when the incoming archives contained only Warehouse or Accounting files.

The workflow was modified to exclude the problem case when the fork link leads to the join. The workflow was enhanced to reflect this resolution.
        
This issue was tracked as Item #154099.
        
Adjusted PACE_MASTER.DBI.ORCH_REQUEST_DEF Table to Reflect Left Outer Join

An error occured due to an inner join on the PACE_MASTERDBO.ORCH.REQUEST_PARAMS table. The issue manifested in the case of a consolidated TSR being set as a a global parameter for the workflow.

The code was updated to reflect a change from an inner join to a left outer join for the PACE_MASTER.DBO.ORCH_REQUEST_DEF table.
        
This issue was tracked as Item #154643.