EagleML Release Notes - July 2017

This release uses XSD Revision 263.

Enhancements

EagleML

Set SMF Generic Tag 12 to SHARES for SWCOTR SMFs
The logic resolution was updated for SMF Generic tag 12 so that it is now set to SHARES for SWCOTR SMF. The eagle_default\in\xml\include\generic_smf_fields_logic.inc include was adjusted to reflect this change.

This issue was tracked as Item #137956.

Added Fields from ESTAR.AVERAGE_NET_ASSETS Table to WRHSNAVMULTIEXTRACT
Fields were added from ESTAR.AVERAGE_NET_ASSETS to WRHSNAVMULTIEXTRACT based upon data in the AUDIT status. The join is by PORTFOLIO_ACCT, SHARE_CLASS, VALUATION_DATE, NAV_SEQUENCE and STATUS (in AUDIT status). A new RTR parameter IncludeAverageNetAssets was added to include and exclude average net assets extensions. The extract_warehouse_nav_multi.inc include, t_warehouse_nav_multi.xml rule, t_warehouse_nav_multi_streaming.xml rule, and t_warehouse_nav_multi_streaming.inc include were enhanced to address this.

This issue was tracked as Item #151714.

Enabled Ability to Publish Date Parameters
EagleML was enhanced to include the ability for extracts to capture and publish the dates sent with the RTR on the final output. To reflect this enhancement, a new model has been added to the XSD. There are two options to enable this logic.
1. You can specify the parameter in RTR for single extract – IncludeCommonTransactionsInfo = Y, N (N by default):

2. You can specify global list of FeedType which should include this information in the w_config_custom.inc:

Note: When passing in a date range in the RTR, both values from***date and to***date should be passed in or a singular ***date. For example: RTR passes fromaccountingdate, fromeffectivedate, and toeffectivedate. The value in extract will be: reportType = EffectiveDate and reportBeginDate and reportEndDate corresponding with the effective date range.

This issue was tracked as Item #152056.

Enabled Support for Additional Identified Calculations for SUBSCRIBEREEDEMEXTRACT
New calculated fields including GrossAmountBase, GrossAmountLocal, NetAmountBase, NetAmountLocal, ParValue, PaidInSurplusBase, PaidInSurplusLocal, and NavImpactPerShare were added to the Subscribe Redeem Extract. To enable these fields, the parameter IncludeSubscribeReedemCalcFields must be added to the RTR:

This issue was tracked as Item #152262.

Changed eagleml_smf_notes XSLT and xml-ref_secnotes Rule to Use Batch Logic
The code was enhanced to reflect changes in the SEC Notes feeds in batch loading. Logic was implemented for removal of all records for specific security_alias fields for the batch. A new rule batch load xml-ref_secnotes_mi.xml and new template eagleml_smf_notes_batch.xsl was created for this enhancement.

This issue was tracked as Item #152339.

Adjusted Process for Multiple Xref Identifiers Loading from smf_generic Rule
The ability to load multiple XrefIdentifier out of the SMD generic rule was unavailable. Only the eagle_default/in/xml/xml-ref_issuexref.xml rule was available, which loaded only one identifier per file.
The addition of a new rule for type SMFIdentifiers allowed for execution of full security resolution. The code was enhanced in the xml-ref_objects.rsf with the addition of:

The XSD was enhanced which reflected an addition of elements to the eagleml-ref-2-0.xsd and a defining of elements to groups in eagleml-sec-shared-2-0.xsd.

This issue was tracked as Item #152344.

Enhanced Wash Sale Activity to be Reflected in the Warehouse Tables
The ability to utilize Wash Sale functionality was previously unavailable from the TRADESDBO.TRADE_WITH_WS and HOLDINGDBO.LOT_LEVEL_WS_POSITION tables. Access to Position Analytics, Open Lot, and Trade was provided for use in extracts.
The following files were created or updated as part of this enhancement: t_warehouse_openlot_streaming.xml, t_warehouse_openlot_streaming.inc, extract_warehouse_openlot.xml , extract_warehouse_washsale_trade.xml, extract_warehouse_washsale_trade.inc, t_warehouse_washsale_trade_streaming.inc, extracts.rsf and t_eagleml-2-0_streaming.rsf.

This issue was tracked as Item #152633.

Updated Generic Issue Analytics Data Object
The following fields in the SECURITY_ANALYTICS_USER table were added to the Generic Issue Analytics Data outbound object: USER_SA_CHAR16, USER_SA_CHAR17, USER_SA_CHAR18, USER_SA_CHAR19, USER_SA_CHAR20, USER_SA_CHAR21. USER_SA_CHAR22, USER_SA_CHAR23, USER_SA_CHAR24, USER_SA_CHAR25, USER_SA_CHAR26, USER_SA_CHAR27, USER_SA_CHAR28. USER_SA_CHAR29 and USER_SA_CHAR30.
Mapping of these fields have been added into the t_generic_issue_analytic.xml outbound rule. Note: To leverage this EagleML change, you must also have the Eagle core version where these fields are available.

This issue was tracked as Item #152651.

Enabled Calculation to Include Formula for the Transfer Agent Mill Rate and Calculation for Raw Nav Change
The calculation to produce the Transfer Agent Mill Rate was previously unavailable for the WRHSNAVMULTIEXTRACT. The transferAgentMillRate and rawNavChange fields were added to the WarehouseNavMulti interface. The reporting of these fields is done by IncludeReportDetails and the enablecalculation flags.

This issue was tracked as Item #152680.

Added Additional Fields from ESTAR.AVERAGE_NET_ASSETS Table to WRHSNAVMULTIEXTRACT
The code was enhanced to include additional fields for the ADJ_PTD_CUMULATIVE, PRIOR_DAY_NET_ASSETS, and PD_PTD_CUMULATIVE columns into the ESTAR.AVERAGE_NET_ASSETS table. The extract_warehouse_nav_multi.inc include, t_warehouse_nav_multi.xml rule, t_warehouse_nav_multi_streaming.xml rule, t_warehouse_nav_multi_streaming.inc include, and the extract_fields_query.inc include were modified to reflect these changes.

This issue was tracked as Item #152687.

Expanded the NAV Daily Extract to Include Additional Fields from the Distribution Log

EagleML Warehouse NAV Daily required further expansion to include corresponding Distribution Rates, Distribution Shares, and Distribution Type to the previously included Distribution Amounts consisting of Category, Tax Indicator, and Type from the Estar Distribution Log by EntityID, Accounting Date, and Share Class.

The code was updated to reflect changes in the extract_nav_multi_details_fields.inc and extract_nav_multi_details_tables.inc includes, as well as the new mapping added to the t_warehouse_nav_multi.xml, t_warehouse_nav_multi_streaming.xml rules, and eagle_ml-2-0_default_out_q, eagle_ml-2-0_default_out_t, eagle_ml-2-0_default_out_streaming_t streams.

This issue was tracked as Item #152791.

Revised Traded Cash Extract to Default the GAIN_CURRENCY and LOSS_CURRENCY to 0

The GAIN_CURRENCY and LOSS_CURRENCY fields were being populated in the ESTAR.ESTAR_TRADED_CASH_ACTIVITY table. The currency gain or loss was only recognized once the activity is settled, as it was the currency impact from the fx rate change from trade date to settle date.

The logic parameter EnableSignageLogic was updated and Traded Cash Extract was revised to default GAIN_CURRENCY and LOSS_CURRENCY to 0. To enable this feature on the Traded Cash Extract, the EnableSignageLogic set to B must be included on the RTR:

This issue was tracked as Item #152844.

Expanded Security and Entity Extended Model to Include FWD_BROKER_CODE and MFF_ENTITY

The Security extended model was expanded to include the Forward Broker Code (FWD_BROKER_CODE) and the Entity Extended Model was expanded to include the Master Feeder Entity flag (MFF_ENTITY). The fields are SECURITYDBO.SECURITY_MASTER_DETAIL.FWD_BROKER_CODE and RULESDBO.ENTITY_STAR_PROCESSING.MFF_ENTITY.

This issue was tracked as Item #153135.

Extended IncludeExtendedDetails Content for Additional Field Elements

The SECURITYDBO.FIXED_INCOME and SECURITYDBO.SECURITY_MASTER tables were populated with the Issue Date, First Coupon Date, Last Coupon Date, Private Placement, and Default Date elements in the Security Common model.

This issue was tracked as Item #153408.

Enhanced Message Center Stream To Handle Multiple Overrides

The ability to drop files with a multitude of a different override records, update the rdc_overrides table, and recomposite affected data elements was required for handling overrides quickly in the system.

The Start Date and Begin Date logic was updated to reflect rdc_overrides record inserts, and checked for overlapping on begin_date and end_date. The data is submitted to be recomposited for the affected securities whose effective date is in the range of the begin and end date of the override.

This issue was tracked as Item #153687.

Spelling Error Corrected on reinvPostInsicator in Distribution Activity Extract

The ElementName and XPath were corrected to reflect proper spelling of the reinvPostIndicator and EagleML/AccountingTransaction/distributionActivity/reinvPostIndicator fields respectively.

The XSD schema /eagleml-acc-shared-2-0.xsd was updated, along with rule changes in the /eagleml/out/xml/t_acct_distribution_activity.xml, /eagleml/out/xml/t_acct_distribution_activity_streaming.xml, and /eagleml/out/xml/t_acct_distribution_activity_streaming.inc files.

This issue was tracked as Item #153840.

EJM

Implemented Process Termination in EJM for Single Stream

EJM was enhanced with the ability to terminate running extracts, as well as inbound and outbound single stream on the database server.
The following parameters are available forthe RTR for Cancel Single Stream:

• ActionType = ‘CANCEL’ (required)

• StreamName = the name of stream, where cancel steps for single stream will be implemented (required). The exact stream name will be known when the development is complete

• CancelStreamName = The name of stream to cancel (required)

• CancelMsgStatId = Message Stat Id of stream to cancel (optional). If empty, all running streams with name CancelStreamName will be cancelled.

For this enhancement, the eagle_ml-2-0_default_cm_control_process stream was created. 

It contains the following:

• control_process.rsf

• control_process.xml

• control_restart_services.inc

• control_workflow_paused.inc

• control_cancel_process.inc

• control_cancel_stream.inc

• control_get_params.inc rules and includes

Also, the execute_wrkfl_event.xml rule was updated for eagle_ml-2-0_default_cm_execwkfl stream.

Below is an example of a Cancel Single Stream RTR:

This issue was tracked as Item #142658.

Provided Retry Capability for Message Delivery Failures

Previously, EJM did not have the ability to attempt to deliver a message (extract, ack, tsr) via JMS when the destination queue is unavailable. This resulted in the the message delivery to fail and no retry executed.

The code was enhanced by adding retry logic to the /eagle_ml-2-0_cm/reporter_dist_methods.inc include, the retry attempts and wait time of which can be adjusted in w_config_custom.inc in the W_CONFIG_JMS_TSR_RE_TRY_CNT and TSR_RE_TRY_WAIT fields respectively. Support was added for a generated error in case the stream was not able to send the message to the delivery JMS stream. Support to identify delivery method as JMS for TSR and to run mixed pay load processes for asynchronous execution directly to the inbound stream was also reflected in an update to the /eagle_ml-2-0_cm/in/xml/eagleml_load.xml rule.

This issue was tracked as Item #148064.

Enhanced Success Condition for TSR When No Data Returned

Calling the eagle_ml-2-0_default_out_q stream returned a success in the final TSR if the extract had returned no data.

The following flags were introduced in order to reflect the ability of defining settings for global and separate message execution:

W_RETURN_ERROR_NO_DATA_CASE – May be set as Y/N. In the case of Y, each time extract returns NO_DATA, the editcheck would fail the q stream inside the loop. In the case of N (N would be default), the record would return success and a simple NO_DATA message.

SetNoDataAsError – New task parameter for extract CM which can be set as Y or N. It is the same for the w_config_custom flag, but should be set in execute CM and override flag in w_config_custom.

The extract2t.inc include was modified for this enhancement. In addition, the SQL code was modified for collecting files in consolidated TSR creation.

This issue was tracked as Item #150579.

Enhanced EJM for Compatibility with RDC Single Security Composite Feature

EJM was enhanced to be compatibility with the RDC Single Security Composite Feature.  Note: The EJM feature is dependent on having the necessary Eagle core version where the RDC Single Composite Feature is available.
In certain workflow scenarios, you may have the need to run the RDC process for either a single security, or a list of several securities.  For example, when applying trades to new securities where you first need to make a vendor call to get the SMF information and run the security through RDC before releasing the trade. In order to help the execution of the RDC portion of the workflow to be more efficient, Eagle has a core stored procedure that can be used to submit the RDC run for specific securities. Note: This procedure should be used for submitting single or several securities at once. It is not intended for a large number of  securities in a single submit.

A sample of the procedure execution and each individual input is shown below:

Input descriptions:

• in_process_name: This is a s user-defined name for the submitted event.  It will appear in the Schedule Name field in the Monitor tab in Automation Center

• in_event_type: Set this to ‘RDC_MAIN’

• in_category_inst: This represents the RDC data category that you are submitting the event for.  It is used interchangeably with the in_category_desc parameter.  If you supply a value for in_category_inst then in_category_desc should be NULL.  The valid values are 1, 2, 3, 4, 5, 6, 7  (1 – Security; 2 – Legal Entity; 3 – Analytics; 4 – Ratings; 5 – Variable Rates; 6 – 3rd Party Cash Flows; 7 – Prepayment Time Series)

• in_category_dec: This represents the RDC data category that you are submitting the event for.  It is used interchangeably with the in_category_inst parameter.  If you supply a value for in_category_desc then in_category_inst should be NULL.  The valid values are: Security; Legal Entity;Analytics; Ratings; Variable Rates; 3rd Party Cash Flows; Prepayment Time Series

• In_data_key_inst: This represents the security_alias, or list of security_alias’ to run the event for (or Legal Entity alias).  If running for multiple securities, the list of securities is delimited by the tilde.  For example: '680021955~680021956'

• in_effective_date: The effective date to submit the RDC event.  This will presumably be the current date if you are requesting the latest SMF from the vendor.  Pass the value in DATE format

• in_eglrdc_pr_mode: This is used to determine whether RDC should look for vendor changes for the security within the lookback days, or whether to run the security for the specified date.  If there is a possibility that the vendor data will be loaded historically, then pass ‘DELTA’, otherwise pass the parameter as NULL

• in_eglrdc_sys_item_44: This is used in conjunction with the in_eglrdc_pr_mode parameter.  If in_eglrdc_pr_mode is set to ‘DELTA’, you can set the number of days to look back for vendor SMF changes.  If you pass the value as NULL, it will default to 7

• in_update_source: This is the user id that will appear in the submitted schedule and will be the update_source represented in all RDC processing results.  This MUST be a valid Eagle user in the pace_masterdbo.pace_users table.

• in_monitor_switch: This is a Y/N switch that determines whether you want to submit the RDC processing synchronously and wait for it to complete before moving to the next stage in the workflow, or submit it asynchronously and continue with subsequent processes regardless of RDC completion or event status.  If you set the switch to ‘Y’ we will monitor the status of the submitted event on a 10 second interval.  The monitoring event has a 5 minute time-out if the RDC event does not complete.

• out_execution_status: This will return the status of the submitted event.  The possible outputs are: D = Done (completed); E = Error (failed); P = in Progress; K = Killed;W = Waiting (not yet scheduled to run); U = Unable to run; I = Inactive (disabled and will not run); ? = Unknown/Unexpected

• out_execution_message: This will return any error message that appears on the parent RDC event.  If there is a failure in the underlying rule event, this will not be returned in this output.  It will need to be viewed through Automation Center.

This issue was tracked as Item #152480.

Software Fixes

EagleML

Multiple Mappings for SMF Generic Tag 1777

There was some questions around the mapping of SMF Generic Tag 1777 , as it was mapped in three files:  smfobjects.ini,  generic_smf_base_model.inc, and fixedincomesmf_model.inc.
The smfobjects.inc include is not actually used for Generic SMF in either Accounting or non-Accounting mode.
The generic_smf_base_model.inc  is present in xml-smf.xml inbound rule, which helps us to upload Accounting SMF objects.  The tag mapping in this include is <COL TAG="1777" EXPRESSION="|pIKIndicator|"/>   
The fixedincomesmf_model.inc is present in xml-smf.xml, after generic_smf_base_model.inc, overwritting tag 1777 for Accounting SMF. For non-Accounting SMF, the generic_smf_base_model.inc is not used, only fixedincomesmf_model.inc. The mapping in fixedincomesmf_model.inc  is<COL EXPRESSION="|prplFlag|" TAG="1777"/> 

This issue was tracked as Item #133135.

Added Delete from TRADESDBO.CORP_ACT_HIST as Part of xml-ref_schedule_mi.xml Rule

The xml-ref_schedule_mi.xml rule file only deleted from SECURITYDBO.SCHEDULE table. However, the ADD_SCHEDULE from ESTAR_PACE package contains an  insert into the TRADESDBO.CORP_ACT_HIST table also and therefore data from this table should be removed before insert as well.

The required logic was added to xml-ref_schedule_mi.xml to reflect this change. The data will now be removed from the TRADESDBO.CORP_ACT_HIST table prior to insert.

This issue was tracked as Item #148018.

Issue with Duplicate Records in Sec Notes Table

The eagle_default/in/xml/xml-ref_secnotes.xml rule had incorrect condition for FOREACH which is responsible for update for Sec Notes. This resulted in an insert of duplicate records into the Sec Notes table.

The code was updated to reflect changes in the update cycle for secnotes in rule xml-ref_secnotes.

This issue was tracked as Item #148020.

Multibasis Activity Extract Duplicating in TRADEDCAEXTRACT and SETTLEDCAEXTRACT

The EagleML Estar Traded Cash and Estar Settled Cash Activity extracts did not have the ability to filter/target a specific basis or the Primary Basis, which posed an issue for entities set up with Multibasis (i.e. USTAX, GAAP, STAT, etc.) which replicate Cash Activity on secondary, tertiary, etc. basis. All activity that was extracted from respective tables per the RTR execution, was receiving the basis that is contained on the RULESDBO.ENTITY table. The value from the Traded/Settled Cash tables from the ACCT_BASIS did not persist, and those values were being over-ridden by the Entity Common Model values.

Code changes were made to the Traded Cash Activity and Settled Cash Activity outbound rules. The common include extract_entity_fields.inc was adjusted to include mapping for the ACCT_BASIS field to the ESTAR_SETTLED_CASH_ACTIVITY table and mapping in T rules. The code was adjusted to add an inner join for the RULESDBO.ENTITY_ACCT_BASIS table, to add a filter by Accounting Basis to Traded Cash and Settled Cash extracts, and add a filter for Use Primary Basis Only. The UsePrimaryBasisOnly task parameter can hold possible values 'Y' and 'N'.

This issue was tracked as Item #151202.

Issues with Extension That Use eagle_ml-2-0_default_in_xml_warehouse_preproc

You could not create extensions that used the eagle_ml-2-0_default_in_xml_warehouse_preproc.

The code was updated to resolve this issue.  Note: Changes were made for Position, Performance and Open Lot only. Also, this change is for Oracle only.

This issue was tracked as Item #151757.

Added Settlement Currency - Tag 63 - to the ML Accounting Trade Stream

The functionality available for adding settlement currency (tag 63) in the panel was unavailable in the ML Accounting Trade stream.

The XSD already allowed for settlement currency, but the trade_aliases.inc include needed to be adjusted to  support tag 63.

This issue was tracked as Item #152533.

Added Mapping for SourceComments (Tag 5669) to the IssuePcPrice Reference Object

The SourceComments field (Tag 5669) was added to the ML reference stream and object type IssuePcPrice. The code was enhanced to reflect mapping updates of pan-price_exchange_upload.htm in pan-pc_security_pricing_upload_eagle and in_source_detail of PRICING_PACE.PRICE_UPLOAD.

This issue was tracked as Item #152562.

Misspelling of Redemption Shares in XSD Element

The field for Redemption Shares was mislabeled as redeptionShares across the element name, xpath, and dbfield. The misspelling also occurred in the HOLDINGDBO.NAV_MULTI.REDEPTION_SHARES table.

The XSD, t_warehouse_nav_multi.xml, and t_warehouse_nav_multi_streaming.xml T rules were updated to reflect proper spelling of redemptionShares.

This issue was tracked as Item #152612.

Resolved XSD Element Conflict for IncludeExtendedDetails and NAV sharesOutstanding

The sharesOutstanding element was duplicated in the Warehouse NAV Multi XSD type.

Changes were made to the code to address this conflict. The sharesOutstanding element in WarehouseNAVMulti.model was renamed to navSharesOutstanding. In addition, the Extended Asset Details were included into the NAV Multi extract in the Q Rule, allowing the inclusion of Extended Details to the extract via the variable. The extract_warehouse_nav_multi.xml, t_warehouse_nav_multi.xml, t_warehouse_nav_multi_streaming.xml rules were enhanced.

This issue was tracked as Item #152613.

Removed Client Specific Content from WRHSNAVMULTIEXTRACT

It was identified that the WRHSNAVMULTIEXTRACT had logic specific to a client as there is activity which is configurable per client and the code used was client specific. This dependency has been removed to improve the query performance.

This issue was tracked as Item #152671.

Several Warehouse Extracts Fail to Return Extended Model When IncludeExtendedDetails is Passed

The Warehouse Extracts WRHSNAVMULTIEXTRACT, WRHSNAVMONTHENDEXTRACT, and WRHSGLDETAILPOSTINGEXTRACT failed to execute the Extended Model when the parameter IncludeExtendedDetails is passed.

The code was updated to resolve this issue.  However, there is also a XSD conflict in WarehouseNAVMonthEnd object, which does not allow you to include security-related Extended Details into WAREHOUSENAVMONTHENDEXTRACT. That issue will be addressed in a future release and is tracked as BT #154159.

This issue was tracked as Item #152690.

Working Trial Balance Extract Missing Element

The EagleML Working Trial Balance Extract was missing the cust ledger source (tag 16420 on STAR Report).
The following was added: EagleML/accountingTransaction/workingTrialBalance/custLedgerSource to address this issue.

This issue was tracked as Item #152696.

Added Ability to Create Separate TSR and Data Files for Synchronous Extracts

The code was enhanced to reflect the ability to create separate TSR and Data Files for synchronous extracts. The eagle_default/in/xml/xml-ref_secnotes.xml rule was adjusted to reflect correct condition for FOREACH and resolve issue of duplicate insertion of records into the Sec Notes table. Support was also added for the EmbeddedExtractData parameter.

This issue was tracked as Item #152804.

Updated xml-ref_issuerrole_common.inc resolution logic

The logic mapping in the xml-ref_issuerrole_common.inc was updated for proper resolution of match all and addition of new records cases.

This issue was tracked as Item #153484.

Adjusted eagle_ml-2-0_default_cm_datafile_control PRF Fileset to Resolve Slow Processing Times

The sequential processing of multiple files took a considerable amount of time.

To address this issue and improve performance, the  eagle_ml-2-0_default_cm_datafile_control prf fileset processing was changed from sequential to parallel processing.

This issue was tracked as Item #153514.

Modified END_DATE and START_DATE Reference Objects to Avoid Being Rewritten

The code was enhanced to prevent END_DATE and START_DATE values from being rewritten. This was an issue in Ref object Schedule and issuer_role.

The extract_schedule.xml and extract_issuer_role.xml files was updated for this change.

This issue was tracked as Item #153891.

EJM

Adjust eagle_wrf_generic_load processing TRD CANCEL files in Warehouse Objects

The eagle_generic_load_warehouse_objects sub process from eagle_wrf_generic_load could only process WAREHOUSETRADE and WAREHOUSECASHACTIVITY files, which neglected the situation in which the file had the original trade and the cancel. This would cause the cancel to be loaded before the trade in error. Further, the warehouse_preproc stream was loading records in parallel mode, which impacted performance.

To resolve this issue, the WAREHOUSECANCELTRADE and WAREHOUSECANCELCASHACTIVITY files in the eagle_generic_load_warehouse_objects were enhanced to follow the load_warehouse_cas_activity and load_warehouse trade tasks.

This issue was tracked as Item #148017.