This release uses XSD Revision 294

Enhancements

Enterprise

EagleML

Expense Activity Extract was not Supported

Expense Activity Extract was not supported.

A new accounting extract (Expense Activity Extract) was created. Support for this extract is available on ORACLE environments only.
We changed XSD.
Following rules were created:
- extract_expense_activity.xml
- extract_expense_activity.inc
- t_acct_expense_activity_streaming.xml
- t_acct_expense_activity_streaming_streaming.inc
Following rules and files were changed:
- w_config.inc
- extracts.rsf
- t_eagleml-2-0_streaming.rsf

This issue was tracked as Item #141895.

Created the Weighted Average Outbound Interface

The new Accounting extract Weighted Average was created.

A corresponding outbound interface was created for the Weighted Average extract. The WeightedAverage (Accounting type) object was added to the XSD. The feed type for the extract is: WEIGHTEDAVERAGEEXTRACT.

This issue was tracked as Item #153459.

TotalNumberShareClasses was Revised to Exclude Closed Share Classes

An enhancement to the Extended Entity Model was required. The totalNumberShareClass field has to exclude any closed Share Classes which was not happening.

A new statement was added for counting the Share Classes correctly: CLASS_STATUS <> 'INACTIVE'
(SELECT COUNT(sc.ENTITY_ID)
 FROM RULESDBO.ENTITY_SHARE_CLASS sc 
 WHERE sc.ENTITY_ID = ENTITY.ENTITY_ID
   and sc.CLASS_STATUS <> 'INACTIVE'
) TOTAL_COUNT_SHARE_CLASSES.

Files changed:
- extract_entity_fields.inc.

This issue was tracked as Item #158990.

IncludeRecordErrorDetails parameter introduced for TSR generation

Client requested an option to avoid unnecessary information in TSR. <record> nodes should be avoided with special flag.

The new IncludeRecordErrorDetails flag was created to dictate how the TSR is generated. If the incoming RTR has IncludeRecordErrorDetails set to N all the <record> nodes will be removed from the TSR unless the extract returns no data.

Files changed:
task_reporter.xml
task_reporter_generate_task_unit.inc

This issue was tracked as Item #159103.

Added Missing Elements to Calendar Holiday Extract

The following elements were missing from the EagleML Calendar Holiday Extract:
- calendarDescription;
- countryCode;
- countryName;
- expirationYear;
- calendarId.

These elements were added to the extract by changing the following files: t_ref_calendar_holiday and t_ref_calendar_holiday_streaming.

This issue was tracked as Item #159957.

Error Handlers Were Added to eagle_ml-2-0_default_cm_w_setup Stream

The eagle_ml-2-0_default_cm_w_setup stream was showing a success status despite critical processing issues.

The w_setup_eagle_version.xml file was updated. Error handlers were added to all read\write and XSL transform operations in the stream.

This issue was tracked as Item #160218.

Updated Condition for ratingTypeIndicator Lookup in the xml-ref_issuerating_all to Use noLoadValidationFlag Switch

There is a lookup in xml-ref_issuerating_all.xml rule, which validates <ratingTypeIndicator> field. We need a flag that can disable this validation, if desired.

The <noLoadValidationFlag> element was added to support this functionality. If set to “Y” the validation will be skipped.

List of files changed:
eagle_default\in\xml\xml-ref_issuerating_all.xml

This issue was tracked as Item #160448.

Added the Ability to Configure and Override Multiple Streams for MQS Replies

The ability to override more than one stream for MQS replies was unavailable. Currently, the ACKs, TSRs, and Extracts can only be configured on a global level and do not support multi-team environments which may expect replies in various queues and through various modes of delivery. The current override diverts all but one team's ACKs and TSRs to an incorrect queue or location.

EJM logic was enhanced. The replyTo and ackTo nodes should define the Reply MQS and ACK MQS delivery streams respectively. The following parameters were also implemented to specify MQS delivery streams as RTR task parameters:
- MQSExtractDeliveryStream – defines MQS delivery stream for Extract, has higher priority than configuration parameter W_MQS_DEF_EXTRACT_STREAM
- MQSAckDeliveryStream – defines MQS delivery stream for Ack, has higher priority than configuration parameter W_MQS_DEF_ACK_STREAM
- MQSReplyDeliveryStream – defines MQS delivery stream for Reply, has higher priority than configuration parameter W_MQS_DEF_REPLY_STREAM
- MQSDeliveryStream – defines MQS delivery stream for Extract/Ack/Reply, in case parameters above are not specified

This issue was tracked as Item #160494.

Underlying Upload/Update logic improved

The old logic was deleting existing underlying SMF records. Instead this logic should close the existing time periods rather than performing a deletion.

The new xml-smf_underling_common.inc file was created to contain common logic related to the Insert/Update of Underlying records.

This issue was tracked as Item #160527.

Added Handling Extension for Generic Entity Object

Generic Entity can't handle extensions.

The entity extension include was changed to include logic for mappings to the history tables.

Files changed:
xml-ent_entitygeneric.xml
entity_custom_fields.inc

This issue was tracked as Item #160989.

Added Support for Bind Approach for NG load in Entity Analytic

The use of auto-generated binds was introduced to the Entity Analytic interface for the NG load. This was done to improve performance.

Rules were updated to use auto-generated binds and requests, which are generated based on metadata info. This change increased performance threefold.

Files changed:
ester/tpe/dynamic/msgcenter/eagle_default/in/xml/xml-ent_entity_analytics.xml

This issue was tracked as Item #161770.

EJM

Logic For Files Renaming was Moved to eagle_ml-2-0_default_cm_datafile_task stream

The logic for renaming files was working slowly in the task reporter stream, especially for files with long names.

Logic for renaming files was moved to the eagle_ml-2-0_default_cm_datafile_task stream. This change helped reduce the average processing time for the task reporter stream.

Files changed:
data_file_task.xml
task_reporter.xml

This issue was tracked as Item #160934.

Added Configuration Parameter to Disable Writing to the Workflow Processing Log Files

EJM works slower when a lot of files are processed in parallel. This is happening because the execwkfl stream reads\writes log files, which are actually unnecessary in most cases.

A configuration parameter was created to allow disabling the writing to the workflow processing log files.
 
This feature will be enabled in the w_config_custom.inc file with the following code:
<COL TAG="w_write_workflow_log" EXPRESSION="'0'" />

Files changed:
eagle_ml-2-0_cm\wrkfl_add_log_entry.inc
eagle_ml-2-0_cm\wrkfl_cancel_log_entries.inc
eagle_ml-2-0_cm\wrkfl_get_processed_events_count.inc
eagle_ml-2-0_cm\wrkfl_read_log_file.inc
eagle_ml-2-0_cm\wrkfl_write_log_file.inc

This issue was tracked as Item #162152.

Software Fixes

Enterprise

EagleML

Element skipIfExistFlag Added to EagleML Schema

The skipIfExistFlag element was missing from the EagleML XSD.

The SkipIfExistFlag.model model group was added to the schema with the XSD Revision 287. The skipIfExistFlag element is part of this model.

Files changed:

eagleml-shared-2-0.xsd

This issue was tracked as Item #149923.

Purge & Archive on MC Data [failure on eagle_ml-2-0_default_cm_purge_archive]

When a non-existing directory was specified for eagle_ml-2-0_default_cm_purge_archive, the purge failed, but the error would not be reflected in the TSR. This issue occurred on Linux platforms only.

The eagle_ml-2-0_cm\out\xml\eagle_ml-2-0_default_cm_purge_archive.xml file was changed to show the error(s) in the TSR.

This issue was tracked as Item #153426.

Added Feature to Setup '‘noLoadValidationFlag’ Node Value in Generic SMF Extract

Loading a cash security through EagleML is resulting in failures in the case that the new currency is loaded with validation. To turn off this validation and allow the loads a new parameter was added. Setting the noLoadValidationFlag to “Y” will turn validation off.

For I2I processing it is possible to set a task parameter in the RTR that will extract an SMF with the noLoadValidationFlag set to “Y” in the resulting data. This is controlled by sending the SetNoLoadValidationFlagValue with the desired value in the RTR use to execute the extract.

Example:

 <taskParameter>
 <name>SetNoLoadValidationFlagValue</name>
 <dataType>S</dataType>
 <value>Y</value>
</taskParameter>

The extract files will have the following row as part of their contents
<noLoadValidationFlag>Y</noLoadValidationFlag>

Remapping for the following fields was added to the generic SMF rules in the case validation is skipped:
<COL TAG="85" EXPRESSION="IF StringLength(:tag85:)=0 AND :tag3931:='CRCRCR' AND :noLoadValidationFlag:='Y' THEN :assetCurrency:" CONTENT="CURRENCY" />
<COL TAG="1186" EXPRESSION="IF StringLength(:tag1186:)=0 AND :tag3931:='CRCRCR' AND :noLoadValidationFlag:='Y' THEN :incomeCurrency:" />
<COL TAG="63" EXPRESSION="IF StringLength(:tag63:)=0 AND :tag3931:='CRCRCR' AND :noLoadValidationFlag:='Y' THEN :settlementCurrency:" />

Files changed:
estar\tpe\dynamic\msgcenter\eagle_ml-2-0_cm\out\tagvalue\extract_smf.inc
estar\tpe\dynamic\msgcenter\eagle_ml-2-0_cm\out\xml\t_smf_streaming.inc
estar\tpe\dynamic\msgcenter\ eagle_ml-2-0_cm\out\xml\t_smf_streaming_eml.inc
estar\tpe\dynamic\msgcenter\eagle_default\in\xml\include\generic_smf_fields_logic.inc
estar\tpe\dynamic\msgcenter\eagle_default\in\xml\include\generic_smf_validations.inc

This issue was tracked as Item #158778.

SMF MI Underlying upload/update logic improved

The SMF MI stream was not processing underlying records correctly.

There is now logic that will let the underlying record be uploaded only once and any subsequent load will count as an update.

Files chagned:
xml-smf_underling.xml
xml-smf_generic.xml
underlying_link_ins_upd.inc

This issue was tracked as Item #158843.

Held filter is Fixed for Zero Balance Securities

The InsuranceVal Extract was returning securities with zero balance despite Held filter with the 'non-zero' balance parameter being used.

If for example two different had positions in the same security, and one had a zero balance, the InsuranceVal extract would have returned both accounts. Thanks to the changes made, the account with a zero balance no longer makes the extract data.

The link between accounts and securities is added into the extract SQL in order to make the functionality work as expected:
inner join TMP_HELD_SECURITY on
  TMP_HELD_SECURITY.SECURITY_ALIAS = ISTAR_LOT.ASSET_ID AND
  TMP_HELD_SECURITY.PORTFOLIO_ACCT = ISTAR_LOT.PORTFOLIO_ACCT.

Files changed:
- extract_warehouse_istarlot.inc
- extract_filter_held.inc.

This issue was tracked as Item #158942.

Added Performance Improvements to WarehouseGLBalances Inbound Interface

WarehouseGlBalance records are taking a particularly long time to load in batch mode due to the security resolution logic. The security alias is resolved in a loop for each individual record in the batch.

1. Changes in the outbound rule:
- the logic for batch key has been updated: the GLDP_ID field has been added to the batch key. As a result one warehouse transaction will only contain a single warehouse GL balance record.

2. Changes in the inbound rule:
- the new WarehouseGLBalances inbound rule has been created
- new rule supports loading only one <warehouseGLBalances> record in a <warehouseTransaction> batch
- added auto generate binds functionality for the WarehouseGLBalances type

This issue was tracked as Item #159011.

Added Logic for Waiting While File will be Fully Loaded to FTP\SFTP

Generic Load process doesn't check the file size before downloading. In some cases it could download a file which was not fully loaded and as a result cause the workflow to fail.

Logic for checking the file size was added. It checks the file size for the FTP\SFTP\FILE delivery methods and if the file size is the same across two checks, the workflow will accept and process this file.
For the FILE delivery method - Time between two check is 5 seconds
For the FTP\SFTP delivery method - Time between two check is 20 seconds

Files changed:
runtaskrequest.xml

RTR parameter to enable functionality:
Required flag:
  <taskParameter>
   <name>WaitForFileUpload</name>
   <dataType>S</dataType>
   <value>Y</value>
  </taskParameter>
Option flag:
This flag allows to set time which file will be waited for.
<taskParameter>
   <name>WaitForFileUploadTime</name>
   <dataType>S</dataType>
   <value>660</value>
  </taskParameter>

This issue was tracked as Item #159012.

Set ConsolidatedTSR = Y by default for Remote Process in I2I

After the I2I remote process (load or extract) a consolidated TSR must be send back to the user. The TSR should show all the steps that occurred on the remote region for error tracking purposes.

The Consolidated TSR functionality was set to be enabled by default for I2I Workflows which are launched remotely.

Files changed:
task_reporter.xml

This issue was tracked as Item #159014.

Updated Logic To Check Warehouse Trade Record Existance Based On Master Ticket Number Or Event Id in Single Mode

The EagleML Warehouse Trade load requires Master Ticket Number (tag 761) to have a value. The load fails if the field is not populated. But for a trade the unique ID can be either the Master Ticket Number (tag 761) or the Event Id (tag 25). If one of them is present, the trade should be loaded.

For Warehouse Trade in Single Mode the logic to check record existance was updated as following:

1. Check if master ticket number or event id (tag25) are present in incoming message. If both are empty, show error.
2. If master ticket number is not null, resolve the record by this field.
3. Otherwise if master ticket number is null, but transition id (tag 25) not, try to resolve records by this field and fill the master ticket number with the same value as transition id (tag 25), because SP will use it anyway for resolution


estar\tpe\dynamic\msgcenter\eagle_default\in\xml\include\xml-dbdirect_common.inc
estar\tpe\dynamic\msgcenter\eagle_default\in\xml\xslt\wrh_egl_gen_interface3.xslt

This issue was tracked as Item #159059.

Added Support Delivery to SFTP and JMS in Generic Extract Workflow

The Generic Extract workflow was only able to deliver an archive file to an FTP site.

Support for SFPT and JMS delivery methods was added. Files can now be delivered without mandatory compression as well.

Files changed:
i2i_common_include.inc
eagle_default_cm_file_distribution.xml
eagle_default_cm_packer.xml

This issue was tracked as Item #159382.

Fixed STAR Schedules Returning "Loopback Completion Status check failure"

STAR Schedules were returning a "Loopback Completion Status check failure" error.

The following logic was created to handle this error:

<COL TAG="varDescr_result" EXPRESSION="replace(:varDescr_result:,'|=|Checking the loopback[XMLDBAPI] completion status. Failure.','')"/>

Files changed:
eagle_ml-2-0_cm\get_sched_event_detailed_log.inc
eagle_ml-2-0_cm\get_sched_event_log.inc

This issue was tracked as Item #160456.

Interface check for existing Issuer uses Upper(issuer_id) which is out of synch with Issuer Organization Uploader.

When checking for users an interface will always convert the value of the issuer Id to uppercase beforehand. The issuer organization uploader adds the issuer Id value as is. This is problematic because this allows for different capitalizations of the same issuer id to pass the search criteria, whereas only one should. This will cause the process of deleting issuer roles to fail.

The Issuer Alias resolution was changed:
1. Removed direct Issuer Alias Resolution by sub-query
2. Used issuer_xref_resolution.inc include for Issuer Alias resolution by IssuerID or Issuer xRefs.

Workflow changed:
eagle_wrf_generic_delete

This issue was tracked as Item #160754.

Adjusted wrh_egl_gen_interface3.xsl to Allow Empty EntityID or EntityXreference Details

The eagle_ml-2-0_default_in_xml_warehouse_preproc XSLT Transformation was failing when there was no entityID or EntityXreference details in the incoming message.

The logic was updated, to allow no entityID or EntityXreference details in the incoming message.

Files changed:
\eagle_default\in\xml\xslt\wrh_egl_gen_interface3.xsl

This issue was tracked as Item #160847.

Made WRHSCASHACTIVITYEXTRACT valid against XSD

The WRHSCASHACTIVITYEXTRACT was not valid against XSD. The "Element [transactionType]. This element is not expected" error was being raised.

The transactionType node in the t_warehouse_cash_activity_streaming_eml.inc and t_warehouse_cash_activity_streaming.inc was moved to the correct position to make WRHSCASHACTIVITYEXTRACT valid against XSD.

This issue was tracked as Item #160911.

Fixed ISSUERORGEXTRACT running in held mode.

When running the ISSUERORGEXTRACT in held mode with the heldeffectivedate filter the results included unexpected redundant records. This was due to the query used for this extract not having a distinct keyword.

The extract_issuer_org.inc and extract_upd_timestamp were updated. When running the extract in held mode the SQL statement now uses the DISTINCT keyword. Additionally, the extract_upd_timestamp does not add the ROWNUM field for this query.

This issue was tracked as Item #160958.

Price Source and Price Type are Fixed When Parameter IncludePortfolioValuation is Used in Warehouse Extracts

The Price Source and Price Type elements were not being populated in the portfolio valuation consistently. We have to change the join to get these two fields from a different table: RULESDBO.PRICE_RULE_SECURITIES.

An existing parameter, IncludePortfolioValuation, was used to accomplish this. A join to the 'RULESDBO.PRICE_RULE_SECURITIES' table was added and fields Price Type and Price Source get populated as a result of using this join.

Files changed:
- extract_holding_ext_port_valuation_fields.inc.

This issue was tracked as Item #161321.

Forward Points XSD updated to add toAsset model and support new fields

New fields and structure were requested for the forward points object. Changes to this object were made in the XSD.

Extract rules were updated in order to complete the new XSD.

Files changed:
t_forward_points_streaming.xml
extract_forwardpoints.xml

This issue was tracked as Item #161566.

Performance Improvements of IssueRatings Extract

Slow performance of the IssueRatings extract was detected when using a multiple values in the SourceName parameter.

Logic for the source filter was changed to address this issue.

Files changed:
- extract_rating.xml
- extract_rating.inc
- extract_start.inc
- extract_filter_sources.inc.

This issue was tracked as Item #161779.

Adjusted EagleML Header Generating Logic

After a transformation the EagleML headers were being generated with empty values in some nodes.

Logic was updated, to work as expected. The EagleML header are now generated with all expected values filled in.

Files changed:
\eagle_ml-2-0_cm\extract_upd_header.inc

This issue was tracked as Item #161794.

Fixed Regression Issue for CustomExtractLocation for Non-Compressed Files in Generic Extract Workflow

In the March release we introduced the issue with the CustomExtract location for Non-Compressed files (Generic Extract Workflow).

Logic for the case when Compress flag = N was fixed.

Files changed:
i2i_common_include.inc
eagle_default_cm_file_distribution.xml

This issue was tracked as Item #161845.

Added Required Field Margin Price Source to Entity Accounting Basis Inbound and Outbound Interfaces

The inbound Entity Accounting Basis rule was failing on environments with version 17.2 because the new required Margin Price Source tag (18041) was added to the Add Accounting Basis panel.

The new field <marginPriceSource> has been added to the XSD EntityAccountingBasis type, xml-ent_entityaccountingbasis.xml inbound rule and t_entity_accounting_basis.xml and t_entity_accounting_basis_streaming.inc outbound T rules.
The mapping from the <marginPriceSource> element to the panel tag 18041 has been added in the inbound rule as well.

This issue was tracked as Item #162039.

EJM

Fix Runpreprocessor 'COPY_TO_STAGING' in 'eagle_ml-2-0_default_cm_datafile_control'  Stream to Avoid Creating Files With Name of Staging Folder Instead of Copying File to This Folder

The 'eagle_ml-2-0_default_cm_datafile_control' stream was failing with MC error 'File exists'. This was occurring when the staging folder did not exist. As a result the ‘COPY_TO_STAGING’ preprocessor would not copy the files to the staging folder but rather name the files with the value passed as the folder name.

This error was addressed to properly create the staging folder instead of naming the file with the name of the desired folder.

Files changed:
estar\tpe\dynamic\msgcenter\eagle_ml-2-0_cm\out\xml\cmw_datafile_control.prf

This issue was tracked as Item #157094.

Added Correct Logic of Rename Long File Name

The renaming logic for the boot process was incorrect and would result in an incorrect status being reflected in the TSR.

The rename logic from adjust_datafile_name.inc was reused in the data_file_task.inc. This allowed the removal of the adjust_datafile_name.inc from the task_reporter.inc file.

This issue was tracked as Item #158888.