EagleML Release Notes - June 2020 (Second)

Owned by Olga Galis (Deactivated)
Jun 30, 2020
6 min read

Enhancements:

SDP-38024: Post-load validation stream supports pre-extract validation

The post-load validation stream eagle_ml-2-0_default_cm_profiled_load_validation may be also used as pre-extract validation step going forward.
It is enough to specify the profile name and define the reprocess_interface via the Profile Loader. It may be required to define the reprocess_feedtype if multiple feed types are loaded.
The post-load validation step also takes into account the data reprocessed or modified manually after the load.

The option is a framework and may require additional interfaces to handle required validations of the data loaded or source file altered.

Files changed:
eagle_ml-2-0_cm/unpacker/profiled_load_validation.xml

SDP-37791: ENTITY COMPOSITE LIST EXTRACT was enhanced to allow to extract data from the RULESDBO.ENTITY_DETAIL_HISTORY database table using the historyOnlyFlag

The ENTITYCOMPOSITELISTEXTRACT was enhanced to support the History Only and all effective date filters like fromeffectivedate, maxeffectivedate , toeffectivedate.
If historyOnlyFlag set to N, then the data will be extracted from RULESDBO.ENTITY_DETAIL database and update_date field will be used.
If historyOnlyFlag set to Y, then the data will be extracted from RULESDBO.ENTITY_DETAIL_HISTORY database table and effective date field fill be used.

For Example:
<taskParameter>
<name>FeedType</name>
<dataType>S</dataType>
<value>ENTITYCOMPOSITELISTEXTRACT</value>
</taskParameter>
<name>fromeffectivedate</name>
<dataType>S</dataType>
<value>2000-01-01</value>
</taskParameter>
<taskParameter>
<name>toeffectivedate</name>
<dataType>S</dataType>
<value>2020-06-01</value>
</taskParameter>
<taskParameter>
<name>historyOnlyFlag</name>
<dataType>S</dataType>
<value>Y</value>
</taskParameter>

File changed:
eagle_ml-2-0_cm/out/tagvalue/extract_entity_composite_list.inc

SDP-37551: ExchangePrice rule was enhanced to fail backdated prices

The ExchangePrice rule was enhanced to fail backdated prices. When the allowPastRate is set to N and the effective date and current date do not match, then the load will not be allowed.

Files changed:
eagle_default\in\xml\xml-ref_exchangeprice.xml

SDP-37550: allowPastRate element was excluded from the ExchangePrice load when the archiveFlag set to Y

The allowPastRate element was excluded from the ExchangePrice load when executing in archive mode.

Files changed:
metadata/bind/eagleml_ref_exchangeprice.xml
eagle_default/in/xml/include/xml-ref_security_pricing_common.inc

SDP-37511: Added logic for the Dividend Reinvest Corporate Action for the Transaction Type = 'REINVEST' to the Accounting mode of Corporate Action inbound interface if the accountingValidationFlag set to Y

Added logic to resolve the initial Cash Dividend Corporate Action record when the Transaction Type is REINVEST. The same logic was available for Corporate Action records when the Transaction Type is DIVREINVEST.

Supported steps:
1 - Load Corporate Action record with Transaction Type CASHDIV
2 - Load Dividend Reinvestment (DRIP) Corporate Action record with Transaction Type REINVEST or DIVREINVEST

The batchEventId of the Reinvest record is resolved from the batchEventId of the Cash Dividend record as they should be the same for both the Cash Dividend and Reinvest. The instance of initial Cash Dividend record is mapped to the field cpTyp of the Reinvest record. The cpCashCrncy and exDate fields are resolved by Cashdiv record and mapped to Reinvest record.

Files changed:
eagle_default/in/xml/xml-ref_ca.xml

SDP-37508: Voluntary CA Elections Interface was enhanced to support multiple corporate actions for a given exDate

The Voluntary CA Elections was not loading correctly when there were multiple voluntary corporate action announcements on the same day.
The issue was resolved by adding the tagNoChangeCase element to the Corporate Action lookup used in the Voluntary CA Elections load to correctly resolve the issue with multiple instances of elections returned as of the Corporate Action resolution during the Voluntary CA Elections load process.

Files changed:
eagle_default/in/xml/xml-ref_voluntarycaelections.xml

SDP-37337: Exchange Price Inbound interface was enhanced to support additional fields

The ExchangePrice interface was updated to allow the load of additional elements into RULESDBO.PRICE_STATUS table:
EagleML/referenceTransaction/exchangePrice/oidYield
EagleML/referenceTransaction/exchangePrice/userFee1
EagleML/referenceTransaction/exchangePrice/userFee2
EagleML/referenceTransaction/exchangePrice/userFee3
EagleML/referenceTransaction/exchangePrice/userNav1
EagleML/referenceTransaction/exchangePrice/userNav2
EagleML/referenceTransaction/exchangePrice/userNav3
EagleML/referenceTransaction/exchangePrice/userNav4
EagleML/referenceTransaction/exchangePrice/userNav5

File changed:
eagle_default\xml\in\xml-ref_exchangeprice.xml

SDP-37266: Microsoft SQL Server: DeltaBased filter was enhanced for Warehouse Trade Extract

By default, the WRHSTRADEEXTRAXT returns all records within a Batch (trades with same Batch Key: Entity+Source+Effective Date+Security), even if only one trade from the Batch was updated since the prior run.
An additional DeltaBased global parameter was introduced in order for WRHSTRADEEXTRACT to return transactions loaded only after the prior run.

<CODE>:W_TRADE_DELTA_BASED_FILTER_MODE: := 'WAREHOUSE';</CODE>

Recently this Enhancement was implemented for the Oracle database. This enhancement is specific for the Microsoft SQL Server.

Files changed:
eagle_ml-2-0_cm/out/tagvalue/extract_warehouse_trade.inc

SDP-36841: Voluntary CA Election Extract was updated to support additional fields

The fields that were previously added to the Inbound Voluntary Corporate Action interface are now supported by the Voluntary Corporate Action Election Extract - VOLCAELECTIONEXTRACT:

EagleML/referenceTransaction/voluntaryCAElection/longShortIndicator
EagleML/referenceTransaction/voluntaryCAElection/babyBondPrice
EagleML/referenceTransaction/voluntaryCAElection/reorgFee
EagleML/referenceTransaction/voluntaryCAElection/reorgFeeBase
EagleML/referenceTransaction/voluntaryCAElection/prorataCreditNoncreditLossIndicator
EagleML/referenceTransaction/voluntaryCAElection/expiryDate
EagleML/referenceTransaction/voluntaryCAElection/expiryDateFlag

Files changed:
eagle_ml-2-0_cm\out\xml\t_voluntary_ca_election_streaming.inc

SDP-36567: Generic Corporate Action Inbound Interface was updated to support additional fields

The XSD schema and the mapping for the Generic Corporate Action rules were updated to support additional fields:

Tag16770 |db field in_business_offset_days | EagleML/referenceTransaction/genericCA/businessOffsetDays
Tag12876 |db field in_expiry_date | EagleML/referenceTransaction/genericCA/expiryDate
Tag16987 |db filed in_payment_override_fact | EagleML/referenceTransaction/genericCA/paymentOverrideFact
Tag16994 |db filed in_paydown_trade_flat | EagleML/referenceTransaction/genericCA/paydownTradedInterestFlag
Tag1670 |db field in_ca_inc_repay_num | EagleML/referenceTransaction/genericCA/caIncRepayNum
Tag2290 |EagleML/referenceTransaction/genericCA/incomePrepaymentFeeTreatment

The incomePrepaymentFeeTreatment element is not stored in the database.

Files changed:
eagle_default/in/xml/xml-ref_ca.xml
eagle_default/in/xml/xml-ref_generic_ca.xml

SDP-36563: Generic SMF load interface was updated to support additional database fields

The Generic SMF Inbound Interface was updated to allow the load of additional elements into SECURITYDBO.SECTOR.

EagleML/referenceTransaction/genericSMF/bicsLevel1SectorCode
EagleML/referenceTransaction/genericSMF/bicsLevel1SectorName
EagleML/referenceTransaction/genericSMF/bicsLevel2IndustryGroupCode
EagleML/referenceTransaction/genericSMF/bicsLevel2IndustryGroupName
EagleML/referenceTransaction/genericSMF/bicsLevel3IndustryCode
EagleML/referenceTransaction/genericSMF/bicsLevel3IndustryName
EagleML/referenceTransaction/genericSMF/bicsLevel4SubIndustryCode
EagleML/referenceTransaction/genericSMF/bicsLevel4SubIndustryName
EagleML/referenceTransaction/genericSMF/bicsLevel5SegmentCode
EagleML/referenceTransaction/genericSMF/bicsLevel5SegmentName
EagleML/referenceTransaction/genericSMF/bicsLevel6SegmentCode
EagleML/referenceTransaction/genericSMF/bicsLevel6SegmentName
EagleML/referenceTransaction/genericSMF/bicsLevel7SegmentCode
EagleML/referenceTransaction/genericSMF/bicsLevel7SegmentName
EagleML/referenceTransaction/genericSMF/naceClassCode
EagleML/referenceTransaction/genericSMF/naceClassName
EagleML/referenceTransaction/genericSMF/naceSectorCode
EagleML/referenceTransaction/genericSMF/naceSectorName
EagleML/referenceTransaction/genericSMF/naceDivisionCode
EagleML/referenceTransaction/genericSMF/naceDivisionName
EagleML/referenceTransaction/genericSMF/naceGroupCode
EagleML/referenceTransaction/genericSMF/naceGroupName

The new fields are supported for the v17 Oracle platform.

Files changed:
eagle_default/in/xml/xslt/smf_egl_gen_interface_template

SDP-36491: Updated Security Resolution logic for Match All with Use First As Main or Use Exist Xref Only modes

Before the fix, the Use First As Main with Match All or Match All + Use Exist Xref Only modes all Security Xreference identifiers that exist in an incoming message were included into the Asset Resolution Option. After the change, all the Security Xreference identifiers that should be part of Asset Resolution Option should be specified in the <assetResolutionOption> node (or specified as 'ALL').
For example:
If the incoming file has a XREF identifier IDXEXCHSYMBOL, but it should be excluded from Asset Resolution, the following options may be used followed by other IDENTIFIERS and other options like ASSET_CURRENCY or XREF_EXCHANGE
<assetResolutionOption>MATCH_ALL,USE_FIRST_AS_MAIN,USE_EXIST_XREF_ONLY</assetResolutionOption>
<assetResolutionOption>MATCH_ALL,,USE_EXIST_XREF_ONLY</assetResolutionOption>

Files changed:
eagle_default/in/xml/include/security_resolution_generic.inc

SDP-36147: Multileg SMF load process was updated to support derivChar26 element

The MultilegSMF Inbound Interface was updated to correctly support the derivChar26 element for each leg. The element derivChar26 was mapped for each leg separately (contract, pay, receive)

Files changed:
eagle_default/in/xml/include/contract_leg_alliases
eagle_default/in/xml/include/contract_mapping
eagle_default/in/xml/include/pay_leg_alliases
eagle_default/in/xml/include/pay_mapping
eagle_default/in/xml/include/receive_leg_alliases
eagle_default/in/xml/include/receive_mapping
eagle_default/in/xml/include/ins_upd_smf_req

SDP-36058: Improve w_setup stream to store the REDIS port in configuration

A new feature was implemented for the eagle_ml-2-0_default_cm_w_setup stream to calculate and save the REDIS port in the w_config_version.inc file like:
<COL TAG="g_RedisPort" EXPRESSION="'20438'"/>
This value is used in the pyrulservice during python script execution.

SDP-35618: SMF MI Inbound Interface was updated to support Issuer Organization GICS fields

SMF MI Inbound Interface was updated to load Issuer Organization GICS fields.
The following fields are now supported:

tag16809|DB Field SECURITYDBO.ISSUER_ORGANIZATION.GICS_SECTOR| EagleML/referenceTransaction/issuerOrganization/gicsSector
tag16810|DB Field SECURITYDBO.ISSUER_ORGANIZATION.GICS_INDUSTRY_GROUP| EagleML/referenceTransaction/issuerOrganization/gicsIndustryGroup
tag16811|DB Field SECURITYDBO.ISSUER_ORGANIZATION.GICS_INDUSTRY| EagleML/referenceTransaction/issuerOrganization/gicsIndustry
tag16812|DB Field SECURITYDBO.ISSUER_ORGANIZATION.GICS_SUB_INDUSTRY| EagleML/referenceTransaction/issuerOrganization/gicsSubIndustry

Files changed:
eagle_default/in/xml/xml-ref_issuerorganization.xml
eagle_default/in/xml/xslt/eagleml_issuer_org.xsl

SDP-35537: Warehouse Trade Interface was updated to support additional elements

New fields were added to the WarehouseTrade object and the XSD schema. The following fields are now supported by the Inbound and Outbound interfaces:

EagleML/warehouseTransaction/warehouseTrade/income
EagleML/warehouseTransaction/warehouseTrade/incomeAmountBase

Files changed:
eagle_default/in/xml/xslt/wrh_egl_gen_interface3.xsl
eagle_ml-2-0_cm/out/xml/t_warehouse_trade_streaming.inc

SDP-33644: Generic SMF Interface was updated to support the additional element resetLockoutPeriod

Added support of the element resetLockoutPeriod for corresponding tag18083/reset_lockout_period via the Generic SMF load and extract:

EagleML/referenceTransaction/genericSMF/FixedIncomeModel/resetLockoutPeriod
Element resetLockoutDays was already supported:

EagleML/referenceTransaction/genericSMF/FixedIncomeModel/resetLockoutDays

Files changed:
eagle_default/in/xml/include/fixedincomesmf_model.inc
eagle_default/in/xml/include/smf_aliases.inc

SDP-32077: Added additional option to cancel scheduled workflow at the specific time via EJMRunCancelTime parameter

The option to cancel the event at the specific time was added for the scheduled workflow.
The EJMRunCancelTime parameter allows to specify time in a format HH:MM for the schedule or EJM profile to stop specific scheduled events at the specified time.
A new global optional parameter, W_SCHD_EJM_RUN_CANCEL_TIME, may be set via the w_config_custom.inc, which allows to setup a specific time to end all orphaned scheduled workflow executions.

Files changed:
eagle_ml-2-0_cm/out/tagvalue/workflow_restart.xml
eagle_ml-2-0_cm/schedule/workflow_restart.sch

Production Bugs:

SDP-37842: Option to control ACK and Reply File names

A prior change SDP-35483, introduced the issue when the parameter W_REPLYCM_MODE was set to ON/Y. The issue was corrected by supporting of the additional value ONuseExactFileName. This option will maintain the original file name by excluding the Correlation Id as the prefix from the ACK and the TSR filenames if the ACKStrictPattern and TSRStrictPattern parameters are defined.

<CODE>
:W_REPLYCM_MODE: :='ONuseExactFileName';
</CODE>

Please use wiki for additional information:

Tutorial: Setting Custom TSR/ACK Names and Folders
EJM Job Notification Overview
Advanced Configuration Options

File changed:
reporter_dist_path.inc

SDP-37618: Issue Analytic interface was updated to correctly resolve the issueName

The error, 'Error 961 Issue Name: Value required' occurred when the issueName is not provided on the incoming file, but the security successfully resolved.
The issue was fixed as the ISSUEANALYTIC rule was corrected and issueName is now resolved upon the insert of a new record from the security resolution.

Files changed:
eagle_default/in/xml/xml-ref_issueanalytic.xml

SDP-37492: eagle_ml-2-0_default_cm_nuggetizer_recover stream was corrected and repackaged

There was no Installer changes, therefore updates are available only after Fresh Install.

Previously, the eagle_ml-2-0_default_cm_nuggetizer_recover stream was packaged incorrectly with two stream categories assigned to it:
<fld name="Stream Category 1">EGL_EJM</fld>
<fld name="Stream Category 2">MC2</fld>
It led to the MC2 instance assigned to the stream during package deployment. The issue was corrected and the Eagleml core package was repackaged.
The stream description was also updated.

SDP-37447: Security resolution was corrected on the load of Generic Corporate Action in Accounting mode

Previously the toSecurityAlias resolution in the to_security_alias lookup was working incorrectly if the accountingValidationFlag was set to Y. The issue was fixed.

File changed:
eagle_default\in\xml\xml-ref_ca.xml

SDP-37105: Post-processing logic for the T-Stream was improved to deliver archives to the FTP Server

The eagle_ml-2-0_default_out_streaming_t stream was reporting an error trying to access the FTP server.
The problem was solved by correcting the post processing logic for the FTP/SFTP delivery method.

File changed:
eagle_ml-2-0_cm/reporter_dist.inc
eagle_ml-2-0_cm/reporter_dist_methods.inc

SDP-37102: EJM Packer was improved to properly handle an archive filename with spaces

The extract file was not archived on the EJM packer step if the filename had a space. In this case, the command would not recognize the path correctly and an archive was not created.
The issue was resolved. Files and archives are properly supported now for filenames with spaces.

File changed:
eagle_ml-2-0_cm/compress_extract.inc

SDP-36743: SOAP calls were enhanced to support encrypted passwords with special characters

The DateRule requests were failing because of user / password authentication failures. The MC Service encryption/decryption process was causing issues when encrypted password contained special symbols. Provided a solution by enhancing SOAP requests for Date Rule when the encrypted user password contains special characters.

File changed:
eagle_ml-2-0_cm/calc_date_from_rule.inc

SDP-36647: Extract process ended without a file distribution when the T-stream is not available

In case of failure at the Q-stream (query stream), the extract still was generated and distributed even if the T-stream was disabled. The issue was corrected.
The extract file will be created or distributed if the T-stream is not available and the corresponding error message is returned:

  • Stream NAME is disabled.

  • Stream NAME is deleted.

  • Stream NAME is not found.

Files changed:
eagle_ml-2-0_cm/extract2t.inc
eagle_ml-2-0_cm/get_stream_folder.inc

SDP-36484: Sweep Date Logic for Generic CA was corrected

The Sweep Date was previously incorrectly overwriting by System Date. The Generic CA rule was improved to fix this issue.

Files changed:
eagle_default/in/xml/xml-ref_ca.xml