EagleML Release Notes - January 2020

Owned by Olga Galis (Deactivated)
Last updated: Jan 14, 2020
4 min read

Enhancements:

SDP-31777: Implemented an additional security check via a validation script for incoming messages against SQL Injections

Each initial message of the process for either the RTR or PayLoad file is validated against the possible SQL injection and if the message contains a value that matches with one of RegEx checks, the file is refused by the process.

The script is written on Python 2.7 which is part of the default installation on each environment and does not require an additional component installation. The script uses regular expressions stored in the configuration file:
eagle/estar/tpe/dynamic/metadata/core/securitypolicy.json

The set of regular expression may be easily adjusted. The functionality is turned off by default for 2020 January Release.

The Python script was incorporated in the following Eagleml rules:

Classic MC:
eagleml_load.xml
realtime_rtr.xml
runtaskrequest.xml


Files changed:

eagle_ml-2-0_cm/in/xml/eagleml_load.xml
eagle_ml-2-0_cm/in/xml/realtime_rtr.xml
eagle_ml-2-0_cm/in/xml/runtaskrequest.xml
eagle_ml-2-0_cm/out/xml/task_reporter.xml
eagle_ml-2-0_cm/upd_orch_req_state.inc
eagle_ml-2-0_cm/validate_msg_safety.inc
dynamic/pyrules/eagleinvsys/eagleutils/parse_validate_indata.py
dynamic/metadata/core/ebs/securitypolicy.json
dynamic/msgcenter/eagle_ml-2-0_cm/cm_checks.inc
dynamic/msgcenter/eagle_ml-2-0_cm/run_python.inc
dynamic/pyrules/eagleinvsys/eagleutils/parse_and_validate_indata.py

SDP-31882: Added support for MC2 rules to call Python Validation Script which removes potentially harmful characters for incoming RTR messages

Added support to call a python validation script that catches potentially harmful SQL injections to the following MC2 streams that process RTRs:

* eagle_ml-2-0_extract_service_control_message
* eagle_ml-2-0_default_cm_wrkfl_listener
* eagle_ml-2-0_default_cm_execebs

The validation is performed only for MC2 RTRs.  For payload messages, the same functionality will be added in a future release. Additionally, the following issues were addressed:

* TSR filename for MC WebService to return correct response
* Reporting other errors except validation from python script
* Processing namespaces generated by the MC WebService

The functionality is turned off by default in the 2020 January Release.

Files changed:

/eagle_ml-2-0_cm/cm_checks.inc
/eagle_ml-2-0_cm/mc2/ejm/workflow/eagleml_listener.xml
/eagle_ml-2-0_cm/mc2/ejm/workflow/workflow_listener.xml
/eagle_ml-2-0_cm/rtr_map_params.inc
/eagle_ml-2-0_cm/in/xml/eagleml_load.xml
/eagle_ml-2-0_cm/in/xml/realtime_rtr.xml
/eagle_ml-2-0_cm/in/xml/runtaskrequest.xml
/eagle_ml-2-0_cm/out/xml/task_reporter.xml
/eagle_ml-2-0_cm/reporter_dist_methods.inc
/eagle_ml-2-0_cm/validate_msg_safety.inc
/metadata/core/securitypolicy.json
/pyrules/eagleinvsys/eagleutils/parse_validate_indata.py

SDP-31181: Delete statement was corrected to escape a concurrency issue when two or more Warehouse Records run in Parallel (Batch Mode)

When the Warehouse Open Lot and Position transactions were executed in parallel to DELETE at the same time, some records could not be deleted when working with large amounts of data.  The Delete statement was corrected to escape the concurrency issue when two or more records run in parallel, by locking the current Position record while the first transaction performs its changes.

Files changed:

eagle_default/in/xml/include/xml-dbdirect_wrhs_delete_statement.inc
eagle_default/in/xml/include/xml-warehouse_common.inc

SDP-31084: Business Key for Corporate Action Records loaded via Accounting CA inbound interface was updated

The business and split key were updated when processing inbound corporate actions.  

  1. A new, revised business key for Corporate Action record was created:
    SECURITY_ALIAS + TRANS_TYPE + CP_COUNTER_PARTY + CP_EX_DT + CP_SUB_PRIORITY +
    TO_SECURITY_ALIAS + CP_TERM_FLAG.
  2. The split key for Accounting Corporate Action inbound interface was added:
    elements transType + primaryAssetId

Files changed:
eagle_default/in/xml/xml-ref_ca.xml
eagle_default/in/xml/xml-ref_objects.rsf
eagle_default/in/xml/xml-ref_objects.spl


SDP-30887: Field toIssueName is auto-populated based on the database value

The mapping in the rule was improved to auto-populate the toIssueName field as tag1087 when the issue name for the toAsset is null as part of the incoming data file.

Files changed:
xml-ref_ca.xml

SDP-30089: Introduced a new mode to support the Issuer Relationship History data

A new data mode was introduced for the Issuer Relationship object. It may be activated by processing the option ISSUER_RELATIONSHIP_HISTORY_MODE - Y or as a global flag W_ISSUER_RELATIONSHIP_GENERATE_HISTORY_RECORDS - Y.

The Issuer Relationship history mode allows emulating period by period ownership information in the SECURITYDBO.ISSUER_RELATIONSHIP for each record based on the key SOURCE + FROM ISSUER + ROLE TYPE. Periods will be processed based on an effective date.

Files changed:
eagle_default/in/xml/xml-ref_issuerrelationship.xml

SDP-9298: The ability to specify meaningful information about the region was added via sentBy

The sentBy output had not included meaningful information about the region. By default the value is the URL. The ability to specify meaningful information about the region was added. In order to set meaningful information about the region, end users need to set the DisplayEnvNameInSentBy parameter to ‘Y’ in the RTR or set the W_DISPLAY_ENVNAME_IN_SENTBY to ‘Y’ in the config file. The RTR parameter has a priority over the config parameter. By default, the option is disabled.
Example of RTR parameter:
<taskParameter>
<name>DisplayEnvNameInSentBy</name>
<dataType>S</dataType>
<value>Y</value>
</taskParameter>
Example of config parameter
<CODE>:W_DISPLAY_ENVNAME_IN_SENTBY: :='Y'</CODE>

Files changed:
eagle_mc/installer/out/csv/eagle_mc_server_support_log_out.xml
eagle_ml-2-0_cm/create_cm_load.inc
eagle_ml-2-0_cm/create_tsr.inc
eagle_ml-2-0_cm/extract_upd_header.inc
eagle_ml-2-0_cm/mc2/ejm/reporter/mc2_build_ack.inc
eagle_ml-2-0_cm/mc2/ejm/reporter/mc2_tsr_header.inc
eagle_ml-2-0_cm/mc2/healthcheck/mc2_healthcheck.xml
eagle_ml-2-0_cm/mc2/mc2_create_cm.inc
eagle_ml-2-0_cm/out/tagvalue/execute_wrkfl_event.xml
eagle_ml-2-0_cm/out/tagvalue/wrkfl_reporter.xml
eagle_ml-2-0_cm/out/xml/extract_service_csv.inc
eagle_ml-2-0_cm/out/xml/extract_service_tsr.inc
eagle_ml-2-0_cm/out/xml/t_eagle_header_fields.inc
eagle_ml-2-0_cm/out/xml/task_reporter.xml
eagle_ml-2-0_cm/out/xml/taskacknowledgement.xml
eagle_ml-2-0_cm/send_alert.inc
eagle_ml-2-0_cm/taskstatus_reply.inc

Production Bugs:

SDP-31023: Corrected issue with eagle_default_in_csv_warehouse_t file name

The stream eagle_default_in_csv_warehouse_t was incorrectly mapping the filename variable for the Warehouse Performance records leading to failures during loads.
The logic was enhanced to correctly process the filename variable.

File changed:
|eagle_default/out/csv/performance.xml

SDP-30993: Corrected issue in case the data mask is passed via an email alert

The stream eagle_ml-2-0_default_cm_monitoring was incorrectly processing data mask files from profiles on the first run leading to incomplete TSR.
The logic was enhanced to correctly process the data mask and correct TSR.

File changed:
eagle_ml-2-0_cm/out/xml/read_profiles.inc

SDP-30952: Stored Procedure generated on the fly updates only fields specified in the input message

The problem was that existing values were overwritten by NULLs in the generated stored procedure. The issue is fixed by changing the update logic to add the NVL function for updates. In case the NULL value is passed in the incoming message for a specific field, it stays as is in the database without update.

Logic changed for following Interfaces:
EntityAnalytic
EntityPeerGroupRanking
EntityRating
IssueFxRateDemand
PricingSystemSetting
PaceSystem
LineOfCreditFacility
EntitySecurityLending
genericIssueAnalytic
IssuerEntityRelationship
ExchangePrice
EntityLiquidity
IssuerAnalytic

File changed:
eagle_ml-2-0_cm/generate_sp_on_fly.inc

SDP-30929: Addressed issues with the incorrect load of records via GroupedSingle mode and W_WRHS_IGNORE_POS_FLAG enabled

The Warehouse Position and OpenLot records loaded incorrectly via sub-batches in GroupedSingle mode with the  W_WRHS_IGNORE_POS_FLAG enabled. Only the last sub-batch or few sub-batches were loaded as result of incoming file processing. The issue was fixed.

Files changed:
xml-warehouse_common.inc
xml-dbdirect_wrhs_delete_statement.inc

SDP-30370: Schedule Interfaces were updated to support additional fields

Three new fields were added to the XSD for the Schedule object:
SECURITYDBO.SCHEDULE.FIXING_DATE
SECURITYDBO.SCHEDULE.CASH_PAYMENT_DATE
SECURITYDBO.SCHEDULE.RESET_DATE
The inbound and outbound rules were updated to include the new fields.

Files changed:
eagleml-ref-shared-2-0.xsd
metadata/bind/eagleml_ref_schedule.xml
eagle_default/in/xml/xml-ref_schedule.xml
eagle_ml-2-0_cm/out/xml/t_schedule_streaming.inc

SDP-30850: Added mapping for the field isMakeWholeCalled (loaded via GenericSMF) to the Accounting Corporate Actions inbound interface

The mapping for the field Make Whole Call Flag (tag10346) was added to fix the error, 'SECURITY DOES NOT HAVE CALL/PUT/REFUND/MAKE WHOLE CALL FLAG SET OR LOTTERY BOND PROVISIONS OR PRIVATE PLCMNT' when loading the Accounting Corporate Actions with the "Make Whole Call Flag" set to Y in version 2017.

Files changed:
eagle_default/in/xml/xml-ref_ca.xml