MC2 Release Notes - July 28 2020

Latest build: 2020.07.28.3

SDP-39201 MC2 Service Installer Fails To Install Services On All Nodes

As of version 2020.07.28.3

Corrected Issues With MC2 Services Installer On Environments With Multiple Nodes

MC2 Services Installer was failing to install the services on all nodes on environments with several nodes. To correct this issue the following changes were implemented:

SDP-38015 Revise 5 EDS Interfaces to have correct EML Elements for Cloud Ingestion

5 interfaces were fixed

Codevalue interface:
"ebsCodeCategoryInstance" and "ebsCodeValueInstance" columns were removed from the codevalue interface.
Genericsmf interface:
It will be fixed when new metadata will be generated.
Reportdictionary interface:
"reportDictionarySrc" and "Type" columns were removed from the reportdictionary interface.
Warehouseperformance interface:
SRC_INTFC_INST srcIntfcInst dictionaeryItemId dictionaryItemId
Also fixed type of dictionaryDetailId and of all above fields as a number.
Issuer interface:
There are columns that were removed from the issuer interface:BOND_DUMMY_1CITY_1EFFECTIVE_DATE_1EQUITY_DUMMY_1FROM_ISSUER_ALIASINSTANCE IRD_CHAR2IRD_UPDATE_SOURCEISSUER_ALIAS_1ISSUER_ALIAS_2ISSUER_ALIAS_3 LEILEI_1PERCENT_OWNED_1RELEASE_DATERELEASE_DATE_1RELEASE_STATUSRELEASE_STATUS_1 ROLE_ALIASROLE_TYPE_1SRC_INTFC_INSTSRC_INTFC_INST_1STATE_1 TO_ISSUER_ALIASUPDATE_DATE_1UPDATE_DATE_2UPDATE_DATE_3UPDATE_SOURCE_1UPDATE_SOURCE_2UPDATE_SOURCE_3UPD_DATEUPD_SOURCE
Renamed fields:
ALPHA_RATING -> alphaRating
COUNTRY_CODE -> countryCode
ENTITY_STATUS -> entityStatus
ID_ASSIGNED_DATE -> idAssignedDateID_CERTIFICATION_STATE -> idCertificationStateID_DISABLED_DATE -> idDisabledDateID_LAST_UPDATE_DATE -> idLastUpdateDateID_RECORD_STATE -> idRecordState
ISSUER_PHONE -> phone
LEGAL_FORM -> legalForm
LEGAL_NAME -> legalName
STREET -> street
ZIP -> zip
ISSUER_PHONE -> phone

SDP-37273 Create EDS Dataset and Taxonomy for performdbo.perf_sec_rollup_relation table called perfsecrolluprelation

Created EDS Dataset and Taxonomy for performdbo.perf_sec_rollup_relation table called perfsecrolluprelation

We created EDS Dataset and Taxonomy for performdbo.perf_sec_rollup_relation table called perfsecrolluprelation.

SDP-38495 Source preprocessing needs ability of setting mask for column

Add new t-item SourcePreproc in EDS definition

Add new t-item SourcePreproc in EDS definition. This t-item performs replace of incoming values in specified columns according to specified rules.
Example:"t_items_def": [["source_row_col_value_replace", "subst_values", [{"column": "alpha_", "map":
{"ALPHA1": "ALPHA_#1"}
, "use_expr": false},{"column": "score_", "map":
{"N/A": "-500", "N/R": "-600"}
, "use_expr": false},{"column": "rating", "map":
{"N/A": "-123", "N/R": "-456"}
, "use_expr": false},{"column": "col", "map":
{"N/A": "-888", "N/R": "-999"}
, "use_expr": false}]]],
Make replace in columns with mask alpha_ - replace ALPHA1 by ALPHA_#1Make replace in columns with mask score_ - replace N/A by "-500" and replace N/R by -600 Make replace in columns with mask rating - replace N/A by -123 and replace N/R by -456 Make replace in columns with mask col - replace N/A by -888 and replace N/R by -999

SDP-38451 Add support basic Editcheck logic for EBS definition

Add basic version of EditCheck t-item

New t-item (EditCheck) implementation was added in EDS definition. This item raise an error if condition return True.There are two options for this item:1. Stop processing with status fail2. Keep processing with just registering error in logger
Definition of the item consist of the following parameter1. Type = editcheck(const)2. Name3. Condition of the EditCheck If TRUE than FAIL4. Error message, which will be mentioned in log.5. Error code6. Edict check type:a. Keep processing with logging error. (Default)b. 1 - Stop processing in case of ErrorExample:["editcheck", "entity_xreference_editcheck", "stringlength(|xrefAccountId|) != 0 and stringlength(|xrefAccountIdType|) != 0 and stringlength(:entity_xreference_lookup_entity_id = 0","'EntityId can not be resolved!'", "'-728'", "1"]

SDP-38423 Root node auto definition

Added ability to define root node automatically in incoming XML file

Added ability to define root node automatically. It allows user not to set "rootnodename" in "source_format_dialect". This is helpful when user has similar incoming files with different root nodes.

SDP-37841 Create Interface for issuerentityrelationship

Create Interface for issuerentityrelationship

We created an EDS interface for issuerentityrelationship. This will pull data from the SECURITYDBO.ISSUER_ENTITY_RELATION table.

SDP-37031 Optimize method for creation XML\JSON output with nested nodes

Methods for generation JSON\XML output were optimized

Methods for generation JSON\XML output were optimized and logic for calculating nested nodes was moved to initialize step instead of calculating it during processing each row. Now these methods are simpler and slightly faster 10-15% than it was before. Also support of nested nodes was added for JSON output.For CSV output new option in target format dialect was added: encode_header = Y\N ( Default = N)When this option is Y header in CSV file will encode / symbols to _ .<taskParameter><name>OutputFormatDialect</name><value>
{"delimiter":",","rowdelimiter":"\n","hasheader":"Y", "encode_header":"Y"}
</value></taskParameter>
Example:"encode_header":"N"

• sourceName,entityId,accountingBasis,market/marketValueLocal,market/marketValueBase"encode_header":"Y"

• sourceName,entityId,accountingBasis,market_marketValueLocal,market_marketValueBase

SDP-27760 Profile inheritance - getting full profile stack

Profile inheritance - getting full profile stack

We have added the use of parameter EliminateDuplicates where when set to "N", then the full profile would not remove the duplicates and return them in the output.

SDP-38753 Disable distribution attempt in eagle_default_ruleservice by mc2_distribution

Issue with 2 attempts at distribution with distribution profile

Corrected issue with 2 attempts to distribute file when distribution profile specified in RTR.

SDP-38205 Update environment name selection logic in HealthCheckDbAPI

HealthCheckDbApi now uses lower case of DBName value in db_connection.ini file.

HealthCheckDbApi now uses lower case of DBName value in db_connection.ini file.Previously the value of instance column in V$Thread system table was used.

SDP-38204 Update environment name selection logic in kafka service

kafka service now uses lower case of DBName value in db_connection.ini file.

kafka service now uses lower case of DBName value in db_connection.ini file.Previously the value of instance column in v$thread system table was used.

SDP-38117 Do not ignore EJM-aware task statuses on MC2 workflow execution

Updated the MC2 workflow executor to not ignore status from executed streams

Updated the MC2 workflow executor to not ignore status from EJM aware streams. Previously, the executor assigned a status of Success if the stream was executed,even if the stream set its own status to NO_DATA.

SDP-37217 issue with kafka-service

Corrected intermittent issue with zookeeper startup

Corrected intermittent issue with zookeeper startup

SDP-37189 Update environment name selection in installer / DR installer

Updated the source of environment name used by services.

Updated the MC2 Services installer to take the environment name used by the services from DBName value in db_connection.ini file. Previously the name was retrieved from the DB (instance column in v$thread table).

SDP-37188 Update environment name selection logic in MC2 service

MC2 service now uses lower case of DBName value in db_connection.ini file as environment name.

MC2 service now uses lower case of DBName value in db_connection.ini file as the environment name.Previously the value used was instance column of V$thread system table.

SDP-36041 Enable publishing raw and translated inbound files

Enabled publishing raw and translated inbound files to sftp or cloud in MC Classic generic load.

Enabled publishing raw or translated inbound data files from MC Classic Generic Loader.The destinations may be sftp or cloud and are identified in distribution profiles.Details: https://eagledocs.atlassian.net/wiki/spaces/MC2/pages/1389559809/Publishing+Inbound+Data+Files+In+Generic+Loader

SDP-35590 EQL requests must have more details in console (DB)

Added request identifiers to ReST API EQL endpoint Business Task Id

Updated the EQL endpoint of ReST API to populate the Business Task ID,as seen in EJM Tasks monitor and Monitoring Tool, to be REST#eql#<feedtype>#timestamp.For example:REST#eql#warehouseposition#202007020900

SDP-34625 Expose kafka status to HealthCheck

Added status of kafka service to HealthCheck

Added status of kafka messaging consumers and publishers to HealthCheck

SDP-29398 MC2 Requests were Failed in MSG Detail but Success in EJM Tasks

Corrected issue in MC2 where EJM task reported success but underlying steps had an error

Corrected issue in MC2 where EJM task reported success but underlying steps had an error

SDP-37479 Improve handling items in exclude list with prefix

Improved handling parameters with a prefix in exclude list for SQL Injection

There was code changes made to the parse_invalidate_indata.py to allow the use of a parameter with a prefix to be excluded from the check. Before this, when using a parameter with a prefix, such as "ebs:warehouseposition:OutputFilterExp"r, even the excluded parameters for SQL Injection checks were invalidated.

SDP-35824 CorrelationIdPrefix not supported for EDS WORKFLOWS (1)

Added CorrelationIdPrefix parameter

CorrelationIdPrefix parameter was added. It can be configured via RTR for EJM tasks executed via the Generic Load and Generic Extract workflows.
<taskParameter><name>CorrelationIdPrefix</name><dataType>S</dataType><value>XXXXX</value></taskParameter>
The maximum length for the prefix set to 5 symbols.

SDP-37458 Enhance possibility to exclude file from wait events queue based on workflow name

Added an option to remove all files from wait events queue based on the workflow name

Previously, it was possible to remove only one file via one control message from waitevents queue. Now RTR with CancelWorkflowName parameter and ActionType 'CancelFile' allow to cancel all files from waitevents queue and specific workflow.
<taskParameter><name>CancelWorkflowName</name><dataType>S</dataType><value>sdp37458_wrf_generic_load</value></taskParameter><taskParameter><name>ActionType</name><dataType>S</dataType><value>CANCELFILE</value></taskParameter>
Files changed:eagle_ml-2-0_cm/out/tagvalue/execute_wrkfl_event.xmleagle_ml-2-0_cm/cancel_global_event_wait.inc

SDP-37066 Update Swagger API docs version to OpenAPI 3

Update ReST endpoint swagger API documents to OpenAPI 3.0

Updated the ReST endpoint swagger API documents to comply with OpenAPI 3.0 specification.Previously, the swagger API documents complied with Swagger 2.0 specification.Client code generators will need to be compliant with OpenAPI 3.0.

SDP-38319 Support Currency Conversion Overrides in REST API for Fetching FX Rates

Supporting FX Rates Fetch for Currency Conversion in OLAP REST API

New feature for overriding Forex Spot rate fetching for the report effective date is supported in OLAP REST API. This allows the user to override the number of prior days to go back and fetch the Forex spot rate which in turn be used for either benchmark or entity's base currencies. This is a similar functionality supported in OLAP reports.

SDP-38318 Support Currency Conversion Overrides in REST API for First and Second alternate Currencies

Supporting Base Entity's Currency override in OLAP REST API

New feature for overriding entity's base currencies either to first or second currency is supported in OLAP REST API. This allows the user to override profile entity's base currencies to any alternate currencies. This is similar functionality supported in OLAP Reports.