MC2 Release Notes - February 09 2021

SDP-44149 EDS Interfaces - New Metadata - entitypolicy

Created entitypolicy Ontology

New ontology of issuerrelationship was developed: \estar\tpe\dynamic\metadata\core\ontology\entity\entitypolicy.json \estar\tpe\dynamic\metadata\core\ontology\entity\entitypolicy-models.json
entitypolicy.json contains sql table joins definition, filters definition, extract rules and etc. entitypolicy-models.json contains fields mapping definition.

SDP-46549 Add IncludedExtendedDetails to Swagger for some EML APIs

Added IncludeExtendedDetails to Swagger for Multiple EML APIs

We added the IncludeExtendedDetails parameter to the Swagger for the following EML APIs:
intraday-navs
master-feeder
preferred-shares
entities
accounting-status
cash-activities
wash-sale

SDP-46376 Ontology: Review and update Numeric fields, change format to 26,6

Ontology: Review and update Numeric fields, change format to 26,6

We changed in the EDS Ontology any numeric fields which can have more them two decimal digits to be 26,6 format.
elem_per_resource (1).json

SDP-45587 exception with filters when using sql-based data object

Fixed error for sql-based datasets with empty filters

Errors occurred when trying to refresh dataset cache for sql-based dataset with empty filters. The same error occurred during interface execution based on this dataset. We revised EDS filter initialization and application.

SDP-45573 Add ability to send logs to another interface during processing

Added ability to send logs to another interface

New property log_process_def was added to interface. It should contain name of EDS interface which is going to process logs.
Examples of usage:
"log_process_def": [{ "_id": "app_logs_sender"}],
Logs being passed as base64 in parameter in RTR log_statuses.
Logs being sent in async mode in following cases:
1 First log captured
2 Captured count of logs % 500 = 0
3 At the end of processing

SDP-44943 MD: expand xrefidentifiers xId1-xId20 in processing rules generation

Enhanced processing rules generator to expand correctly nested nodes and xrefIdentifiers in vocabulary

The processing rules generator was enhanced to expand correctly nested nodes and xrefIdentifiers in vocabulary part of load dataset. This allows correctly processing elements from nested nodes during the EDS load.
Example:
"vocabulary": { "xrefIdentifiers": { "path": "xrefIdentifiers", "datatype": "dict" } , "xrefIdentifiers_xId1": { "path": "xrefIdentifiers/xId1", "datatype": "dict" } , "xrefIdentifiers_xId1_xId": { "path": "xrefIdentifiers/xId1/xId" } , "xrefIdentifiers_xId1_xIdType": { "path": "xrefIdentifiers/xId1/xIdType" } , "xrefIdentifiers_xId2": { "path": "xrefIdentifiers/xId2", "datatype": "dict" } , "xrefIdentifiers_xId2_xId": { "path": "xrefIdentifiers/xId2/xId" } , "xrefIdentifiers_xId2_xIdType": { "path": "xrefIdentifiers/xId2/xIdType" } , "xrefIdentifiers_xId3": { "path": "xrefIdentifiers/xId3", "datatype": "dict" } }
Also there was updated generation of resolution service item to support xId1-xId20 identifiers as ids for resolution. Example:
[ "resolution_service", "defaultassetresolutionservice", { "services": [ { "id": "defaultassetresolutionservice", "type": "Asset", "output": { "columns": "security_alias=securityAlias", "errormessage": "error_message" } , "condition": "", "instructions": { "mode": "'indata'", "ids": [ [ "|primaryAssetId|", "|primaryAssetType|", "'primaryAssetId'" ], [ "|cusip|", "'CUSIP'", "'cusip'" ], [ "|xrefIdentifiers_xId1_xId|", "|xrefIdentifiers_xId1_xIdType|", "'xId1'" ], [ "|xrefIdentifiers_xId2_xId|", "|xrefIdentifiers_xId2_xIdType|", "'xId2'" ], ... [ "|xrefIdentifiers_xId20_xId|", "|xrefIdentifiers_xId20_xIdType|", "'xId20'" ] ], "primaryExchange": "|primaryExchange|", "securityAlias": "|securityAlias|", "assetCurrency": "|assetCurrency|", "localCurrency": "|localCurrency|", "issueCountryCode": "|issueCountryCode|", "maturityDate": "|maturityDate|", "battingOrder": "|battingOrder|", "options": "|assetResolutionOption|" } } ] } ]

SDP-46599 pyruleservice install error: "find: missing argument to `-exec'"

Corrected issue with pyruleservice install

Corrected an issue with pyruleservice install due to a missing terminator in an operating system command

SDP-46506 EagleML Installer Fails If There Are Compressed/Binary Files in logs/starweb/msgcenter

Corrected Installer issue with zipped files in logs/starweb/msgcenter directory

Corrected an issue with the EagleML installer trying to parse a compressed log file in the logs/starweb/msgcenter/ directory

SDP-46154 EDS extract - Provide ability to have custom header and sorted data

Enabled a user defined header and sorted data in EDS output

Enabled user defined headers and trailers along with sorted data in EDS output

SDP-46139 Change time to live from 3 hours to 1 hour for userGroups cache for ReST call

Changed time to live from 3 hours to 1 hour for userGroups cache for ReST calls

The BusinessGroup of front end users is cached within MC2 for ReST API extracts. If the front end user is not a super user, the user's business group is included in the request as an entity filter. The time to live for the cache was changed to 1 hour. Changes to the front end user's business group will be reflected within MC2 after 1 hour. The cache can be reinitialized on demand by reinitializing the MC2 rule cache. The time to live can also be set to a custom value in estar/tpe/cfg/eaglemc2s/extractservice/application.yml with an entry as below
(see application.yml.template):
eaglemlrestapi:
services:
EXTRACT:
usergroupscache:
#lifetime of every group name in cache separately
lifetime:
#set hours to 0 if only minutes parameter is needed
hours: 0
minutes: 1

SDP-46054 Incorrect cleanup period of waitevents

Corrected issue with the cleanup of waitevents

Corrected issue with the cleanup of waitevents: increased the interval for cleanup of queue waitevents

SDP-45619 Change workflow_restart rule to avoid delays on MC2 workflow start

Updated MC2 to enable concurrent start of multiple scheduled workflows

Updated MC2 to route RTRs to eagle_ml-2-0_extract_service_control_message stream to enable concurrent start of multiple scheduled workflows

SDP-45266 EDS: OutputFormat shouldn't be case sensitive in rdf (new ontology) extracts

updated EDS extracts with new metadata to make outputFormat case insensitive

updated EDS extracts with the new metadata to make outputFormat parameter case insensitive

SDP-44169 Expression parser API uses SimpleDateFormat in a potentially thread unsafe mode

Updated MC expression parser to use classes from java.time package

SDP-43725 Use Consistent Time for all healthcheck monitoring type alerts

Updated Health Check to use local time instead of UTC

SDP-41331 Exceptions in the MC2 log with generic extract with nuggetizer

Corrected MC2 issue with nuggetizer used with generic extract

Corrected MC2 issue with nuggetizer when used with generic extract

SDP-45951 EDS: enable display of onStart and onEnd rows when no data and allowEmptyFiles = Y

enabled the display of onStart header and onEnd trailer with no data found

In EDS, enabled the display of user defined onStart header and onEnd trailer in the result when no data is found and allowEmptyFiles parameter = Y.

SDP-45950 EBS: Data header not displayed with source mashup and allowEmptyFiles ='Y' in case of no data

Corrected EDS issue when the data header was not in the output

Corrected EDS issue when the data header was not in the output as expected under these conditions:
interface includes source mashup,
allowemptyfiles is Y,
outputformat is CSV with header,
and no data is found

SDP-45355 Update caching policy for the EDS reportdictionary object to FULL from NONE

Updated caching policy for the reportdictionary object

Changed caching mode in the reportdictionary dataset.
"cache_mode": "none" -> "cache_mode": "full"
Also changed property that defines where the cache is saved.
"cache_provider": "cache-manager-file" -> "cache_provider": "cache-manager-default"

SDP-45253 Change 'hasheader' behavior to not affect OnStart Headers

EDS: updated the control of data versus custom headers

EDS: updated to allow independent control of the display of column (data) header versus custom (onStart) header

SDP-45353 Ontology: Improve TaskStatus Resource to add information about errors

EDS TaskStatus extract enhancements

EDS TaskStatus extract has been enhanced to return additional information, including the error description of the task. The TaskStatus ontology resource supports all common filters as well as several additional filters such as queryoriginalcorrelationid, queryparentcorrelationid. The Task Status resource was updated to support new fields: PACE_MASTERDBO.ORCH_QUEUE.ORIG_CORRELATION_ID PACE_MASTERDBO.ORCH_QUEUE.PROCESS_CORRELATION_ID PACE_MASTERDBO.ORCH_QUEUE.BUS_TASK_ID PACE_MASTERDBO.ORCH_QUEUE.CREATE_DATE PACE_MASTERDBO.ORCH_QUEUE.UPDATE_DATE PACE_MASTERDBO.ORCH_QUEUE.ORCH_STATE_CLOB
Xpath:
EagleML/monitoringTransaction/taskStatus/originalCorrelationId EagleML/monitoringTransaction/taskStatus/parentCorrelationId EagleML/monitoringTransaction/taskStatus/workflowName EagleML/monitoringTransaction/taskStatus/taskName EagleML/monitoringTransaction/taskStatus/startDateTime EagleML/monitoringTransaction/taskStatus/endDateTime EagleML/monitoringTransaction/taskStatus/errorDescription Files changed: metadata/core/ontology/definition/eagle-vocab-definition-common.json metadata/core/ontology/eagle-vocab-common.json metadata/core/ontology/eagle-vocab-enumerations.json metadata/core/ontology/monitoring/taskstatus.json metadata/core/ontology/monitoring/taskstatus-models.json

SDP-46035 Validate user entitlements for Report Rule, Source Rule, Field Rule and few other GET methods of OLAP REST API (via eagle/report/v1/ PACEOLAP REST endpoints)

Unified user Entitlement approach in OLAP REST API GET methods-Phase 2

The GET methods for Report Rule, Field Rule, Source Rule, Benchmark Definition, Grouping rule Range and Filter Fields of the OLAP REST API endpoints have been enhanced to support the unified user Entitlement approach. The method used is the same as what is supported with NXG user entitlements. The results of the GET method extracts will be based on the user profile configurations. The entitlements define the level of access for each user profile based on Center Role, Pace Role, Business User configurations and User's functional permissions. Prior to this enhancement all these GET endpoints did not verify user entitlements prior to fetching the respective data.