MC2 Release Notes - April 06 2021

Owned by Marius Navickas
Last updated: Jun 06, 2024 by Pamela Totas (Deactivated)
4 min read

Secure REDIS Connection

Starting with the April 2021 release MC2 supports connecting to REDIS using TLS.
This is not enabled by default and requires manual configuration.
See: MC2+Redis+TLS+Configuration

SDP-40013 Add logic to register logical and unique key for each data set

Added ability to define additional physical keys

Added ability to define additional physical keys along with unique key in dataset and use them further in mashup. Additional keys set is placed in "index" element: "unique_key": [ {"field": "XREF_SECURITY_ALIAS"} ], "index": [ { "_id": "id_type", "key": [ {"field": "XREF_SECURITY_ID"} , {"field": "XREF_TYPE"} ] } ],

SDP-40737 MC2 execution of EDS requests using Kafka messaging

MC2 now executes EDS requests via Kafka.

MC2 now executes EDS requests via Kafka messaging. If request is sent via RTR the following line will appear in Message Status Detail:
Sending the python rule execution request for processing on Kafka endpoint with IP x.x.x.x.
For details see: MC2+execution+of+EDS+requests+through+Kafka

SDP-44553 Development: enhance processing rule generator to get identifier for resolution from ontology

Enhanced processing rules generator to get identifiers and resolution option from ontology

The generator of processing rules was enhanced to get identifiers and resolution option from ontology eagle-resolution.json instead of using predefined set. The eagle-resolution.json ontology was developed in SDP-46940. The identifiers and options are used for preparing resolution service item definition in processing rules. Example of generated resolution service item definition for asset resolution: [ "resolution_service", "defaultassetresolutionservice", { "services": [ { "id": "defaultassetresolutionservice", "type": "Asset", "output": { "columns": "security_alias=securityAlias", "status_columns": "asset_error_message=errorMessage" } , "condition": ":gvarDestFilter: = 'N'", "instructions": { "mode": "'indata'", "ids": [ ... [ "|cusip|", "'CUSIP'", "'cusip'" ], ... [ "|xrefIdentifiers_xId1_xId|", "|xrefIdentifiers_xId1_xIdType|", "'xId1'" ], ... ], "assetCurrency": "|assetCurrency|", "battingOrder": "|battingOrder|", "maturityDate": "|maturityDate|", "nraTaxCountry": "|issueCountryCode|", "optionCode": [ { "assetAll": "'ALL'" } , { "assetCurrency": "'ASSET_CURRENCY'" } , { "assetMatchAll": "'MATCH_ALL'" } , { "assetMatchIter": "'MATCH_ITER'" } , { "maturityDate": "'MATURITY_DATE'" } , { "nraTaxCountry": "'NRA_TAX_COUNTRY'" } , { "xrefExchange": "'XREF_EXCHANGE'" } ], "options": "|assetResolutionOption|", "xrefExchange": "|primaryExchangeCode|" } } ] } ],

SDP-45609 Enhance match mashup for datasets with different headers

Improved match mashup for datasets with different headers

Changed match expression generation. As a result match mashup for datasets with different headers works correctly and does not contain any errors.

SDP-46092 New Metadata - paceattribute

Created paceattribute ontology

The Ontology for paceattribute resource was created. Files added: estar/tpe/dynamic/metadata/core/ontology/reference/paceattribute.json estar/tpe/dynamic/metadata/core/ontology/reference/paceattribute-models.json
Extract method is supported for paceattribute resource.
Available filters : sourcename fromdate/todate fieldattributeid Mashups used: interfaces

SDP-47067 Allow users to retrieve daterule value in EDS runtime

Enabled EDS runtime to evaluate daterules

Enabled EDS runtime to evaluate daterules, e.g. inside an interface definition:

"t_start_items_def": [

[ "daterule", "start_daterule", "'Yesterday'", "", "1" ],

[ "gvariable", "evaluated_daterule", ":start_daterule_RESULT:"],

[ "gvariable", "daterule_errormsg", ":start_daterule_ERROR_MSG:"] ]

SDP-47274 Create new Metadata/Ontology for SMFidentifiers interface

Created SMFidentifiers ontology

The Ontology for smfidentifiers resource was created. Also added support for history mode. Files added:
estar/tpe/dynamic/metadata/core/ontology/reference/smfidentifiers.json
estar/tpe/dynamic/metadata/core/ontology/reference/smfidentifiers-models.json
Extract method is supported for smfidentifiers resource.
Available filters : sourcename fromdate/todate securityalias processingsecuritytype securitytype investmenttype releasestatus primaryassetid ticker xrefid xrefidtype securitysubtype

SDP-47718 Change the MC2 installer to preserve TLS parameters in redis.conf and sentinel.conf

Updated MC2 install to preserve redis TLS parameters if present

Updated MC2 install to preserve TLS related parameters if present in the REDIS configuration:
tls-port
tls-cert-file
tls-key-file
tls-ca-cert-file
tls-ca-cert-dir
tls-dh-params-file
tls-auth-clients
tls-replication
tls-cluster
tls-protocols
tls-ciphers
tls-ciphersuites
tls-prefer-server-ciphers
tls-session-caching
tls-session-cache-size
tls-session-cache-timeout

SDP-47720 Change MC2 packager and installer to install REDIS 6

MC2 now deployed with REDIS 6.2.1

MC2 now deployed with REDIS 6.2.1

SDP-47722 Change MC2 Pyruleservice starter to pass TLS related parameters to pyruleservice

Updated pyruleservice starter to pass parameters necessary to support secure REDIS connections

For instructions for manually enabling secure REDIS connections see: MC2+Redis+TLS+Configuration

SDP-47754 When disabling the MC2JMS stream the consumers are not removed from the queue.

Updated MC2 for inbound JMS streams to stop reading messages from queue when the stream is disabled.

Updated MC2 so that when an inbound MC2 JMS stream is disabled or removed, MC2 stops consuming messages from the JMS queue. Subsequent messages will accumulate within the JMS queue.

SDP-47898 Enhance Ontology Canonical Model Output for source_type_dialect

Enhanced ontology canonical model output for source_type_dialect

Enhanced code to fill source_type_dialect for mashup and absent in DB elements with number and date types.

SDP-47926 Change the MC2 installer to preserve password in REDIS config files if it is present

Update MC2 Installer to preserve REDIS password if enabled

Update MC2 Installer to preserve REDIS password if the REDIS configuration file requires and specifies a password. Note: this story applies to extractservice only install. AllServices install was updated in a prior story.

SDP-47980 Create updated JRE11 distributable for MC2

Updated MC2 jre version to 11.0.10

MC2 is now shipping with openjdk version "11.0.10" 2021-01-19

SDP-48016 Profiled generic load stays in In Progress status

Corrected an issue with profiled generic load

Corrected an issue python wrf file generator.

SDP-48076 Correct clean up to avoid incorrect linking in load subprocesses in profiled generic load

Corrected an issue with multiple steps inside a profiled generic load

Corrected an issue with multiple steps inside a profiled generic load, when the load would be stuck on the first step and never proceed to the next step.

SDP-48221 Add parser for PDF to EDS code

Add parser for PDF to EDS code

Added python packages openpyxl and pdfminer to pypy in order to be able to parse PDF's and Excel formats into an EDS output.

SDP-48268 Create Generic CA EDS Feed

Created issuefxrate EDS Resource

The Ontology for issuefxrate resource was created. Files added:

estar/tpe/dynamic/metadata/core/ontology/reference/issuefxrate.json

estar/tpe/dynamic/metadata/core/ontology/reference/issuefxrate-models.json


Extract method is supported for issueprice resource.
Available filters : todate fromdate effectivedate fromeffectivedate toeffectivedate processingsecuritytype securitytype investmenttype releasestatus primaryassetid xrefid ticker securitysubtype securityalias sourcename heldonlyfilter
Mashups used: interfaces genericsmf

SDP-48388 Adjust generated ontology for Warehouse resources

Generated Full Ontology for Warehouse Resources

We generated the full Ontology for the Warehouse Resources available from the EML XSD.

SDP-48391 Adjust generated ontology for Reference resources

Adjust generated ontology for Reference

We generated the full Ontology for the Reference Resources available from the EML XSD.

SDP-48434 Adjust generated ontology for Entity resources

Adjusted generated ontology for Entity resources

We generated the full Ontology for the Entity Resources from the EML XSD.

SDP-48575 Change the MC2 installer to preserve additional TLS parameters in sentinel.conf

Updated MC2 install to preserve additional redis TLS parameters if present

Updated MC2 install to preserve TLS related parameters if present in the REDIS configuration:

tls-port
tls-cert-file
tls-key-file
tls-ca-cert-file
tls-auth-clients
tls-replication