Secure REDIS Connection
Starting with the April 2021 release MC2 supports connecting to REDIS using TLS.
This is not enabled by default and requires manual configuration.
See: MC2+Redis+TLS+Configuration
SDP-40013 Add logic to register logical and unique key for each data set
Added ability to define additional physical keys
Added ability to define additional physical keys along with unique key in dataset and use them further in mashup. Additional keys set is placed in "index" element: "unique_key": [ {"field": "XREF_SECURITY_ALIAS"} ], "index": [ { "_id": "id_type", "key": [ {"field": "XREF_SECURITY_ID"} , {"field": "XREF_TYPE"} ] } ],
SDP-40737 MC2 execution of EDS requests using Kafka messaging
MC2 now executes EDS requests via Kafka.
MC2 now executes EDS requests via Kafka messaging. If request is sent via RTR the following line will appear in Message Status Detail:
Sending the python rule execution request for processing on Kafka endpoint with IP x.x.x.x.
For details see: MC2+execution+of+EDS+requests+through+Kafka
SDP-44553 Development: enhance processing rule generator to get identifier for resolution from ontology
Enhanced processing rules generator to get identifiers and resolution option from ontology
The generator of processing rules was enhanced to get identifiers and resolution option from ontology eagle-resolution.json instead of using predefined set. The eagle-resolution.json ontology was developed in SDP-46940. The identifiers and options are used for preparing resolution service item definition in processing rules. Example of generated resolution service item definition for asset resolution: [ "resolution_service", "defaultassetresolutionservice", { "services": [ { "id": "defaultassetresolutionservice", "type": "Asset", "output": { "columns": "security_alias=securityAlias", "status_columns": "asset_error_message=errorMessage" } , "condition": ":gvarDestFilter: = 'N'", "instructions": { "mode": "'indata'", "ids": [ ... [ "|cusip|", "'CUSIP'", "'cusip'" ], ... [ "|xrefIdentifiers_xId1_xId|", "|xrefIdentifiers_xId1_xIdType|", "'xId1'" ], ... ], "assetCurrency": "|assetCurrency|", "battingOrder": "|battingOrder|", "maturityDate": "|maturityDate|", "nraTaxCountry": "|issueCountryCode|", "optionCode": [ { "assetAll": "'ALL'" } , { "assetCurrency": "'ASSET_CURRENCY'" } , { "assetMatchAll": "'MATCH_ALL'" } , { "assetMatchIter": "'MATCH_ITER'" } , { "maturityDate": "'MATURITY_DATE'" } , { "nraTaxCountry": "'NRA_TAX_COUNTRY'" } , { "xrefExchange": "'XREF_EXCHANGE'" } ], "options": "|assetResolutionOption|", "xrefExchange": "|primaryExchangeCode|" } } ] } ],
SDP-45609 Enhance match mashup for datasets with different headers
Improved match mashup for datasets with different headers
Changed match expression generation. As a result match mashup for datasets with different headers works correctly and does not contain any errors.
SDP-46092 New Metadata - paceattribute
Created paceattribute ontology
The Ontology for paceattribute resource was created. Files added: estar/tpe/dynamic/metadata/core/ontology/reference/paceattribute.json estar/tpe/dynamic/metadata/core/ontology/reference/paceattribute-models.json
Extract method is supported for paceattribute resource.
Available filters : sourcename fromdate/todate fieldattributeid Mashups used: interfaces
SDP-47067 Allow users to retrieve daterule value in EDS runtime
Enabled EDS runtime to evaluate daterules
Enabled EDS runtime to evaluate daterules, e.g. inside an interface definition:
"t_start_items_def": [
[ "daterule", "start_daterule", "'Yesterday'", "", "1" ],
[ "gvariable", "evaluated_daterule", ":start_daterule_RESULT:"],
[ "gvariable", "daterule_errormsg", ":start_daterule_ERROR_MSG:"] ]
SDP-47274 Create new Metadata/Ontology for SMFidentifiers interface
Created SMFidentifiers ontology
The Ontology for smfidentifiers resource was created. Also added support for history mode. Files added:
estar/tpe/dynamic/metadata/core/ontology/reference/smfidentifiers.json
estar/tpe/dynamic/metadata/core/ontology/reference/smfidentifiers-models.json
Extract method is supported for smfidentifiers resource.
Available filters : sourcename fromdate/todate securityalias processingsecuritytype securitytype investmenttype releasestatus primaryassetid ticker xrefid xrefidtype securitysubtype
SDP-47718 Change the MC2 installer to preserve TLS parameters in redis.conf and sentinel.conf
Updated MC2 install to preserve redis TLS parameters if present
Updated MC2 install to preserve TLS related parameters if present in the REDIS configuration:
tls-port
tls-cert-file
tls-key-file
tls-ca-cert-file
tls-ca-cert-dir
tls-dh-params-file
tls-auth-clients
tls-replication
tls-cluster
tls-protocols
tls-ciphers
tls-ciphersuites
tls-prefer-server-ciphers
tls-session-caching
tls-session-cache-size
tls-session-cache-timeout
SDP-47720 Change MC2 packager and installer to install REDIS 6
MC2 now deployed with REDIS 6.2.1
MC2 now deployed with REDIS 6.2.1
SDP-47722 Change MC2 Pyruleservice starter to pass TLS related parameters to pyruleservice
Updated pyruleservice starter to pass parameters necessary to support secure REDIS connections
For instructions for manually enabling secure REDIS connections see: MC2+Redis+TLS+Configuration
SDP-47754 When disabling the MC2JMS stream the consumers are not removed from the queue.
Updated MC2 for inbound JMS streams to stop reading messages from queue when the stream is disabled.
Updated MC2 so that when an inbound MC2 JMS stream is disabled or removed, MC2 stops consuming messages from the JMS queue. Subsequent messages will accumulate within the JMS queue.
SDP-47898 Enhance Ontology Canonical Model Output for source_type_dialect
Enhanced ontology canonical model output for source_type_dialect
Enhanced code to fill source_type_dialect for mashup and absent in DB elements with number and date types.
SDP-47926 Change the MC2 installer to preserve password in REDIS config files if it is present
Update MC2 Installer to preserve REDIS password if enabled
Update MC2 Installer to preserve REDIS password if the REDIS configuration file requires and specifies a password. Note: this story applies to extractservice only install. AllServices install was updated in a prior story.
SDP-47980 Create updated JRE11 distributable for MC2
Updated MC2 jre version to 11.0.10
MC2 is now shipping with openjdk version "11.0.10" 2021-01-19
SDP-48016 Profiled generic load stays in In Progress status
Corrected an issue with profiled generic load
Corrected an issue python wrf file generator.
SDP-48076 Correct clean up to avoid incorrect linking in load subprocesses in profiled generic load
Corrected an issue with multiple steps inside a profiled generic load
Corrected an issue with multiple steps inside a profiled generic load, when the load would be stuck on the first step and never proceed to the next step.
SDP-48221 Add parser for PDF to EDS code
Add parser for PDF to EDS code
Added python packages openpyxl and pdfminer to pypy in order to be able to parse PDF's and Excel formats into an EDS output.
SDP-48268 Create Generic CA EDS Feed
Created issuefxrate EDS Resource
The Ontology for issuefxrate resource was created. Files added:
estar/tpe/dynamic/metadata/core/ontology/reference/issuefxrate.json
estar/tpe/dynamic/metadata/core/ontology/reference/issuefxrate-models.json
Extract method is supported for issueprice resource.
Available filters : todate fromdate effectivedate fromeffectivedate toeffectivedate processingsecuritytype securitytype investmenttype releasestatus primaryassetid xrefid ticker securitysubtype securityalias sourcename heldonlyfilter
Mashups used: interfaces genericsmf
SDP-48388 Adjust generated ontology for Warehouse resources
Generated Full Ontology for Warehouse Resources
We generated the full Ontology for the Warehouse Resources available from the EML XSD.
SDP-48391 Adjust generated ontology for Reference resources
Adjust generated ontology for Reference
We generated the full Ontology for the Reference Resources available from the EML XSD.
SDP-48434 Adjust generated ontology for Entity resources
Adjusted generated ontology for Entity resources
We generated the full Ontology for the Entity Resources from the EML XSD.
SDP-48575 Change the MC2 installer to preserve additional TLS parameters in sentinel.conf
Updated MC2 install to preserve additional redis TLS parameters if present
Updated MC2 install to preserve TLS related parameters if present in the REDIS configuration:
tls-port
tls-cert-file
tls-key-file
tls-ca-cert-file
tls-auth-clients
tls-replication