MC2 Release Notes - March 2020

Owned by Pamela Totas (Deactivated)
Last updated: Jun 05, 2024
4 min read

SDP-33782 Python XML2XML: Add possibility to map nodes under the header node after objectType

Add possibility to map message header and transaction header nodes

Add possibility to map message header and transaction header nodes. Previously header could only be mapped in Delimited or Fixed formats using dest_items. For the dialect output where taxonomy element is used handling of elements "txn_h" and "h" in dest_items was also implemented.
Example:
"dest_items":

{ "h": [ [[ ["'custom_element_value_h'", "", false, null, "", "custom_header_node/custom_header_element_1"], [":investmentType:", "", false, null, "", "custom_header_node/custom_header_element_2"], ["get_instruction_param('custom_header_element_3', :GVAR_RTR:)", "", false, null, "", "custom_header_node/custom_header_element_3"], ["'custom_header_element_5_value'", "", false, null, "", "custom_header_element_5"]], "True", null] ], "txn_h": [ [[ ["'Test value for objectDescription'", "", false, null, "", "objectDescription"], ["'EffeectiveDate~Today'", "", false, null, "", "calcDates"], ["'N'", "", false, null, "", "checkActivity"], [":putCallIndicator:", "", false, null, "", "custom_transaction_header_node/custom_header_element_7"], ["'custom_header_element_8_value'", "", false, null, "", "custom_transaction_header_node/custom_header_element_8"], ["get_instruction_param('custom_header_element_9_value', :GVAR_RTR:)", "", false, null, "", "custom_transaction_header_node/custom_header_element_9"], ["convert(Date(|accrualDate|,'mm/dd/yyyy'), 'yyyy-mm-dd')", "", false, null, "", "accrualDate"]], "True", null] ] }

,

"taxonomy": [
...
],

SDP-33748 Installer error without description for corrupted file unzip and artifactory mc2.zip download time out error

Updated Installer to provide error message and exit with error in case of download timeout or corrupted archive

When artifactory download timed out for the installer, the stream was not failing and the time out error was hidden. The code was changed to fail the stream when artifactory download timed out to display the error.

Also, when the installer unzip was corrupted and failed there wasn't an error description. There was code added to display an error description when the unzip step failed.

SDP-33578 distribution profiles with ForceExtractFileCreation work incorrectly for SourceMashups

The distribution profiles issues have been fixed for EBS SourceMashup extracts.

The distribution profile issues are resolved for EBS source mashup requests with distribution profile configured.

SDP-33558 Create interfaces dataset that joins by the instance

Create interface dataset that joins by instance

Originally the EDS interfaces dataset was joining by source_name, but now it will join using instance instead.

SDP-33532 MC2 Generic Load profiles: add DECRYPT rule element to MC2

Added ability to decrypt inbound data in MC2

Added the ability to decrypt incoming gpg/pgp files in dfa stream and also made these changes backward compatible with classic mc generic load profile.

SDP-33507 issue with the effectivedaterule and ExtractNamePatternEx parameters for EBS extracts.

The ExtractNamePatternEx effectivedaterule issue for EBS extracts has been resolved.

If effectivedaterule and ExtractNamePatternEx with @effectivedate pattern parameters are specified, the extract file will contain effectivedate as a part of file name.

SDP-33479 MC2 Generic Load profiles: eagle_ml-2-0_default_cm_dfa has to support load profile datamap files

The MC2 eagle_ml-2-0_default_cm_dfa stream supports generic load profiles

As part of MC2 generic load profiles support we improved the MC2 DFA stream to parse and process generic load profiles.

SDP-33449 ForceExtractFileCreation does not work correctly - there is no headers in result files, while testing interfaces that contain source mashups

The ForceExtractFileCreation parameter issue has been fixed for EBS source mashups extracts

The EBS source mashup now creates empty file for if status is no_data and ForceExtractFileCreation is enabled.

SDP-32908 EDS Held Only filter fails to use the entity selection parameters

Corrected EDS Held Only filter

Corrected the EDS Held only filter to work in conjunction with the temp table used by the Entity filter.

SDP-32484 EntityXrefs were not loaded with GenericEntity mc2 loader

EntityXrefs were not loaded with GenericEntity mc2 loader

When using the MC2 loader for GenericEntity the EntityXrefs are now correctly being applied on the inbound.

SDP-34270 Change Python Deployment To Not Update The Python PATH In The User Profile

No longer updating bash_profile on python deployment

SDP-33793 Add support for using custom datasets with default query profiles

Support for Default Query Profile in Datasets

Default Query Profile is used to reduce number of fields in dataset query. One of the main goals of using default query profiles is for caching data for such static core datasets like GenericSMF, Codevalue, etc. (which do not change every day and fully cached) . For example, some clients do not need all Generic SMF data (~2100 fields) cached and use in mashups only less than 10% of fields. And in order not to cache all fields and make cache reload process faster, the default query profile will be introduced.

File Name, Id and location convention for default query profile:

  • File Name: _ {dataframe_name}

default.json Example: “codevalue_default.json”

  • Id: For default profile should be the the same as file name

  • Location: “eagle/estar/tpe/dynamic/matadata/custom/dataset/profile”

Default query profile will be applied during the initial request generated, when source query policy is created.
For current stage if default query profile exists in region, it will be used by default.

Example of profile:
{
"_id": "codevalue_default",
"sourceTable":
[

{ "resource": "eagledb:REFERENCE.CODE_VALUES", "selectColumns": "CODE_INST,LONG_DESC,SHORT_DESC,UPD_OPER" }

,

{ "resource": "eagledb:REFERENCE.INTERFACES", "selectColumns": "SHORT_DESC=SOURCE_NAME" }

]
}

SDP-32125 MC2-kafka messaging: implement java logic to work with kafka eagle streams configuration

Updated MC2 java to work with kafka streams

SDP-33380 Add the x-eagle-business-task-id to the Swagger UI for EQL

Added x_eagle_business_task_id parameter to EQL interface in swagger UI

The business task id parameter was not available in Swagger UI, so we added x_eagle_business_task_id parameter,

SDP-33315 Reply file not created with unique name in response to EQL request with error

Updated eql ReST endpoint to create response file with unique name

When there was an EQL request that failed, the TSR created was not a unique file name. The code was updated so that a unique file name was created even when an EQL request failed.

SDP-33127 Create ESTAR_PORTFOLIO_VALUATION EDS dataset

Create ESTAR_PORTFOLIO_VALUATION EDS

Created portfoliovaluation EDS dataset that pulls data from the ESTAR_PORTFOLIO_VALUATION table in Eagle. Can be used as an EDS Dataset call or used in Source Mashup.

SDP-33112 EQL endpoint returns incorrect Content-Type header value

Updated eql ReST endpoint to return correct content-type in response header

Content-Type header wasn’t set correspondingly to the response for EQL endpoint, so it had the default value application.json. This was corrected to show the appropriate file type according to the request.

SDP-31195 Ability to run multi processing transformation in EBS

Added ability to run multi processing transformation

Added option to transform data using multiprocessing to reduce transformation time.
It works for main data set and for additional destinations. This option can be enable via RTR parameters.
<taskParameter>
<name>multiproc</name>
<value>Y</value>
</taskParameter>
<taskParameter>
<name>partitioncount</name>
<value>2</value>
</taskParameter>
partitioncount can be any integer value and all result files are split on the several parts (chunk count = partitioncount).

SDP-31192 Add new cache mode - BLOB

New cache mode "blob"

Added new cache mode "blob" for data which does not have unique key.
Data and header are saved in cache with one common key which is caclulated with RTR filters.

To use this mode set in dataset file:
"source_query_policy" : {
"cache_mode": "blob",
...

Files changed:
def_enums.py
idsdataframe.py