MC2 Release Notes - December 15 2020

Owned by Marius Navickas
Last updated: Jun 06, 2024 by Pamela Totas (Deactivated)
8 min read

SDP-41679 Change the EQL endpoint to use Kafka to communicate with EDS directly from MC2 LoadBalancer

Add ability for MC2 to process EQL endpoint REST requests through kafka

Added the ability for MC2 to process EQL endpoint REST requests directly through kafka for improved throughput.

Performance tests indicate a 5x improvement in performance of EDS requests via the EQL REST endpoint.

Note: REST requests to the EQL endpoint will no longer be visible in EJM Tasks tab as the message detail is no longer written to database.

The new functionality may be fully reverted by adding to or creating estar/tpe/dynamic/mc2/cfg/extractservice.yml with the below and restarting extractservicelb:

---
spring:
profiles: extractservice-lb
application:
name: extractservice-lb

eagle.eaglemlrestapi.services.EQL.directexecution: false



Details on configuring logging level of the new functionality:

EQL Rest Requests Configuration

SDP-44624 Asynchronous Task update does not remove the task information from REDIS, leading to a memory leak

Updated asynchronous status updates to guarantee removal of entries from Redis cache

Updated asynchronous status updates to prevent a memory leak by guaranteed removal of entries from Redis cache. This includes adding a time to live value for these cache entries, taskStatusTTLSeconds, configured in application.yml.

SDP-44598 Update installer to backup and remove folder estar/tpe/dynamic/pyrules/eagleinvsys/

updated installer to backup and remove estar/tpe/dynamic/pyrules/eagleinvsys/

updated installer to backup and remove the folder estar/tpe/dynamic/pyrules/eagleinvsys/

SDP-44548 Increase java heap max size to 1.5 GB for extractservice_lb

Increased java heap max size to 1.5 GB for extractservice_lb

Setting the parameter "-Xmx1500m" in process_guard.xml.template

SDP-44431 MetaData: Add ability to validate output data

Added data validation for eds extract output. (eds resources)

Based on requirements to the data quality, output data validation was added. This data validation can be enabled for any output format via RTR parameter (outputvalidationlevel).It works ONLY for the resources which are generated from Ontology. Following levels of validation are supported. 1. Enumeration (1)2. Type Validation (2) - For now we support only validation for Numeric values.3. ALL (3)
RTR parameter:<taskParameter><name>outputvalidationlevel</name><dataType>S</dataType><value>3</value></taskParameter>
Important note: Record(s) which don't pass data validation will be excluded from result output.

SDP-44427 MetaData: update existing objects and add missing elements - Code Changes

Added extended models to entity and smf mashups

We added extended fields to generic entity and smf extract mashups.
The extended fields can be extracted by applying a flag IncludeExtendedDetails = ALL in rtr:
<taskParameter>
<name>IncludeExtendedDetails</name>
<dataType>S</dataType>
<value>ALL</value>
</taskParameter>
Files added:

  1. estar\\tpe\\dynamic\\metadata\\core
    ontology eagle-report-extended-models

SDP-44340 GraphQL endpoint is not working when used through IIS

Corrected an issue with graphql service when invoked through IIS

Corrected an issue with graphql service when invoked through IIS

SDP-44300 Reduce the amount of disk space used by Kafka

Reduced the disk space used by kafka transaction logs

Added compression and updated the configuration to reduce disk space used by kafka log files.

SDP-44178 New Meta Data issue (EBS_ResourceName=ebs:codevalue,ebs:issuerrating)

Update Delta Based logic for EDS extract with resource

1. Updated logic to store correct value in ORCH table for the single EDS extract. 2. Improve Generic Extarctor workflow to handle DeltaBased and DeltaBasedUID parameters with eds prefix 3. Update cloud workflow so set correct delta parameters

SDP-44167 Change the startup for Python Rule Service to set the environment variable ADAL_PYTHON_SSL_NO_VERIFY=1

Change the startup for Python Rule Service to set the environment variable ADAL_PYTHON_SSL_NO_VERIFY=1

Setting the environment variable ADAL_PYTHON_SSL_NO_VERIFY=1 in application.yml to prevent an issue with connecting to Azure.

SDP-44166 Need to implement ability to compare strings with pattern list in eds interface

Created match t-item

Match class was created to compare string with patterns and file masks.
Here is a Match t-item: ["match", "filename_match",
{"result_is_boolean": false,"dynamic_pattern": false,"pattern_sep": "," ,"pattern_is_file_mask": true}
, ["|filename|"], [":patterns:"], ["stringlength(:patterns:)!=0"]]

  1. result_is_boolean can be set as true or false (true is default value). Allows to setup return value. When parameter is true then returns value can be Y or N value, if it false then match returns the 1st pattern value that was defined in Item 4.

  2. dynamic_pattern can be set as true or false (false is default value). Allows to use calculated pattern in eds interface when it is defined as true. When it is false then as pattern will be use the 1st pattern value.

  3. pattern_sep can be define as char or several chars. Allows to use string as list patterns that is divided by character separator.

  4. pattern_is_file_mask can be set as true or false (false is default value). Allows using file mask special characters in patterns when it is true. If it is false then pattern should be used with regular expression special characters.

  5. Item 3 is a string that has to be compared with pattern.

  6. Item 4 is a string pattern value.

  7. Item 5 is condition for execution.

SDP-43884 Need to add ADDITIONAL_DATA parameter in executeeds

Added ability to get results of executeeds execution

Added ability to get results of executeeds in
{executeeds_name}
_DATA variable when "storedataintovar" task parameter is defined as 'Y' in RTR.

SDP-43833 Additional improvements for kafka configuration

Additional improvements for kafka configuration

Additional improvements to make kafka more resilient to configuration errors.

SDP-43813 Update mashup fields using ConditionalDirectMapping

Fields expressions are applied dynamically

Extended entity and smf elements mapping was updated by adding new condition element:
{
"@id": "eagle-mashups:IncludeExtendedDetailsCondition",
"@type": "eagle-t-base:Condition",
"eagle-t-base:hasConditionParameter": { "@id": "eagle-interfaces:includeextendeddetails" },
"eagle-t-base:hasSatisfyingValue" : ["ALL", "Y"]
},
If includeextendedetails flag is added to rtr then expression will be applied dynamically.
Example of maturityDate element mapping in eagle-mashups.json:
before changes:
{
"eagle-t-base:hasModelElement": { "@id": "eagle-vocab:maturityDate" },
"eagle-t-base:hasExpressionString": "|{alias}.MAT_DATE| if get_instruction_param('IncludeExtendedDetails', :GVAR_RTR:) in ('ALL', 'Y') else ''"
},
after changes:
{
"eagle-t-base:hasModelElement": { "@id": "eagle-vocab:maturityDate" },
"eagle-t-base:hasCondition": { "@id": "eagle-mashups:IncludeExtendedDetailsCondition"},
"eagle-t-base:hasExpressionString": "|{alias}.MAT_DATE|"
},

SDP-43799 Performance improvement for Eagle Streams query

Performance improvement for MC2 stream query

Updated the sql used in MC2 when polling for stream changes

SDP-43715 Exception parsing POST body for simple EagleML extracts via REST

Corrected an issue with parsing json in POST body for REST request to simple EagleML extract

Corrected an issue with parsing json in POST body to REST call having parameters in a single line, e.g.
{\"maxRows\":\"10\",\"outputFormat\":\"json\"}

SDP-43652 Ability to run independently from MC2 workers and in continuous polling run

Ability to run independently from MC2 workers and in continuous polling run

Developed EDS simple orchestrator, which allows running a single logical unit of work continuously. Single unit of work stored in a configuration file, which when parsed is made into RTR. The EDS orchestrator reads the file, creates RTR and sends it to py workers for execution. It then waits until the RTR is executed after which it repeats the process.

SDP-43582 Allow user to define several group footer rows

Ability to define several effective header, footer, group header and group footer rows was added

Now it is possible to define several header, and footer, and group header, and group footer rows in the same manner as it was possible to define several output rows for incoming record on per-row level.

SDP-43519 Add check of POST for simple EagleML extract REST endpoint to TechCert

Added a check of POST method to REST endpoint in the health check stream

Added a check of POST method to REST endpoint in the health check performed by eagle_ml-2-0_default_cm_mc2_healthcheck stream.

SDP-43281 msgexchange has to write file in variable

Added ability to read file for msgexchange t-item

Output expression field was added in MsgExchange class and reading file logic was added in _file_oper method of MsgExchange class.
Here is an example how to read data from file in variable:["msgexchange", "read_data",
{"delivery_method": "file", "target": "expression", "operation": "copy", "source": "file"}
, ["get_temp_file_location()+|file_name|"], [":data:"], ["''"], ["True"]]

  1. Operation can be set as copy only

  2. Source can be set as file only

  3. Delivery method can be set as file only

  4. Target can be set as expression only

  5. Item 3 is a path to existed file

  6. Item 4 is a name of variable that will store data from file

  7. Item 5 is a credentials that can be empty

  8. Item 6 is condition to execute

SDP-43144 POST Body Not Passing OutputFormat Correctly to EagleML extracts

Corrected issue with outputFormat parameter in POST body of REST calls

Updated the MC2 REST API to recognize the outputFormat parameter when submitted within the POST body of a REST call to a simple EagleML extract.

SDP-42986 New Metadata - warehouseopenlot

Created warehouseopenlot ontology

The Ontology for warehouseopenlot interface was created.
Files added:

  1. estar\\tpe\\dynamic\\metadata\\core
    ontologywarehouse warehouseopenlot.json

  2. estar\\tpe\\dynamic\\metadata\\core
    ontology warehouse warehouseopenlot-models.json

SDP-42781 Need ability to archive in t items

Archive component was developed to be able to create tar (or tar.gz) and zip file based on file list and/or file masks with ability to remove source files and base64 encoding of result file. It also allows to generate file without actual compressing. Samples of usage:
["archive", "save_to_zip_mask",
{"method": "zip", "operation": "pack", "source": "file", "compression_codec": "zlib",
"remove_source_files": true, "base64_encode": false, "use_mask": true}, ["get_temp_file_location() + 'archive_1.txt'", "get_temp_file_location() + 'archive_2.doc?'"],
["'archive_test_mask.zip'"], ["True"]
]
["archive", "save_to_gz_b64",
{"method": "tar", "operation": "pack", "source": "file", "compression_codec": "gzip",
"remove_source_files": false, "base64_encode": true, "use_mask": false},
["GetDirRoot() + 'data/msgcenter/cmw/in/' + 'archive_gzip_1.txt'", "GetDirRoot() + 'data/msgcenter/cmw/in/' + 'archive_gzip_2.txt'"],
["'archive_test_b64.tar.gz'"],
["True"]
]

SDP-42647 Add support for mashup definition in EDS extracts to enable AsOf

Added EDS mashup definition layer to support AsOf functionality

Enabled the use of mashup definitions to specify which underlying mashup detail data object should be used under which conditions.The mashup definition includes the dataframe section which identifies 2 or more data sets and their respective conditions.Example:
"dataframe": {"genericsmf_asof":
{ "condexpr": ":asof_date: != today() and get_global_setting('generic_smf_asof_mode') = 'full'", "cache_update_mode": "dynamic" }
,"genericsmf_asof_delta":
{ "condexpr": "asof_date != today() and get_global_setting('generic_smf_asof_mode') = 'delta'", "effective_for": ":asof_date:" }
,"genericsmf":
{ "condexpr": ":asof_date: = today()" }
}
(asof_date is either user specified effective_date or defaulted to current day)
If specified asof date is not current day and smf is fully updated every day, use genericsmf_asof data object:extract data via db query from security history table for the specified effective date and source
If specified asof date is not current day and smf is updated in delta mode, use genericsmf_asof_delta data object:used cached security history table for the specified date as logical asof date
If specified asof date is current day, use genericsmf dataobject: latest full cache snapshot of security master table

SDP-42430 Large TSR not displayed correctly in EJM MT

Added ability to generate a TSR with large number of file names

Added the ability to generate a TSR with a large number of extract file names, as when using Generic Extract with several extract types along with the nuggetizer.

SDP-42364 Need ability to move/copy files from (s)ftp

Ability to move/copy files from (s)ftp in t-item

To msg exchange component was added feature to read file from (s)ftp with ability to move file to another (s)ftp folder before download. Samples of usage:
["msgexchange", "save_message_from_ftp",
{"delivery_method": "ftp", "target": "file", "operation": "copy", "source": "file"},
["|FTP_FOLDER| + '/' + |FTP_FILENAME|"],
["|FTP_FILENAME|"],
["|FTP_USER| + ':' + |FTP_PASSWORD| + '@' + |FTP_URL| + ':21'"],
["True"]
]
["msgexchange", "save_message_from_sftp",
{"delivery_method": "sftp", "target": "file", "operation": "move", "source": "file"},
["|SFTP_FOLDER| + '/' + |SFTP_FILENAME|"],
["|SFTP_FILENAME|"],
["|SFTP_USER| + ':' + |SFTP_PASSWORD| + '@' + |SFTP_URL| + ':22'"],
["True"]
]

SDP-42303 Need to add copy function for files in msgexchange

Added ability to copy file for msgexchange t-item

Added ability to copy file by msgexchange t-item in data/mc2/temp/ folder.
Copy file logic was added in _file_oper method of MsgExchange class.
Here is an example how to copy files by msgexchange.
["msgexchange", "copy_file",
{"delivery_method": "file", "target": "file", "operation": "copy", "source": "file"}
, ["get_temp_file_location()+|file_name|"], ["'copied_file.txt'"], ["''"], ["True"]]

  1. Operation can be set as copy only

  2. Source can be set as file only

  3. Delivery method can be set as file only

  4. Target can be set as file only

  5. Item 3 is a path to existed file

  6. Item 4 is a file name that will be saved in estar/tpe/data/mc2/temp/ folder

  7. Item 5 is a credentials that can be empty

  8. Item 6 is condition to execute

SDP-41714 EDS Interfaces - New Metadata - warehousecashactivity

Created Ontology for warehousecashactivity

We created the Ontology for warehousecashactivity interface.Files added:1) estar\tpe\dynamic\metadata\core\ontology\warehouse\warehousecashactivity.json2) estar\tpe\dynamic\metadata\core\ontology\warehouse\warehousecashactivity-models.json3) estar\tpe\dynamic\metadata\core\ontology\warehouse\warehousecashactivity_in.json

SDP-41693 Vault Interfaces - New Metadata - entityshareclass

 Created entityshareclass ontology

The Ontology for entityshareclass interface and entityshareclass in the history mode was created.
Files added:

  1. estar/tpe/dynamic/metadata/core/ontology/entity/entityshareclass.json

  2. estar/tpe/dynamic/metadata/core/ontology/warehouse/entityshareclass-models.json

  3. estar/tpe/dynamic/metadata/core/ontology/warehouse/entityshareclass_in.json



SDP-37394 Python XML2XML: Add possibility to use incoming value from parameter outfilepattern in settings of the additional destination

Added ability to use expressions in file name definition

Added new metadata parameter - outfilepatternexpr - to use expressions in file name definition. Expression can contain any string or symbol, RTR-parameter call, conditions, file pattern expressions. Example of usage:
"outfilepatternexpr" : "IIF(StringLength(get_instruction_param('OutputFormat', :GVAR_RTR: != 0), get_instruction_param('OutputFormat', :GVAR_RTR:,get_instruction_param('maxrows', :GVAR_RTR) + '@CORRID@FeedType_MAINEXPR_.@EXT'"

SDP-30845 Enable cancelling profiled generic load when a new instance has been started

A profiled generic load workflow will now cancel previous instances when a new instance is started

A profiled generic load workflow will now cancel all previous instances when a new workflow instance is started. This functionality is enabled by creating a new schedule for the generated ...wrf_generic_load_schd workflow.

SDP-43693 Allow users to define EDS based translation for profiled generic load

Enable EDS based translators in profiled generic load

Added the ability to reference EDS translators in a Generic Load Profile.

Example:

given an EDS transform tpe/dynamic/metadata/custom/ebs/sdp43693_smf.json,

that transform can be referenced within the Translator attribute of the Generic Load Profile:

eds_load~sdp43693_pos/eds_translate~sdp43693_smf/sdp43693_ent