SDP-41679 Change the EQL endpoint to use Kafka to communicate with EDS directly from MC2 LoadBalancer
Add ability for MC2 to process EQL endpoint REST requests through kafka
Added the ability for MC2 to process EQL endpoint REST requests directly through kafka for improved throughput.
Performance tests indicate a 5x improvement in performance of EDS requests via the EQL REST endpoint.
Note: REST requests to the EQL endpoint will no longer be visible in EJM Tasks tab as the message detail is no longer written to database.
The new functionality may be fully reverted by adding to or creating estar/tpe/dynamic/mc2/cfg/extractservice.yml with the below and restarting extractservicelb:
---
spring:
profiles: extractservice-lb
application:
name: extractservice-lb
eagle.eaglemlrestapi.services.EQL.directexecution: false
Details on configuring logging level of the new functionality:
EQL Rest Requests Configuration
SDP-44624 Asynchronous Task update does not remove the task information from REDIS, leading to a memory leak
Updated asynchronous status updates to guarantee removal of entries from Redis cache
Updated asynchronous status updates to prevent a memory leak by guaranteed removal of entries from Redis cache. This includes adding a time to live value for these cache entries, taskStatusTTLSeconds, configured in application.yml.
SDP-44598 Update installer to backup and remove folder estar/tpe/dynamic/pyrules/eagleinvsys/
updated installer to backup and remove estar/tpe/dynamic/pyrules/eagleinvsys/
updated installer to backup and remove the folder estar/tpe/dynamic/pyrules/eagleinvsys/
SDP-44548 Increase java heap max size to 1.5 GB for extractservice_lb
Increased java heap max size to 1.5 GB for extractservice_lb
Setting the parameter "-Xmx1500m" in process_guard.xml.template
SDP-44431 MetaData: Add ability to validate output data
Added data validation for eds extract output. (eds resources)
Based on requirements to the data quality, output data validation was added. This data validation can be enabled for any output format via RTR parameter (outputvalidationlevel).It works ONLY for the resources which are generated from Ontology. Following levels of validation are supported. 1. Enumeration (1)2. Type Validation (2) - For now we support only validation for Numeric values.3. ALL (3)
RTR parameter:<taskParameter><name>outputvalidationlevel</name><dataType>S</dataType><value>3</value></taskParameter>
Important note: Record(s) which don't pass data validation will be excluded from result output.
SDP-44427 MetaData: update existing objects and add missing elements - Code Changes
Added extended models to entity and smf mashups
We added extended fields to generic entity and smf extract mashups.
The extended fields can be extracted by applying a flag IncludeExtendedDetails = ALL in rtr:
<taskParameter>
<name>IncludeExtendedDetails</name>
<dataType>S</dataType>
<value>ALL</value>
</taskParameter>
Files added:
- estar\\tpe\\dynamic\\metadata\\core
ontology eagle-report-extended-models
SDP-44340 GraphQL endpoint is not working when used through IIS
Corrected an issue with graphql service when invoked through IIS
Corrected an issue with graphql service when invoked through IIS
SDP-44300 Reduce the amount of disk space used by Kafka
Reduced the disk space used by kafka transaction logs
Added compression and updated the configuration to reduce disk space used by kafka log files.
SDP-44178 New Meta Data issue (EBS_ResourceName=ebs:codevalue,ebs:issuerrating)
Update Delta Based logic for EDS extract with resource
1. Updated logic to store correct value in ORCH table for the single EDS extract. 2. Improve Generic Extarctor workflow to handle DeltaBased and DeltaBasedUID parameters with eds prefix 3. Update cloud workflow so set correct delta parameters
SDP-44167 Change the startup for Python Rule Service to set the environment variable ADAL_PYTHON_SSL_NO_VERIFY=1
Change the startup for Python Rule Service to set the environment variable ADAL_PYTHON_SSL_NO_VERIFY=1
Setting the environment variable ADAL_PYTHON_SSL_NO_VERIFY=1 in application.yml to prevent an issue with connecting to Azure.
SDP-44166 Need to implement ability to compare strings with pattern list in eds interface
Created match t-item
Match class was created to compare string with patterns and file masks.
Here is a Match t-item: ["match", "filename_match",
{"result_is_boolean": false,"dynamic_pattern": false,"pattern_sep": "," ,"pattern_is_file_mask": true}
, ["|filename|"], [":patterns:"], ["stringlength(:patterns:)!=0"]]
- result_is_boolean can be set as true or false (true is default value). Allows to setup return value. When parameter is true then returns value can be Y or N value, if it false then match returns the 1st pattern value that was defined in Item 4.
- dynamic_pattern can be set as true or false (false is default value). Allows to use calculated pattern in eds interface when it is defined as true. When it is false then as pattern will be use the 1st pattern value.
- pattern_sep can be define as char or several chars. Allows to use string as list patterns that is divided by character separator.
- pattern_is_file_mask can be set as true or false (false is default value). Allows using file mask special characters in patterns when it is true. If it is false then pattern should be used with regular expression special characters.
- Item 3 is a string that has to be compared with pattern.
- Item 4 is a string pattern value.
- Item 5 is condition for execution.
SDP-43884 Need to add ADDITIONAL_DATA parameter in executeeds
Added ability to get results of executeeds execution
Added ability to get results of executeeds in
{executeeds_name}
_DATA variable when "storedataintovar" task parameter is defined as 'Y' in RTR.
SDP-43833 Additional improvements for kafka configuration
Additional improvements for kafka configuration
Additional improvements to make kafka more resilient to configuration errors.
SDP-43813 Update mashup fields using ConditionalDirectMapping
Fields expressions are applied dynamically
Extended entity and smf elements mapping was updated by adding new condition element:
{
"@id": "eagle-mashups:IncludeExtendedDetailsCondition",
"@type": "eagle-t-base:Condition",
"eagle-t-base:hasConditionParameter": { "@id": "eagle-interfaces:includeextendeddetails" },
"eagle-t-base:hasSatisfyingValue" : ["ALL", "Y"]
},
If includeextendedetails flag is added to rtr then expression will be applied dynamically.
Example of maturityDate element mapping in eagle-mashups.json:
before changes:
{
"eagle-t-base:hasModelElement": { "@id": "eagle-vocab:maturityDate" },
"eagle-t-base:hasExpressionString": "|{alias}.MAT_DATE| if get_instruction_param('IncludeExtendedDetails', :GVAR_RTR:) in ('ALL', 'Y') else ''"
},
after changes:
{
"eagle-t-base:hasModelElement": { "@id": "eagle-vocab:maturityDate" },
"eagle-t-base:hasCondition": { "@id": "eagle-mashups:IncludeExtendedDetailsCondition"},
"eagle-t-base:hasExpressionString": "|{alias}.MAT_DATE|"
},
SDP-43799 Performance improvement for Eagle Streams query
Performance improvement for MC2 stream query
Updated the sql used in MC2 when polling for stream changes
SDP-43715 Exception parsing POST body for simple EagleML extracts via REST
Corrected an issue with parsing json in POST body for REST request to simple EagleML extract
Corrected an issue with parsing json in POST body to REST call having parameters in a single line, e.g.
{\"maxRows\":\"10\",\"outputFormat\":\"json\"}
SDP-43652 Ability to run independently from MC2 workers and in continuous polling run
Ability to run independently from MC2 workers and in continuous polling run
Developed EDS simple orchestrator, which allows running a single logical unit of work continuously. Single unit of work stored in a configuration file, which when parsed is made into RTR. The EDS orchestrator reads the file, creates RTR and sends it to py workers for execution. It then waits until the RTR is executed after which it repeats the process.
SDP-43582 Allow user to define several group footer rows
Ability to define several effective header, footer, group header and group footer rows was added
Now it is possible to define several header, and footer, and group header, and group footer rows in the same manner as it was possible to define several output rows for incoming record on per-row level.
SDP-43519 Add check of POST for simple EagleML extract REST endpoint to TechCert
Added a check of POST method to REST endpoint in the health check stream
Added a check of POST method to REST endpoint in the health check performed by eagle_ml-2-0_default_cm_mc2_healthcheck stream.
SDP-43281 msgexchange has to write file in variable
Added ability to read file for msgexchange t-item
Output expression field was added in MsgExchange class and reading file logic was added in _file_oper method of MsgExchange class.
Here is an example how to read data from file in variable:["msgexchange", "read_data",
{"delivery_method": "file", "target": "expression", "operation": "copy", "source": "file"}
, ["get_temp_file_location()+|file_name|"], [":data:"], ["''"], ["True"]]
- Operation can be set as copy only
- Source can be set as file only
- Delivery method can be set as file only
- Target can be set as expression only
- Item 3 is a path to existed file
- Item 4 is a name of variable that will store data from file
- Item 5 is a credentials that can be empty
- Item 6 is condition to execute
SDP-43144 POST Body Not Passing OutputFormat Correctly to EagleML extracts
Corrected issue with outputFormat parameter in POST body of REST calls
Updated the MC2 REST API to recognize the outputFormat parameter when submitted within the POST body of a REST call to a simple EagleML extract.
SDP-42986 New Metadata - warehouseopenlot
Created warehouseopenlot ontology
The Ontology for warehouseopenlot interface was created.
Files added:
- estar\\tpe\\dynamic\\metadata\\core
ontologywarehouse warehouseopenlot.json - estar\\tpe\\dynamic\\metadata\\core
ontology warehouse warehouseopenlot-models.json
SDP-42781 Need ability to archive in t items
Archive component was developed to be able to create tar (or tar.gz) and zip file based on file list and/or file masks with ability to remove source files and base64 encoding of result file. It also allows to generate file without actual compressing. Samples of usage:
["archive", "save_to_zip_mask",
{"method": "zip", "operation": "pack", "source": "file", "compression_codec": "zlib",
"remove_source_files": true, "base64_encode": false, "use_mask": true}, ["get_temp_file_location() + 'archive_1.txt'", "get_temp_file_location() + 'archive_2.doc?'"],
["'archive_test_mask.zip'"], ["True"]
]
["archive", "save_to_gz_b64",
{"method": "tar", "operation": "pack", "source": "file", "compression_codec": "gzip",
"remove_source_files": false, "base64_encode": true, "use_mask": false},
["GetDirRoot() + 'data/msgcenter/cmw/in/' + 'archive_gzip_1.txt'", "GetDirRoot() + 'data/msgcenter/cmw/in/' + 'archive_gzip_2.txt'"],
["'archive_test_b64.tar.gz'"],
["True"]
]
SDP-42647 Add support for mashup definition in EDS extracts to enable AsOf
Added EDS mashup definition layer to support AsOf functionality
Enabled the use of mashup definitions to specify which underlying mashup detail data object should be used under which conditions.The mashup definition includes the dataframe section which identifies 2 or more data sets and their respective conditions.Example:
"dataframe": {"genericsmf_asof":
{ "condexpr": ":asof_date: != today() and get_global_setting('generic_smf_asof_mode') = 'full'", "cache_update_mode": "dynamic" }
,"genericsmf_asof_delta":
{ "condexpr": "asof_date != today() and get_global_setting('generic_smf_asof_mode') = 'delta'", "effective_for": ":asof_date:" }
,"genericsmf":
{ "condexpr": ":asof_date: = today()" }
}
(asof_date is either user specified effective_date or defaulted to current day)
If specified asof date is not current day and smf is fully updated every day, use genericsmf_asof data object:extract data via db query from security history table for the specified effective date and source
If specified asof date is not current day and smf is updated in delta mode, use genericsmf_asof_delta data object:used cached security history table for the specified date as logical asof date
If specified asof date is current day, use genericsmf dataobject: latest full cache snapshot of security master table
SDP-42430 Large TSR not displayed correctly in EJM MT
Added ability to generate a TSR with large number of file names
Added the ability to generate a TSR with a large number of extract file names, as when using Generic Extract with several extract types along with the nuggetizer.
SDP-42364 Need ability to move/copy files from (s)ftp
Ability to move/copy files from (s)ftp in t-item
To msg exchange component was added feature to read file from (s)ftp with ability to move file to another (s)ftp folder before download. Samples of usage:
["msgexchange", "save_message_from_ftp",
{"delivery_method": "ftp", "target": "file", "operation": "copy", "source": "file"},
["|FTP_FOLDER| + '/' + |FTP_FILENAME|"],
["|FTP_FILENAME|"],
["|FTP_USER| + ':' + |FTP_PASSWORD| + '@' + |FTP_URL| + ':21'"],
["True"]
]
["msgexchange", "save_message_from_sftp",
{"delivery_method": "sftp", "target": "file", "operation": "move", "source": "file"},
["|SFTP_FOLDER| + '/' + |SFTP_FILENAME|"],
["|SFTP_FILENAME|"],
["|SFTP_USER| + ':' + |SFTP_PASSWORD| + '@' + |SFTP_URL| + ':22'"],
["True"]
]
SDP-42303 Need to add copy function for files in msgexchange
Added ability to copy file for msgexchange t-item
Added ability to copy file by msgexchange t-item in data/mc2/temp/ folder.
Copy file logic was added in _file_oper method of MsgExchange class.
Here is an example how to copy files by msgexchange.
["msgexchange", "copy_file",
{"delivery_method": "file", "target": "file", "operation": "copy", "source": "file"}
, ["get_temp_file_location()+|file_name|"], ["'copied_file.txt'"], ["''"], ["True"]]
- Operation can be set as copy only
- Source can be set as file only
- Delivery method can be set as file only
- Target can be set as file only
- Item 3 is a path to existed file
- Item 4 is a file name that will be saved in estar/tpe/data/mc2/temp/ folder
- Item 5 is a credentials that can be empty
- Item 6 is condition to execute
SDP-41714 EDS Interfaces - New Metadata - warehousecashactivity
Created Ontology for warehousecashactivity
We created the Ontology for warehousecashactivity interface.Files added:1) estar\tpe\dynamic\metadata\core\ontology\warehouse\warehousecashactivity.json2) estar\tpe\dynamic\metadata\core\ontology\warehouse\warehousecashactivity-models.json3) estar\tpe\dynamic\metadata\core\ontology\warehouse\warehousecashactivity_in.json
SDP-41693 Vault Interfaces - New Metadata - entityshareclass
Created entityshareclass ontology
The Ontology for entityshareclass interface and entityshareclass in the history mode was created.
Files added:
- estar/tpe/dynamic/metadata/core/ontology/entity/entityshareclass.json
- estar/tpe/dynamic/metadata/core/ontology/warehouse/entityshareclass-models.json
- estar/tpe/dynamic/metadata/core/ontology/warehouse/entityshareclass_in.json
SDP-37394 Python XML2XML: Add possibility to use incoming value from parameter outfilepattern in settings of the additional destination
Added ability to use expressions in file name definition
Added new metadata parameter - outfilepatternexpr - to use expressions in file name definition. Expression can contain any string or symbol, RTR-parameter call, conditions, file pattern expressions. Example of usage:
"outfilepatternexpr" : "IIF(StringLength(get_instruction_param('OutputFormat', :GVAR_RTR: != 0), get_instruction_param('OutputFormat', :GVAR_RTR:,get_instruction_param('maxrows', :GVAR_RTR) + '@CORRID@FeedType_MAINEXPR_.@EXT'"
SDP-30845 Enable cancelling profiled generic load when a new instance has been started
A profiled generic load workflow will now cancel previous instances when a new instance is started
A profiled generic load workflow will now cancel all previous instances when a new workflow instance is started. This functionality is enabled by creating a new schedule for the generated ...wrf_generic_load_schd workflow.
SDP-43693 Allow users to define EDS based translation for profiled generic load
Enable EDS based translators in profiled generic load
Added the ability to reference EDS translators in a Generic Load Profile.
Example:
given an EDS transform tpe/dynamic/metadata/custom/ebs/sdp43693_smf.json,
that transform can be referenced within the Translator attribute of the Generic Load Profile:
eds_load~sdp43693_pos/eds_translate~sdp43693_smf/sdp43693_ent