Page Comparison

SDP-

...

Updated asynchronous status updates to guarantee removal of entries from Redis cache

Updated asynchronous status updates to prevent a memory leak by guaranteed removal of entries from Redis cache. This includes adding a time to live value for these cache entries, taskStatusTTLSeconds, configured in application.yml.

SDP-44598 Update installer to backup and remove folder estar/tpe/dynamic/pyrules/eagleinvsys/

...

41679 Change the EQL endpoint to use Kafka to communicate with EDS directly from MC2 LoadBalancer

Add ability for MC2 to process EQL endpoint REST requests through kafka

Added the ability for MC2 to process EQL endpoint REST requests directly through kafka for improved throughput.

Performance tests indicate a 5x improvement in performance of EDS requests via the EQL REST endpoint.

Note: REST requests to the EQL endpoint will no longer be visible in EJM Tasks tab as the message detail is no longer written to database.

The new functionality may be fully reverted by adding to or creating estar/tpe/dynamic/

...

mc2/

...

updated installer to backup and remove the folder estar/tpe/dynamic/pyrules/eagleinvsys/

SDP-44548 Increase java heap max size to 1.5 GB for extractservice_lb

Increased java heap max size to 1.5 GB for extractservice_lb

Setting the parameter "-Xmx1500m" in process_guard.xml.template

SDP-44431 MetaData: Add ability to validate output data

Added data validation for eds extract output. (eds resources)

Based on requirements to the data quality, output data validation was added. This data validation can be enabled for any output format via RTR parameter (outputvalidationlevel).It works ONLY for the resources which are generated from Ontology. Following levels of validation are supported. 1. Enumeration (1)2. Type Validation (2) - For now we support only validation for Numeric values.3. ALL (3)
RTR parameter:<taskParameter><name>outputvalidationlevel</name><dataType>S</dataType><value>3</value></taskParameter>
Important note: Record(s) which don't pass data validation will be excluded from result output.

SDP-44427 MetaData: update existing objects and add missing elements - Code Changes

Added extended models to entity and smf mashups

We added extended fields to generic entity and smf extract mashups.
The extended fields can be extracted by applying a flag IncludeExtendedDetails = ALL in rtr:
<taskParameter>
<name>IncludeExtendedDetails</name>
cfg/extractservice.yml with the below and restarting extractservicelb:

<sup>---
spring:
profiles: extractservice-lb
application:
name: extractservice-lb

eagle.eaglemlrestapi.services.EQL.directexecution: false</sup>

Details on configuring logging level of the new functionality:

EQL Rest Requests Configuration

...

SDP-44624 Asynchronous Task update does not remove the task information from REDIS, leading to a memory leak

Updated asynchronous status updates to guarantee removal of entries from Redis cache

Updated asynchronous status updates to prevent a memory leak by guaranteed removal of entries from Redis cache. This includes adding a time to live value for these cache entries, taskStatusTTLSeconds, configured in application.yml.

...

SDP-44598 Update installer to backup and remove folder estar/tpe/dynamic/pyrules/eagleinvsys/

updated installer to backup and remove estar/tpe/dynamic/pyrules/eagleinvsys/

updated installer to backup and remove the folder estar/tpe/dynamic/pyrules/eagleinvsys/

...

SDP-44548 Increase java heap max size to 1.5 GB for extractservice_lb

Increased java heap max size to 1.5 GB for extractservice_lb

Setting the parameter "-Xmx1500m" in process_guard.xml.template

...

SDP-44431 MetaData: Add ability to validate output data

Added data validation for eds extract output. (eds resources)

Based on requirements to the data quality, output data validation was added. This data validation can be enabled for any output format via RTR parameter (outputvalidationlevel).It works ONLY for the resources which are generated from Ontology. Following levels of validation are supported. 1. Enumeration (1)2. Type Validation (2) - For now we support only validation for Numeric values.3. ALL (3)
RTR parameter:<taskParameter><name>outputvalidationlevel</name><dataType>S</dataType><value>3</value></taskParameter>
Important note: Record(s) which don't pass data validation will be excluded from result output.

...

SDP-44427 MetaData: update existing objects and add missing elements - Code Changes

Added extended models to entity and smf mashups

We added extended fields to generic entity and smf extract mashups.
The extended fields can be extracted by applying a flag IncludeExtendedDetails = ALL in rtr:
<taskParameter>
<name>IncludeExtendedDetails</name>
<dataType>S</dataType>
<value>ALL</value>
</taskParameter>
Files added:

1. estar\\tpe\\dynamic\\metadata\\core
   ontology eagle-report-extended-models

...

SDP-44340 GraphQL endpoint is not working when used through IIS

...

Match class was created to compare string with patterns and file masks.
Here is a Match t-item: ["match", "filename_match",
{"result_is_boolean": false,"dynamic_pattern": false,"pattern_sep": "," ,"pattern_is_file_mask": true}
, ["|filename|"], [":patterns:"], ["stringlength(:patterns:)!=0"]]

1. result_is_boolean can be set as true or false (true is default value). Allows to setup return value. When parameter is true then returns value can be Y or N value, if it false then match returns the 1st pattern value that was defined in Item 4.

2. dynamic_pattern can be set as true or false (false is default value). Allows to use calculated pattern in eds interface when it is defined as true. When it is false then as pattern will be use the 1st pattern value.

3. pattern_sep can be define as char or several chars. Allows to use string as list patterns that is divided by character separator.

4. pattern_is_file_mask can be set as true or false (false is default value). Allows using file mask special characters in patterns when it is true. If it is false then pattern should be used with regular expression special characters.

5. Item 3 is a string that has to be compared with pattern.

6. Item 4 is a string pattern value.

7. Item 5 is condition for execution.

...

SDP-43884 Need to add ADDITIONAL_DATA parameter in executeeds

...

Extended entity and smf elements mapping was updated by adding new condition element:
{
"@id": "eagle-mashups:IncludeExtendedDetailsCondition",
"@type": "eagle-t-base:Condition",
"eagle-t-base:hasConditionParameter": { "@id": "eagle-interfaces:includeextendeddetails" },
"eagle-t-base:hasSatisfyingValue" : ["ALL", "Y"]
},
If includeextendedetails flag is added to rtr then expression will be applied dynamically.
Example of maturityDate element mapping in eagle-mashups.json:
before changes:
{
"eagle-t-base:hasModelElement": { "@id": "eagle-vocab:maturityDate" },
"eagle-t-base:hasExpressionString": "|{alias}.MAT_DATE| if get_instruction_param('IncludeExtendedDetails', :GVAR_RTR in :) in ('ALL', 'Y') else ''"
},
after changes:
{
"eagle-t-base:hasModelElement": { "@id": "eagle-vocab:maturityDate" },
"eagle-t-base:hasCondition": { "@id": "eagle-mashups:IncludeExtendedDetailsCondition"},
"eagle-t-base:hasExpressionString": "|{alias}.MAT_DATE|"
},

...

Output expression field was added in MsgExchange class and reading file logic was added in _file_oper method of MsgExchange class.
Here is an example how to read data from file in variable:["msgexchange", "read_data",
{"delivery_method": "file", "target": "expression", "operation": "copy", "source": "file"}
, ["get_temp_file_location()+|file_name|"], [":data:"], ["''"], ["True"]]

1. Operation can be set as copy only

2. Source can be set as file only

3. Delivery method can be set as file only

4. Target can be set as expression only

5. Item 3 is a path to existed file

6. Item 4 is a name of variable that will store data from file

7. Item 5 is a credentials that can be empty

8. Item 6

...

1. is condition to execute

...

SDP-43144 POST Body Not Passing OutputFormat Correctly to EagleML extracts

...

The Ontology for warehouseopenlot interface was created.
Files added:

1. estar\\tpe\\dynamic\\metadata\\core
   ontologywarehouse warehouseopenlot.json

2. estar\\tpe\\dynamic\\metadata\\core
   ontology warehouse warehouseopenlot-models.json

...

SDP-42781 Need ability to archive in t items

Archive component was developed to be able to create tar (or tar.gz) and zip file based on file list and/or file masks with ability to remove source files and base64 encoding of result file. It also allows to generate file without actual compressing. Samples of usage:
["archive", "save_to_zip_mask",
{"method": "zip", "operation": "pack", "source": "file", "compression_codec": "zlib",
"remove_source_files": true, "base64_encode": false, "use_mask": true}, ["get_temp_file_location() + 'archive_1.txt'", "get_temp_file_location() + 'archive_2.doc?'"],
["'archive_test_mask.zip'"], ["True"]
]
["archive", "save_to_gz_b64",
{"method": "tar", "operation": "pack", "source": "file", "compression_codec": "gzip",
"remove_source_files": false, "base64_encode": true, "use_mask": false},
["GetDirRoot() + 'data/msgcenter/cmw/in/' + 'archive_gzip_1.txt'", "GetDirRoot() + 'data/msgcenter/cmw/in/' + 'archive_gzip_2.txt'"],
["'archive_test_b64.tar.gz'"],
["True"]
]

...

To msg exchange component was added feature to read file from (s)ftp with ability to move file to another (s)ftp folder before download. Samples of usage:
["msgexchange", "save_message_from_ftp",
{"delivery_method": "ftp", "target": "file", "operation": "copy", "source": "file"},
["|FTP_FOLDER| + '/' + |FTP_FILENAME|"],
["|FTP_FILENAME|"],
["|FTP_USER| + ':' + |FTP_PASSWORD| + '@' + |FTP_URL| + ':21'"],
["True"]
]
["msgexchange", "save_message_from_sftp",
{"delivery_method": "sftp", "target": "file", "operation": "move", "source": "file"},
["|SFTP_FOLDER| + '/' + |SFTP_FILENAME|"],
["|SFTP_FILENAME|"],
["|SFTP_USER| + ':' + |SFTP_PASSWORD| + '@' + |SFTP_URL| + ':22'"],
["True"]
]

...

Added ability to copy file by msgexchange t-item in data/mc2/temp/ folder.
Copy file logic was added in _file_oper method of MsgExchange class.
Here is an example how to copy files by msgexchange.
["msgexchange", "copy_file",
{"delivery_method": "file", "target": "file", "operation": "copy", "source": "file"}
, ["get_temp_file_location()+|file_name|"], ["'copied_file.txt'"], ["''"], ["True"]]

1. Operation can be set as copy only

2. Source can be set as file only

3. Delivery method can be set as file only

4. Target can be set as file only

5. Item 3 is a path to existed file

6. Item 4 is a file name that will be saved in estar/tpe/data/mc2/temp/ folder

7. Item 5 is a credentials that can be

...

1. empty

2. Item 6 is condition to execute

...

SDP-41714 EDS Interfaces - New Metadata - warehousecashactivity

...

The Ontology for entityshareclass interface and entityshareclass in the history mode was created.
Files added:

1. estar/tpe/dynamic/metadata/core/ontology/entity/entityshareclass.json

2. estar/tpe/dynamic/metadata/core/ontology/warehouse/entityshareclass-models.json

3. estar/tpe

...

SDP-41679 Change the EQL endpoint to use Kafka to communicate with EDS directly from MC2 LoadBalancer

Add ability for MC2 to process EQL endpoint REST requests through kafka

...

1. /dynamic/metadata/core/ontology/warehouse/entityshareclass_in.json

...

SDP-37394 Python XML2XML: Add possibility to use incoming value from parameter outfilepattern in settings of the additional destination

...

Added new metadata parameter - outfilepatternexpr - to use expressions in file name definition. Expression can contain any string or symbol, RTR-parameter call, conditions, file pattern expressions. Example of usage:
"outfilepatternexpr" : "IIF(StringLength(get_instruction_param('OutputFormat', :GVAR_RTR: != 0), get_instruction_param('OutputFormat', :GVAR_RTR:,get_instruction_param('maxrows', :GVAR_RTR) + '@CORRID@FeedType_MAINEXPR_.@EXT'"

SDP-30845 Enable cancelling profiled generic load when a new instance has been started

A profiled generic load workflow will now cancel previous instances when a new instance is started

A profiled generic load workflow will now cancel all previous instances when a new workflow instance is started. This functionality is enabled by creating a new schedule for the generated ...wrf_generic_load_schd workflow., get_instruction_param('OutputFormat', :GVAR_RTR:,get_instruction_param('maxrows', :GVAR_RTR) + '@CORRID@FeedType_MAINEXPR_.@EXT'"

...

SDP-30845 Enable cancelling profiled generic load when a new instance has been started

A profiled generic load workflow will now cancel previous instances when a new instance is started

A profiled generic load workflow will now cancel all previous instances when a new workflow instance is started. This functionality is enabled by creating a new schedule for the generated ...wrf_generic_load_schd workflow.

...

SDP-43693 Allow users to define EDS based translation for profiled generic load

Enable EDS based translators in profiled generic load

Added the ability to reference EDS translators in a Generic Load Profile.

Example:

given an EDS transform tpe/dynamic/metadata/custom/ebs/sdp43693_smf.json,

that transform can be referenced within the Translator attribute of the Generic Load Profile:

eds_load~sdp43693_pos/eds_translate~sdp43693_smf/sdp43693_ent

...