SDP-44149 EDS Interfaces - New Metadata - entitypolicy
Created entitypolicy Ontology
New ontology of issuerrelationship was developed: \estar\tpe\dynamic\metadata\core\ontology\entity\entitypolicy.json \estar\tpe\dynamic\metadata\core\ontology\entity\entitypolicy-models.json
entitypolicy.json contains sql table joins definition, filters definition, extract rules and etc. entitypolicy-models.json contains fields mapping definition.
SDP-46549 Add IncludedExtendedDetails to Swagger for some EML APIs
Added IncludeExtendedDetails to Swagger for Multiple EML APIs
We added the IncludeExtendedDetails parameter to the Swagger for the following EML APIs:
intraday-navs
master-feeder
preferred-shares
entities
accounting-status
cash-activities
wash-sale
SDP-46376 Ontology: Review and update Numeric fields, change format to 26,6
Ontology: Review and update Numeric fields, change format to 26,6
We changed in the EDS Ontology any numeric fields which can have more them two decimal digits to be 26,6 format.
elem_per_resource (1).json
SDP-45587 exception with filters when using sql-based data object
Fixed error for sql-based datasets with empty filters
Errors occurred when trying to refresh dataset cache for sql-based dataset with empty filters. The same error occurred during interface execution based on this dataset. We revised EDS filter initialization and application.
SDP-45573 Add ability to send logs to another interface during processing
Added ability to send logs to another interface
New property log_process_def was added to interface. It should contain name of EDS interface which is going to process logs.
Examples of usage:
"log_process_def": [{ "_id": "app_logs_sender"}],
Logs being passed as base64 in parameter in RTR log_statuses.
Logs being sent in async mode in following cases:
1 First log captured
2 Captured count of logs % 500 = 0
3 At the end of processing
SDP-44943 MD: expand xrefidentifiers xId1-xId20 in processing rules generation
Enhanced processing rules generator to expand correctly nested nodes and xrefIdentifiers in vocabulary
The processing rules generator was enhanced to expand correctly nested nodes and xrefIdentifiers in vocabulary part of load dataset. This allows correctly processing elements from nested nodes during the EDS load.
Example:
"vocabulary": { "xrefIdentifiers": { "path": "xrefIdentifiers", "datatype": "dict" } , "xrefIdentifiers_xId1": { "path": "xrefIdentifiers/xId1", "datatype": "dict" } , "xrefIdentifiers_xId1_xId": { "path": "xrefIdentifiers/xId1/xId" } , "xrefIdentifiers_xId1_xIdType": { "path": "xrefIdentifiers/xId1/xIdType" } , "xrefIdentifiers_xId2": { "path": "xrefIdentifiers/xId2", "datatype": "dict" } , "xrefIdentifiers_xId2_xId": { "path": "xrefIdentifiers/xId2/xId" } , "xrefIdentifiers_xId2_xIdType": { "path": "xrefIdentifiers/xId2/xIdType" } , "xrefIdentifiers_xId3": { "path": "xrefIdentifiers/xId3", "datatype": "dict" } }
Also there was updated generation of resolution service item to support xId1-xId20 identifiers as ids for resolution. Example:
[ "resolution_service", "defaultassetresolutionservice", { "services": [ { "id": "defaultassetresolutionservice", "type": "Asset", "output": { "columns": "security_alias=securityAlias", "errormessage": "error_message" } , "condition": "", "instructions": { "mode": "'indata'", "ids": [ [ "|primaryAssetId|", "|primaryAssetType|", "'primaryAssetId'" ], [ "|cusip|", "'CUSIP'", "'cusip'" ], [ "|xrefIdentifiers_xId1_xId|", "|xrefIdentifiers_xId1_xIdType|", "'xId1'" ], [ "|xrefIdentifiers_xId2_xId|", "|xrefIdentifiers_xId2_xIdType|", "'xId2'" ], ... [ "|xrefIdentifiers_xId20_xId|", "|xrefIdentifiers_xId20_xIdType|", "'xId20'" ] ], "primaryExchange": "|primaryExchange|", "securityAlias": "|securityAlias|", "assetCurrency": "|assetCurrency|", "localCurrency": "|localCurrency|", "issueCountryCode": "|issueCountryCode|", "maturityDate": "|maturityDate|", "battingOrder": "|battingOrder|", "options": "|assetResolutionOption|" } } ] } ]
SDP-46599 pyruleservice install error: "find: missing argument to `-exec'"
Corrected issue with pyruleservice install
Corrected an issue with pyruleservice install due to a missing terminator in an operating system command
SDP-46506 EagleML Installer Fails If There Are Compressed/Binary Files in logs/starweb/msgcenter
Corrected Installer issue with zipped files in logs/starweb/msgcenter directory
Corrected an issue with the EagleML installer trying to parse a compressed log file in the logs/starweb/msgcenter/ directory
SDP-46154 EDS extract - Provide ability to have custom header and sorted data
Enabled a user defined header and sorted data in EDS output
Enabled user defined headers and trailers along with sorted data in EDS output
SDP-46139 Change time to live from 3 hours to 1 hour for userGroups cache for ReST call
Changed time to live from 3 hours to 1 hour for userGroups cache for ReST calls
The BusinessGroup of front end users is cached within MC2 for ReST API extracts. If the front end user is not a super user, the user's business group is included in the request as an entity filter. The time to live for the cache was changed to 1 hour. Changes to the front end user's business group will be reflected within MC2 after 1 hour. The cache can be reinitialized on demand by reinitializing the MC2 rule cache. The time to live can also be set to a custom value in estar/tpe/cfg/eaglemc2s/extractservice/application.yml with an entry as below
(see application.yml.template):
eaglemlrestapi:
services:
EXTRACT:
usergroupscache:
#lifetime of every group name in cache separately
lifetime:
#set hours to 0 if only minutes parameter is needed
hours: 0
minutes: 1
SDP-46054 Incorrect cleanup period of waitevents
Corrected issue with the cleanup of waitevents
Corrected issue with the cleanup of waitevents: increased the interval for cleanup of queue waitevents
SDP-45619 Change workflow_restart rule to avoid delays on MC2 workflow start
Updated MC2 to enable concurrent start of multiple scheduled workflows
Updated MC2 to route RTRs to eagle_ml-2-0_extract_service_control_message stream to enable concurrent start of multiple scheduled workflows
SDP-45266 EDS: OutputFormat shouldn't be case sensitive in rdf (new ontology) extracts
updated EDS extracts with new metadata to make outputFormat case insensitive
updated EDS extracts with the new metadata to make outputFormat parameter case insensitive
SDP-44169 Expression parser API uses SimpleDateFormat in a potentially thread unsafe mode
Updated MC expression parser to use classes from java.time package
Updated MC expression parser to use classes from java.time package
SDP-43725 Use Consistent Time for all healthcheck monitoring type alerts
Updated Health Check to use local time instead of UTC
Updated Health Check to use local time instead of UTC
SDP-41331 Exceptions in the MC2 log with generic extract with nuggetizer
Corrected MC2 issue with nuggetizer used with generic extract
Corrected MC2 issue with nuggetizer when used with generic extract
SDP-45951 EDS: enable display of onStart and onEnd rows when no data and allowEmptyFiles = Y
enabled the display of onStart header and onEnd trailer with no data found
In EDS, enabled the display of user defined onStart header and onEnd trailer in the result when no data is found and allowEmptyFiles parameter = Y.
SDP-45950 EBS: Data header not displayed with source mashup and allowEmptyFiles ='Y' in case of no data
Corrected EDS issue when the data header was not in the output
Corrected EDS issue when the data header was not in the output as expected under these conditions:
interface includes source mashup,
allowemptyfiles is Y,
outputformat is CSV with header,
and no data is found
SDP-45355 Update caching policy for the EDS reportdictionary object to FULL from NONE
Updated caching policy for the reportdictionary object
Changed caching mode in the reportdictionary dataset.
"cache_mode": "none" -> "cache_mode": "full"
Also changed property that defines where the cache is saved.
"cache_provider": "cache-manager-file" -> "cache_provider": "cache-manager-default"
SDP-45253 Change 'hasheader' behavior to not affect OnStart Headers
EDS: updated the control of data versus custom headers
EDS: updated to allow independent control of the display of column (data) header versus custom (onStart) header
SDP-45353 Ontology: Improve TaskStatus Resource to add information about errors
EDS TaskStatus extract enhancements
EDS TaskStatus extract has been enhanced to return additional information, including the error description of the task. The TaskStatus ontology resource supports all common filters as well as several additional filters such as queryoriginalcorrelationid, queryparentcorrelationid. The Task Status resource was updated to support new fields: PACE_MASTERDBO.ORCH_QUEUE.ORIG_CORRELATION_ID PACE_MASTERDBO.ORCH_QUEUE.PROCESS_CORRELATION_ID PACE_MASTERDBO.ORCH_QUEUE.BUS_TASK_ID PACE_MASTERDBO.ORCH_QUEUE.CREATE_DATE PACE_MASTERDBO.ORCH_QUEUE.UPDATE_DATE PACE_MASTERDBO.ORCH_QUEUE.ORCH_STATE_CLOB
Xpath:
EagleML/monitoringTransaction/taskStatus/originalCorrelationId EagleML/monitoringTransaction/taskStatus/parentCorrelationId EagleML/monitoringTransaction/taskStatus/workflowName EagleML/monitoringTransaction/taskStatus/taskName EagleML/monitoringTransaction/taskStatus/startDateTime EagleML/monitoringTransaction/taskStatus/endDateTime EagleML/monitoringTransaction/taskStatus/errorDescription Files changed: metadata/core/ontology/definition/eagle-vocab-definition-common.json metadata/core/ontology/eagle-vocab-common.json metadata/core/ontology/eagle-vocab-enumerations.json metadata/core/ontology/monitoring/taskstatus.json metadata/core/ontology/monitoring/taskstatus-models.json
SDP-46035 Validate user entitlements for Report Rule, Source Rule, Field Rule and few other GET methods of OLAP REST API (via eagle/report/v1/ PACEOLAP REST endpoints)
Unified user Entitlement approach in OLAP REST API GET methods-Phase 2
The GET methods for Report Rule, Field Rule, Source Rule, Benchmark Definition, Grouping rule Range and Filter Fields of the OLAP REST API endpoints have been enhanced to support the unified user Entitlement approach. The method used is the same as what is supported with NXG user entitlements. The results of the GET method extracts will be based on the user profile configurations. The entitlements define the level of access for each user profile based on Center Role, Pace Role, Business User configurations and User's functional permissions. Prior to this enhancement all these GET endpoints did not verify user entitlements prior to fetching the respective data.