MC2 Release Notes - October 20 2020

Owned by Marius Navickas
Last updated: Jun 06, 2024 by Pamela Totas (Deactivated)
8 min read

New version of Pace ReportService client

New version is available: pace-reportservice-1.0.7.jar
The jar must be installed separately; refer to this page for instructions to install the client jar file:
Installing Pace Report Service

SDP-42293 cache policy to reload cache for extract workflow for Cloud

Setup default cache policies for interfaces, genericsmf, genericentity, issuerorganization datasets

Added default cache policies for the following datasets:

  • interfaces

  • genericsmf

  • genericentity

  • issuerorganization

That means that after package installation cache reload for above datasets will be performed automatically in case data was updated. If cache for these datasets does not exist, it will be fully loaded. The default interval for checking updated records is set to 5 minutes. Default cache policies can be overwritten with custom one if the specific check and update cache settings are required.

SDP-41157 Revise the Generic Extractor Workflow for New EDS Ontology Feeds

Added ability to use new EDS ontology in generic_extract workflow

Added ability to use new ontology in generic_extract workflow.RTR example with new parameters:
<taskParameter> <name>ActionType</name> <dataType>S</dataType> <value>EXECUTE</value> </taskParameter> <taskParameter> <name>WorkflowName</name> <dataType>S</dataType> <value>eagle_wrf_generic_extract</value> </taskParameter> <taskParameter><name>EBS_ServiceName</name><value>ebs:EBS</value></taskParameter><taskParameter><name>EBS_ServiceMethodName</name><dataType>S</dataType><value>ebs:extract</value></taskParameter><taskParameter><name>EBS_ResourceName</name><value>ebs:genericsmf</value></taskParameter><taskParameter> <name>ebs:genericsmf:primaryassetid</name> <dataType>S</dataType> <value>ASSET_ID_VALUE</value> </taskParameter>

SDP-41084 Create New EDS Metadata for performdbo.perf_sec_rollup_relation table called perfsecrollup

New Metadata - perfsecrollup

New ontology of perfsecrollup was developed:
estar\\\\tpe\\\\dynamic\\\\metadata\\\\core\\\\ontology\\\\warehouse\\\\perfsecrollup.json
estar\\\\tpe\\\\dynamic\\\\metadata\\\\core\\\\ontology\\\\warehouse\\\\perfsecrollup-models.json
perfsecrollup.json contains sql table joins definition, filters definition, extract and load rules and etc.
perfsecrollup-models.json contains fields mapping definition.
New perfsecrollup metadata can be used to extract data.

SDP-41083 New Metadata - issuerRating

Added issuerrating EDS Ontology

New ontology of issuer rating was added:
estar/tpe/dynamic/metadata/core/ontology/reference/issuerrating.json
estar/tpe/dynamic/metadata/core/ontology/reference/issuerrating-models.json
issuerrating.json contains sql table joins definition, filters definition, extract and load rules and etc.
issuerrating-models.json contains fields mapping definition.
New issuerrating metadata can be used to extract data.

SDP-40808 New EDS Metadata - issuerOrganization

New Metadata - issuerOrganization/hist

New ontology of issuer organization and issuer organization in the history mode was added:
estar/tpe/dynamic/metadata/core/ontology/reference/issuerorganization.json
estar/tpe/dynamic/metadata/core/ontology/reference/issuerorganization-models.json
issuerorganization.json contains sql table joins definition, filters definition, extract and load rules and etc.
issuerorganization-models.json contains fields mapping definition.
New issuerorganization metadata can be used to extract data.

SDP-40698 New EDS Metadata - entitydetailhistory

New Metadata - entitydetailhistory

New ontology of entity detail and entity detail in the history mode was developed:
estar\\tpe\\dynamic\\metadata\\core\\ontology
entity entitydetail.json
estar\\tpe\\dynamic\\metadata\\core\\ontology
entity entitydetail-models.json
entitydetail.json contains sql table joins definition, filters definition, extract and load rules and etc.
entitydetail-models.json contains fields mapping definition.
New entitydetail metadata can be used to extract data.

SDP-42384 csv_char_strip parsing dialect loses last value in csv row if row has no trailing delimiter

csv_char_strip Parsing Dialect Loses Last Value in CSV Load if File is missing Trailing Delimiter

There was an issue when parsing a CSV file with an EDS load when the final data element would not be loaded if there was no trailing delimiter. We changed the code so that we can load delimited files correctly even if the final data element does not have a delimiter following it.

SDP-39310 Design and Implement JSON-LD Metadata Discovery API

Add Discovery API to Return Canonical Model of the EDS Resource

As result of this story we can call API and get back canonical model for the specified EDS resource (interface). Dialect of canonical model can be changed through the parameters: includedescription - Y\N - default Y. Include\exclude description(skos:definition) for the elements includelabel - Y\N - default Y. Include\exclude label for the elements modeloutputdilect - asDict, asList. Default(asList) when asList option is used all structure will be like this:
AsList Example:{"resources": [{"resource":
{ "id": "warehousePosition", "description": "", "type": "dataset", "elements_label": ["Object Type element", "Object Id element", "Object Description element",...], "elements_id": ["objectType", "objectId", "objectDescription", ...], "elements_datatype": ["string", "string", "string", ...], "elements_datatype_dialect": [...], "elements_path": ["objectType", "objectId", "objectDescription", ...], "elements_description": ["A classification of the type of object.", "An object identifier allocated by a customer.", "", ...], "provenance": "official" }
}]}
AsDict Example:{"resources": [{"resource": {"id": "warehousePosition","description": "","type": "dataset","elements": [{"element_id": "objectType","path": "objectType","type": "string","type_dialect":
{ "enum": [ "IssuerRegulatory", "Entitlement", "ConsolidatedEarnedIncome", ..... ] }
,"label": "Object Type element","definition": "A classification of the type of object."},{"element_id": "objectId","path": "objectId","type": "string","type_dialect": {},"label": "Object Id element","definition": "An object description allocated by a customer. "},{"element_id": "objectDescription","path": "objectDescription","type": "string","type_dialect":
{...}
,"label": "Object Description element","definition": "" },.....],"provenance": "official"}}]}

SDP-42469 Fix ExtractNamePatternEx Parameter to be able to match Current Cloud Workflow

Fixed @resourcename pattern replacement in ExtractNamePatternEx

1. Fixed @resourcename pattern replacement in ExtractNamePatternEx in eagle_wrf_generic_extract processing.2. Added ability to execute old metadata (FeedType) and new metadata (EBS_ServiceName, EBS_ServiceMethodName, EBS_ResourceName) modes in one RTR in eagle_wrf_generic_extract processing

SDP-42427 Taxonomy should contain only existing fields

EDS Taxonomy Should Contain Only Existing Fields

We added code to check if tables from the Ontology were present in a region before trying to extract them. This will improve efficiency by only querying for tables that actually exist in a region.

SDP-42371 EDS: Correct smfidentifiers data object definition to contain xrefs only for existing securities

Changed the join of XREFERENCE and SECURITY_MASTER to an inner join, so only identifiers for existing securities are returned.

SDP-42053 Implement extract results grouping by effective date, entity Id, and extract type

Updated MC2 to allow grouping of results by effective date, entity Id, and extract type for the WarehousePosition and WarehouseOpenLot extracts
(WRHSPOSITIONEXTRACT, WRHSOPENLOTEXTRACT).
Enabled the below parameters:
GroupbyExtractKey: parameter in RTR for a single extract in extractservice
ALLEXTRACTS_GroupbyExtractKey: parameter in RTR for eagle_wrf_generic_extract worklfow
W_GROUPBY_EXTRACT_KEY: global parameter in w_config_custom.inc for all extracts for the WarehousePosition and WarehouseOpenLot feedtypes


For full details see: https://eagledocs.atlassian.net/wiki/spaces/MC2/pages/1660321794/Grouping+extract+results+with+nuggetizer

SDP-41935 Python XML2XML: Incorrect FileNotFoundError error during transformation

Python XML2XML: Incorrect FileNotFoundError error during transformation

Added code that will check and create temp folder if it doesn't exist so that an error isn't raised when an in-memory EDS file is large enough that it begins to be file-based.

SDP-41662 REST endpoint for EDS metadata

REST endpoint for EDS metadata

We created code to expose our EDS metadata through new endpoints. Below are the current examples of endpoints that we have utilized:
This request returns ALL available resources in ontology and URL for their calling: URL/eagle/v2/metadata/models/issuerrating
Below can be used to get canonical model for single resource. This request returns canonical model for single resource in JSON format. URL/eagle/v2/metadata/models/codevalue?modeloutputdialect=asDict&includedescription=N&includelabel=N
This request returns canonical models for ALL available resources in ontology: URL/eagle/v2/metadata/models?modeloutputdialect=asDict&includedescription=N&includelabel=N
This request returns ALL available resources in ontology and URL for their calling: URL/eagle/v2/metadata/resources/codevalued?includedescription=N&includelabel=N&includedbinfo=N
This method returns ALL resources info in CSV format: URL/eagle/v2/metadata/resources?includedescription=N&includelabel=N&includedbinfo=N
This method return information about single vocabulary element in ontology: URL/eagle/v2/metadata/vocab/autoAddBasis
This request returns information about ALL vocab elements: URL/eagle/v2/metadata/vocab

SDP-41625 Add smfidentifiers_currency core data object

Add smfidentifiers_currency core data object

We added smfidentifiers_currency to core EDS package to be utilized in a resolution mashup. This makes security resolution more specific because it includes asset currency.

SDP-41623 Add Refresh All Option for EDS Ontology

Add Refresh All Option for EDS Ontology

In September release we added the EDS Ontology and at the time we had to refresh each resource independently. The below RTR will now refresh all available resources which in turn will generate necessary rules and definitions for the resources to be used.
<EagleML xmlns="http://www.eagleinvsys.com/2011/EagleML-2-0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="RunTaskRequest" eaglemlVersion="2-0" xsi:schemaLocation="http://www.eagleinvsys.com/2011/EagleML-2-0 eagleml-main-2-0.xsd" eaglemlType="RunTaskRequest"><header><messageId>E30X5FTN4N317JFF</messageId><sentBy>MC2EJM</sentBy><sendTo>http://www.eagleinvsys.com/eagle_default_ruleservice</sendTo><creationTimestamp>2018-04-13T09:52:39.73-05:00</creationTimestamp><expiryTimestamp/><expiryLoadFlag/></header><taskIdentifier><processingSequenceNumber/><correlationId>CONVER_META_ALL</correlationId><businessTaskId>CONVER_META_ALL</businessTaskId><sequenceNumber/></taskIdentifier><taskTypeEnum>NEW</taskTypeEnum><synchronousExecution>yes</synchronousExecution><taskParameters><taskParameter><name>ServiceName</name><value>Metadata</value></taskParameter><taskParameter><name>ServiceMethodName</name><dataType>S</dataType><value>Refresh</value></taskParameter><taskParameter><name>ResourceName</name><value>ALL</value></taskParameter><taskParameter><name>SaveOutput</name><value>Y</value></taskParameter></taskParameters></EagleML>

SDP-41156 Fix ExtractNamePatternEx Parameter to be able to use @resourcename

Added @resourcename pattern in ExtractNamePatternEx

Added @resourcename pattern in ExtractNamePatternEx task parameter when new ontology is used. @resourcename works the same way as @feedtype.

SDP-41046 Improve CSV output transformation

Improve CSV Output Transformation for EDS Extracts

We enhanced the code to allow the transformation to CSV output to occur more efficiently using EDS Extracts.

SDP-40655 Issue with TSR when running XML REST extract and getting NO_DATA

Corrected the response for an XML output ReST request in a no data scenario so that a valid TSR is returned.

SDP-40011 Caching strategy

EDS caching strategy

EDS caching strategy adds a possibility to override EDS datasets cache_mode. It is defined by a JSON file in eagle/estar/tpe/dynamic/metadata/custom/profiles/cache_strategy.json
There, you can specify custom cache_mode for a dataframe, group, or all interfaces with the specific cache_mode. Example of the cache_mode override for all interfaces with group = Warehouse (from any to blob):
{ "group_name": {
"Warehouse": {
"cache_mode_override": "blob"
}}}
Example of the cache_mode override to blob for all interfaces with none, for except of warehousetrade that is set to none:
{
"cache_mode_override": {
"none": "blob"
},
"dataframes": {
"warehousetrade": {
"cache_mode_override": "none"
}
}}

SDP-38194 EDS: Improve fetch from DB for EDS extracts. Part 3

EDS: Improve Fetch from DB for EDS Extracts

We have implemented a new Oracle library to improve Fetch call from DB. This will help improve EDS extract efficiency.

SDP-36878 Add ability to enable fields only when filter for this table is used

Added Ability to Conditionally Add Joins Based on Filter Inputs for EDS Request

Added support for conditional join for filters on EDS requests. Conditional join allow to reduce unnecessary joins in query in case fields from these tables should not be extract and tables used only for applying filters. Conditional join will be added to query only if filter is applying for extract (defined in RTR)The example of conditional :
"sourcename": {"filter_type": "bind","description": "source name filter","disableCache": true,"parameters": [
{ "name": "sourcename" }
],"bindings": [
{ "sourceProvider": "eagledb://oracle", "filter": "INTERFACES.short_desc @(:sourcename)" }
,
{ "sourceProvider": "eagledb://mssql", "filter": "INTERFACES.short_desc @(:sourcename)" }
],"conditionaljoin":
{ "resource": "eagledb:REFERENCE.INTERFACES", "mashupMode": "INNER", "expression": "MASTERTABLE.SRC_INTFC_INST = INTERFACES.INSTANCE" }
},

SDP-33481 MC2 Generic Load profiles: modify python wrf generator to support MC2 EJM

Fixed generation of workflow with postprocessing step not being redefined in profile

Fixed generation of workflow with postprocessing step not being redefined in profile

SDP-42315 Implement delivery of HealthCheck alerts via REST

Added ability for HealthCheck to send an alert to a ReST endpoint

SDP-42040 MC2 generates duplicate correlation ids for REST calls

Modified generation of Correlation Ids to prevent duplicates

Modified generation of Correlation Ids to prevent duplicates

SDP-41635 The USERGROUPS parameter is not working in EDS

Corrected an issue with the usergroups parameter

Corrected an issue in EDS with the usergroups parameter not being applied to the database query.

SDP-41049 Enable topic level configuration in kafka

Added ability to configure kafka at topic level

Added the ability to configure number of partitions, log segment retention hours, and log segment bytes at the topic level in kafka.

SDP-40787 Nuggetizer not returning expected results on File Server or TSR

Corrected issue with MC2 nuggetizer not delivering all nugget files

Corrected issue with MC2 nuggetizer not delivering all nugget files when custom file name or custom file location specified.

SDP-38943 PACEREPORT output format not working with distribution option of report profile

Corrected issue with PACEREPORT output format with report profile distribution option

Corrected issue with output format of the PACEREPORT REST API when the the report profile distribution option is used.

SDP-41055 Generate BusinessTaskId for additional REST endpoints

Generating BusinessTaskId for REST endpoints in addition to the EQL endpoint

System generated Business Task Id for REST endpoints now contains "REST#" + request details + timestamp.

Examples:

Securities endpoint: REST#securities#202010061659

EQL endpoint for warehouseposition EDS extract: REST#eql#warehouseposition#202010061655

PACEREPORT adhoc report: REST#pace#adhocReport#202010061701

Generic extract requests have "REST#" + timestamp + workflow step:

REST#202010061706#genericextract

REST#202010061706#eagle_generic_extract_entity_objects

REST#202010061706#extract_generic_entity

REST#202010061706#eagle_ml20_default_cm_packer

REST#202010061706#eagle_ml20_default_cm_file_distribution

SDP-39928 REST endpoint for EDS metadata

Implemented REST endpoints for EDS metadata

Implemented the METADATA ReST API with endpoints for returning EDS metadata.The endpoints are visible via the swagger-ui.

SDP-39841 Additional parameters to GET METHOD of Pace Report Service

Added additional parameters to GET method of adhocReport REST endpoint

Added support for parameters benchmarkToPortfolio and constituentsAndBenchmarkToComposite to the GET method of the adhocReport endpoint.

SDP-42035 REST API GET methods for Report rule and Field Rule Override

GET methods supporting Report Rule and Field Rule overrides in OLAP REST API

New GET methods on Report rule, Field Rule, Source rule. field Rule by Id and Source Rule Id are exposed to get the list. This ease end user to get the rules and its respective Ids to construct the request payload.

SDP-41645 OLAP REST API output format Validation

OLAP REST API Output Formats

OLAP REST API provides the output files supporting in all formats like standard CSV and TSV, Flat CSV (FCSV) and TSV (FTSV), Excel (XLS), extended CSV (XCSV), JSON and JSON2. For JSON and JSON2 output formats, distribution option of Report profile needs to be switched OFF.

SDP-38693 OLAP REST API supporting Report rule and Field Rule Override

Supporting Report Rule and Field Rule overrides in OLAP REST API

New feature added in OLAP REST API which allows to override the Report Rule, Field rule and fields for OLAP reports supporting the same. You can use the Report and field rule override to change the fields and with different grouping in other report rules at run time to perform 'what-if' analysis. Respective new GET methods on Report rule and Field Rules exposed to get the list.