New version of Pace ReportService client
New version is available: pace-reportservice-1.0.7.jar
The jar must be installed separately; refer to this page for instructions to install the client jar file:
Installing Pace Report Service
SDP-42293 cache policy to reload cache for extract workflow for Cloud
Setup default cache policies for interfaces, genericsmf, genericentity, issuerorganization datasets
Added default cache policies for the following datasets:
- interfaces
- genericsmf
- genericentity
- issuerorganization
That means that after package installation cache reload for above datasets will be performed automatically in case data was updated. If cache for these datasets does not exist, it will be fully loaded. The default interval for checking updated records is set to 5 minutes. Default cache policies can be overwritten with custom one if the specific check and update cache settings are required.
SDP-41157 Revise the Generic Extractor Workflow for New EDS Ontology Feeds
Added ability to use new EDS ontology in generic_extract workflow
Added ability to use new ontology in generic_extract workflow.RTR example with new parameters:
<taskParameter> <name>ActionType</name> <dataType>S</dataType> <value>EXECUTE</value> </taskParameter> <taskParameter> <name>WorkflowName</name> <dataType>S</dataType> <value>eagle_wrf_generic_extract</value> </taskParameter> <taskParameter><name>EBS_ServiceName</name><value>ebs:EBS</value></taskParameter><taskParameter><name>EBS_ServiceMethodName</name><dataType>S</dataType><value>ebs:extract</value></taskParameter><taskParameter><name>EBS_ResourceName</name><value>ebs:genericsmf</value></taskParameter><taskParameter> <name>ebs:genericsmf:primaryassetid</name> <dataType>S</dataType> <value>ASSET_ID_VALUE</value> </taskParameter>
SDP-41084 Create New EDS Metadata for performdbo.perf_sec_rollup_relation table called perfsecrollup
New Metadata - perfsecrollup
New ontology of perfsecrollup was developed:
estar\\\\tpe\\\\dynamic\\\\metadata\\\\core\\\\ontology\\\\warehouse\\\\perfsecrollup.json
estar\\\\tpe\\\\dynamic\\\\metadata\\\\core\\\\ontology\\\\warehouse\\\\perfsecrollup-models.json
perfsecrollup.json contains sql table joins definition, filters definition, extract and load rules and etc.
perfsecrollup-models.json contains fields mapping definition.
New perfsecrollup metadata can be used to extract data.
SDP-41083 New Metadata - issuerRating
Added issuerrating EDS Ontology
New ontology of issuer rating was added:
estar/tpe/dynamic/metadata/core/ontology/reference/issuerrating.json
estar/tpe/dynamic/metadata/core/ontology/reference/issuerrating-models.json
issuerrating.json contains sql table joins definition, filters definition, extract and load rules and etc.
issuerrating-models.json contains fields mapping definition.
New issuerrating metadata can be used to extract data.
SDP-40808 New EDS Metadata - issuerOrganization
New Metadata - issuerOrganization/hist
New ontology of issuer organization and issuer organization in the history mode was added:
estar/tpe/dynamic/metadata/core/ontology/reference/issuerorganization.json
estar/tpe/dynamic/metadata/core/ontology/reference/issuerorganization-models.json
issuerorganization.json contains sql table joins definition, filters definition, extract and load rules and etc.
issuerorganization-models.json contains fields mapping definition.
New issuerorganization metadata can be used to extract data.
SDP-40698 New EDS Metadata - entitydetailhistory
New Metadata - entitydetailhistory
New ontology of entity detail and entity detail in the history mode was developed:
estar\\tpe\\dynamic\\metadata\\core\\ontology
entity entitydetail.json
estar\\tpe\\dynamic\\metadata\\core\\ontology
entity entitydetail-models.json
entitydetail.json contains sql table joins definition, filters definition, extract and load rules and etc.
entitydetail-models.json contains fields mapping definition.
New entitydetail metadata can be used to extract data.
SDP-42384 csv_char_strip parsing dialect loses last value in csv row if row has no trailing delimiter
csv_char_strip Parsing Dialect Loses Last Value in CSV Load if File is missing Trailing Delimiter
There was an issue when parsing a CSV file with an EDS load when the final data element would not be loaded if there was no trailing delimiter. We changed the code so that we can load delimited files correctly even if the final data element does not have a delimiter following it.
SDP-39310 Design and Implement JSON-LD Metadata Discovery API
Add Discovery API to Return Canonical Model of the EDS Resource
As result of this story we can call API and get back canonical model for the specified EDS resource (interface). Dialect of canonical model can be changed through the parameters: includedescription - Y\N - default Y. Include\exclude description(skos:definition) for the elements includelabel - Y\N - default Y. Include\exclude label for the elements modeloutputdilect - asDict, asList. Default(asList) when asList option is used all structure will be like this:
AsList Example:{"resources": [{"resource":
{ "id": "warehousePosition", "description": "", "type": "dataset", "elements_label": ["Object Type element", "Object Id element", "Object Description element",...], "elements_id": ["objectType", "objectId", "objectDescription", ...], "elements_datatype": ["string", "string", "string", ...], "elements_datatype_dialect": [...], "elements_path": ["objectType", "objectId", "objectDescription", ...], "elements_description": ["A classification of the type of object.", "An object identifier allocated by a customer.", "", ...], "provenance": "official" }
}]}
AsDict Example:{"resources": [{"resource": {"id": "warehousePosition","description": "","type": "dataset","elements": [{"element_id": "objectType","path": "objectType","type": "string","type_dialect":
{ "enum": [ "IssuerRegulatory", "Entitlement", "ConsolidatedEarnedIncome", ..... ] }
,"label": "Object Type element","definition": "A classification of the type of object."},{"element_id": "objectId","path": "objectId","type": "string","type_dialect": {},"label": "Object Id element","definition": "An object description allocated by a customer. "},{"element_id": "objectDescription","path": "objectDescription","type": "string","type_dialect":
{...}
,"label": "Object Description element","definition": "" },.....],"provenance": "official"}}]}
SDP-42469 Fix ExtractNamePatternEx Parameter to be able to match Current Cloud Workflow
Fixed @resourcename pattern replacement in ExtractNamePatternEx
1. Fixed @resourcename pattern replacement in ExtractNamePatternEx in eagle_wrf_generic_extract processing.2. Added ability to execute old metadata (FeedType) and new metadata (EBS_ServiceName, EBS_ServiceMethodName, EBS_ResourceName) modes in one RTR in eagle_wrf_generic_extract processing
SDP-42427 Taxonomy should contain only existing fields
EDS Taxonomy Should Contain Only Existing Fields
We added code to check if tables from the Ontology were present in a region before trying to extract them. This will improve efficiency by only querying for tables that actually exist in a region.
SDP-42371 EDS: Correct smfidentifiers data object definition to contain xrefs only for existing securities
Changed the join of XREFERENCE and SECURITY_MASTER to an inner join, so only identifiers for existing securities are returned.
SDP-42053 Implement extract results grouping by effective date, entity Id, and extract type
Updated MC2 to allow grouping of results by effective date, entity Id, and extract type for the WarehousePosition and WarehouseOpenLot extracts
(WRHSPOSITIONEXTRACT, WRHSOPENLOTEXTRACT).
Enabled the below parameters:
GroupbyExtractKey: parameter in RTR for a single extract in extractservice
ALLEXTRACTS_GroupbyExtractKey: parameter in RTR for eagle_wrf_generic_extract worklfow
W_GROUPBY_EXTRACT_KEY: global parameter in w_config_custom.inc for all extracts for the WarehousePosition and WarehouseOpenLot feedtypes
For full details see: https://eagledocs.atlassian.net/wiki/spaces/MC2/pages/1660321794/Grouping+extract+results+with+nuggetizer
SDP-41935 Python XML2XML: Incorrect FileNotFoundError error during transformation
Python XML2XML: Incorrect FileNotFoundError error during transformation
Added code that will check and create temp folder if it doesn't exist so that an error isn't raised when an in-memory EDS file is large enough that it begins to be file-based.
SDP-41662 REST endpoint for EDS metadata
REST endpoint for EDS metadata
We created code to expose our EDS metadata through new endpoints. Below are the current examples of endpoints that we have utilized:
This request returns ALL available resources in ontology and URL for their calling: URL/eagle/v2/metadata/models/issuerrating
Below can be used to get canonical model for single resource. This request returns canonical model for single resource in JSON format. URL/eagle/v2/metadata/models/codevalue?modeloutputdialect=asDict&includedescription=N&includelabel=N
This request returns canonical models for ALL available resources in ontology: URL/eagle/v2/metadata/models?modeloutputdialect=asDict&includedescription=N&includelabel=N
This request returns ALL available resources in ontology and URL for their calling: URL/eagle/v2/metadata/resources/codevalued?includedescription=N&includelabel=N&includedbinfo=N
This method returns ALL resources info in CSV format: URL/eagle/v2/metadata/resources?includedescription=N&includelabel=N&includedbinfo=N
This method return information about single vocabulary element in ontology: URL/eagle/v2/metadata/vocab/autoAddBasis
This request returns information about ALL vocab elements: URL/eagle/v2/metadata/vocab
SDP-41625 Add smfidentifiers_currency core data object
Add smfidentifiers_currency core data object
We added smfidentifiers_currency to core EDS package to be utilized in a resolution mashup. This makes security resolution more specific because it includes asset currency.
SDP-41623 Add Refresh All Option for EDS Ontology
Add Refresh All Option for EDS Ontology
In September release we added the EDS Ontology and at the time we had to refresh each resource independently. The below RTR will now refresh all available resources which in turn will generate necessary rules and definitions for the resources to be used.
<EagleML xmlns="http://www.eagleinvsys.com/2011/EagleML-2-0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="RunTaskRequest" eaglemlVersion="2-0" xsi:schemaLocation="http://www.eagleinvsys.com/2011/EagleML-2-0 eagleml-main-2-0.xsd" eaglemlType="RunTaskRequest"><header><messageId>E30X5FTN4N317JFF</messageId><sentBy>MC2EJM</sentBy><sendTo>http://www.eagleinvsys.com/eagle_default_ruleservice</sendTo><creationTimestamp>2018-04-13T09:52:39.73-05:00</creationTimestamp><expiryTimestamp/><expiryLoadFlag/></header><taskIdentifier><processingSequenceNumber/><correlationId>CONVER_META_ALL</correlationId><businessTaskId>CONVER_META_ALL</businessTaskId><sequenceNumber/></taskIdentifier><taskTypeEnum>NEW</taskTypeEnum><synchronousExecution>yes</synchronousExecution><taskParameters><taskParameter><name>ServiceName</name><value>Metadata</value></taskParameter><taskParameter><name>ServiceMethodName</name><dataType>S</dataType><value>Refresh</value></taskParameter><taskParameter><name>ResourceName</name><value>ALL</value></taskParameter><taskParameter><name>SaveOutput</name><value>Y</value></taskParameter></taskParameters></EagleML>
SDP-41156 Fix ExtractNamePatternEx Parameter to be able to use @resourcename
Added @resourcename pattern in ExtractNamePatternEx
Added @resourcename pattern in ExtractNamePatternEx task parameter when new ontology is used. @resourcename works the same way as @feedtype.
SDP-41046 Improve CSV output transformation
Improve CSV Output Transformation for EDS Extracts
We enhanced the code to allow the transformation to CSV output to occur more efficiently using EDS Extracts.
SDP-40655 Issue with TSR when running XML REST extract and getting NO_DATA
Corrected the response for an XML output ReST request in a no data scenario so that a valid TSR is returned.
SDP-40011 Caching strategy
EDS caching strategy
EDS caching strategy adds a possibility to override EDS datasets cache_mode. It is defined by a JSON file in eagle/estar/tpe/dynamic/metadata/custom/profiles/cache_strategy.json
There, you can specify custom cache_mode for a dataframe, group, or all interfaces with the specific cache_mode. Example of the cache_mode override for all interfaces with group = Warehouse (from any to blob):
{ "group_name": {
"Warehouse": {
"cache_mode_override": "blob"
}}}
Example of the cache_mode override to blob for all interfaces with none, for except of warehousetrade that is set to none:
{
"cache_mode_override": {
"none": "blob"
},
"dataframes": {
"warehousetrade": {
"cache_mode_override": "none"
}
}}
SDP-38194 EDS: Improve fetch from DB for EDS extracts. Part 3
EDS: Improve Fetch from DB for EDS Extracts
We have implemented a new Oracle library to improve Fetch call from DB. This will help improve EDS extract efficiency.
SDP-36878 Add ability to enable fields only when filter for this table is used
Added Ability to Conditionally Add Joins Based on Filter Inputs for EDS Request
Added support for conditional join for filters on EDS requests. Conditional join allow to reduce unnecessary joins in query in case fields from these tables should not be extract and tables used only for applying filters. Conditional join will be added to query only if filter is applying for extract (defined in RTR)The example of conditional :
"sourcename": {"filter_type": "bind","description": "source name filter","disableCache": true,"parameters": [
{ "name": "sourcename" }
],"bindings": [
{ "sourceProvider": "eagledb://oracle", "filter": "INTERFACES.short_desc @(:sourcename)" }
,
{ "sourceProvider": "eagledb://mssql", "filter": "INTERFACES.short_desc @(:sourcename)" }
],"conditionaljoin":
{ "resource": "eagledb:REFERENCE.INTERFACES", "mashupMode": "INNER", "expression": "MASTERTABLE.SRC_INTFC_INST = INTERFACES.INSTANCE" }
},
SDP-33481 MC2 Generic Load profiles: modify python wrf generator to support MC2 EJM
Fixed generation of workflow with postprocessing step not being redefined in profile
Fixed generation of workflow with postprocessing step not being redefined in profile
SDP-42315 Implement delivery of HealthCheck alerts via REST
Added ability for HealthCheck to send an alert to a ReST endpoint
SDP-42040 MC2 generates duplicate correlation ids for REST calls
Modified generation of Correlation Ids to prevent duplicates
Modified generation of Correlation Ids to prevent duplicates
SDP-41635 The USERGROUPS parameter is not working in EDS
Corrected an issue with the usergroups parameter
Corrected an issue in EDS with the usergroups parameter not being applied to the database query.
SDP-41049 Enable topic level configuration in kafka
Added ability to configure kafka at topic level
Added the ability to configure number of partitions, log segment retention hours, and log segment bytes at the topic level in kafka.
SDP-40787 Nuggetizer not returning expected results on File Server or TSR
Corrected issue with MC2 nuggetizer not delivering all nugget files
Corrected issue with MC2 nuggetizer not delivering all nugget files when custom file name or custom file location specified.
SDP-38943 PACEREPORT output format not working with distribution option of report profile
Corrected issue with PACEREPORT output format with report profile distribution option
Corrected issue with output format of the PACEREPORT REST API when the the report profile distribution option is used.
SDP-41055 Generate BusinessTaskId for additional REST endpoints
Generating BusinessTaskId for REST endpoints in addition to the EQL endpoint
System generated Business Task Id for REST endpoints now contains "REST#" + request details + timestamp.
Examples:
Securities endpoint: REST#securities#202010061659
EQL endpoint for warehouseposition EDS extract: REST#eql#warehouseposition#202010061655
PACEREPORT adhoc report: REST#pace#adhocReport#202010061701
Generic extract requests have "REST#" + timestamp + workflow step:
REST#202010061706#genericextract
REST#202010061706#eagle_generic_extract_entity_objects
REST#202010061706#extract_generic_entity
REST#202010061706#eagle_ml20_default_cm_packer
REST#202010061706#eagle_ml20_default_cm_file_distribution
SDP-39928 REST endpoint for EDS metadata
Implemented REST endpoints for EDS metadata
Implemented the METADATA ReST API with endpoints for returning EDS metadata.The endpoints are visible via the swagger-ui.
SDP-39841 Additional parameters to GET METHOD of Pace Report Service
Added additional parameters to GET method of adhocReport REST endpoint
Added support for parameters benchmarkToPortfolio and constituentsAndBenchmarkToComposite to the GET method of the adhocReport endpoint.
SDP-42035 REST API GET methods for Report rule and Field Rule Override
GET methods supporting Report Rule and Field Rule overrides in OLAP REST API
New GET methods on Report rule, Field Rule, Source rule. field Rule by Id and Source Rule Id are exposed to get the list. This ease end user to get the rules and its respective Ids to construct the request payload.
SDP-41645 OLAP REST API output format Validation
OLAP REST API Output Formats
OLAP REST API provides the output files supporting in all formats like standard CSV and TSV, Flat CSV (FCSV) and TSV (FTSV), Excel (XLS), extended CSV (XCSV), JSON and JSON2. For JSON and JSON2 output formats, distribution option of Report profile needs to be switched OFF.
SDP-38693 OLAP REST API supporting Report rule and Field Rule Override
Supporting Report Rule and Field Rule overrides in OLAP REST API
New feature added in OLAP REST API which allows to override the Report Rule, Field rule and fields for OLAP reports supporting the same. You can use the Report and field rule override to change the fields and with different grouping in other report rules at run time to perform 'what-if' analysis. Respective new GET methods on Report rule and Field Rules exposed to get the list.