MC2 Release Notes - May 05 2020
- Marius Navickas
- Pamela Totas (Deactivated)
SDP-35817 Add dataset definition and taxonomy for genericentityhistory
Added EDS Dataset and Taxonomy for genericentityhistory
We created a genericentityhistory EDS Interface that can be called to extract data from the history tables that mirror the output of the genericentity dataset.
SDP-16198 Add dataset definition and taxonomy for entitymanagerrelationship
SDP-35767 Introduced general output view of processing result in pyruleservice
Introduced general output view of processing result in common methods of pyruleservice
The new view was introduced as the default output of all services executed in pyruleservice via RTR (EBS, graphql, py transform, registerdataset, generateworkflow, execute, extract).
SDP-35629 Missed records when transforming large xml files
Improved reading xml chunks in buffer
When performing the xml2xml transform we were experiencing special case where at the end we were missing one record. This was an issue with XMLDESC_BUFFERSIZE variable (mc2py/xml_enum.py). When we increased the buffersize, the issue was resolved. Reading xml chunk was improved to be more sustained for the case when chunk ends in the middle of xml nodes.
SDP-35504 Allow OR to be used in outputfilterexpr
Improved detection of SQL injection in RTR
outputfilterexpr was added to exception list along with interfacejsondef, because these parameters can contain syntax similar to SQL.
SDP-35465 Missing expression error ORA-00936 if onlyheld=Y and heldpositionsources filter is used
SQL query is formed correctly if heldpositionsources filter applied in held only mode
Line break typo fixed related to effective date query of the heldpositionsources filter.
SDP-35407 Add ability to use param map for the vendor setting
Added ability to use parameter map for the vendor setting
We added the ability to be able to call RTR parameters from a parameter map in an EDS Interface. For example:
This gives you the option of using case_x in your RTR request parameters as per below:
<taskParameter><name>battingOrder_TestCase</name><value>case_1</value></taskParameter><taskParameter><name>customElement_TestCase</name><value>case_2</value></taskParameter>
SDP-35299 Create new EBS Dataset for Entity Detail - Specific to Master Sector Resolution
Create new EBS Dataset for Entity Detail
Created a new dataset called entitydetailmastsect which is specific to master/sector resolution in an EDS Mashup. There was no interface so it cannot be called independently since we already have entitydetail dataset.
SDP-35268 EDS: Add ability to define dataset\dataframe in RTR
Add ability to specify dataset\dataframe in RTR
Ability to define dataset\dataframe value was implemented and default value in ebs definition can be overwritten.<taskParameter><name>dataset</name><value>msci_esg_delimited</value></taskParameter><taskParameter><name>dataframe</name><value>msci_esg_delimited</value></taskParameter>So for example we have core ebs interface which load data into data base and we have two vendors who have incoming CSV file in different format. All we need is to create dataset files with vocabulary definition and for the load we can use CORE definition. Note: In vocabulary it is important to use EagleML dialect.
SDP-35163 Enhance transform service functions to collect timings
Added collecting 't' timings during python transformation
Collecting 't' timings was missed during python transformation. Added (transform) timing to generic transform function to report it in processing result.
SDP-35138 Create EDS Dataset and taxonomy for genericissueanalytic
Create EDS Dataset and taxonomy for genericissueanalytic
Created EDS Dataset and taxonomy for genericissueanalytic.
SDP-35010 Enhancement to add retry logic to EDS distribution profile
Retry logic of profiled EDS extract
Using Distribution Profile, there are retries for any distribution.
SDP-34454 ORA-01000: maximum open cursors exceeded when many lookup triggers
Fixed issue with ORA-01000: maximum open cursors exceeded when many lookup triggers error
For a large file we were receiving an error "ORA-01000: maximum open cursors exceeded" when there were multiple lookups in a SP. Fixed by correctly closing the descriptor in the python code.
SDP-34152 Misleading exception in REST EQL call
Misleading exception in REST EQL call
When a failure was generated for a REST EQL call, the TSR was not able to be parsed, so the user was presented with a response that did not call out the specific failure. The TSR is now able to be parsed so that the failure message in the response is more accurate.
SDP-33834 Publish the MC2 Service Installer as a separate artifact.
Publishing the MC2 Services (DR) installer as a separate artifact
SDP-32575 EDS - disable dataset cache if the heldonly filter is applied
EDS cache disabled when using held only filter
We changed the code so that if you are using the heldonly filer on the main dataset in an EDS interface, you don't have to add invalidateCache parameter. InvalidateCache parameter is applied by default.
SDP-36100 Installer: increase unzip_win runprocessor timeout to 3000 seconds
 Increased unzip_win runprocessor timeout to 3000 seconds
Increased unzip_win runprocessor timeout to 3000 seconds, to match unzip_unix, to prevent timeouts on install in windows environments.
SDP-35901 python translator fails on empty source file
Updated python translator to process empty files correctly
Modified python translator to not fail on empty source file
SDP-35550 Defect: exponential notation in JSON output of EagleML extract
Update made to EagleML extracts for format of numbers in JSON output.
Numbers in JSON format will not be rounded or truncated,and will not be displayed in scientific notation.Zeroes will not be added;the one exception being values such as .6 will be displayed as 0.6 to satisfy the JSON validator.Values will still be displayed in JSON even if they are not a valid number.
SDP-35466 Return an informative message if pace report service is not installed
Updated ReST call to pace report service
If a ReST call is made to pace-reportservice, and the service has not been installed in the region, the reply will include the message "Pace report service is not installed".
SDP-35204 EDS: Add ability to clean older cache entries for dataset snapshots
Added the ability to replace previous cache entries in EDS cache snapshots.
For redis cache entries, we clean older cache entries on dataset cache reload to reduce memory consumption.
SDP-34430 Enable secure messaging between kafka and MC2
Enabled secure messaging between MC2 and an external kafka instance
Enabled secure messaging between MC2 and an external kafka instance with SSL encryption
SDP-33013 MC2 issue loading the same EntityBankRelationship file multiple times
Updated MC2 to run panel CENTBANKREL for existing entity-bank relationship
Updated MC2 to run panel CENTBANKREL for existing entity-bank relationship for both tpesendpanproc and panprocV2 panel processing APIs.
SDP-35597 DEV-Redesign existing logic for Xrefs identifiers mapping in XMLDBAPI format
All hard coded EagleML elements moved to meta data files and to enumeration files
All hard coded EagleML elements were moved to meta data files or to enumeration files, which also should be reviewed later and replaced by metadata as well.
SDP-35101 Remove global statuses variable and introduce processing_context to Base Classes
Enhanced collecting processing statuses on the level of individual record
The implementation of collecting processing statuses for MC in pyrules was enhanced. Statuses were added into processing context to collect information on the level of individual record and processing context was introduced into Bases Classes and made a part of processing result.