SDP-39201 MC2 Service Installer Fails To Install Services On All Nodes
As of version 2020.07.28.3
Corrected Issues With MC2 Services Installer On Environments With Multiple Nodes
MC2 Services Installer was failing to install the services on all nodes on environments with several nodes. To correct this issue the following changes were implemented:
The extraction of MC2 Services Installer was changed so that it occurs only once.
The MC2 Service Installer logs will be written in separate files for each of the nodes in the environment
The extraction of installation files from the installer was changed to update the timestamp of the files to current time,
to prevent the purge processes from removing the installation files before the installer completed the installation.
SDP-38015 Revise 5 EDS Interfaces to have correct EML Elements for Cloud Ingestion
5 interfaces were fixed
Codevalue interface:
"ebsCodeCategoryInstance" and "ebsCodeValueInstance" columns were removed from the codevalue interface.
Genericsmf interface:
It will be fixed when new metadata will be generated.
Reportdictionary interface:
"reportDictionarySrc" and "Type" columns were removed from the reportdictionary interface.
Warehouseperformance interface:
SRC_INTFC_INST srcIntfcInst dictionaeryItemId dictionaryItemId
Also fixed type of dictionaryDetailId and of all above fields as a number.
Issuer interface:
There are columns that were removed from the issuer interface:BOND_DUMMY_1CITY_1EFFECTIVE_DATE_1EQUITY_DUMMY_1FROM_ISSUER_ALIASINSTANCE IRD_CHAR2IRD_UPDATE_SOURCEISSUER_ALIAS_1ISSUER_ALIAS_2ISSUER_ALIAS_3 LEILEI_1PERCENT_OWNED_1RELEASE_DATERELEASE_DATE_1RELEASE_STATUSRELEASE_STATUS_1 ROLE_ALIASROLE_TYPE_1SRC_INTFC_INSTSRC_INTFC_INST_1STATE_1 TO_ISSUER_ALIASUPDATE_DATE_1UPDATE_DATE_2UPDATE_DATE_3UPDATE_SOURCE_1UPDATE_SOURCE_2UPDATE_SOURCE_3UPD_DATEUPD_SOURCE
Renamed fields:
ALPHA_RATING -> alphaRating
COUNTRY_CODE -> countryCode
ENTITY_STATUS -> entityStatus
ID_ASSIGNED_DATE -> idAssignedDateID_CERTIFICATION_STATE -> idCertificationStateID_DISABLED_DATE -> idDisabledDateID_LAST_UPDATE_DATE -> idLastUpdateDateID_RECORD_STATE -> idRecordState
ISSUER_PHONE -> phone
LEGAL_FORM -> legalForm
LEGAL_NAME -> legalName
STREET -> street
ZIP -> zip
ISSUER_PHONE -> phone
SDP-37273 Create EDS Dataset and Taxonomy for performdbo.perf_sec_rollup_relation table called perfsecrolluprelation
Created EDS Dataset and Taxonomy for performdbo.perf_sec_rollup_relation table called perfsecrolluprelation
We created EDS Dataset and Taxonomy for performdbo.perf_sec_rollup_relation table called perfsecrolluprelation.
SDP-38495 Source preprocessing needs ability of setting mask for column
Add new t-item SourcePreproc in EDS definition
Add new t-item SourcePreproc in EDS definition. This t-item performs replace of incoming values in specified columns according to specified rules.
Example:"t_items_def": [["source_row_col_value_replace", "subst_values", [{"column": "alpha_", "map":
{"ALPHA1": "ALPHA_#1"}
, "use_expr": false},{"column": "score_", "map":
{"N/A": "-500", "N/R": "-600"}
, "use_expr": false},{"column": "rating", "map":
{"N/A": "-123", "N/R": "-456"}
, "use_expr": false},{"column": "col", "map":
{"N/A": "-888", "N/R": "-999"}
, "use_expr": false}]]],
Make replace in columns with mask alpha_ - replace ALPHA1 by ALPHA_#1Make replace in columns with mask score_ - replace N/A by "-500" and replace N/R by -600 Make replace in columns with mask rating - replace N/A by -123 and replace N/R by -456 Make replace in columns with mask col - replace N/A by -888 and replace N/R by -999
SDP-38451 Add support basic Editcheck logic for EBS definition
Add basic version of EditCheck t-item
New t-item (EditCheck) implementation was added in EDS definition. This item raise an error if condition return True.There are two options for this item:1. Stop processing with status fail2. Keep processing with just registering error in logger
Definition of the item consist of the following parameter1. Type = editcheck(const)2. Name3. Condition of the EditCheck If TRUE than FAIL4. Error message, which will be mentioned in log.5. Error code6. Edict check type:a. Keep processing with logging error. (Default)b. 1 - Stop processing in case of ErrorExample:["editcheck", "entity_xreference_editcheck", "stringlength(|xrefAccountId|) != 0 and stringlength(|xrefAccountIdType|) != 0 and stringlength(:entity_xreference_lookup_entity_id = 0","'EntityId can not be resolved!'", "'-728'", "1"]
SDP-38423 Root node auto definition
Added ability to define root node automatically in incoming XML file
Added ability to define root node automatically. It allows user not to set "rootnodename" in "source_format_dialect". This is helpful when user has similar incoming files with different root nodes.
SDP-37841 Create Interface for issuerentityrelationship
Create Interface for issuerentityrelationship
We created an EDS interface for issuerentityrelationship. This will pull data from the SECURITYDBO.ISSUER_ENTITY_RELATION table.
SDP-37031 Optimize method for creation XML\JSON output with nested nodes
Methods for generation JSON\XML output were optimized
Methods for generation JSON\XML output were optimized and logic for calculating nested nodes was moved to initialize step instead of calculating it during processing each row. Now these methods are simpler and slightly faster 10-15% than it was before. Also support of nested nodes was added for JSON output.For CSV output new option in target format dialect was added: encode_header = Y\N ( Default = N)When this option is Y header in CSV file will encode / symbols to _ .<taskParameter><name>OutputFormatDialect</name><value>
{"delimiter":",","rowdelimiter":"\n","hasheader":"Y", "encode_header":"Y"}
</value></taskParameter>
Example:"encode_header":"N"
- sourceName,entityId,accountingBasis,market/marketValueLocal,market/marketValueBase"encode_header":"Y"
- sourceName,entityId,accountingBasis,market_marketValueLocal,market_marketValueBase
SDP-27760 Profile inheritance - getting full profile stack
Profile inheritance - getting full profile stack
We have added the use of parameter EliminateDuplicates where when set to "N", then the full profile would not remove the duplicates and return them in the output.
SDP-38753 Disable distribution attempt in eagle_default_ruleservice by mc2_distribution
Issue with 2 attempts at distribution with distribution profile
Corrected issue with 2 attempts to distribute file when distribution profile specified in RTR.
SDP-38205 Update environment name selection logic in HealthCheckDbAPI
HealthCheckDbApi now uses lower case of DBName value in db_connection.ini file.
HealthCheckDbApi now uses lower case of DBName value in db_connection.ini file.Previously the value of instance column in V$Thread system table was used.
SDP-38204 Update environment name selection logic in kafka service
kafka service now uses lower case of DBName value in db_connection.ini file.
kafka service now uses lower case of DBName value in db_connection.ini file.Previously the value of instance column in v$thread system table was used.
SDP-38117 Do not ignore EJM-aware task statuses on MC2 workflow execution
Updated the MC2 workflow executor to not ignore status from executed streams
Updated the MC2 workflow executor to not ignore status from EJM aware streams. Previously, the executor assigned a status of Success if the stream was executed,even if the stream set its own status to NO_DATA.
SDP-37217 issue with kafka-service
Corrected intermittent issue with zookeeper startup
Corrected intermittent issue with zookeeper startup
SDP-37189 Update environment name selection in installer / DR installer
Updated the source of environment name used by services.
Updated the MC2 Services installer to take the environment name used by the services from DBName value in db_connection.ini file. Previously the name was retrieved from the DB (instance column in v$thread table).
SDP-37188 Update environment name selection logic in MC2 service
MC2 service now uses lower case of DBName value in db_connection.ini file as environment name.
MC2 service now uses lower case of DBName value in db_connection.ini file as the environment name.Previously the value used was instance column of V$thread system table.
SDP-36041 Enable publishing raw and translated inbound files
Enabled publishing raw and translated inbound files to sftp or cloud in MC Classic generic load.
Enabled publishing raw or translated inbound data files from MC Classic Generic Loader.The destinations may be sftp or cloud and are identified in distribution profiles.Details: https://eagledocs.atlassian.net/wiki/spaces/MC2/pages/1389559809/Publishing+Inbound+Data+Files+In+Generic+Loader
SDP-35590 EQL requests must have more details in console (DB)
Added request identifiers to ReST API EQL endpoint Business Task Id
Updated the EQL endpoint of ReST API to populate the Business Task ID,as seen in EJM Tasks monitor and Monitoring Tool, to be REST#eql#<feedtype>#timestamp.For example:REST#eql#warehouseposition#202007020900
SDP-34625 Expose kafka status to HealthCheck
Added status of kafka service to HealthCheck
Added status of kafka messaging consumers and publishers to HealthCheck
SDP-29398 MC2 Requests were Failed in MSG Detail but Success in EJM Tasks
Corrected issue in MC2 where EJM task reported success but underlying steps had an error
Corrected issue in MC2 where EJM task reported success but underlying steps had an error
SDP-37479 Improve handling items in exclude list with prefix
Improved handling parameters with a prefix in exclude list for SQL Injection
There was code changes made to the parse_invalidate_indata.py to allow the use of a parameter with a prefix to be excluded from the check. Before this, when using a parameter with a prefix, such as "ebs:warehouseposition:OutputFilterExp"r, even the excluded parameters for SQL Injection checks were invalidated.
SDP-35824 CorrelationIdPrefix not supported for EDS WORKFLOWS (1)
Added CorrelationIdPrefix parameter
CorrelationIdPrefix parameter was added. It can be configured via RTR for EJM tasks executed via the Generic Load and Generic Extract workflows.
<taskParameter><name>CorrelationIdPrefix</name><dataType>S</dataType><value>XXXXX</value></taskParameter>
The maximum length for the prefix set to 5 symbols.
SDP-37458 Enhance possibility to exclude file from wait events queue based on workflow name
Added an option to remove all files from wait events queue based on the workflow name
Previously, it was possible to remove only one file via one control message from waitevents queue. Now RTR with CancelWorkflowName parameter and ActionType 'CancelFile' allow to cancel all files from waitevents queue and specific workflow.
<taskParameter><name>CancelWorkflowName</name><dataType>S</dataType><value>sdp37458_wrf_generic_load</value></taskParameter><taskParameter><name>ActionType</name><dataType>S</dataType><value>CANCELFILE</value></taskParameter>
Files changed:eagle_ml-2-0_cm/out/tagvalue/execute_wrkfl_event.xmleagle_ml-2-0_cm/cancel_global_event_wait.inc
SDP-37066 Update Swagger API docs version to OpenAPI 3
Update ReST endpoint swagger API documents to OpenAPI 3.0
Updated the ReST endpoint swagger API documents to comply with OpenAPI 3.0 specification.Previously, the swagger API documents complied with Swagger 2.0 specification.Client code generators will need to be compliant with OpenAPI 3.0.
SDP-38319 Support Currency Conversion Overrides in REST API for Fetching FX Rates
Supporting FX Rates Fetch for Currency Conversion in OLAP REST API
New feature for overriding Forex Spot rate fetching for the report effective date is supported in OLAP REST API. This allows the user to override the number of prior days to go back and fetch the Forex spot rate which in turn be used for either benchmark or entity's base currencies. This is a similar functionality supported in OLAP reports.
SDP-38318 Support Currency Conversion Overrides in REST API for First and Second alternate Currencies
Supporting Base Entity's Currency override in OLAP REST API
New feature for overriding entity's base currencies either to first or second currency is supported in OLAP REST API. This allows the user to override profile entity's base currencies to any alternate currencies. This is similar functionality supported in OLAP Reports.