SDP-16198 Add dataset definition and taxonomy for entitymanagerrelationship
Created EDS dataset for entitymanagerrelationship
We created the EDS dataset & Taxonony for entitymanagerrelationship.
SDP-37177 Edit genericsmfhistory taxonomy to use EML Elements
Edit genericsmfhistory taxonomy to use EML Elements
We noticed that the XREF Identifiers in the EDS genericsmfhistory taxonomy were not using EML elements but were the database column names. We adjusted the taxonomy to correctly reflect EML elements.
SDP-36657 GL Balances data object data missing on except outer join data extraction
GL Balances data object data missing on except outer join data extraction
We found that data was missing on operation of except outer join in an EDS interface. This exception is caused by the query when we have two fields with the same alias. In this case, we found that it was the TICKER field, because we grab one field form SECURITY_MASTER Table and another one we grab from XREFERENCE table.
The below bind files were edited:• /dynamic/metadata/bind/eagleml_ent_entityshareclass.xml• /dynamic/metadata/bind/eagleml_ref_issueprice.xml• /dynamic/metadata/bind/eagleml_ref_smf.xml• /dynamic/metadata/bind/eagleml_ref_smf_hist.xml• /dynamic/metadata/bind/eagleml_wrhs_warehouseglbalances.xml
As a solution all meta data with SMF reference was reviewed and TICKER field from SECURITY_MASTER was added in exclude list and added join with XREFERENCE table.
SDP-36500 Add ability to use a different entity_id filter than the mastertable for an EDS dataset/interface
Add ability to use a different table for entity_id filter than the mastertable for an EDS dataset/interface
We added changes to python code to be able to use a different table than the mastertable in an EDS dataset to filter on entity_id.
SDP-36349 Add an API to return information on stored procedure based datasets
EDS interface based on a EDS dataset can be created and information viewed
We made code changes so that an EDS interface based on a EDS dataset that is a stored procedure based bind file can be opened. We also added realizedgl EDS interface to the core package.
SDP-36152 Added processing statistics rollup_key_count and key_file_name to translator statistics
Added 2 fields "rollup_key_count" and "key_file_name" to statistic output of processing result to display fields calculated for rollup. Fields were added to each destination in "destinations" section.
SDP-36021 Add ability to return a new format of statistics in additionalData node of TSR
We added the ability to return new addtionalData in the TSR.
Please see example below:
<EagleML eaglemlVersion="2-0" eaglemlType="TaskStatusResponse" xsi:schemaLocation="http://www.eagleinvsys.com/2011/EagleML-2-0 eagleml-main-2-0.xsd" xsi:type="TaskStatusResponse" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.eagleinvsys.com/2011/EagleML-2-0"><header> <messageId>A9D87988CBCD4F16</messageId><sentBy>http://vgd-d023-le06.eagleaccess.com/PID=37595/WORKER=3</sentBy><sendTo>www.client.com</sendTo><creationTimestamp>2020-04-27T08:32:38-04:00</creationTimestamp></header><statusItem><taskIdentifier><correlationId correlationIdScheme="correlationIdScheme">C00EHJ23EDLPYDRX</correlationId><businessTaskId correlationIdScheme="businessTaskIdScheme">C00EHJ23DHSMVMIQ</businessTaskId></taskIdentifier><status>SUCCESS</status><severityCode>0</severityCode><reason><reasonTypeEnum>INFO</reasonTypeEnum><reasonCode>0</reasonCode><additionalData><string>*<![CDATA[{"partitions": [{"partition_id": "1", "number_of_records": 2,"destinations": [
{ "destination_id": "dis_aladdin_smf_2", "number_of_records": 2, "file_name": "files/estar/tpe/data/msgcenter/cmw/extracts/HSK2WOVM31EA21HS_GENERICSMF_DATA_p1.xml" }
]}],"summary": {"number_of_records": 2,"destination_summaries": [
{ "destination_id": "dis_aladdin_smf_2", "number_of_records": 2 }
],"processing_stats": {"processes": [{"process_id": "1","timings": [
{ "processing_stage": "e", "total_time": 1.277711919, "cpu_time": 0.015625 }
,
{ "processing_stage": "t", "total_time": 3.2561647370000008, "cpu_time": 0.421875 }
]}]},"worker_id": "DKRAVCHENKO7400/PID=16824/WORKER="}}]]>*</string></additionalData></reason></statusItem></EagleML>
SDP-35968 python translator with file mashup fails on empty source file
Python translator with file mashup fails on empty source file
The python translator was failing when the mashup was done with an empty source file. Code was added so that the process status was successful even though the source file was blanks.
SDP-35917 Send file to Cloud Dropdir Even if no_data
Empty files are distributed to the cloud
We created a parameter called allowemptyfiles to allow files that produce NO_DATA to be distributed to the cloud. This way they can have an indicator on their side whether the process of extraction data has completed but there was no data, rather than have no notification to determine if there was an issue or not.
SDP-35585 Change pace_masterdbo.interfaces join and sourceName filter from entitydetail, entitydetailhistory
Changed interfaces join for entitydetail and entitydetailhistory datasets
We were using an inner join to the pace_masterdbo.interfaces table for the entitydetail and entitydetailhistory EDS datasets, but most of the time the src_intfc_inst did not line up between the rulesdbo.entity and rulesdbo.entity_detail or rulesdbo.entity_detail_history tables. We changed this join to a left join so that the sourceName filter can be used if the src_intfc_inst matches, but will also return full data when not trying to filter by src_intfc_inst.
SDP-35538 Create clear error message when EDS Core Filters not working because the fields are not included in the taxonomy
Added clear error message when core filter is not being utilized
There is now a clearer error message when an EDS interface fails to extract when a core filter is used, and the field needed for the filter is not provided in the taxonomy.
SDP-34928 Enhancement to add FTP/SFTP retry logic to EDS (EBS) MC2 extracts
Enhancement to add FTP/SFTP retry logic to EDS (EBS) MC2 extracts
We added support of 4 new variables in w_config_custom.inc:
W_FTP_EXTRACT_RETRIES number of retries for FTP distributionW_FTP_EXTRACT_RETRYINTERVAL retry interval for FTP distributionW_SFTP_EXTRACT_RETRIES number of retries for SFTP distribution W_SFTP_EXTRACT_RETRYINTERVAL retry interval for SFTP distribution
Example:<CODE>:W_SFTP_EXTRACT_RETRIES: := 5;:W_SFTP_EXTRACT_RETRYINTERVAL: := 10;</CODE>
Specifies to retry up to 5 times, trying every 10 seconds.
SDP-29217 Implement Enhanced TSR Request to augment the nuggetizer to be able to group the file name output to include the Account
We enhanced the TSR when using the nuggetizer so that it will display the entityId for each nugget's filename when the underlying extract is utilizing the "GroupFeedType" parameter. The entity id can then be placed in the filename using "ExtractNamePatternEx" parameter. There will also be a file produced for each entity from the extracts using the "GroupFeedType" parameter.
SDP-16159 Add dataset definition and taxonomy for closelot
EDS Interface for closelot added
We created an EDS dataset and taxonomy to mirror the WRHSCLOSELOTEXTRACT which calls the tradesdbo.disposal_lots table.
SDP-36510 Update DR Installer to retrieve DB name from database
Updated DR Installer to use DB name from database
Updated DR Installer to use DB name retrieved from database, the same way as MC2 service does.
SDP-36342 SendAdapter shows all ssl information in MCC status
Updated MC2 SendAdapter rule item
Updated MC2 SendAdapter rule item to not include private information when logging an exception
SDP-35964 Refactor FTPChannel msg exchange item
Refactored the java code for MC2 FTP channel
Refactored the java code for the MC2 FTP channel to conform with static code checks.
SDP-35341 Update MC2 to get the list of MC2 nodes from a file
Update MC2 to read mc2-nodes.txt file
MC2 will attempt to read the file estar/tpe/cfg/eaglemc2s/mc2-nodes.txtin order to identify the nodes for Redis, Hazelcast and Kafka Service.If the file is not found, the nodes are read from the database.The file is generated by the MC2 Services (D.R.) installer.
SDP-35340 update Kafka service to get the list of kafka nodes from a file
Updated MC2 kafka to try to get list of nodes from file first
Updated MC2 kafka to try to get list of nodes from file first.If file is not found, the service will get the list from DB, as it has been doing.
SDP-35311 Configure HTTP server for Pyruleservice to use as REST-endpoint for MC2
Running pyruleservice through Gunicorn http server
Updated MC2 to invoke pyruleservice through Gunicorn server for increased throughput.
This new method is not enabled by default.
SDP-34991 Change the communication between MC2 and Pyruleservice to use HTTP calls
Updated MC2 to invoke pyruleservice via http service
Updated MC2 to invoke pyruleservice via http server (Gunicorn) for increased throughput.
Not enabled by default.
SDP-34040 Add dataset definition and taxonomy to mirror ENTITYMANAGEREXTRACT
Add dataset definition and taxonomy to mirror ENTITYMANAGEREXTRACT
We added a dataset and taxonomy that mirrors the EML ENTITYMANAGEREXTRACT which is the entitymanager EDS Interface.
SDP-34826 Support OpenApi 3
MC2 updated to support OpenAPI 3 for Swagger UI
MC2 updated to support OpenAPI 3 specification for RESTful interfaces in the Swagger UI