EagleML Release Notes - February 2019

SDP-15754: Add support of WarehouseNavMonthEnd object to eagle_wrf_generic_load workflow.

The Generic Load workflow was enhanced to support the WarehouseNavMonthEnd objects.

Files changed:
msgcenter\eis\ejmwf\eagle_wrf_generic_load.ejmwf
msgcenter\eagle_ml-2-0_workflows\eagle_wrf_generic_load.wrf

SDP-14563: Pricing Center - Parent event showing complete prior to child event completing

When a Pricing Center Event consists of one rule, the sub-event that is executed for this rule runs independently. The Pricing Center event did not wait for its completion and reported that the event had finished. When the Pricing Center event is executed via EJM, the process was marked as complete even if the sub-event was still running.

EJM was enhanced to check statuses of Pricing Center sub-events and wait for their completion.

Files changed:
exec_pace_event.xml
exec_pace_event_pricing_check.inc

SDP-15206: Generic Loader change to process the SMF Composite only to execute when loading SMF data

The Generic Load workflow has a sub-process, “eagle_generic_load_smf_composite,” which may be used to call a Pace Event, RDC or SRM event.
When one of the following parameters (PACEEventName, W_ExecuteSRM or W_ExecuteRDC) was set (for example in w_config_custom.inc) this sub process was launched.
This sub-process was triggered whether or not SMF files were present in the incoming files.

The condition to trigger this step was updated to check if the SMF file is part of the incoming files.

Files changed:
eagle_wrf_generic_load.ejmwf
eagle_wrf_generic_load.wrf

SDP-14278: Process error descriptions in the TSR from the reason node processed by i2i_parse_errors_from_hub.xsl/taskstatusresponse.xml
During the the Instance2Instance process, two environments exchange RTR's \TSR's with each other. If an error occurs on the remote server, a failed TSR is sent. The change is for the local environment to be able to process the TSR and show the correct error description message on the local environment.

The XSL Transformation file has been adjusted to process all error descriptions present in the incoming TSR file.

Files changed:
- /msgcenter/eagle_ml-2-0_cm/in/xml/taskstatusresponse.xml
- /msgcenter/eagle_ml-2-0_cm/in/xml/xslt/i2i_parse_errors_from_hub.xsl

SDP-10920: Issue with query panel/tag55 logic in the xml-ref_issueprice.xml

The issuePrice has been adjusted that in case of an invalid effectiveDate or sourceName fields in the incoming message, the logic has been updated to default to insert mode, instead of update mode as it was before the modification.

Files changed:
- msgcenter\eagle_default\in\xml\xml-ref_issueprice.xml

SDP-14566: Enhance EagleML in the AccountingAssetCash coverage to support mapping of the Cash Event Type

The AccountingAssetCash Interface was enhanced to support the cashEventType element for MISC Income and Expense Asset Cash message types (Asset Cash Type = AMISCINC or AMISCEXP).
Possible values of Cash Event Type: DISB, RECPT.
<cashEventType>DISB</cashEventType> or <cashEventType>RECPT</cashEventType>

Files changed:

\dynamic\msgcenter\eagle_default\in\xml\ xml-acct_cash_asset.xml
\dynamic\msgcenter\eagle_default\in\include\csv-cash_misc_inc_exp.inc
\dynamic\eagleml\schema\eagleml-acc-2-0.xsd

SDP-1700: EntityAnalytic load issue

The EntityAnalytic interface failed during load. The problem was in the auto-generated stored procedure which in some cases had incorrect conversion logic for dates. The include file, which generates the stored procedure, has been fixed to handle these scenarios correctly.

Changed file:
eagle_ml-2-0_cm\generate_sp_on_fly.inc

SDP-15851: Entitytocomposite object fails the record when the same record is loaded in

The Entitytocomposite object fails the record when the same record is reloaded instead of ignoring it or or updating the record.

There was a problem with parallel loading of several entities to one composite. This happens due to the logic of stored procedure that updates the entire composite even if the effective date for one entity was updated.

A condition in the split rule was updated to process entities added to one composite with the same effective date to be processed sequentially.

Files Changed:
eagle_default\in\xml\xml-ent_objects.rsf
eagle_default\in\xml\ xml-ent_entitygeneric.spl

SDP-14038: Issue with incorrect TSR (missed identifiers) in case of enabled auto-reprocess streams

There were no error details in the final TSR for messages that were processed by the auto-reprocess stream. These messages have specific status value in MSGCENTER_DBO.MSG_DETAIL.STATUS like 10 or 11. To collect error details for messages with such statuses for the TSR, the requests were updated in the eagle_ml-2-0_default_cm_task_reporter stream.

Files changed:
task_reporter_generate_task_unit.inc
task_reporter_gen_task_unit_realtime.inc

SDP-17249: ISSUERRATINGEXTRACT data source filter and grouping issues

The Issuer Rating Extracts were enhanced to support the source filter and the grouping was updated to include source field in group key.
Correct grouping statement is Issuer Alias + Alpha Rating + Source Instance.

<taskParameter>
<name>datasources</name>
<value>MOODY</value>
</taskParameter>

Changed files:
- extract_issuer_rating.inc
- t_issuer_rating_streaming.inc

SDP-15083: Enable support when loading a Forward security to load the Alt Asset ID on Both Currency Legs

The Forward model of GenericSMF Interface was enhanced to support unique Asset Identifiers in legAsset node and lookThroughIndicator, lookThroughValue fields for second leg (buy or sell) in one message.

Sample RTR:

<EagleML eaglemlVersion="2-0" eaglemlType="ReferenceTransactionMessage" xsi:type="LoadReference" xmlns="http://www.eagleinvsys.com/2011/EagleML-2-0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<header>
<messageId>EA12345</messageId>
<sentBy>CSMEagleAdapter</sentBy>
<sendTo>http://www.eagleinvsys.com/eagle_ml-2-0_default_cm_control_message</sendTo>
<creationTimestamp>2017-06-19T08:15:36-05:00</creationTimestamp>
<replyTo>http://www.eagleinvsys.com/g_os_csm_jmscore_eml_response</replyTo>
<ackTo>http://www.eagleinvsys.com/g_os_csm_jmscore_eml_response</ackTo>
</header>
<taskIdentifier>
<correlationId correlationIdScheme="correlationIdScheme">FWDLOAD_EX001</correlationId>
<businessTaskId businessTaskIdScheme="businessTaskIdScheme">FWDLOAD</businessTaskId>
</taskIdentifier>
<referenceTransaction>
<header>
<action>ADD</action>
<objectType>GenericSMF</objectType>
</header>
<genericSMF>
<accountingValidationFlag>Y</accountingValidationFlag>
<sourceName>EAGLE PACE</sourceName>
<updateSource>USER</updateSource>
<effectiveDate>2018-12-05</effectiveDate>
<primaryAssetId>SMPLFWD1171199144</primaryAssetId>
<primaryAssetType>INTERNAL</primaryAssetType>
<altAssetIdAndType>
<instrumentId>EMLFW74</instrumentId>
<instrumentIdType>COMMON</instrumentIdType>
</altAssetIdAndType>
<processingSecurityType>FWXXXX</processingSecurityType>
<investmentType>FWD</investmentType>
<issueName>USD/TRY FWD 20190131</issueName>
<issueDescription>USD/TRY FWD 20190131</issueDescription>
<maturityDate>2019-01-31</maturityDate>
<quantityType>CURRENCY</quantityType>
<priceMultiplier>1</priceMultiplier>
<quantityScale>1</quantityScale>
<buysell>S</buysell>
<ForwardModel>
<buyCurrency>TRY</buyCurrency>
<pricingCurrency>USD</pricingCurrency>
<matchBrokerIndicator>N</matchBrokerIndicator>
<forwardBrokerCode>BIC</forwardBrokerCode>
<sellCurrency>USD</sellCurrency>
<legAsset>
<altAssetIdAndType>
<instrumentId>EMLFW39</instrumentId>
<instrumentIdType>COMMON</instrumentIdType>
</altAssetIdAndType>
</legAsset>
</ForwardModel>
</genericSMF>
</referenceTransaction>
</EagleML>

Changed files:

eagle_default\in\xml\xml-smf_generic.xml
eagle_default\in\xml\include\smf_forwrd.inc
eagle_default\in\xml\include\security_resolution_generic.inc
eagle_default\in\xml\include\get_xrefs_list.inc
eagle_default\in\xml\include\smf_aliases.inc
eagle_default\in\xml\include\forwardsmf_model.inc
eagle_default\in\xml\xslt\get_smf_all_xrefs_list.xsl
eagle_default\in\xml\xslt\eagleml_smf_gen_fields.xsl

New files:
eagle_default\in\xml\include\forward_identifiers_remap.inc
eagle_default\in\xml\include\smf_identifiers_map.inc

SDP-14318: Duplicate identifier in database after SMF update

A duplicate identifier was posting to the database during SMF updates. The duplicated Xref Identifiers were added to the same security when the Accounting Validation Flag is passed with a value of Y.

In order to resolve the issue, the variables are set to be reinitialized in the rule.

Changed files:
eagle_default\in\xml\include\security_resolution_generic.inc

SDP-12456: Accounting Trade loads through EagleML requires support of the tagNochangecase capability

Accounting Trade and Multiple Leg Accounting Trade interfaces were enhanced to support the tagNochangecase element. This element is mapped to tag 4637 in the Trade panels and specifies which tag elements should maintain their original case.

Sample RTR:

<EagleML xsi:type="AccountingTransactionMessage" eaglemlVersion="2-0" eaglemlType="AccountingTransactionMessage" xsi:schemaLocation="http://www.eagleinvsys.com/2011/EagleML-2-0 eagleml-main-2-0.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.eagleinvsys.com/2011/EagleML-2-0">
<header>
<messageId>EML-In_TRADE</messageId>
<sentBy>user</sentBy>
<sendTo>http://www.eagleinvsys.com/eagle_ml-2-0_default_cm_control_message</sendTo>
</header>
<taskIdentifier>
<correlationId>EML-In_Conversion_{GUID}</correlationId>
<businessTaskId>EML-In_TRADE</businessTaskId>
</taskIdentifier>
<accountingTransaction>
<header>
<identifier>
<transactionId>ticket4</transactionId>
</identifier>
<objectType>AccountingTrade</objectType>
</header>
<accountingTrade>
<accountingValidationFlag>Y</accountingValidationFlag>
<buySellIndicator>BUY</buySellIndicator>
<mcModeSwitch>N</mcModeSwitch>
<entityId>SMPLE001</entityId>
<accountingBasis>USTAX</accountingBasis>
<primaryAssetId>EQJNZ5516</primaryAssetId>
<primaryAssetType>INTERNAL</primaryAssetType>
<assetResolutionOption>MATCH_ALL</assetResolutionOption>
<tradeDate>20190102</tradeDate>
<settlementDate>20190102</settlementDate>
<sharesParQuantity>300000.0</sharesParQuantity>
<price>100.00</price>
<brokerCode>EAGLE</brokerCode>
<originalTradeTicketNumber>ticketAaBb</originalTradeTicketNumber>
<longShortIndicator>L</longShortIndicator>
<calculateLocalAmountsFlag>Y</calculateLocalAmountsFlag>
<calculateBaseAmountsFlag>Y</calculateBaseAmountsFlag>
<settleInstructions1>xsettleontrade</settleInstructions1>
<tagNochangecase>5032,26,457</tagNochangecase>
</accountingTrade>
</accountingTransaction>
</EagleML>

Files changed:

\dynamic\msgcenter\eagle_default\in\include\csv-buy_columns.inc
\dynamic\msgcenter\eagle_default\in\include\csv-cancel_columns.inc
\dynamic\msgcenter\eagle_default\in\include\csv-capfundinvestments_columns.inc
\dynamic\msgcenter\eagle_default\in\include\csv-conversion_columns.inc
\dynamic\msgcenter\eagle_default\in\include\csv-deliver.inc
\dynamic\msgcenter\eagle_default\in\include\csv-fxfwd_columns.inc
\dynamic\msgcenter\eagle_default\in\include\csv-fxspot.inc
\dynamic\msgcenter\eagle_default\in\include\csv-receive.inc
\dynamic\msgcenter\eagle_default\in\include\csv-sell_columns.inc
\dynamic\msgcenter\eagle_default\in\include\trade_aliases.inc
\dynamic\msgcenter\eagle_default\in\xml\xml-accttrade_multipleleg.xml
\dynamic\msgcenter\eagle_default\in\xml\include\multiplelegtrade_aliases.inc
\dynamic\msgcenter\eagle_default\in\xml\include\multiplelegtrade_common_columns.inc

SDP-10683: Create new EagleML extract to retrieve a listing of AEW events

A new outbound interface was created for the new ControlCenterEventLog EagleML object.

Feed type: CONTROLCENTEREVENTLOGEXTRACT
Main table: CTRLCTR.EVENT_GROUP
Additional tables: CTRLCTR.EVENT_GROUP_DETAILS, CTRLCTR.QUERY_EVENT_GROUP_TAGS

Supported date filter for extract:
- UpdateDate
Supported own filters for extract:
- eventgroupname (filters on the 'eventGroupName' field, DB field name: EVENT_GROUP.EVENT_NAME)
- eventtaskid (filters on the 'eventTaskId' field, DB field name: EVENT_GROUP.TASK_ID)
- activeflag (filters on the 'activeFlag' field, DB field name: EVENT_GROUP.ACTIVE_FLAG)
- completedflag (filters on the 'completedFlag' field, DB field name: EVENT_GROUP.COMPLETED_FLAG)
- coregroup (filters on the 'coreGroup' field, DB field name: EVENT_GROUP.CORE_GROUP)
- statisticcategory (filters on the 'statisticCategory' field, DB field name: EVENT_GROUP.STATISTICS_CATEGORY)
- statisticname (filters on the 'statisticName' field, DB field name: EVENT_GROUP.STATISTICS_NAME)
- statistictype (filters on the 'statisticType' field, DB field name: EVENT_GROUP.STATISTICS_TYPE)
- businessprocessinggroup (filters on the 'businessProcessingGroup' field, DB field name: EVENT_GROUP.BUS_PROCESS_GRP)

The extract has been added to the eagle_wrf_generic_extract workflow.

Sample RTR:

<EagleML xsi:type="RunTaskRequestSync" eaglemlType="RunTaskRequest" eaglemlVersion="2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<header>
<messageId>CC59ADD40E1E069</messageId>
<sentBy>http://www.client.com</sentBy>
<sendTo>http://www.eagleinvsys.com/eagle_ml-2-0_default_cm_control_message</sendTo>
<replyTo>http://www.client.com/Eagle Control Messages</replyTo>
</header>
<taskIdentifier>
<correlationId>SDP10683_CCE_EXTRACT_${=import java.text.SimpleDateFormat ; new SimpleDateFormat("YYMMddHHmmssS").format(new Date())}</correlationId>
<businessTaskId>AEW_Extract_SC_ALL</businessTaskId>
</taskIdentifier>
<taskTypeEnum>LOAD</taskTypeEnum>
<synchronousExecution>no</synchronousExecution>
<taskParameters>
<taskParameter>
<name>ActionType</name>
<dataType>S</dataType>
<value>EXTRACT</value>
</taskParameter>
<taskParameter>
<name>StreamName</name>
<dataType>S</dataType>
<value>eagle_ml-2-0_default_out_q</value>
</taskParameter>
<taskParameter>
<name>FeedType</name>
<dataType>S</dataType>
<value>CONTROLCENTEREVENTLOGEXTRACT</value>
</taskParameter>

<taskParameter>
<name>eventgroupname</name>
<dataType>S</dataType>
<value>CC-COST-INC-CHG-EXCP-PROCESS</value>
</taskParameter>





<taskParameter>
<name>completedflag</name>
<dataType>S</dataType>
<value>Y</value>
</taskParameter>

<taskParameter>
<name>coregroup</name>
<dataType>S</dataType>
<value>N</value>
</taskParameter>

<taskParameter>
<name>statisticcategory</name>
<dataType>S</dataType>
<value>M-ACCT-PD-CLOSE</value>
</taskParameter>

<taskParameter>
<name>statisticname</name>
<dataType>S</dataType>
<value>RATES</value>
</taskParameter>

<taskParameter>
<name>statistictype</name>
<dataType>S</dataType>
<value>ALL</value>
</taskParameter>

<taskParameter>
<name>businessprocessinggroup</name>
<dataType>S</dataType>
<value>ALL</value>
</taskParameter>
</taskParameters>
</EagleML>

Added files:
extract_int_control_center_event_log.inc
extract_int_control_center_event_log.xml
t_int_control_center_event_log_streaming.inc
t_int_control_center_event_log_streaming.xml

Modified files:
extract_sql_schema_name.inc
extracts.rsf
t_eagleml-2-0_streaming.rsf
w_config.inc

SDP-14727: ENTITYENTITLEMENTLISTEXTRACT - Missing ObjectDescription in Output for MC Classic

The objectDescription field has been added in the extract transform rule for EntityEntitlementList.

Changed files:
t_ref_entity_entitlement_list_streaming.xml
t_ref_entity_entitlement_list_streaming.inc

SDP-14354: EntityShareClass, EntityStatistics, EntityRating - fix minor output and grouping issues

The ENTITYSHARECLASSEXTRACT, ENTITYSTATISTICEXTRACT and ENTITYRATINGEXTRACT were all updated to correct identified output and to improve the grouping of the data.

ENTITYSHARECLASSEXTRACT - EntityShareClass:
The EntityShareClass extract contained an empty heldAsset node.
The transform rule has been updated to not produce an empty node if there is no information inside.

ENTITYRATINGEXTRACT - Entity Rating:
The EntityRating extract transform rule did not group records by source, as a result invalid EagleML could be created against what is defined in the XSD.
The source field has been added in the grouping for the EntityRating interface in the extract transform rule.

ENTITYSTATISTICEXTRACT - EntityStatistics
The EntityStatistic extract transform rule did not group records by source, as a result invalid EagleML could be created against what is defined in the XSD.
The effectiveDate field was updated to include mapping.
The source field has been added in the grouping for the EntityStatistic interface in the extract transform rule.

Changed files:
t_entity_share_class_streaming.inc
t_entity_rating_streaming.inc
t_entity_statistics_streaming.inc

SDP-15920: Workflow_restart stream not failing if the underlying workflow is failed

Logic was enhanced to ensure that the corresponding event and schedule for the schedule-based workflow get the correct status when the underlying workflow fails.

Changed files:

task_reporter.xml
workflow_restart.xml

SDP-10820: EJM: Adjust eagle_ml-2-0_default_cm_sequencer to have correct out of order logic

The sequencer logic was enhanced to sort the queue of workflow instances waiting for processing.

This allows processing out of order instances in correct sequence from smaller to larger.

Modified file:
sequencer_orch.xml

SDP-15048: EagleML: Source interface has an issue with long sourceName

Added logic to check that the value of the sourceName element does not exceed 15 characters during the Source Interface load.

Files changed:
xml-ref_source.xml

SDP-12457: Incorrect error message for WarehousePosition

When the warehouse position load fails in the Batch mode, incorrect security identifiers may be displayed in the TSR.

The error processing for batch mode was revised. The error returned from the database will be displayed in the TSR with the correct identifiers.

Modified file:
xml-dbdirect_error_msg.inc

SDP-9276: Curl error during moving several extract files

When split extract file logic is applied (For example MicroBatchKeyRecordCount parameter) and the deliveryMethod is configured as (S) FTP, several files may be generated as a result of the extract. In this case there was a problem that only one file was sent to the destination.

The logic for the delivery extracted files to (S) FTP server was updated to be able to send more than one file to the destination.

Changed files:

reporter_dist.inc

SDP-15121: Incorrect dynamic entity list resolution in extracts when the definition includes the Accounting System Description

The extract rules were not correctly resolving dynamic entity lists when the list definition included the Accounting System Description field.

A new type of EntityList resolution has been added, it uses dynamic query instead of stored procedure.
Added a new value for the parameter 'entityselectiontype' for switching between Entity List resolution modes.

'entityselectiontype' parameter values:
- EntityList - default mode. Stored procedure is used for Entity list resolution (for both EntityList types dynamic or static).
- DynamicPortfolioList - new mode. Dynamic query will be generated and used. Static Entity list in this mode will be resolved by the stored procedure (as in default mode).

Changed files:
- extract_filter_entity.inc
- extract_filter_entity_list_resolution.inc

SDP-9266: Allow EJM profiles to use date rules

Support for using date rules in EJM profiles was added. The date rule can be relative to the current date or a specified date.

To use this functionality a parameter has to be added to the profile with the following structure: [date parameter name] + “DateRule” (ex: effectivedateDateRule). The value for this parameter should be the name of the date rule that will be used.

If using the date rule relative to the current date, including the above parameter on its own is enough. The resulting RTR will then have just the date parameter with a value that was calculated based on the date rule (i.e. if effectivedateDateRule is used, the RTR will just contain effectivedate with a value taken from the date rule calculation).

When using this functionality relative to a specified date, then the starting value should be added to the profile in that date parameter. The RTR will contain that same parameter but the original value will be replaced with the calculated one. (I.e. effectivedate [value 2019-01-01] effectivedateDateRule set to next day: the RTR will contain effectivedate [value 2019-01-02])

Changed files:

collect_task_parameters.inc

SDP-15962: Fix 2017R2 (WIN) Error in Insert and Update Daily Cycles in QA envs. in REF_013_Generic_Issue_Analytics_Insert.xml

There were a number of fields that were missing in the Generic Issue Analytic extract.

The Generic Issue Analytic interface has been updated to extract these fields.

The following models were changed:

analyticsUserModel (SECURITY_USER_ANALYTICS table) - userSaChar16 - userSaChar30 were not loaded on MSSQL environments version 2017
Meta information has been updated for this model, to load these fields on MSSQL environments.

ePSEquityModel (SECURITY_EPS_EQUITY table) - only epsEst1X and epsEst2X elements were supported on Oracle environments. The model was not supported on MSSQL environments.Added processing of ePSEquityModel for MSSQL. Added all fields for the model described in XSD for Oracle and MSSQL to load and extract.

priceEarnEquityModel (SECURITY_PRICE_EARN_EQUITY) - The model was not supported on MSSQL environments.
Added processing of priceEarnEquityModel for MSSQL.

Changed files:
call_eagleml_dbml_request.inc
eagleml_ref_genericissueanalytic.xml
generate_sp_on_fly.inc
t_generic_issue_analytic.xml
t_generic_issue_analytic_streaming.inc
xml-ref_genericissueanalytic.xml

SDP-12935: Provide a possibility to exclude files from WAITEVENTS.DAT queue

User were not able to remove records about files waiting to be processed.
Information about waiting workflows and files waiting for the launched workflows stored in WAITEVENTS.DAT file or
in the database (if parameter W_WAITEVENTS_DATABASE_MODE = Y in the w_config_custom.inc) are now available.

Functionality to cancel wait event based on the file name (FileName) or WaitEventCorrelationId has been added.
When a user sends a RTR with WaitEventCorrelationId or CancelRemoteFileName, this value match will wait for the event state record,
and the waiting file for the workflow will be canceled.

Added the ability to update the file statistic information for the canceled files when the file statistic is available
(W_ENABLE_EXTENDED_FILE_STATISTICS = Y is set in w_config_custom.inc) was added.
if W_WAITEVENTS_DATABASE_MODE = Y then canceled files will be transferred from {workflow_name} _wait_queue (queue for waiting events) to {workflow_name} _error_queue (queue for canceled events)

RTR taskParameter:

ActionType = CANCELFILE
StreamName=eagle_ml-2-0_default_cm_execwkfl
CancelWorkflowName – Workflow Name to cancel event wait (optional parameter, should be used when W_WAITEVENTS_DATABASE_MODE=Y and W_ENABLE_EXTENDED_FILE_STATISTICS=N)
To cancel event wait for run workflow should use one from the next parameters:
WaitEventCorrelationId – CorrelationID of the file to cancel event wait
FileName – a name of the file to cancel event wait

Files Changed:
eagle\estar\tpe\dynamic\msgcenter\eagle_ml-2-0_cm\cancel_global_event_wait.inc
eagle\estar\tpe\dynamic\msgcenter\eagle_ml-2-0_cm\get_file_stat.inc
eagle\estar\tpe\dynamic\msgcenter\eagle_ml-2-0_cm\out\tagvalue\execute_wrkfl_event.xml
eagle\estar\tpe/dynamic\msgcenter\eagle_ml-2-0_cm\cleanup_orch_wait_state.inc

SDP-17577: WRHSOPENLOTEXTRACT and WRHSPOSITIONEXTRACT - MSTARDIRECT source with IncludePortfolioValuation set to Y or ALL failing

When the Source was passed in as 'MSTARDIRECT', the extract SQL query was generated incorrectly when the IncludePortfolioValuation was included.

Fixed the error of executing of Warehouse Position and OpenLot extracts with parameter 'IncludePortfolioValuation' = Y or ALL and Source filter = 'MSTARDIRECT'.
Changed rules:
- extract_holding_ext_port_valuation_fields.inc

SDP-15207: EagleML: Warehouse Rollback functionality should not use the CorrId from the file name(XMLDBAPI)
SDP-15191: EagleML: Warehouse Rollback functionality should not use the CorrId from the file name(EagleML)

The usage of the correlation id from the file name for checking which sub batches had failed on the load and run a roll back for the whole batch was proven reliable because the correlationId should be in exact place in the file name.

Only the correlationId in the file name can link key and data files related to the one process instance.
This approach was replaced by adding correlationId value on the Python transformation step in the files itself.
Now key and data files have an additional value CORRELATION_ID(XMLDBAPI) and taskIdentifier/correlationId (EagleML).
It helps to link key and data files related to the one process more reliable, because each process has a unique correlationId.

Changed files:
include/xml-warehouse_common.inc
xml-dbdirect_invar.xml
xml-dbdirect_invar_rollup.xml
xml-warehouse_v2.xml

SDP-9851: issue in loading extended generic SMF (Xpanels extension and fieldCompareFlag=Y bug)

There was a problem with extension processing when fieldCompareFlag = Y, because the insert\update into the extension table happens in a separate database call resulting in the extension and core tables not being in sync.

The new implementation allows to extend the core stored procedure to support loading extension fields which will keep the extension and core tables in sync.
To activate this logic and switch to the new processing some manual steps are required to be setup as a one time activity. Request additional information located on the confluence page defining extensions.

Changed files:

eagle_default/in/xml/include/custom_fields.inc
eagle_default/in/xml/include/msgxchg_sec_core.inc
eagle_default/in/xml/include/ng_exec_sp.inc
eagle_ml-2-0_cm/generate_sp_on_fly.inc

SDP-10059: Issues with EXPENSEACTIVITYEXTRACT

The Expense Activity Extract did not report an error when the required filters reportStartDate and reportEndDate were not specified.

A parameter check was added to return an error message if the filters are not part of the request.

File changed:
extract_acct_expense_activity.inc

SDP-17578: WRHSOPENLOTEXTRACT and WRHSPOSITIONEXTRACT - Malformed XML

There was an issue identified in grouping for the Warehouse Position and OpenLot extracts when using the ‘IncludePortfolioValuation’ with a value set to Y or ALL. The data appear as an outlier, but to handle this inconsistency, the PortfolioValuation logic has been enhanced and a new repeated node pvAssetPrice was added in the extendedPortValuationModel to address when more than one price / source combination exists for the specific position.

This model will describe all prices which exist for the specific position.

Changed rules:
- extract_holding_ext_port_valuation_fields.inc
- t_holding_portfolio_valuation_fields_streaming.inc
- extract_save_and_execute_query.inc