Enhancements

EagleML

Added the List of the Possible Global Parameters and their Default Values
The list of all possible global parameters is not obvious, neither are the parameters’ default values.

A list of global parameters was added in the w_config_custom_example.inc for information purposes. It was inserted into the comment section.

This issue was tracked as Item #153491.

Three Global Parameters Have Been Renamed
Some global parameters have negative meanings which was causing some confusion.  The values were renamed to improve the user experience.
Three parameters were renamed and redefined with set defaults.

This issue was tracked as Item #153493. 

Option for Custom SP Usage was Added for the Issuer Role
The Issuer Role was updated to enable configuration in the case of the custom stored procedure /version setup.

A new parameter named W_ISSUERROLE_CUSTOM_SP was added with the valid values of Y/N. The parameter allows the process to be manually set to indicate which version of the SP is used - the newer (15.2 or 17.2 and higher) or the historical version. The value of Y is to select the new procedure and N is for the historical version. By default, the stored procedure version is set according to the region version.

This issue was tracked as Item #153734. 

The Consolidated Earned Income and Unsettled Transactions are Returning the Maturity Date as 19000101
The EagleML Objects for the Consolidated Earned Income - CONSEARNEDINCOMEEXTRACT and the Unsettled Transactions - UNSETTLEDTRANSACTIONSEXTRACT were updated, as the Maturity date is retrieved by the stored procedure. If the date is NULL then the value ‘19000101’ is returned and presented as the MaturityDate.

The change was to evaluate the field <maturityDate> and remove it from the extract if it is empty.

This issue was tracked as Item #154254.

Added a Feature to Retrieve the Begin Date / End Date from the FloaterModel in eagleml_smf_gen_underlying_tagvalue.xsl
eagleml_smf_gen_underlying_tagvalue has no logic to retrieve beginDate/endDate from the eagleml:FloaterModel complex node.

Changed XSLT (eagleml_smf_gen_underlying_tagvalue.xsl) to support the begin date / end date in FloaterModel.

This issue was tracked as Item #154580.

Shareholder Activity Model was Enhanced to Include the Rest of the Elements of the Database Table
The Shareholder Activity Model contained only a part of database fields. More fields from the table are used and should be supported.

The following FeedTytes were enhanced:
1. WRHSNAVMULTIEXTRACT
2. WRHSNAVMONTHENDEXTRACT
3. SUBSCRIBEREDEEMEXTRACT
4. ENTITYSHARECLASSEXTRACT

The EntityShareClass.model was changed and is now used in place of the ShareholderActivity.model.

The full list of fields:
CLASS_CODE,CLASS_NAME,TRANSFER_AGENCY,TA_CODE, NASDAQ_SYMBOL, CLASS_STATUS,CLASS_TERMINATION_DT,
DAILY_FIXED_DIST_RATE,MONTHLY_DIST_RATE, INCEPTION_NAV, SECURITY_IDENTIFIER, NASDAQ_LISTING_START_DATE,
NASDAQ_LISTING_END_DATE,LOAD_CHARGE, LOAD_EFFECTIVE_DATE, USER_CHAR1, USER_CHAR2, IND_12B1,
NAV_CURRENCY,CLASS_INCEPTION_DATE,HELD_AS_SECURITY, NAV_PRICE_PRECISION, CUSIP, TICKER,
EXCHANGE, ISIN, SECURITY_ALIAS, BID_SPREAD_FACTOR, BID_FORMULA, OFFER_SPREAD_FACTOR,OFFER_FORMULA,
EURO_EQUAL_METHOD,BASE_CLASS,CONTRACTUAL_EXP_RATIO,SHARE_CLASS_ID,MINIMUM_YIELD,NII_DISTRIBUTION_METHOD

Additional compatibility was added for the NasdaqSymbol.  The element name has been updated to shareClassNasdaqSymbol instead of nasdaqSymbol in the EntityShareClass.model.

This issue was tracked as Item #155098.

GenericSMF Extensions History Table Logic
This issue is related to defect 152039 which updated only one include file for the GenericSMF extension. The fix was applied to the custom core fields only and did not affect any other extension tables.
This addressed the other include files related to other EagleML feed types or models.
The issue was causing problems with History tables whenever any extensions, except the limitedPartnershipModel, are uploaded.
The corresponding logic was changed to fix this item.

This issue was tracked as Item #155485.

MillRate Fields Being Populated for Non-Daily Distributions
Fields specific to mill Rate/Daily Distributions are being populated for non-mill rate Distributions.

The logic was refined to include the listed fields only if the Distribution Category is not equal to (PERIODIC or PERIODICAJ).
These fields were added into SQL with the following statement (example for 'MILL_RATE'):
CASE WHEN (ESTAR_DIST_LOG.CATEGORY = 'PERIODIC' OR ESTAR_DIST_LOG.CATEGORY = 'PERIODICAJ')
   THEN NULL
   ELSE ESTAR_NAV.MILL_RATE
END MILL_RATE,

This issue was tracked as Item #155580.

Added Four New Parameters for the I2I Process
New parameters were requested for I2I:
SPOKE_UID
PROCESS_UID
FEED_TYPE

These are internal parameters for the I2I Tool.

Added the following parameters for the I2I process: SPOKE_SID, I2I_PROCESS_UID, FEED_TYPE, PROCESS_UID.

Changes were applied to the extract_delta_orch.inc include file.

This issue was tracked as Item #155588. 

PERF_SUM_INST, SECURITY_ALIAS, PERF_ROLLUP_RETURNS_ID Set as Key Fields for Extension Table
Key fields for the extension table are security_alias,perf_freq_code, dictionary_id, perf_summary_type, entity_id , effective_date, src_intfc_inst. Must be PERF_SUM_INST, SECURITY_ALIAS, PERF_ROLLUP_RETURNS_ID.

1. Added resolution logic in wrhs_ext_custom_fields.inc for PERF_SUM_INST and PERF_ROLLUP_RETURNS_ID
2. Added changes for entity in xml-dbdirect_common.inc
3. Added elements in warehousePerformance.xsl for key fields resolutions

This issue was tracked as Item #156011. 

Tags 14 and 1432 are Passed to the Issuer Pc Price Panel to Load if not Specified
Tags 14 (Primary Asset Id) and 1432 (Primary Asset Id Type) are required in the Issue Pc Price load panel. These tags are not resolved by Security Resolution Logic.

Added tags 14 and 1432 to Security Resolution Logic core include.

This issue was tracked as Item #156032.

Ampersand Processing Corrected for Extension Tables
The “&” from file is being stored as “&amp;” in the extension table.
This causes problems with the "&" in extension fields whenever the ampersand symbol is present in the extension fields.

Support for processing of the ampersand symbol was included.

This issue was tracked as Item #156124.

A New EagleML Interface ExpectedCreditLoss was Created

A new EagleML object for the pan-collectivelevel_grp.htm panel was created.
1. New accounting based data object ExpectedCreditLoss description was added in XSD
2. The rule "xml-acct_expectedcreditloss.xml" was created

This issue was tracked as Item #156139.

Created a New ISSUER_REGULATORY Object and Added New Fields to Issuer Organization Object and the ISSUERREGULATORYEXTRACT Extract
Added support for new fields of the ISSUER_ORGANIZATION table to XSD, inbound and outbound rules.
Added new EagleML type: IssuerRegulatory.  The ObjectName is ISSUERREGULATORYEXTRACT.
Added support of this type to the XSD, Generic Loader and Generic Extractor.
Created inbound and outbound rules and added NG support in the inbound rule.

This issue was tracked as Item #156144.

Created new ENTITY_REGULATORY object and the ENTITYREGULATORYEXTRACT extract and Added New Fields to Entity Object
Added support of new fields for ENTITY table to XSD, inbound and outbound rules.
Added new EagleML type: EntityRegulatory.  The ObjectName is ENTITYREGULATORYEXTRACT.
Added support of this type to XSD, Generic Loader and Generic Extractor.
Created inbound and outbound rules. Added NG support in the inbound rule.

This issue was tracked as Item #156147.

Created a New EntitySecurityLending Object and ENTITYSECLENDINGEXTRACT Extract
An interface for the ENTITY_SEC_LENDING table was requested.
The new XSD object, Inbound and Outbound rules created.  The ObjectName is ENTITYSECLENDINGEXTRACT.

This issue was tracked as Item #156148.

Created the Entity Collateral Summary Object and ENTITY_COLLATERAL_SUMM Extract.
Enabled the load and extract of the data from/to database table ENTITY_COLLATERAL_SUMM should be supported by EagleML Core.
Added new EagleML type: EntityCollateralSumm.  The ObjectName is ENTITY_COLLATERAL_SUMM.
Added support of this type to XSD, Generic Loader and Generic Extractor.
Created inbound and outbound rules. Added NG support in the inbound rule. 

This issue was tracked as Item #156149.

Created the EntityStatistic Object and ENTITYSTATISTICEXTRACT Extract
A new interface for the ENTITY_STATISTICS table was created.
Added the EntityStatistic XSD type, inbound and outbound rules.  The ObjectName is ENTITYSTATISTICEXTRACT.

This issue was tracked as Item #156150. 

Removed the Timestamp from File Names to Handle Long Names
Long file name were causing issue when moving across destination paths.
Added logic to delete the TimeStamp from the file name before moving the file.

This issue was tracked as Item #156170.

Adjusted Possibility to Populate Position Detail Analytics Table
As part of the bug 153500, the parameter in_pos_det_anl_switch was introduced in the EGL_GEN_HOLDING package to the POS_EGL_GEN_INTERFACE procedure. This parameter must be included to populate Analytics table with a value of ALL or ANL.

The logic was updated to work as expected.  The file wrh_egl_gen_interface3.xsl was changed.
Since defect 153500 fix was implemented only on 2015R2, this parameter and logic work only on 2015R2 environments.

This issue was tracked as Item #156297. 

Created the Entity Analytics Object and ENTITYANALYTICEXTRACT extract.
The loading and extract data from/to DB tables ENTITY_ANALYTICS and ENTITY_PEER_GRP_ANALYTICS should be supported by EagleML Core.

Added new EagleML type: EntityAnalytics.  The ObjectName is ENTITYANALYTICEXTRACT.
Added support for this type to XSD, Generic Loader and Generic Extractor.
Created inbound and outbound rules. Added NG support in the inbound rule.

This issue was tracked as Item #156317.

EJM

Added the Feature to Retrieve DBName from the db_connection.ini
Added the possibility to retrieve the DBName from the db_connection.ini in case the primary environment is down. For example, to make a DR environment work instead of a PROD environment.

Logic was added to retrieve the dbname from db_connection.ini

This issue was tracked as Item #155581.

I2I

Confirmed Support of performdbo.perf_sec_universe in the Warehouse Performance
performdbo.perf_sec_universe should be updated in I2I process.
No change was required. performdbo.perf_sec_universe is already supported by Warehouse Performance.

This issue was tracked as Item #154856.

Updated Logic for the Delete Cash Records on the Spoke for I2I
Updated logic for the Delete Cash records on the Spoke for I2I.
If the Effective Date was updated in the Cash Table on the HUB, orphaned records appeared on the SPOKE. As a result, the HUB and SPOKE were not in sync.
Refined the logic and added delete statement for Original Transaction Number because this field is unique.

This issue was tracked as Item #155019.

Updated Logic for Entity Fields on the Extract Side for I2I
1. Extracting data from different sources (environments) and loading them into different partitions was requested.
2. Using Entity Resolution logic with Entity Xref map was requested.
Changed the load for the Generic Entity interface and added a new include file named entity_partition_mapping.inc
This logic is similar to Source Mapping. Created configuration in which Partition should be assigned Entities from specific source(HUB) on the LOAD side

For example:
<COL TAG="W_ENTITY_PARTITION_MAPPING" EXPRESSION="'HUBID|TESTENV_1|PARTITION|1|\n
HUBID|TESTENV_2|PARTITION|2|OVERWRITE|Y|\n
FROMSOURCENAME|TESTSOURCE_1|PARTITION|1|OVERWRITE|N|\n
FROMSOURCENAME|TESTSOURCE_3|PARTITION|2|OVERWRITE|Y|\n
'"/>

For the entity resolution logic a new RTR parameter was introduced:
<taskParameter>
 <name>EntityXrefMapOption</name>
 <dataType>S</dataType>
 <value>XREF4,XREF4_@SentBy,XREF2,XREF2_NEW</value>
</taskParameter>
 
How it works:
Specify the initial value and the replacement value and split these values using a comma.  Add mnemonic logic, for now only one mnemonic @SentBy is supported, if this string is used, it will be replaced by HUBID.  Add some options for Entity common fields

• AccountXrefTypeSelection – this parameter determines which TYPE will be in the root node, works only for streaming extracts
<taskParameter>
 <name>AccountXrefTypeSelection</name>
 <dataType>S</dataType>
 <value>XREF3</value>
</taskParameter>
 
• XrefAccountTypeOutput  - in this parameter Entity Account ID type can be selected for extraction. As a result, some Entity Xref Types can be excluded (just do not specify them).
<taskParameter>
 <name>XrefAccountTypeOutput</name>
 <dataType>S</dataType>
 <value>XREF4,XREF2</value>
</taskParameter>

• SetEntityResolutionOptionValue - populates the entityResolutionOption field in the extract file (available only for Entity Extract)
<taskParameter>
  <name>SetEntityResolutionOptionValue</name>
  <dataType>S</dataType>
  <value>USE_ENTITY_ID_AS_XREF,PERFML_HUB141</value>
</taskParameter>

• SetEntityBattingOrderValue - populates entityBattingOrder field in the extract file (available only for Entity Extract)
<taskParameter>
  <name>SetEntityBattingOrderValue</name>
  <dataType>S</dataType>
  <value>PERFML_HUB141,PERFML_HUB142,PERFML_HUB143,PERFML_HUB144</value>
</taskParameter>

• SetXrefMapOptionValue - populates the entityXrefMap field in the extract file (available only for Entity Extract)
<taskParameter>
  <name>SetXrefMapOptionValue</name>
  <dataType>S</dataType>
  <value>SetXrefMapOptionValue, XREW0-1,XREF_2,odfo,3rs</value>
</taskParameter>

• SetLoadXrefOptionsValue - populates the loadXrefOptions field in the extract file (available only for Entity Extract)
<taskParameter>
  <name>SetLoadXrefOptionsValue</name>
  <dataType>S</dataType>
  <value>SetLoadXrefOptionsValue,TESt1,TEst2,Test3</value>
</taskParameter>

This issue was tracked as Item #155639.

Updated the Logic for Security Resolution for I2I
Ability to use only one specific identifier for Security Resolution for Multi hub case was requested.
New Security Resolution option , USE_ONLY_SEC_ALIAS_AS_XREF, was added.
When <assetResolutionOption>USE_ONLY_SEC_ALIAS_AS_XREF</assetResolutionOption> only Security Alias will be used as identifier and if resolution cannot find a Security, new Security will be created.

In case of I2I the following option in the RTR for I2I workflow can be added:
 <taskParameter>
  <name>SetAssetResolutionOptionValue</name>
  <dataType>S</dataType>
  <value>USE_ONLY_SEC_ALIAS_AS_XREF</value>
 </taskParameter>

so, result files will have
<assetResolutionOption>USE_ONLY_SEC_ALIAS_AS_XREF</assetResolutionOption>

This issue was tracked as Item #155914.

Added the Feature to Use xreftype Based on HUB Mnemonic
Using different values for autogenerated xreftype in the multi hub I2I workflow for SecurityAlias value was requested.

Logic for the TypeSecurityAliasAsXREF parameter was updated.
Substring @SentBy can be used - it will be replaced by HUBID(Environment name) value from Data Base.
  <taskParameter>
   <name>ALLEXTRACTS_AddSecurityAliasAsXREF</name>
   <dataType>S</dataType>
   <value>Y</value>
  </taskParameter>
  <taskParameter>
   <name>ALLEXTRACTS_TypeSecurityAliasAsXREF</name>
   <dataType>S</dataType>
   <value>@SentBy_SECALIAS</value>
  </taskParameter>

This issue was tracked as Item #155915.

Enhanced All Data Loads for the Warehouse and Accounting Data to Have the Ability to Resolve the Entity Via a Specified XREF
Current data loaders for Accounting Data used only the first entity cross reference in the incoming data to resolve the entity. Accounting data loaders were enhanced to support entity resolution by all xref. For compatibility purposes, the support of entity resolution by all xref is switched off by default in the Accounting and Warehouse interfaces. It may be enabled by setting PERFORM_FULL_ENTITY_RES.

This parameter can be defined in the incoming message in the <entityResolutionOption>PERFORM_FULL_ENTITY_RES</entityResolutionOption> node or as a global parameter W_PERFORM_FULL_ENTITY_RES = ‘Y’ in the w_config_custom.inc.

Also as a part of this defect the following enhancements were implemented:
1. Added support for the entity xref map. It allows to remap incoming entity xref types into other configured types. Entity xref map contains pairs source_type_1,targert_type_1,source_type_2,target_type_2, …
This setting can be defined in the incoming message in the <entityXrefMap> source_type_1,targert_type_1,source_type_2,target_type_2</entityXrefMap> node or as a global parameter W_ENT_XREF_MAP= ‘source_type_1,targert_type_1,source_type_2,target_type_2 in w_config_custom.inc.

2. Added support for entity load xref options. It allows to decide which incoming entity xref types will be inserted/updated. Entity load xref options works in the same way as security xref options and contain a list of xref types for update.
This setting can be defined in the incoming message in <loadXrefOptions> type1,type2</loadXrefOptions> node or as a global parameter W_ENT_LOAD_XREF_OPTIONS = ‘type1,type2’ in w_config_custom.inc.
3. Introduced global parameters for entity Batting Order, entity Resolution Option, entity Xref Map and Entity Load Xref Options. Appropriate global parameters are:
• W_ENT_BATTING_ORDER
• W_ENT_RES_OPTION
• W_ENT_XREF_MAP
• W_ENT_LOAD_XREF_OPTIONS
They can be set in w_config_custom.inc.
4. Added the RESOLVE_ENTITY_BY_XREFS option. It switches on entity xref resolution even if the entity id is defined. This setting can be defined in the incoming message in the <entityResolutionOption>RESOLVE_ENTITY_BY_XREFS</entityResolutionOption> node or as a global parameter W_ENT_RES_OPTION = ‘RESOLVE_ENTITY_BY_XREFS’ in the w_config_custom.inc.
5. Added the SKIP_INCOMING_ENTITY_ID option. It skips the incoming entity id value and to allow to generate the entity id if it was not resolved. This setting can be define in the incoming message in the <entityResolutionOption> SKIP_INCOMING_ENTITY_ID </entityResolutionOption> node or as a global parameter W_ENT_RES_OPTION = ‘SKIP_INCOMING_ENTITY_ID’ in w_config_custom.inc.

This issue was tracked as Item #155944.

Software Fixes

EagleML

Added “.pgp” or “.gpg” Extension Applied for Encrypted TSR Files
Issue with the encrypted TSR extension: “TSR files though encrypted do not have the “.pgp” or “.gpg” extension at all”.

The following file was patched for the fix: reporter_dist_methods.inc.

This issue was tracked as Item #144744.

Fixed the Security Xreference Identifiers Load when <loadXrefOptions> Element is Specified
Two issues were reported:
1. If <loadXrefOptions> contains LP_ID identifier (specified in xId1) and != 'ALL' then LP_ID is not loaded
2. Ticker is loaded though not specified in <loadXrefOptions>

The first issue was already fixed in XSLT transformation for Issuer Xreference identifiers
The second issue was not fixed because the Ticker is loaded by Core stored procedure INSERT_UPDATE_SECURITY_INFO. The Ticker is loaded to SECURITY_MASTER DB table and it can't be removed.

This issue was tracked as Item #154909. 

Created the cashSegregations interface and New Tags Usage for Entity Bank Relationship was Added  
1. Missing tag7001 and tag2283 mappings for Entity Bank Relationship.
2. New objects Cash Account and Cash Account Relationship should be created on enviroments with 2017 version only.

1. In xml-ent_entitybankrelationship.xml mappings for tag7001 and tag2283  were added.
2. Created panel overlay pan-addentbankrel.custom.xml.
3. Created rules for new objects.
4. New Entity based objects descriptions were added in XSD.

This issue was tracked as Item #155050.

Updated the Logic for Generic Entity Load
The GenericEntity EagleML load process is creating duplicate Entities in the DB when the file has multiple records for the same Entity, but with different effective dates.

If parallel loading is enabled and a file with multiple records for the same Entity is sent, duplicate entities are created.

1. Split rules were added for the generic entity to handle processing of multiple records for the same entity sequentially.
2. Changed the logic for the load of entities with the same xref. Now the load of the entity and xrefAccountIds occurs in one transaction, and if at least one xrefAccountIds fails, the entity rolls back.
3. The logic for resolving by xrefAccountId was added in the deletion rule for generic entity.

This issue was tracked as Item #155557.

Corrected the Mapping for Schedule Model
The element 1250 (scheduleType) is mapped to scheduleType in the table and 3678 (scheduleSubType) is mapped to scheduleSubType in the table. According to the panel, 1250 is schedule type on the panel but is mapped to schedule_sub_type in the table and 3678 is schedule sub type in the panel but is mapped to schedule_type in the table.

These mappings were changed.

This issue was tracked as Item #155560. 

Added the Substring WRHS for the Feed Type of Insurance Valuation
The INSURANCEVALUATIONLOTEXTRACT is a Warehouse object that does not have the substring WRHS in the Feed Type. Within the generic Warehouse extract code/includes there is logic that is looking at the Feed Type and searching for the substring 'WRHS'.  As such, the logic is failing to be applied to the Insurance Extracts. 

Searching 'INSURANCEVALUATIONLOTEXTRACT' string was added in some rules where logic was checking for the 'WRHS' prefix.
Example of revised code:
iif(Find(:varFeedType:,'WRHS') >= 0 or :varFeedType: = 'INSURANCEVALUATIONLOTEXTRACT', ...

This issue was tracked as Item #155562. 

Prior Fields Returned as Zero When Run the Day After a Non-Business Day
Prior fields (PRIOR_MILL_RATE, etc.) are defined within the existing stored procedure with the parameter value: ACCOUNTING_DATE-1.
If the prior day is a non-business day, the stored procedure returns 0. It occurs for ACCOUNTING_DATE = MONDAY for example.

The max prior date is resolved in the table ‘ESTAR.ESTAR_DIST_LOG’ instead of using prior date (as ACCOUNTING_DATE - 1).

This issue was tracked as Item #155578. 

Filtering Parameters are Missing in the Unsettled Transaction Extract
The parameter 'ReportView' (tag 7009) sends the value into the stored procedure parameter 'in_rpt_view', but the value of the 'ReportView' parameter is prepared by the value of the 'TransactionType' parameter and is not present in the RTR to receive custom values.

Support of the 'ReportView' parameter was added into RTR to receive the values of A, T, P or M.

This issue was tracked as Item #156335.

WRHSOPENLOTEXTRACT and WRHSPOSITIONEXTRACT with Extended Portfolio Valuation Model has an Invalid Output
WRHSOPENLOTEXTRACT and WRHSPOSITIONEXTRACT with Extended Portfolio Valuation Model had an Invalid Output which required an additional join on the Security Alias with the temporary table for the portfolio valuation.

This issue was tracked as Item #156926.

EJM

W_USE_STREAM_FAILURE_TOLERANCE Usage for More than 1000 Errors
The USE_STREAM_FAILURE_TOLERANCE did not work in case of 1000 or more errors.

1. USE_STREAM_FAILURE_TOLERANCE parameter was renamed to W_USE_STREAM_FAILURE_TOLERANCE.  The USE_STREAM_FAILURE_TOLERANCE is still supported for compatibility
2. The “task_reporter_generate_task_unit.inc” was added to process cases with more than 1000 errors

This issue was tracked as Item #156431.

I2I

Updated logic for the Underlying SMF Resolution when the W_USE_SEC_ALIAS_AS_XREF parameter is set to YES
Security Resolution was resolving the incorrect SecurityAlias for Underlying SMF when the W_USE_SEC_ALIAS_AS_XREF parameter set to YES.
Parameter is set W_USE_SEC_ALIAS_AS_XREF=YES in w_config_custom.inc.
In security_resolution_generic.inc, the logic was updated when the :UseSecAliasAsXREF: = 'YES'.

This issue was tracked as Item #155414. 

Added Passing OnlyHeld Filter to Issuer Organization Extract in Generic Extract Workflow
The OnlyHeld filter value was not passed to the Issuer Organization Extract step in Generic Extract workflow and as a result, the user extracted a lot of unnecessary data, because the OnlyHeld filter was not applied.

This was identified when running I2I workflow in Held mode for Issuer Organization FeedType.
Logic was updated to add the OnlyHeld parameter in the Generic Extract workflow for Issuer Organization.

This issue was tracked as Item #155561.