EagleML Release Notes - June 2016

This release uses XSD Revision 225.

There are four overlay files that are currently packaged as part of the EagleML monthly release. Please refer to About EagleML Overlays for specific details and impact to you.

Enhancements

EagleML

Enabled Streaming for Voluntary CA Election

Streaming support was added for extracting Voluntary CA Elections. There were problems with large volume extracts for Voluntary CA Election data. The extracts could time out and not complete. A new streaming rule was created to allow for extracting large amounts of Voluntary CA Election data quickly. You can enable streaming for this extract in the w_config_custom.inc. In the CODE node of this file you would need to include VOLCAELECTIONEXTRACT.
This issue was tracked as Item #133686.


Introduced Security Type Node in Warehouse Trade Interface

The Warehouse Trade XSD was enhanced to include a <securityType> node. The new SecurityType.model group was created with this change.
This issue was tracked as Item #137316.


New Alternative to Passing SwapType in Security_resolution Includes

Previously, the Swap SMF resolution required the passing of a swapType as a parameter in the incoming message. The logic was enhanced to allow populating the swapType through a variable that could be obtained via the "secdetail" lookup. To accomplish this task the "secdetail" lookup as well as the security_resolution_full.inc and the security_resolution_generic.inc includes had additional logic in the form of new mappings to assign the swapType based on the new :swaptype: variable.
This issue was tracked as Item #137754.


Added Trial Balance

There was no offering within the EagleML product which allowed you to extract the trial balance report summed at a ledger account level. Therefore, you would need to sum the security level balances from the WRHSGLDETAILPOSTINGEXTRACT to obtain this level of information. To address this need, a new object trialBalance with complex type TrialBalance and group TrialBalance.model has been added to the schema.
This issue was tracked as Item #138273.


Added HeldOnly Filter

The HeldOnly filter was added to five EagleML interfaces, including GenericCA, IssuerOrganization, Schedule, Ratings and Issuer. Extracts for all these interfaces now allow filtering data via the heldOnly filter. The extract_generic_issuer.xml rule was changed to allow for the heldOnly filter to be passed in as a task parameter that would be recognized by these interfaces.
This issue was tracked as Item #138603.


New Accounting Distribution Activity Extract

EagleML was enhanced to include a new Accounting Distribution Activity Extract. This extract pulls information from the ESTAR_DIST_LOG and ESTAR_NAV tables. Please note that currently the Accounting Distribution Activity is an outbound interface only. A new model group, object type and new outbound rules were created to support this enhancement.
This issue was tracked as Item #138996.


Fixed Underlying Record Resolution

In the include file underlyingsmf_model.inc, the DB lookup SEC_UNDERLYING_LOOKUP would fool the logic for security resolution and attempt to load a shell record that already existed in the database. This issue would only happen in the case that the underlying record was located in the securitydbo.security_master_hist table instead of securitydbo.security_master. In this case tag 10, the security alias, would have a value but since the SEC_UNDERLYING_LOOKUP would only check the security_master table it would return no data. This would cause the rule to attempt loading the shell record despite there already being one. To avoid this problem with the location of the underlying record in the database the logic of the include file was changed to disallow loading of the shell record if tag 10 was non empty. A non empty tag signifies an existing underlying and therefore shell record and therefore is a sufficient condition for the security resolution.
This issue was tracked as Item #140192.


Custom Security Resolution

Using the parameter var_mid_customsec_res in the w_config_custom file, you can switch from security_resolution_full.inc to the security_resolution_custom.inc include file.The same ability was not available for security_resolution_generic.inc which is used in the include underlying_smf_ins_upd.inc. The code was enhanced in the security_resolution_generic.inc file to make it dependent on the var_mid_customsec_res parameter found in the w_config_custom include file.
This issue was tracked as Item #140286.


Expanded Security Resolution to Use Xreferences to Resolve Swap Securities

When loading swaps sometimes there could be problems with security resolution in the primary asset Id/type of the underlying shell security and the contract leg would not always match. The mismatched primary asset Id/type would not allow the swap security to resolve correctly. To resolve this issue, a W_ALWAYS_USE_XREF parameter was added to the w_config_custom.inc file. This will allow resolution to be executed based on the xreferences rather than the primary asset Id/type. This functionality will only be used if the security type begins with "sw".
This issue was tracked as Item #140337.


Added Ignore Message When no GenericSMF node in File

There was a need for an ignore message in eagleml_smf_schedule_delete.xsl in case there was no genericSMF node in file. The eagleml_smf_schedule_delete.xsl was updated in order to create ignore messages if an incoming file has an ignore node.
This issue was tracked as Item #140344.


Added Table to FLACTIVITYEXTRACT

There was a request to add the ESTAR.ESTAR_LEDGER_NOTES table to the existing "extract_acct_gl_activity.xml" model. This makes data from the table extractable. Elements ledgerNotes\notes and ledgerNotes\notesAlloc have now been added to XSD type GeneralLedgerActivity model LedgerPostModel.model at the end of LedgerPost.model. The database table ESTAR.ESTAR_LEDGER_NOTES has been added to the list of tables for extract in General Ledger Activity.
This issue was tracked as Item #140700.


Additional Date Parameters for WRHSCLOSELOTEXTRACT, TRADEDCAEXTRACT, SETTLEDCAEXTRACT, SUBSCRIBEREDEEEMEXTRACT

WRHSCLOSELOTEXTRACT, TRADEDCAEXTRACT, SETTLEDCAEXTRACT, SUBSCRIBEREDEEEMEXTRACT did not have the same capability to filter extracts by various dates as the WarehouseTrade object. The code was updated and you can now filter by ledgerEffectiveDate, accountingDate, monthEndAccountingDate, postDate, tradeDate and settlementDate. Logic for the outbound rules of the WarehouseCloseLot, TradedCA, SettledCA and SubscribedEM objects was changed to allow for this additional filter. Below is an example of one of the new data parameters that can be used in control messages:

<taskParameter>
<name>postdate</name>
<dataType>S</dataType>
<value>2016-03-30</value>
</taskParameter>

This issue was tracked as Item #141271.


Added Warehouse Date Parameters to WarehousePosition, WarehouseOpenLot, WarehouseGLBalances and WarehouseCashActivity

The WarehousePosition, WarehouseOpenLot, WarehouseGLBalnaces and WarehouseCashActivity objects did not have the same capability to filter extracts by various dates as the WarehouseTrade object. The code was updated and you can now filter by ledgerEffectiveDate, accountingDate, monthEndAccountingDate, postDate, tradeDate and settlementDate. Logic for the outbound rules of the WarehousePosition, WarehouseOpenLot, WarehouseGLBlanaces and WarehouseCashActivity objects was changed to allow for this additional filter. Below is an example of one of the new parameters that can be used in control messages:

<taskParameter>
<name>postdate</name>
<dataType>S</dataType>
<value>2016-03-30</value>
</taskParameter>

This issue was tracked as Item #141273.


Added OnlyHeld Feature to Specified Extracts

Only Held will include securities with a non zero balance. The OnlyHeld functionality was added to the following extracts:

  • GENCORPACTIONS
  • ISSUERORG
  • SCHEDULE
  • RATINGS
  • GENISSUER
  • FACTOR

This issue was tracked as Item #141626.

EJM

Leveraging of Date rule for RTR date parameters

The existing accounting date parameter required you to manually edit the date when sending the same RTR. For processes that are launched every day and have a set cutoff, for example the past seven days, the value in the data parameter would have to be edited every day the process is launched. The date rule uses a calendar input to calculate a date based on certain instructions that are inherent to this rule. For example it can return either the prior business date when given such directions. The calculations are based on the calendar used and the date used as the basis of the calculations is selected by default based on the given calendar. There is also an option to change this basis date by passing the AS_OF_DATE parameter which will allow the date rule to use that date as the basis of calculations instead. Finally it is possible to still use the old date parameters as before. An example of how the date rule parameter and the as_of_date parameter are given below

<taskParameter>
<name>valuationdaterule</name>
<dataType>S</dataType>
<value>Prior Business Date</value>
</taskParameter>

<taskParameter>
<name>valuationdate</name>
<dataType>S</dataType>
<value>2016-05-24</value>
</taskParameter>

Note: To use the second parameter as the AS_OF_DATE both parameters must be passed in the RTR. If only the first parameter is included the AS_OF_DATE will be null. If only the second parameter is passed the regular date parameter will be used and the date rule will not be invoked.
This issue was tracked as Item #141022.

I2I

Added Source Mapping

Source mapping was unavailable for I2I tools. This feature is important in loading history records to a SPOKE source that is different than the HUB source. To accommodate capabilities of choosing the source mapping, the I2I tool source code was updated to allow the use of source mapping parameters by specifying them in the UI.
This issue was tracked as Item #138686.


Delta Mode Extract Now Supports More than One Last Run Time

Previously, the delta mode extracts would save only one last run time. This would create potential issues if two users were to run the same extract in delta mode. For example, the first user could run the extract expecting that the next extract would include all data that was loaded since their first run. Another user could then run a delta extract grabbing data that was loaded since the first user's extract thus making that data unavailable for the first user. To resolve this issue, a Database task parameter was added to the I2I platform which enables users to pass a unique value that will bookmark their specific extracts and therefore save their last run time based on the value provided in this parameter. This will allow users to launch delta based extracts based on their particular last run. If the parameter is omitted the delta based extract will use the over all last run time, otherwise it will check for the last run time that was related to the value in the DeltaBasedUID parameter.
This issue was tracked as Item #138728.


Added Ability to Exclude Empty Fields From I2I Extract

A parameter was added to allow you to prevent I2I extracts from returning fields with a value of 0. With this parameter, if an extract field has an 0 value, that field will not be included in the extract. Since the value 0 has significance in some accounting fields, by default the exclude zero parameter is now off. To leverage this feature, add the parameter of excludezero with a value of Y in your control message.Here is an example from a control message:

<taskParameter>
<name>excludezero</name>
<dataType>S</dataType>
<value>Y</value>
</taskParameter>

This issue was tracked as Item #135038.


Issue with Profiles in Extractor Workflow

The Generic Extractor workflow was not capable of sending extract profiles on the I2I platform by individual feed type and therefore not able to utilize the element reduction functionality. The logic for the generic extractor workflow would cause extracts to fail if a profile was not found for any feed. This would then cause issues if you wanted to conduct a mass extract but did not have functional profiles for each feed type. The code was updated to address this situations. A new parameter, QueryProfileName, was added for all Extracts in Generic Workflow and logic in include extract_profile_query was changed.
This issue was tracked as Item #140004.

Software Fixes

EagleML

EagleML Redemption Currency

There was an issue with tag 1530, the redemption_currency field, not getting populated in the database. The problem pertained to the forward SELL legs of Generic SMF. Having only one of the two forward legs contain the redemption currency data is not consistent. The forwardsmf_model.inc include file was edited to include logic for the SELL leg to include the redemptionCurrency element. The updated code will fill in the redemption currency field depending on the leg of the forward. For the SELL leg, the redemption currency will be filled with the value from the buy currency while for the BUY leg, the redemption currency is mapped from the sell currency.
This issue was tracked as Item #131734.


Filtering by Source

Eagle ML extract did not allow filtering of data via a source parameter. The code was updated to resolve this. The include file "../../extract_filter_sources.inc" was added to the XSD to allow for filtering by the Source parameter. It is now permissible to add the following parameters into RTR control messages to filter by source:

<taskParameter>
<name>datasources</name>
<dataType>S</dataType>
<value>STARDIRECT</value>

This issue was tracked as Item #137638.


Missing Issue Variable Rate Rule Mappings

Several issue variable rate elements were not loaded to the database. In addition, there was a problem resolving the variableResetCode field. The missing element values in the database were caused by missed mappings in the xml-ref_issuevarrate.xml rule. These were fixed by adding the relevant mappings. The absence of the value in the variableResetCode field was the result of the stored procedure lookup not properly working. This lookup was changed to map the value of this field correctly.
This issue was tracked as Item #138615.


Fixed Missing Fields in Multileg SMF extracts

There were several critical fields missing from the Multileg swap SMF extract. These fields are supposed to be in the extracts since they exist in the database and had values loaded through an inbound stream. Any data that is loaded into the database should ideally be available for extracts. The SMF outbound rule was updated with the necessary mappings to allow missed elements to show up in extracts. Additionally two elements were not included in the outbound rule at all and logic for these elements was added as part of this fix.
This issue was tracked as Item #138798.


New elements in the impairments interface

Several elements were added to the impairments interface. Cost Offset Account, Recalculate Yield, Transfer Method and Maturity Price Override fields were all added to the impairments interface.
This issue was tracked as Item #139058.


Incorrect Value in tag 1256 Affected Look up Execution in Entity Bank Relationship Interface

There was an issue with the ENT_CUST_REL_QUERY stored procedure. It is responsible for performing a database lookup to decide whether a Entity Bank Relationship record should be passed to the DB as a new record or sent for updating was not working properly. There were scenarios where the lookup would give the go ahead for the bank relationship record to be loaded despite there not being an entity with an Id that matched the incoming file's entity Id. There was a problem with tag 1256 having a value that did not allow this lookup to be executed properly. For the entity Id clause to be included in the SQL, the value of tag 1256 must be "EXACT'. Under other values that part of the SQL will be skipped, therefore not checking whether that entity exists in the database before trying to load the bank relationship record.
This issue was tracked as Item #139317.


Restricted Units Missing in Warehouse Custody Balance Extract
The Restricted Units element (tag 903) was not included in the Warehouse Custody Balance extracts. To resolve this issue, an additional output-to-XML node and DB-field mapping string were added to the code. The outbound warehouse custody balance t and streaming t rules were both edited to include mapping to the Units field. Now tag 903 to be present in all warehouse custody balance extracts. 
This issue was tracked as Item #139437.


Changes to the Default Workflow Eagle_wrf_generic_purge

Changes were needed for default workflow eagle_wrf_generic_purge. The name of the workflow was changed to eagle_wrf_generic_purge_archive to more properly reflect its purpose. Several specification changes were made such as the addition of the ability to clear the local cmw/in folder as well as sending TSR messages to the cmw/outbound folder rather than the cmw/reply folder as was the case before. There was an issue with timeout errors that was corrected by extending the timeout intervals for two subprocesses that would consistently create scenarios in which the entire workflow would time out. The ability to delete subfolders and maintain the folder structure when archiving all deleted files was created by adding the include_subfolders parameter usable in control messages. Finally there were compatibility issues resolved such as the visibility of the workflow in the EJM processes as well as inconsistency between the capabilities of the workflow and the information on the wiki.
This issue was tracked as Item #139830.


Elements enumerated in XSD

There were some elements that are used in the calculateValuesBySTARIndicator for trade processing that were not reflected in the EagleML XSD. As a result of the omission of these elements the EagleML schema is not consistent with the Eagle system. The elements that are used in panels but were not included in the XSD were traced and added to the XSD enumeration into the eagleml-enum-2-0.xsd file.
This issue was tracked as Item #139944.


Issues with CPA Resolution

The CPA resolution in the ref_generic_ca rule did not resolve the same way as ref_ca and ref_cancelca. The ref_generic_ca rule incorrectly mapped from tag 60 whereas it should have mapped from tag 1965. Because of this issue, you were not able to map tags correctly in a change_ca lookup. The INMAP was changed in the change_ca lookup so that the correct tag would be used to populate the CPA field.
This issue was tracked as Item #140079.


Could Not Extract All Entity Lists that An Entity Id

There was a need to extract all entity lists that contain a specified entity Id. A new interface was added to allow for this functionality. New EntityInEntityLists interface with extract capabilities. A new interface, object type and corresponding rules were added to the EagleML library.
This issue was tracked as Item #140148.


Issue with Replace Condition in ng_exec_sp.inc h

The stored procedure in ng_exec_sp.inc had logic that was responsible for manipulating an incoming parameter into a different format which would sometimes cause the length of the parameter to exceed the limit for the database field. The original value would be compliant with this length limitation, but after processing by the stored procedure it would become too long. The code that was responsible for the parameter manipulation was removed so that the parameter could be passes as is.
This issue was tracked as Item #140844.


Issue with Like Criteria in Security Resolution Logic

The criteria in the logic for sec_resolution_input in security_resolution_full.inc did not resolve all identifiers correctly. This created problems with certain identifiers that were similar enough to each other by name to allow for the resolution logic to use both of them to attempt finding a security. Since the search was not meant to be done with more than one of these identifiers, the resolution will not find the correct records. This issue particularly pertains to CUSIP and BCUSIP both of which the like criteria logic would pick up as required parameters. The logic was changed to exact matching so that no further confusion could occur and all identifiers would have to be matched exactly to be considered for resolution purposes.
This issue was tracked as Item #140872.


Performance Optimization on Warehouse Open Lot Extract

The query for extracting warehouse open lot objects was not optimized. In addition, the query was using LOT_LEVEL_POSITION.effective_date instead of the POSITION.EFFECTIVE_DATE. The query is parsed several times which slows down the performance. The indexes were replaced with bind parameters which allow the filters to be parsed once and therefore maximize efficiency of the query. Additionally the effective date filter was edited to use the correct database field.
This issue was tracked as Item #140941.


Refined Query for Source Interfaces

The source filter in source interfaces used literal values for parameters. This would cause the query to be parsed more than once. To resolve this issue, bind parameters were added instead of the literal values in the source filter for DB extract queries.
This issue was tracked as Item #140956.


Warehouse Trade Extract Missing Nodes

The nodes costLocal and costBase were missing from the Warehouse Trade Extract. These nodes previously existed but were negatively impacted by the May 2015 release. As a result, your extract would not include this data. The code was updated to resolve this issue. The q rule was already picking up on the relevant database fields and therefore all that was necessary to restore these two elements in extracts was the addition of a message mapping in the _t and _streaming_t rules.
This issue was tracked as Item #141069.


Performance Issue with Extract streaming t

After the addition of streaming capabilites to the entity extract, slow performance was reported. The code was updated to address this issue. There was an RTRIM() expression in the query that was included in the rule that was affecting the performance. This part of the eagle_ml-2-0_cm\out\tagvalue\extract_entity.xml rule was edited to remove the RTRIM() which increased the extract speed considerably.
This issue was tracked as Item #141106.


Entity Node Missing in Benchmark Extract

When running a GenericEntity extract, the Entity detail section would not contain the entityType node. The code was updated to resolve this issue. The extract entity_detail.entity_type field was added as part of the extract. Entity details are now included in the benchmark extract as expected.
This issue was tracked as Item #141546.


Updates to W_Config for Streaming

Streaming t capabilities are only supported on certain version of the Eagle regions. In the w_config file that ships as part of core EagleML, the w_eagleml_streaming_t section specifies these versions. The BK and BW releases were not included here but should have been. As a result, you could not use streaming in those releases even though the functionality actually existed there. The w_config file was updated to include the additional releases.
This issue was tracked as Item #141753.


Warehouse Open Lot Extract Returned Duplicates

The Warehouse open lot extract could return duplicates. This issue was a result of the long/short indicator not being included in a join. To resolve the issue, the long/short indicator was added to the join statement to prevent duplicated records showing up in the extracts for open lots.
This issue was tracked as Item #141967.


Streamlined Task_reporter_generate_task_unit.inc File

Code for the TransactionId in task_reporter_generate_task_unit.inc was different from that in task_reporter_generate_task_unit_old.inc. These differences made it necessary to pass extra parameters to determine the TransactionId in the newer version of the rule. The code was changed to reflect the old version of the rule making it streamlined and removing the extra parameters that the user would have otherwise had to pass in.
This issue was tracked as Item #141984.


Timeout Issue For Loading Streams

There was an issue introduced in the May release that could cause timeout for loading streams. A change was made to the /tpe/dyanmic/msgcenter/eagle_ml-2-0_cm/waiting_processor.inc file to address this. This change was delivered as a May install amendment package. It is now officially part of the core EagleML release.
This issue was tracked as Item #141968.


File Name Incorrectly Changed Before Copying to cmw/in Folder
An enhancement introduced in the May release (#140759) negatively impacted some workflows by changing the name of the file before copying it into the cmw/in folder. The code was updated to correct this issue.
This issue was tracked as Item #142120.

I2I

UPDATE_SOURCE No Longer Passed From Hub to Spoke

The SOURCE_NAME and UPDATE_SOURCE were being copied over from the Hub to the Spoke in I2I processes. In the case that there were two users with the same name, one of which was using the hub and the other using the spoke, confusion could arise since it would be hard to track records by user. Furthermore, it would be better to have a standardized value for records sent via I2I to show that the record was updated by the I2I processing rather than a user. To resolve this issue, the source mapping was deleted from the referenceobject.inc file which was mapped the incoming UPDATE_SOURCE from the hub into the UPDATE_SOURCE in the resulting record in the spoke.
This issue was tracked as Item #140182.


I2I introduction of full extracts to match hub and spoke data contents

In a previous release there was a bug that dealt with the functionality to only extract cash securities (CRCRCR) if they were held by an account. In this particular item this scenario has to be ignored for a certain case in order to match up the hub and spoke for delta based extracts. The impact of the problem is that this will allow for matching of the hub and spoke in the case that they were not in tandem. Essentially when the data in the hub and spoke is not necessarily matched up a full extract should be run on both these platforms so that the delta based extracts that follow will be consistent with one another. The previous bug that dealt with the cash security extracts was resolved by creating a filter parameter called includecurrencyassets that would only allow the cash records that had a relevant holding account. For the full extracts that are run to match up data in the hub and spoke require that parameter to be set to "Y" whereas the default value is "N". This functionality was automated to allow the user to run the full extract without having to worry about setting the parameter manually.
This issue was tracked as Item #140183.


Resolved Position Loading Issue

The position loading returned the error:

(ERR) SECURITY FOR THE IDENTIFIER "" DOES NOT EXIST IN EAGLE

when SMF data is not available on the incoming record. There are valid business reasons for this scenario to exist. One example is when a position has been fully sold. Another example might be a test account without positions. To resolve this issue, the inbound warehouse trade rules were updated with new ignore logic for cases like this. If there is no SMF Id found then the record is loaded, rather than failing with an error the deletion process takes place as expected.
This issue was tracked as Item #140592.


Issue with Entity Cross Reference Resolution in Warehouse Open Lot and Positions Extracts

The delta based batch mode extracts in Instance to Instance processing were not returning the entity cross reference. This caused the load side to attempt resolution for the entity based on the source side's entity cross reference, which resulted in the process failing. To resolve this issue, the logic was updated to disallow streaming t capabilities if the format was XMLDBAPI for warehouse position and open lot extracts.
This issue was tracked as Item #141964.


Issue with Selecting EagleML instead of XMLDBAPI Format for Warehouse Position and Open Lot Extracts

The /tpe/dynamic/msgcenter/eagle_ml-2-0_cm/out/xml/t_eagleml-2-0.rsf rule would run into problems selecting the proper format for messages when running in I2I mode.The issue affected Warehouse Position and Open Lot extract specifically. This issue would cause extracts to fail. The conditional statement that was responsible for selecting which format should be applied for the warehouse position and open lot extracts was updated to resolve this issue.
This issue was tracked as Item #141969.