Manage Element Reduction in Extracts via XPanels

This article discusses creating a tag reduction profile and using this profile to remove some elements from your extract.

About Element Reduction

One of the many benefits of EagleML is that it exposes all of the data elements within an object. During implementations, it may be beneficial to extract fewer elements than the default settings. To allow for this, a new configuration was introduced using the task parameter QueryProfileName to enable a reduction in the number of elements requested.

The approach is a column filter where a user may configure what will be included in the extract. The column names must abide by the OWNER.TABLE.COLUMN format. The name of the file and two task parameters in the control message, FeedType and QueryProfileName, work together to make this option possible. This functionality may be created using a text editor or preferably through the application using IWS and accessing the Open XPanels / Extract Element Reduction screens.

Using XPanels for Element Reduction in the Extract

On this page

1 About Element Reduction
2 Using XPanels for Element Reduction in the Extract

Creating a Profile of Selection

Choose the panel you are going to work with from the list of available panels. For example, Reference Calendar:

Go to the Extract element reduction tab:

The window is split into two parts: on the left you can tick/untick an element to add/remove it from the extract; on the right available selection profiles are displayed.
You can either choose an existing profile by clicking it or configure your own selection, click Add New Profile, enter a name and create your specific new profile:

Refresh the list if necessary:

To delete a profile click the icon next to it and confirm the action.

Extracting Entity Identifiers with Element Reduction

This applies since November 2016 (IWS version v1.0.1.711 and higher).

To extract Entity Identifiers, please make sure you have ticked such custom identifier groups as Xref Ids like this

Using the Profile to Extract Data

Element reduction option is enabled by defining the :QueryProfileName: parameter. The name of the profile should either be specified in the control message initiating the extract:

<taskParameter>
    <name>FeedType</name>
    <dataType>S</dataType>
    <value>[FeedType]</value>
</taskParameter>
<taskParameter>
    <name>QueryProfileName</name>
    <dataType>S</dataType>
    <value>[Name]</value>
</taskParameter>

or in {}w_config_custom.inc as a global environmental parameter:

<COL TAG="w_[FeedType]_profile" EXPRESSION="'[Name]'"/>

where [Name] is the name of your profile.
A feed type can have several profiles associated with it. Profiles are formatted as text files named as profile[feedtype]{}[Name].txt (for example, profile_GENERICISSUEANALYTICEXTRACT_test23.txt) with lists of comma-separated DB fields. These fields are consumed in the Q-rule for the extract in the /eagle_ml-2-0_cm/out/tagvalue/extract_sql_fields_app.inc include file before performing the SQL query to resolve the list of DB fields.

Here is an example of such file:

SECURITYDBO.SECURITY_ANALYTICS.CURRENCY,
SECURITYDBO.SECURITY_ANALYTICS.MOD_DURATION,
SECURITYDBO.SECURITY_ANALYTICS.DELTA,
SECURITYDBO.SECURITY_ANALYTICS.GAMMA,
SECURITYDBO.SECURITY_ANALYTICS.THETA,
SECURITYDBO.SECURITY_ANALYTICS.VEGA,
SECURITYDBO.SECURITY_ANALYTICS_EQUITY.BID,
SECURITYDBO.SECURITY_ANALYTICS_EQUITY.ASK,
SECURITYDBO.SECURITY_ANALYTICS_EQUITY.DEBT_CAP,
SECURITYDBO.SECURITY_ANALYTICS_FI.EFFECTIVE_MODIFIED_DURATION,
SECURITYDBO.SECURITY_ANALYTICS_FI.YIELD_TO_WORST_PUT,

These files are stored in the APP directory /estar/dynamic/msgcenter/profile/extract.

Sample Generic Entity Extract (ENTITYEXTRACT) with Element Reduction

RTR filename: RunTaskRequest_EXTRACT_ENTITY.xml

QueryProfileName task parameter in the RTR:

<taskParameter>
    <name>QueryProfileName</name>
    <dataType>S</dataType>
    <value>test10</value>
</taskParameter>

Query Profile filename: profile_ENTITYEXTRACT_test10.txt

RULESDBO.ENTITY.ENTITY_BUILD_FLAG,
RULESDBO.ENTITY.ENTITY_CODE,
RULESDBO.ENTITY.ENTITY_LONG_NAME,
RULESDBO.ENTITY.ENTITY_NAME,
RULESDBO.ENTITY.ENTITY_SUB_TYPE,
RULESDBO.ENTITY.ENTITY_TYPE,
RULESDBO.ENTITY_ACCT_BASIS.ACCT_PERIOD_FREQ,
RULESDBO.ENTITY_ACCT_BASIS.ACCT_PERIOD_WEEK_END_DAY,
RULESDBO.ENTITY_ACCT_BASIS.ACTION_FREQUENCY,

The RTR is sent to eagle_ml-2-0_default_cm_control_message stream.
Message details for eagle_ml-2-0_default_out_q stream:

The extracted record contains key fields specific for the feedtype and fields specified in the profile.

<entityTransaction>
    <header>
        <objectType>GenericEntity</objectType>
    </header>
    <genericEntity>
        <objectType>GenericEntity</objectType>
        <objectId>GenericEntity</objectId>
        <objectDescription>ENTITY</objectDescription>
        <updateTimestamp>2015-02-04T10:20:18-04:00</updateTimestamp>
        <updateSource>TRESKOT</updateSource>
        <entityId>BLTEST</entityId>
        <entityName>SMFUND</entityName>
        <entityType>PORT</entityType>
        <sourceName>EAGLE PACE</sourceName>
        <entityLongName>SMFUND</entityLongName>
        <entityRulesUniqueKey>
            <cashProcessRulesUniqueKey>
                <cashRuleName>DEFAULTBOTH1</cashRuleName>
            </cashProcessRulesUniqueKey>
            <monthEndDateRuleUniqueKey>
                <monthRuleDesc>LAST_CAL_DAY</monthRuleDesc>
            </monthEndDateRuleUniqueKey>
        </entityRulesUniqueKey>
    </genericEntity>
</entityTransaction>