Delta Based Filter

Owned by Natalie Gesin (Deactivated)
Last updated: Oct 25, 2024 by Kinga Huszno
3 min read

Make sure that Delta mode is supported by the extract you are going to use, see the Cross-Reference of Objects and Filters table.

DeltaBased Parameter

Default value is "N".

If this parameter is set to Y then only the data added or updated since the last event of extraction with DeltaBased=Y will be extracted. This way you can extract new records that have been recently added or modified.

Example: Delta Based Filter without any additional options

       <taskParameter>               <name>DeltaBased</name>               <dataType>S</dataType>               <value>Y</value>        </taskParameter>

DeltaBasedUID was added in June 2016 EagleML release

DeltaBasedUID Parameter

DeltaBasedUID parameter has sense only when DeltaBased = Y

This parameter has been designed to store an independent Delta mode state for each user. It works like this:

  1. You decide on an specific string UID you are going to use, say, mydelta1.

  2. Use it with the DeltaBasedUID parameter in the control message.

  3. Other users with other UIDs performing the same extract in Delta mode, will not interfere into your workflow (unless they specify your UID) and you will be getting added or modified records in a more predictable way.

Based on extract feed type AND DeltaBasedUID value, the process stores the “last run date” value and the last instance used for delta extract, so if you run two extracts with the following task parameters: FeedType=SMFEXTRACT, DeltaBased=Y, DeltaBasedUID=mydelta1 AND FeedType=SMFEXTRACT, DeltaBased=Y, DeltaBasedUID=mydelta2, then two separate “last run date” values will be stored: one for mydelta1 and the other for mydelta2.

DeltaBasedUID parameter can be used with common extracts as well as with the Generic Extract workflow (eagle_wrf_generic_extract). But Generic Extract has some specifics:

  • If Generic Extract is launched with DeltaBased=Y and profile without setting the DeltaBasedUID value, profile name will be used as DeltaBasedUID value.

  • You cannot use different DeltaBasedUIDs for different feedtypes within one control message for Generic Extract workflow. All extracts will use the same DeltaBasedUID.

Example: Delta Based Filter with Delta Based UID

       <taskParameter>               <name>DeltaBased</name>               <dataType>S</dataType>               <value>Y</value>        </taskParameter>        <taskParameter>               <name>DeltaBasedUID</name>               <dataType>S</dataType>               <value>mydelta1</value>        </taskParameter>

You can change the way your delta extract handles the last run date value by specifying the increaselastrundatetime task parameter in your RTR for a specific extract (higher priority) or by setting its global version in w_config_custom.inc

DeltaBased increaselastrundatetime Option

Valid values:

  • ALWAYS (default) - always rewrite the last run date value after delta-based extraction

  • NEVER - switches off rewriting last run date

  • ONLY_DATA_EXTRACTED - changes last run date only if some data has been extracted

Example: Delta Based Filter with increaselastrundatetime

       <taskParameter>               <name>DeltaBased</name>               <dataType>S</dataType>               <value>Y</value>        </taskParameter>        <taskParameter>               <name>increaselastrundatetime</name>               <dataType>S</dataType>               <value>ONLY_DATA_EXTRACTED</value>        </taskParameter>

UseInstance Parameter

Toggles instance usage by resolving vDeltaDBInstance - primary key of the DB table - in extract_delta_based_param.inc

       <taskParameter>               <name>UseInstance</name>               <dataType>S</dataType>               <value>Y</value>        </taskParameter>

Delta Timestamp Details

In the parameters below :FeedType: is the feedtype value without the EXTRACT suffix. For example, for SMFEXTRACT  :FeedType: = SMF.

When you run an extract in Delta mode, first of all records are created in PACE_MASTERDBO.ORCH_REQUEST_DEF table wtih 

CORRELATION_ID = 'GWDELTA_'+:DeltaBaseUID:+:FeedType:

  • GWDELTA_ - common prefix for delta process

  • :DeltaBaseUID: - unique identifier of the process

  • :FeedType: - feedType value without the EXTRACT suffix

Example of such Corr Id: GWDELTA_QA15R2LN_1_KK_002

If the extraction was successful, then the PACE_MASTERDBO.ORCH_REQUEST_PARAMS table will receive new values for delta info and the following fields will be updated or added:

ORCH_INSTANCE = PACE_MASTERDBO.ORCH_REQUEST_DEF.INSTANCE

PARAMETER_NAME = LAST_RUN_DATE             | PARAMETER_VALUE = 2017-02-02 01-01-01

PARAMETER_NAME = :FeedType:+'_INSTANCE' | PARAMETER_VALUE = 202812041

This process is different for Generic Extract and I2I Extracts, see the Eagle Supported IWS Solutions section in the IWS wiki for more details.