Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

This article explains how timestamps work with DeltaMode in I2I. 

Info

This information applies to the EagleML March 2017 release and higher.

With I2I run in delta mode, the prior run's timestamp is retrieved and cached.  When the query is complete, the timestamp being cached is saved for the next run.

Therefore, any records updated will be picked up as a part of the following run, thus ensuring that all data gets to Local PACE.

Note: As a result, some data extracted in the first run will also be pulled in the second run. However, at the time of loading into Spoke:

  • If loading warehouse data, existing data will be deleted based on the BATCH_KEY and the new data will re-inserted. This will prevent duplicate data going in.

  • If reference data, the UPDATES may occur twice. For ADD and DELETE type of entries, errors may be noticed on the Spoke when loading because the data is already present or deleted.
    Note:  Any run can miss some data being inserted into the table during the extract time but the situation is corrected by the next run.

  • In order to completely avoid this situation, it would be necessary to suspend any process that writes to the tables from where the I2I Extract pulls the data while the extract process is in progress so that a static set is provided to the query.

...

Processing Dates in Delta Mode

On the first step of this workflow Calculate_date is resolved. A record in the PACE_MASTERDBO.ORCH_REQUEST_DEF and PACE_MASTERDBO.ORCH_QUEUE tables table is created with the following value:

PACE_MASTERDBO.ORCH_REQUEST_DEF.CORRELATION_ID = 'GWDELTA_'+:DeltaBaseUID:

  • GWDELTA_ is the common  prefix for delta processes

  • :DeltaBaseUID: is a unique ID for the process

Fromdate value is  generated or extracted from the ORCH table

...

Code Block
languagexml
PARAMETER_NAME = :FeedType:+'_INSTANCE' | PARAMETER_VALUE = 202812041 
Info

This process is different for common extracts. For more information, see the Delta-Based Filter in the Available Filters section of EagleML and EJM wiki.

Orchestration can be switched off for delta mode by setting in w_config_custom.inc:

Code Block
languagexml
:W_DO_NOT_USE_ORCH_FOR_DELTA: = 'Y'

And old logic will be used. 

Expand
titleClick to read about Last Run Date logic for I2I (obsolete):

What is the Last Run Date?

The last run date is the value of the datetime stamp of the last launch of a certain I2I process. 

Note
Info
title

Despite a similar name, the delta last run time for I2I and a deltaBased filter for extracts are two different concepts.

The last run date value is populated from system date on the Spoke at the moment the TSR from the Hub has arrived (this way it understands that the extraction is complete (it does not matter whether any data was extracted or not)). The following format is applied:

Code Block
Convert( Today(), 'yyyy-mm-dd') +' '+Convert( CurrentTime(), 'hh-mm-ss' )

Where is the Last Run Date Stored?

EJM core takes the current date and time on the Spoke and writes it into CodeValue.  
You can check it in the List Code Value panel (/eagle/star/reference/pan-lcodeval.htm) as shown below:

title
Info
Note

The code category is EAGLE I2I DELTA.

Image RemovedImage Added

The Code Value is 'i2i_d' (const prefix) + SpokeID + I2IProcessUID (set in the related i2i solution in IWS).

As an example, the result of the lookup looks like this:

Image Modified
The Code Value Description (1177) is the value of the datetime stamp of last launch of the I2I process.

The datetime stamp is linked to the I2I workflow (specifically, to the I2I Process UID), and to the UID of the environment (the same as the Spoke ID). It is not related to any users. If two users ran the same I2I workflow at the same time, it would be the same situation as if one user runs the same I2I workflow twice at the same time.

I2I Process UID:

Info
titleNote

This IWS workflow is named eagle_i2i_delta_extract.

Image ModifiedNote
Info
title

Last run datetime is not on the record level. It is related to the I2I workflow Process UID and to the environment in general. In this example, if an SMF delta extract is done with a filter for one source at 1:00pm and then another SMF delta extract is done with a filter for another source at 2:00pm, the last run datetime will be updated twice: after the 1:00pm run and after 2:00pm run.

What if I Need to Change or Reset the Last Run Date?

To change or reset last run datetime stamp, you can use standard Change code value or Delete code value panels to do that.