Warehouse Objects Structure

The diagram below illustrates the Warehouse Objects structure and the relationships between them.

Open

The following Warehouse objects are available:

On this page

• 1 Warehouse Objects Structure
  • 1.1 Common Elements
• 2 Stream Logic
  • 2.1 EagleML Processing Sequence
  • 2.2 CSV Processing Sequence
  • 2.3 Pipe-delimited Processing Sequence
• 3 Load Modes
  • 3.1 Batch Mode
    • 3.1.1 Batch Key
    • 3.1.2 Using Batch Error Mode
  • 3.2 Single Row UPSERT Mode
    • 3.2.1 Procedure Mode
  • 3.3 Grouped Single Mode
  • 3.4 Sub-Batch Load Mode

Common Elements

Stream Logic

EagleML Processing Sequence

All the Warehouse are loaded into DB via eagle_ml-2-0_default_in_xml_warehouse_preproc in two steps:

• XSLT transformation. XSLT rule 'eagle_default/in/xml/xslt/wrh_egl_gen_interface3.xsl' is used.

• Loading data into the database. The 'eagle_default/in/xml/xml-warehouse_v2.xml' rule is used (xml-wh_brokertradequote_v2.xml for Broker Trade Quotes).

Load Modes

Batch Mode

In batch mode multiple rows can be loaded into DB by one EagleML message. In this mode the warehouseTransaction element usually contains several complex elements, specific for each object type, for example, warehouseCloseLot.

The data from each element will be loaded into a separate database row.

Batch Key

The complex objects in the batch are "tied" together by the batch key element.

Example of use:

When you load a Warehouse in batch mode, first of all the stream takes the warehouseTransaction/batchKey element from your incoming EagleML message, resolves a specific DB key from the batch key and deletes all records with the same DB key from DB (if they exist). Only then inserting new records is performed.

Using Batch Error Mode

If some errors occur when you load a Warehouse object in Batch mode, there are two modes to display them:

• Default mode: only the first error is reported in the Task Status Response (TSR).

• Batch Error mode: all errors are reported in the TSR. To enable this mode add the following line to {}w_config_custom.inc:

See a comparison of error reports for Default mode vs Batch Error mode

Single Row UPSERT Mode

The UPSERT mode is used if the Batch Key element (warehouseTransaction/batchKey element) is not present or null in the incoming EagleML message. The message should contain a logical unique Object key for each Warehouse object instead. 

For single row mode the warehouseTransaction node can contain only one specific complex element (e.g. warehouseCloseLot).

Procedure Mode

The mode - INSERT or UPDATE - can be defined automatically or by the Procedure Mode element .

Leave the Procedure Mode element (warehouseTransaction/.../procedureMode) empty and in this case the SP will decide itself whether to update the data in DB tables (if the row already exists) or to insert it (if it does not).
Valid values: 'INSERT', 'UPDATE'.

Grouped Single Mode

Grouped single mode is the load mode when a group of records is sent and loaded to DB separately (unlike Batch mode, when several records are considered as a batch and are sent in one event and are loaded to DB in one event.

• Before load all position details are deleted

• After load RollUp is performed

In EagleML this structure looks like several <warehousePosition> clusters under a single <warehouseTransaction> message. Batch key has no sense, so the header should look like this:

Sub-Batch Load Mode

In Sub-Batch load mode the incoming file is split into several files, one file for each batch (the data for one batch cannot be split into different files!) and each batch is split into sub-batches to decrease its size:

Open

Batch Keys for these sub-batches must be enhanced to express their relation, for example, with security alias numbers (this is important, batch key with 4 elements defines sub-batch mode):

• Batch key: entity1^20150306^EAGLE PACE

• Sub-Batch key: entity1^20150306^EAGLE PACE^212