Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

In this page:

Table of Contents

Streaming translators are used for big source files (100MB or more) and work faster than classic translators. The main benefit is significantly smaller memory consumption. Streaming allows not to load the whole file or even any sizeable chunk into memory.

Streaming Translator

Icon of the streaming translator:

...

Streaming Translator iconImage Added

On this page

Table of Contents

Info

Currently, streaming transformation works only for the following data formats:

  • CSV to CSV

  • CSV to STAR

  • CSV to XML

  • STAR to CSV

  • STAR to STAR

  • STAR to XML.

Here is an example of setting up a streaming translator:

  1. Open IWS and create a simple solution with Task Translator:

...

  1. Task Translator exampleImage Added
  2. Create new transformation (for example, CSV2STAR) in task translator:

    Select Parameters windowImage Modified
  3. Add simple data for both Source and Destination and select Streaming at the top of the window:
    Select Parameters windowImage Modified

    Info
    iconfalse

    Once you have selected a streaming translator, you cannot change it. Also, if you have not toggled Streaming as shown, you will not be able to change it later

  4. Click Apply.

...

  1. Create Streaming Translators workflow example Image Added

    Image Modified

    It’s the

...

  1. Mapping worksheet.

  2. For streaming translators you can use

...

  1. Preproc worksheet (for example, to add a debug status for testing). Logic in the

...

  1. Preproc tab is executed before the incoming file is processed:
    Preproc worksheetImage Modified

    Info
    iconfalse

    This worksheet already has the Data Query shape created for handling incoming parameters. It is created automatically

  2. You can also use the

...

  1. Postproc feature of the worksheet (for example, to add a status shape for debugging). It works like “Result Output translation” for regular translators:

...

  1. Postproc iconImage Added

Python

...

Translator

Prerequisites (please make sure these items are installed on box):

  1. MC square

  2. PyRuleService

Currently, python translators are under development, but they are the only way to create a translator for incoming source files exceeding 2GB.

Restrictions for python translation:

  1. Currently available only CSV2EagleML translations.

  2. Only direct mapping available

When you create a python translator, you should enable Python Translation checkbox in the top-right corner of the box:

...

Python Translation checkboxImage Added