Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 9 Next »

Streaming translators are used for big source files (100MB or more) and work faster than classic translators. The main benefit is significantly smaller memory consumption. Streaming allows not to load the whole file or even any sizeable chunk into memory.

Streaming Translator

Icon of the streaming translator:

Streaming Translator icon

On this page


Currently, streaming transformation works only for the following data formats:

  • CSV to CSV
  • CSV to STAR
  • CSV to XML
  • STAR to CSV
  • STAR to STAR
  • STAR to XML.

Here is an example of setting up a streaming translator:

  1. Open IWS and create a simple solution with Task Translator:
    Task Translator example
  2. Create new transformation (for example, CSV2STAR) in task translator:
    Select Parameters window
  3. Add simple data for both Source and Destination and select Streaming at the top of the window:
    Select Parameters window

    Once you have selected a streaming translator, you cannot change it. Also, if you have not toggled Streaming as shown, you will not be able to change it later

  4. Click Apply.
    Create Streaming Translators workflow example

    It’s the Mapping worksheet.

  5. For streaming translators you can use Preproc worksheet (for example, to add a debug status for testing). Logic in the Preproc tab is executed before the incoming file is processed:
    Preproc worksheet

    This worksheet already has the Data Query shape created for handling incoming parameters. It is created automatically

  6. You can also use the Postproc feature of the worksheet (for example, to add a status shape for debugging). It works like “Result Output translation” for regular translators:
    Postproc icon

Python Translator

Prerequisites (please make sure these items are installed on box):

  1. MC square
  2. PyRuleService

Currently, python translators are under development, but they are the only way to create a translator for incoming source files exceeding 2GB.

Restrictions for python translation:

  1. Currently available only CSV2EagleML translations.
  2. Only direct mapping available

When you create a python translator, you should enable Python Translation checkbox in the top-right corner of the box:

Python Translation checkbox


  • No labels