In this page:

Streaming translators are used for big source files (100MB or more) and work faster than classic translators. The main benefit is significantly smaller memory consumption. Streaming allows not to load the whole file or even any sizeable chunk into memory.

Streaming Translator

Icon of the streaming translator:

Here is an example of setting up a streaming translator:

1. Open IWS and create a simple solution with Task Translator:
   
2. Create new transformation (for example, CSV2STAR) in task translator:
   

3. Add simple data for both Source and Destination and select Streaming at the top of the window:
   

   Once you have selected a streaming translator, you cannot change it. Also, if you have not toggled Streaming as shown, you will not be able to change it later

4. Click Apply.
   
   
   

   It’s the Mapping worksheet.

5. For streaming translators you can use Preproc worksheet (for example, to add a debug status for testing). Logic in the Preproc tab is executed before the incoming file is processed:
   

   This worksheet already has the Data Query shape created for handling incoming parameters. It is created automatically

6. You can also use the Postproc feature of the worksheet (for example, to add a status shape for debugging). It works like “Result Output translation” for regular translators:
   

Python Translator

Prerequisites (please make sure these items are installed on box):

1. MC square
2. PyRuleService

Currently, python translators are under development, but they are the only way to create a translator for incoming source files exceeding 2GB.

Restrictions for python translation:

1. Currently available only CSV2EagleML translations.
2. Only direct mapping available

When you create a python translator, you should enable Python Translation checkbox in the top-right corner of the box: