Streaming translators are used for big source files (100MB or more) and work faster than classic translators. The main benefit is significantly smaller memory consumption. Streaming allows not to load the whole file or even any sizeable chunk into memory.

Streaming Translator

Icon of the streaming translator:

Open

On this page

• 1 Streaming Translator
• 2 Python Translator

Here is an example of setting up a streaming translator:

1. Open IWS and create a simple solution with Task Translator:
   
   

   Open

   

2. Create new transformation (for example, CSV2STAR) in task translator:
   
   

   Open

   

3. Add simple data for both Source and Destination and select Streaming at the top of the window:
   

    

4. Click Apply.
   
   

   Open

   

    

   Open

   

    

   It’s the Mapping worksheet.

5. For streaming translators you can use Preproc worksheet (for example, to add a debug status for testing). Logic in the Preproc tab is executed before the incoming file is processed:
   

    

6. You can also use the Postproc feature of the worksheet (for example, to add a status shape for debugging). It works like “Result Output translation” for regular translators:
   
   

   Open

   

Python Translator

Prerequisites (please make sure these items are installed on box):

1. MC square

2. PyRuleService

Currently, python translators are under development, but they are the only way to create a translator for incoming source files exceeding 2GB.

Restrictions for python translation:

1. Currently available only CSV2EagleML translations.

2. Only direct mapping available

When you create a python translator, you should enable Python Translation checkbox in the top-right corner of the box:

Open