Streaming Translator

Icon of the streaming translator:

Streaming translators used for big source files (100mb and more) and working faster than classic translators.  The main benefit is significantly smaller memory consumption. Streaming allows us not to load the whole file or even any sizeable chunk into memory.

Here example of creating streaming translators:

1. Open IWS and create simple solution with Task Translator:
   
2. Create new transformation (for example CSV2STAR) in task translator:
   

3. Add simple data for both source and select “streaming” in the top of window:
   

   Note

   Oonce u select streaming translator? you cant change it. Also, if you did not select "streaming" in current phase you cant change it later

4. Click “Apply”.
   
   
   

   It’s “mapping” worksheet.

5. For the streaming translators we able to use “Preproc” worksheet (for example we can add a debug status for testing). Logic in “Preproc” tab is executed before the incoming file is processed:
   

   Note

   This worksheet already have the “Data Query” shape for handling incoming parameters. It’s created automatically

6. Also, we can use “Postproc” feature on worksheet(for example add status shape for debugging). It is analogue of “Result Output translation” for regular translators:
   

Python Translators

Prerequisites (this items should be installed on box):

1. MC square
2. PyRuleService

Currently a python translator under development, but it is only way to create a translator for incoming source files exceeding 2 GB.

Restrictions for python translation:

1. Currently available only CSV2EagleML translations.
2. Available only direct mapping

For turn on python translator user need turn on the “Python Translation” checkbox: