Overview of the IWS Solution for CSV

Owned by Alexei Matveev
Last updated: Jul 29, 2024 by Pamela Totas (Deactivated)
2 min read

The workflow for a CSV feed consists of two steps: Preprocessing and Load.

Step 1 - Preprocessing Step

The first step, rs_source_files_handler, is represented by the unpacker shape. It can handle CSV and Bloomberg files in either uncompressed form or “packed” with any standard archiver, such as zip, gz, tar.

Step 2 - Default Load Workflow

The second step is the default load Raw Service workflow. You can view the details by double-clicking the rs_load_source_files shape from the previous diagram. See the following figure. 

image-20240729-192536.png

The Load Workflow consists of four steps:

  1. rs_init_structure. Load model (structure) in Data Base with unique batch_id. Global columns are loaded in this step as well.

  2. rs_main_t. Translate csv data to XMLDBAPI format.

  3. rs_load_source_files. Load xmldbapi data to database using the eagle_ml-2-0_default_in_xml_warehouse message stream.

  4. [vendor name]_[feed name]_rs_create_table. Create view table, update dimension table, and check errors during the load.

After you run the feed from IWS, it automatically displays additional tabs from the Monitoring Tool, where you can see the statuses of all the involved steps, and can drill down into the details.