In this page:
| Table of Contents |
|---|
Generic Load is capable of unpacking and transforming files immediately after they are received, without waiting for download queues. Files for load should be sent via eagle_ml-2-0_default_cm_ndfa
Configure Profiling for Generic Load
First of all, create a profile in the EJM Monitoring Tool
Fill and save parameters in the pop-up window
where:
profileName is the profile name used to generate names of new workflow and other files - must be unique for each profile;
workflowName is the source workflow name used to generate new workflow;
inMask{number} describe incoming file masks. Must be coherent and start with number 1. Each record should contain the following values, separated by “|”:
- incoming file mask;
- translator for incoming mask (if you do not want to use a translator, specify empty string). Must match transform_pairs.inc;
- comma-separated feed types for load available for this translator (should not contain duplicates).
Configure Decompress-Transform
To add custom translator you should create a pair of mask file name and translator stream name. Pairs are represented with a variable matching the following pattern:
:pair{pair_number}: := ‘{file_mask_without_wildcards|tranlation_stream_name}’
Important:
- Pair variables list must be coherent and start with number 1.
- Files mask values can not include wildcards.
- Only the first file matching the mask will be transformed.
Example of setting five pairs:
$ESTAR_HOME/tpe/dynamic/msgcenter/customer/transform_pairs.inc
| Code Block | ||
|---|---|---|
| ||
<CODE>
:pair1: := '_SRM__PROFILED_LOAD_TST|_srm_tr';
:pair2: := '_FUND_PROFILED_LOAD_TST|_fund_profile_5_tr';
:pair3: := '_FUND_ALT_ID_PROFILED_LOAD_TST|_fund_profile_alt_id_tsk';
:pair4: := '_HOLDING_PROFILED_LOAD_TST|_wh_pos_tr';
:pair5: := '_TRANS_PROFILED_LOAD_TST|_wh_cash_activity_tr';
</CODE> |
Create Schedule Event
| Info |
|---|
Please note, that the schedule described below is time-based. This means that incoming files are expected to arrive within the time interval specified. If the files were not received on time, the cycle of pending for them is reset. |
Create workflow_restart stream (must be created once for a region)
1. Open Message Center Editor in Eagle shell
2. Open Message Center Streams tab
3. Streams -> Add Stream
4. Set the following parameters:
5. Reinitialize MC cache 2 -3 times
Create a Schedule Event
1. Open Automation Center
2. Create New -> Schedule
3. Find the stream you created
4. Define tab
5. Time tab
6. Run Parameters tab
- 1166 – profile file name
- 1176 – target workflow name (in workflow name replace ‘eagle’ to profile name)(example for workflow name ‘eagle_wrf_generic_load’ and profile name ‘***_feed’, target workflow name ‘***_feed_wrf_generic_load’)
- 1178 – ‘EnableWrfCreating’
7. Dependency tab - default values
8. Notification tab - default values
...
Creating Generic Load Profile
Overview
Configuring Profiled Generic Load through EJM MT allows user:
- to use a more plain form;
- to generate profiled generic load configuration files here-and-now on Publish;
- to avoid re-generating profiled generic load configuration files on each profile execution;
- to use different EJM profiles for the same profiled generic load instance;
- to escape the need to change customer/filesets.inc and customer/transform_pairs.inc includes manually;
- to have only workflow name as a mandatory schedule execution parameter.
Required Steps
To create Profiled Generic Load:
- Open EJM Monitoring Tool (MT) and choose Activities tab
- Under drop-down menu choose Add Generic Load Profile option
- Enter required parameters.
- Press Publish button to publish the profile.
Configuring Profile Name
Profile name is set in the field between Workflow Name and _wrf_generic_load. Please be informed that profile name will be used as a prefix for profiled generic load, thus it is redundant to add profile extension or word “profile” in this field.
Configuring Data Files Processing
Add File
To add files to the workflow user can click Add File button.
Remove File
To remove a file from the profile use Remove File button with focus set on the row you want to remove.
Please be advised that order of files does not affect processing order or order of load. Files will be consumed by profile executor on arrival to the region and results of translation will be loaded according to dependencies that are determined by eagle_wrf_generic_load workflow.
Configuring file processing options
File mask in File Mask field (please be advised that wildcards are not needed and are not supported. Date rules are supported. Anything after @ symbol is considered as date rule and will be handled accordingly on processing
Name of translator in Translator field. It drops down a list allowing to choose a specific translator from existing list of streams on the region.
Feed type or feed types that are expected as result of translation of incoming file by the specified translator. Please be advised that there is a menu under the three dots symbol which allows to add common EagleML objects.
But user is free to just type in EagleML object names in the Feed Types field. This may be useful for cases when user needs to add EBD files ({EagleML feed type}EDB_DATA, {EagleML feed type}EDB_KEYS) or Cancel ({Feed type group}CANCEL{Feed type name} such as WarehouseCancelTrade, WarehouseCANCELCashActivity) files.
Configuring Post-processing
Post-processing tab allows user to determine steps that should happen after data files load. User is free to configure any post-processing step or sequence of steps.
Step Type – type of post processing. Possible options are: CM, subprocess or task.
Step Name – name of translator/task /workflow which will be used for post-processing.
If step type is CM – step name will be used as name of stream that should be triggered by control message
If step type is task – step name will be used as name of stream for which trigger file will be sent through eagle_ml-2-0_default_cm_control_message stream
If step type is subprocess – step name will be used as name of workflow that should be launched as subprocess.
Condition To Execute – condition when Post Processing step should be executed. Value can be configured to be evaluated on runtime, use EXPR# prefix to set expression.
Parameters – incoming parameters for Post Processing step. Use EXPR# to set parameter values that should be evaluated on runtime based on workflow launch parameters.
Example:
Configuring Custom Parameters
Provides the ability to setup specific parameters for the workflow.
Parameter names can be dynamic. To set dynamic parameters use EXPR# prefix before expressions to evaluate.
Configuring customer/transform_pairs.inc
Publishing Profiled Generic Load definition does not require user actions to fix customer/transform_pairs.inc
EJM MT adds transform pair or pairs required for configured profiled generic load automatically on publish.
Configuring customer/filesets.inc
Publishing Profiled Generic Load definition does not require user actions to fix customer/filesets.inc
EJM MT adds code to customer/filesets.inc if it is required for configured profiled generic load automatically on publish.
Scheduling Generic Load Profile on Eagle17 regions
To configure scheduled execution of profiled generic load use following steps:
| Info |
|---|
If you already have schedule or schedules that are used for profiled generic load – duplicate one of existing schedules and go to step 5. |
- Open Automation Center and create new event based on workflow_restart rule and workflow_restart sch file.
- Event Type field should be set manually (Message Stream Process) or chosen from the expand list;
- Enter an Event Name.
- Choose the found event and click OK
- Enter a schedule name and click Next
If you want to turn on your schedule tick Enabled
- Setup run time and days for workflow schedule and click Next
- Add tag 1166 with value
{name of created profile}_wrf_generic_load
This is required to start profiled generic load workflow on schedule execution.
Other parameters are optional.
Info You can use any profile name with tag 1166 (do not forget profile extension .profile in this case) to use execution parameters from the profile to be applied to profiled generic load workflow execution. Using profiled generic load profile is not mandatory and should be skipped because profiled generic load instance is created on profile publish in EJM MT.
Thus the same profiled generic load once created may be executed with any other EJM profile - Set dependencies if needed.
- Set Notifications if needed.
- Enable schedule.