Generic Load Enrichment Recovery Steps

Once the issue is identified, the recovery steps should be performed, because this behavior will prevent the normal processing of the Generic Load Enrichment workflow.

1. Temporary disable Eagle Default Nuggetizer Schedule ('eagle_nuggetizer_run_schedule').
   Schedule can be done in Automation Center: find and select the necessary schedule in 'Schedule' tab and click 'Disable' (Figure 1: Generic Load Enrichment Recovery Steps).
   
   Figure 1: Generic Load Enrichment Recovery Steps - Eagle Default Nuggetizer Stream in Automation Center

   It is best to disable Eagle Default Nuggetizer Schedule as soon as possible in order not to increase the quantity of hung instances.  

2. Run Now with Overrides ‘eagle_ml-2-0_default_cm_nuggetizer_recover’ stream by selecting ‘Run Now’ -> ‘with Overrides’ option (Figure 6a: Generic Load Enrichment Recovery Steps) in context menu in Message Center Console or Automation Center application, and type in Workflow Name field ‘generic_load_enrichment’ value and click OK (Figure 2b: Generic Load Enrichment Recovery Steps):
   
   Figure 2a: Generic Load Enrichment Recovery Steps - Run Now with Overrides ‘eagle_ml-2-0_default_cm_nuggetizer_recover’ stream in Automation Center
   
   Figure 2b: Generic Load Enrichment Recovery Steps - Run Now with Overrides Workflow Name
3. Wait for ‘eagle_ml-2-0_default_cm_nuggetizer_recover’ execution will be complete. The hung ‘generic_load_enrichment’ instance will be cleanup up and rerun. It will allow other instances to continue processing.
4. After all ‘generic_load_enrichment’ instances will be completed, set Eagle Default Nuggetizer Schedule (‘eagle_nuggetizer_run_schedule’) to Enabled in Automation Center.

5. Check eagle_nuggetizer_run_schedule’, ‘eagle_default_nuggetizer’ and ‘generic_load_enrichment' workflow executed successfully every 10 minutes to ensure the issue is resolved.

6. Running the eagle_ml-2-0_default_cm_nuggetizer_recover stream with the default settings typically resolves the issue. However, if it does not resolve in your case, there are additional steps that you can try:
   
   1. Select the ‘eagle_ml-2-0_default_cm_nuggetizer_recover’ stream again  by selecting ‘Run Now’ -> ‘with Overrides’ option (as shown in Figure 6a: Generic Load Enrichment Recovery Steps) in context menu in Message Center Console or Automation Center application, and type in Workflow Name field ‘generic_load_enrichment’ value as you did in the previous step. 
   2. HOWEVER, before clicking OK, right click in the box and select Debug - Show Hidden Cells.
   3. Change the Mode to be "Recovery" and change the Option to be "All Queue".
   4. Click OK to submit.
      

7. Wait for ‘eagle_ml-2-0_default_cm_nuggetizer_recover’ execution to complete.

   Note

   When the Recovery and All queue mode is selected, all queue will be cleaned and sequence number will be reset to 1. The other instances will not be processed and ‘generic_load_enrichment’ will not be rerun. Just the cleanup will be done. 

   
   After this cleanup the eagle_default_nuggetizer should be executed as usual, which will run the ‘generic_load_enrichment’ with processing sequence number starting with 1.

    After the generic_load_enrichment is run in Recovery/All Queue mode, the non-processed data files associated with those hung instances are not automatically processed. Therefore, if you run the  Recovery and All queue mode and if data files were being receiving when the nuggetizer or generic load enrich hung, it is necessary to resend data files after the recovery.

8. After the cleanup, set Eagle Default Nuggetizer Schedule (‘eagle_nuggetizer_run_schedule’) to Enabled in Automation Center.

9. Check eagle_nuggetizer_run_schedule’, ‘eagle_default_nuggetizer’ and ‘generic_load_enrichment' workflow executed successfully every 10 minutes to ensure the issue is resolved, otherwise contact Eagle for more in depth investigation.