Nuggetizer Recovery Steps

Once you have identified the issue, you will need to manually recover because these behaviors will prevent the normal processing of the Eagle Default Nuggetizer workflow.

Recovery steps for the following issues are as follows:

Recovery Steps for When the Eagle Default Nuggetizer Schedule (‘eagle_nuggetizer_run_schedule’) is Disabled

 Follow these recovery steps:

1. Set Eagle Default Nuggetizer Schedule (‘eagle_nuggetizer_run_schedule’) Enabled.

   This can be done in Automation Center: find and select the necessary schedule in ‘Schedule’ tab and click ‘Enable’ (Figure 1: Nuggetizer Recovery Steps).
   
   Figure 1: Nuggetizer Recovery Steps - Enable Eagle Default Nuggetizer Schedule

2. After enabling, check ‘eagle_nuggetizer_run_schedule’ and ‘eagle_default_nuggetizer’ workflow executed successfully every 10 minutes to ensure the issue is resolved, otherwise contact Eagle.  If necessary, the Eagle representative will contact the EagleML Team directly through our typical escalation procedures.

Recovery Steps for When the Eagle Default Nuggetizer Schedule (‘eagle_nuggetizer_run_schedule’) is Hung

 Follow these recovery steps:

1. Restart Message Center Processes.
   Restart means to stop and then start, as outtlined below.

   Note

   The steps below outline how to do the Restart via Engine Manager, as clients have access to this*.   If you are an Eagle ACCESS person who typically restarts Message Center an alternative way, that is fine.

   *Engine Manager was retired in v2017.  The same tasks can be done in System Management Center under Environment - Services.

   Open  the ‘Engine manager’ application, for necessary web server select THICK client and select ‘Web Controls’ in central page (Figure 2: Nuggetizer Recovery Steps).

   
   Figure 2: Nuggetizer Recovery Steps - Engine Manager
   In the EagleSTAR Web Control Panel tab (Figure 3: Nuggetizer Recovery Steps):
   Click on button “Status MessageCenter Processes” for checking status.
   Click on button “Stop Message Center Processes”.
   Click on button “Status Message Center Processes” for checking status.
   Click on button “Start Message Center Processes”.
   Click on button “Status Message Center Processes” for checking status.
   
   Figure 3: Nuggetizer Recovery Steps - Engine Manager Web Control Panel
2. If Step 1 did not help (which you can determine because the ‘eagle_nuggetizer_run_schedule’ has not been executed in 10 minutes), you would need to next run SQL request for canceling hung instance from the backend.
   Depending on your permissions, you may be able to run these SQL from Engine Manager.  Otherwise, you will need Eagle ACCESS DBA assistance.
   There is no need to stop Message Center to execute these 2 SQL statements.
   In this SQL example, we are using instance 1285602.  But, you would need to find the actual instance value from the ‘Queue Id’ column in the Automation Center Monitor (see Figure 4: Nuggetizer Recovery Steps)
   delete pace_masterdbo.egl_sched_def_details where sq_instance = 1285602;
   update pace_masterdbo.egl_sched_queue set status=3, actual_stop_time = sysdate, update_date = sysdate where instance = 1285602;
   
   Figure 4: Nuggetizer Recovery Steps - Queued in Automation Center
3.  After, check ‘eagle_nuggetizer_run_schedule’ and ‘eagle_default_nuggetizer’ workflow executed successfully every 10 minutes to ensure the issue is resolved, otherwise please contact Eagle.  If necessary, the Eagle representative will contact the EagleML Team directly through our typical escalation procedures.

Recovery Steps for When the Eagle Default Nuggetizer Workflow is Hung

1. Temporary disable Eagle Default Nuggetizer Schedule (‘eagle_nuggetizer_run_schedule’).
   Schedule can be done in Automation Center: find and select the necessary schedule in ‘Schedule’ tab and click ‘Disable’ (Figure 5: Nuggetizer Recovery Steps).
   
   

   Figure 5: Nuggetizer Recovery Steps - Eagle Default Nuggetizer Stream in Automation Center

   It is best to disable Eagle Default Nuggetizer Schedule as soon as possible in order not to increase the quantity of hung instances.  

2. Run ‘eagle_ml-2-0_default_cm_nuggetizer_recover’ stream by selecting ‘Run Now’ -> ‘Selected’ option in context menu in  Automation Center or Message Center Console.

   Note

   The stream will be executed with default values for ‘eagle_default_nuggetizer’ workflow.

   
   Figure 6: Nuggetizer Recovery Steps -  Run ‘eagle_ml-2-0_default_cm_nuggetizer_recover’ Stream in Automation Center

3. Wait for ‘eagle_ml-2-0_default_cm_nuggetizer_recover’ stream execution to complete.  The hung ‘eagle_default_nuggetizer’ instance will be cleanup up and rerun. It will allow other instances to continue processing.
4. After all ‘eagle_default_nuggetizer’ instances are completed,  set the Eagle Default Nuggetizer Schedule (‘eagle_nuggetizer_run_schedule’) to Enabled again in Automation Center.
5. Check ‘eagle_nuggetizer_run_schedule’ and ‘eagle_default_nuggetizer’ workflow to see it executed successfully every 10 minutes to ensure the issue is resolved.
6. Running the eagle_ml-2-0_default_cm_nuggetizer_recover stream with the default settings typically resolves the issue.  However, if it does not resolve in your case, there are additional steps that you can try:
   1. Select the ‘eagle_ml-2-0_default_cm_nuggetizer_recover’ stream again, but this time select  ‘Run Now’ -> ‘with Overrides’ option in context menu in Message Center Console or Automation Center application. 
   2. Leave the stream name defaulted to  ‘eagle_default_nuggetizer’.
   3. Right click in the box and select Debug - Show Hidden Cells.

   4. Change the Mode to be "Recovery" and change the Option to be "All Queue".

      Note

       You will get a warning that you are changing the default settings. This is OK, but you will need to switch back to the default settings after your issue is resolved.
   5. Click OK to submit.
      
      Figure 7: Nuggetizer Recovery Steps - Run Now’ -> ‘with Overrides’ Option
7. Check ‘eagle_nuggetizer_run_schedule’ and ‘eagle_default_nuggetizer’ workflow to see it executed successfully every 10 minutes to ensure the issue is resolved.  If you still have issues, please contact Eagle for further assistance.

After such cleanup the eagle_default_nuggetizer should be executed as usual, which will run the ‘generic_load_enrichment’ with processing sequence number starting with 1.