Sequencer CleanUp

In some rare cases (especially related to emergency situations in env, such as MC crash, MC stopped/restarted, files got stuck) you might find that one or more of launched workflows have “In progress” status, however there are no launched sub-steps (streams). For example:

Please note, that this situation differs from a hanging extract stream (in case of hanging extract stream you would be able to expand a workflow line in MT and see that there is a stream or streams with "Processing" status (not a workflow))

This page with help you cleanup the stuck process using built-in EJM instruments and recover Sequencer functionality.

On this page

1 Reset Sequencer using eagle_ml-2-0_default_cm_sequencer_cleanup
- 1.1 Reset current stuck process (recommended method)
- 1.2 Reset the entire queue
2 Reset Sequencer using eagle_ml-2-0_default_cm_nuggetizer_recover
3 Sequencer states validation

Reset Sequencer using eagle_ml-2-0_default_cm_sequencer_cleanup

Reset current stuck process (recommended method)

1. Create a .txt file with any name with the following line:

1176:CUR_[workflow_name]:1177:[POSTFIX]:

where:

CUR_ – is a prefix used when you want to cleanup only current process (which is stuck),
[workflow_name] – is workflow name,
[POSTFIX] – is defined as:
- the value of <processingSequencerIdentifier> element of the initial RTR (if this node is not empty and exists). For example, on Sequencer Execution in Parallel Mode page:
  […] <taskIdentifier> <processingSequenceNumber>1</processingSequenceNumber> <processingSequencerIdentifier>PROCSEQID</processingSequencerIdentifier> <correlationId correlationIdScheme="correlationIdScheme">correlationId</correlationId> <businessTaskId correlationIdScheme="businessTaskIdScheme">businessTaskId</businessTaskId> </taskIdentifier> […]
  The cleanup command should look like this:
  1176:CUR_g_os_eddf_mc_eagle_extract:1177:PROCSEQID:
- OR the value of the corresponding tag 1177 from related pace event (you can just copy it in Automation Center):
  
  In this case the cleanup command should look as follows:
  1176:CUR_g_os_eddf_mc_eagle_extract:1177:MSGCENT:

2. Drop that file to eagle_ml-2-0_default_cm_sequencer_cleanup stream.
3. Wait eagle_ml-2-0_default_cm_sequencer_cleanup stream to complete the cleanup.
4. Done! You can resume Sequencer, it will continue processing the queue with the next sequence number

The stuck record was not loaded to DB, so we recommend loading this record manually (w/o Sequencer) to keep data integrity.

Reset the entire queue

Please note, that this way you will cleanup not only the current stuck process, but the rest of the queue as well

You should form a message as above, but with ALL_ prefix.

1176:ALL_[workflow_name]:1177:[POSTFIX]:

Example:

1176:ALL_g_os_eddf_mc_eagle_extract:1177:PROCSEQID:

Reset Sequencer using eagle_ml-2-0_default_cm_nuggetizer_recover

Using Nuggetizer is quite similar to unstucking a single process with Sequencer cleanup, but you do not need to form any messages, everything is done via panel. But it cannot work with PROCSEQID

Go to the Message Center Console/Editor.
Find eagle_ml-2-0_default_cm_nuggetizer_recover stream and right click on it
Choose Run Now With Overrides:
In the opened Run Now With Overrides window, right click somewhere in the List Options, press Debug
And tick Show Hidden Cells:
Add the following parameters: Mode = Recovery, Option = All Queue and Workflow Name = eagle_wrf_generic_load:
Press Ok

The sequencer will be unstuck and will continue processing the queue with the next sequence number.

The stuck record was not loaded to DB, so we recommend loading this record manually (w/o Sequencer) to keep data integrity.

Sequencer states validation

Check database with the next query:

select 'STATE' as ACTION,
replace(ORCH_STATE_CLOB, ':', '^') as SEQ_STATE,
t.bus_task_id,
t.correlation_id,
t.proc_status,
o.status,
o.instance Q_INST,
t.instance DEF_INST
from PACE_MASTERDBO.orch_queue o, PACE_MASTERDBO.ORCH_REQUEST_DEF t
where o.orch_req_def_instance = t.instance
and t.bus_task_id like '%#sequencer'

If you see some record with proc_status='FAILED', you need to change it back to 'ACK':

UPDATE PACE_MASTERDBO.ORCH_REQUEST_DEF 
SET PROC_STATUS = 'ACK' 
WHERE PROC_STATUS = 'FAILED' AND BUS_TASK_ID LIKE '%#sequencer';