In some rare cases (especially related to emergency situations in env, such as MC crash, MC stopped/restarted, files got stuck) you might find that one or more of launched workflows have “In progress” status, however there are no launched sub-steps (streams). For example:
...
2. Drop that file to eagle_ml-2-0_default_cm_sequencer_cleanup stream.
3. Wait eagle_ml-2-0_default_cm_sequencer_cleanup stream to complete the cleanup.
4. Done! You can run the processresume Sequencer, it will continue processing the queue with the next sequence number
| Info |
|---|
The stuck record was not loaded to DB, so we recommend loading this record manually (w/o Sequencer) to keep data integrity. |
Reset the entire queue
| Info |
|---|
Please note, that this way you will cleanup not only the current stuck process, but the rest of the queue as well |
...
| Info |
|---|
Using Nuggetizer is quite similar to unstucking a single process with Sequencer cleanup, but you do not need to form any messages, everything is done via panel. But it cannot work with PROCSEQID |
- Go to the Message Center Console/Editor.
- Find eagle_ml-2-0_default_cm_nuggetizer_recover stream and right click on it
- Choose Run Now With Overrides:
- In the opened Run Now With Overrides window, right click somewhere in the List Options, press Debug
- And tick Show Hidden Cells:
- Add the following parameters: Mode = Recovery, Option = All Queue and Workflow Name = eagle_wrf_generic_load:
- Press Ok
The sequencer was now reset in the region.will be unstuck and will continue processing the queue with the next sequence number.
| Info |
|---|
The stuck record was not loaded to DB, so we recommend loading this record manually (w/o Sequencer) to keep data integrity. |
Sequencer states validation
Check database with the next query:
| Code Block | ||
|---|---|---|
| ||
select 'STATE' as ACTION,
replace(ORCH_STATE_CLOB, ':', '^') as SEQ_STATE,
t.bus_task_id,
t.correlation_id,
t.proc_status,
o.status,
o.instance Q_INST,
t.instance DEF_INST
from PACE_MASTERDBO.orch_queue o, PACE_MASTERDBO.ORCH_REQUEST_DEF t
where o.orch_req_def_instance = t.instance
and t.bus_task_id like '%#sequencer' |
If you see some record with proc_status='FAILED', you need to change it back to 'ACK':
| Code Block | ||
|---|---|---|
| ||
UPDATE PACE_MASTERDBO.ORCH_REQUEST_DEF
SET PROC_STATUS = 'ACK'
WHERE PROC_STATUS = 'FAILED' AND BUS_TASK_ID LIKE '%#sequencer'; |