In case of sequencer getting stuck, we suggest checking estar/tpe/cfg/msgcenter/msgservice.ini config file first of all.
SplitBatchSize parameter should not be a large number, preferred value is 2, in some cases, 1 - it is responsible for memory consumption and values like 100 may harm processing.
In case of sequencer getting stuck, our cleanup stream can work in several modes - either remove the entire queue or remove only the current (stuck workflow, including the case when you use PROCSEQID). CommandIn some rare cases (especially related to emergency situations in env, such as MC crash, MC stopped/restarted, files got stuck) you might find that one or more of launched workflows have “In progress” status, however there are no launched sub-steps (streams). For example:
Info |
---|
Please note, that this situation differs from a hanging extract stream (in case of hanging extract stream you would be able to expand a workflow line in MT and see that there is a stream or streams with "Processing" status (not a workflow)) |
This page with help you cleaup the stuck process using built-in EJM instruments and recover Sequencer functionality:
Table of Contents |
---|
Reset Sequencer using eagle_ml-2-0_default_cm_sequencer_cleanup
Reset current stuck process (recommended method)
1. Create a .txt file with any name with the following line:
1176:CUR_[workflow_name]:1177:[POSTFIX]:
where:
- CUR_ – is a prefix used when you want to cleanup only current process (which is stuck),
- [workflow_name] – is workflow name,
- [POSTFIX] – is defined as:
the value of <processingSequencerIdentifier> element of the initial RTR (if this node is not empty and exists). For example, on Sequencer Execution in Parallel Mode page:
Code Block language xml […] <taskIdentifier> <processingSequenceNumber>1</processingSequenceNumber> <processingSequencerIdentifier>PROCSEQID</processingSequencerIdentifier> <correlationId correlationIdScheme="correlationIdScheme">correlationId</correlationId> <businessTaskId correlationIdScheme="businessTaskIdScheme">businessTaskId</businessTaskId> </taskIdentifier> […]
The cleanup command should look like this:
Code Block language text 1176:CUR_g_os_eddf_mc_eagle_extract:1177:PROCSEQID:
OR the value of the corresponding tag 1177 from related pace event (you can just copy it in Automation Center):
In this case the cleanup command should look as follows:Code Block language text 1176:CUR_g_os_eddf_mc_eagle_extract:1177:MSGCENT:
2. Drop that file to eagle_ml-2-0_default_cm_sequencer_cleanup stream.
3. Wait eagle_ml-2-0_default_cm_sequencer_cleanup stream to complete the cleanup.
4. Done! You can resume Sequencer, it will continue processing the queue with the next sequence number
Info |
---|
The stuck record was not loaded to DB, so we recommend loading this record manually (w/o Sequencer) to keep data integrity. |
Reset the entire queue
Info |
---|
Please note, that this way you will cleanup not only the current stuck process, but the rest of the queue as well |
You should form a message as above, but with ALL_ prefix.
1176:ALL_[workflow_name]:1177:[POSTFIX]:
Example:
Code Block | ||
---|---|---|
| ||
1176:CUR_eagle_:ALL_g_os_eddf_mc_eagle_extract:1177:PROCSEQID: |
Reset Sequencer using eagle_ml-2-0_default_cm_nuggetizer_recover
Info |
---|
Using Nuggetizer is quite similar to unstucking a single process with Sequencer cleanup, but you do not need to form any messages, everything is done via panel. But it cannot work with PROCSEQID |
...
- Go to the Message Center Console/Editor.
- Find eagle_ml-2-0_default_cm_nuggetizer_recover stream and right click on it
- Choose Run Now With Overrides:
- In the opened Run Now With Overrides window, right click somewhere in the List Options, press Debug
- And tick Show Hidden Cells:
- Add the following parameters: Mode = Recovery, Option = All Queue and Workflow Name = eagle_wrf_generic_load:
...
- Press Ok
The sequencer will be unstuck and will continue processing the queue with the next sequence number.
Info |
---|
The stuck record was not loaded to DB, so we recommend loading this record manually (w/o Sequencer) to keep data integrity. |
Sequencer states validation
Check database with the next query:
Code Block | ||
---|---|---|
| ||
select 'STATE' as ACTION,
replace(ORCH_STATE_CLOB, ':', '^') as SEQ_STATE,
t.bus_task_id,
t.correlation_id,
t.proc_status,
o.status,
o.instance Q_INST,
t.instance DEF_INST
from PACE_MASTERDBO.orch_queue o, PACE_MASTERDBO.ORCH_REQUEST_DEF t
where o.orch_req_def_instance = t.instance
and t.bus_task_id like '%#sequencer' |
If you see some record with proc_status='FAILED', you need to change it back to 'ACK':
Code Block | ||
---|---|---|
| ||
UPDATE PACE_MASTERDBO.ORCH_REQUEST_DEF
SET PROC_STATUS = 'ACK'
WHERE PROC_STATUS = 'FAILED' AND BUS_TASK_ID LIKE '%#sequencer'; |