Page Comparison

In some rare cases (especially related to emergency situations in env) we can see , such as MC crash, MC stopped/restarted, files got stuck) you might find that one or more of launched workflows have “in “In progress” status, however there are no any launched sub-steps (streams). For example:

Image Removed

Note: please do not confuse it with hanging extract-stream. In that case: we are able to expand Image Added

...

...

This page with help you unstuck cleaup the stuck process using built-in EJM instruments and recover Sequencer functionality:

Reset Sequencer

...

using eagle_ml-2-0_default_cm_sequencer_

...

cleanup 

Reset current stuck process (recommended method)

1. Create a .txt file with any name with the following line:

1176:CUR_[workflow_name]:1177:[POSTFIX]:

where:

• CUR_  – is a prefix used when you want to cleanup only current process

...

Reset the entire queue

In case of sequencer getting stuck, we suggest checking estar/tpe/cfg/msgcenter/msgservice.ini config file first of all.

SplitBatchSize parameter should not be a large number, preferred value is 2, in some cases, 1 - it is responsible for memory consumption and values like 100 may harm processing.

...

• (which is stuck),
• [workflow_name] – is workflow name,
• [POSTFIX] – is defined as:

  • the value of <processingSequencerIdentifier> element of the initial RTR (if this node is not empty and exists). For example, on Sequencer Execution in Parallel Mode page:
    

    Code Block
    language xml
    […] <taskIdentifier>     <processingSequenceNumber>1</processingSequenceNumber>     <processingSequencerIdentifier>PROCSEQID</processingSequencerIdentifier>     <correlationId correlationIdScheme="correlationIdScheme">correlationId</correlationId>     <businessTaskId correlationIdScheme="businessTaskIdScheme">businessTaskId</businessTaskId> </taskIdentifier> […]

    The cleanup command should look like this:

    Code Block
    language text
    1176:CUR_g_os_eddf_mc_eagle_extract:1177:PROCSEQID:

    
    

  • OR the value of the corresponding tag 1177 from related pace event (you can just copy it in Automation Center):
    Image Added
    In this case the cleanup command should look as follows:
    

    Code Block
    language text
    1176:CUR_g_os_eddf_mc_eagle_extract:1177:MSGCENT:

    
    

2. Drop that file to eagle_ml-2-0_default_cm_sequencer_cleanup stream.
3. Wait eagle_ml-2-0_default_cm_sequencer_cleanup stream to complete the cleanup.
4. Done! You can resume Sequencer, it will continue processing the queue with the next sequence number

Reset the entire queue

You should form a message as above, but with ALL_ prefix. 

1176:ALL_[workflow_name]:1177:[POSTFIX]:

Example:

1176:CUR_eagle_:ALL_g_os_eddf_mc_eagle_extract:1177:PROCSEQID: 

Reset Sequencer using eagle_ml-2-0_default_cm_nuggetizer_recover

...

1. Go to the Message Center Console/Editor.
2. Find eagle_ml-2-0_default_cm_nuggetizer_recover stream and right click on it
3. Choose Run Now With Overrides:
   Image Added
   
4. In the opened Run Now With Overrides window, right click somewhere in the List Options, press Debug
   Image Added
   
5. And tick Show Hidden Cells:
   Image Added
   
   
6. Add the following parameters: Mode = Recovery, Option = All Queue and Workflow Name = eagle_wrf_generic_load:

...

1. 
   Image Added
   
   
2. Press Ok

The sequencer will be unstuck and will continue processing the queue with the next sequence number.

Sequencer states validation

Check database with the next query:

select 'STATE' as ACTION,
replace(ORCH_STATE_CLOB, ':', '^') as SEQ_STATE,
t.bus_task_id,
t.correlation_id,
t.proc_status,
o.status,
o.instance Q_INST,
t.instance DEF_INST
from PACE_MASTERDBO.orch_queue o, PACE_MASTERDBO.ORCH_REQUEST_DEF t
where o.orch_req_def_instance = t.instance
and t.bus_task_id like '%#sequencer'

If you see some record with proc_status='FAILED', you need to change it back to 'ACK': 

UPDATE PACE_MASTERDBO.ORCH_REQUEST_DEF 
SET PROC_STATUS = 'ACK' 
WHERE PROC_STATUS = 'FAILED' AND BUS_TASK_ID LIKE '%#sequencer';