Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

The following applies to November 2018 release or later

In this page:

The monitoring stream handles continuous execution check and check for stuck workflows (workflows which execution takes longer than expected).

SPF Profiles

Example of an *.spf profile contents:

…
unit|workflowCheck|workflow|eagle_dkm_source_collector|type|checkPeriodicWorkflow|timeperiod|300|error_prefix|XXX3 Periodic Alert|notificationemail|email_example@eagleinvsys.com|patterns|ON SEND ALERTS
unit|workflowCheck|workflow|eagle_dkm_source_launcher|type|checkPeriodicWorkflow|timeperiod|300|error_prefix|XXX3 Periodic Alert|notificationemail|email_example@eagleinvsys.com|patterns|ON SEND ALERTS
unit|workflowCheck|workflow|eagle_dkm_source_launcher|type|checkStuckWorkflow|timethreshold|600|error_prefix|XXX3 Stuck Workflow Alert|notificationemail|email_example@eagleinvsys.com|patterns|ON SEND ALERTS
unit|workflowCheck|workflow|eagle_dkm_source_collector|type|checkStuckWorkflow|timethreshold|600|error_prefix|XXX3 Stuck Workflow Alert|notificationemail|email_example@eagleinvsys.com|patterns|ON SEND ALERTS
…

where

  • unit is a special parameter, works as a pointer on type of the include in case of new alerts processing;
  • workflow is workflow name;
  • type sets the type of processing ;
  • timeperiod is time period in seconds;
  • timethreshold is processing limit in seconds;
  • error_prefix sets default type of alerts for this workflow;
  • notificationemail is default email for notifications;
  • patterns switches on/off sending alerts

Query

All new code will be in read_workflow_profiles.inc. This include will execute sql query from the start.

 Click to expand:
select
  req_def.INSTANCE          ,
  req_def.CORRELATION_ID    ,
   case
    when INSTR( queue_def.ProcessCorrelationId, 'DATA' ) > 0
    then substr(queue_def.ProcessCorrelationId,1,INSTR( queue_def.ProcessCorrelationId, 'DATA' ))
    else queue_def.ProcessCorrelationId
   end as PROCESS_CORRELATION_ID,
   req_def.BUS_TASK_ID       ,
   req_def.PROC_STATUS       ,
   to_char(req_def.UPDATE_DATE,'YYYYMMDD HH24MISS') UPDATE_DATE,
   queue_def.SCHED_QUEUE_INSTANCE ,
   queue_def.orch_state_clob ORCH_STATE,
   to_char(queue_def.CREATE_DATE,'YYYYMMDD HH24MISS') CREATE_DATE
from
PACE_MASTERDBO.ORCH_REQUEST_DEF req_def,
(select  q.SCHED_QUEUE_INSTANCE ,
   q.ORCH_REQ_DEF_INSTANCE,
   q.orch_state_clob,
   q.CREATE_DATE,
case
when INSTR( q.orch_state_clob, ':ProcessCorrelationId:' ) > 0
then to_char(substr(q.orch_state_clob,INSTR( q.orch_state_clob, ':ProcessCorrelationId:' )+22,INSTR( q.orch_state_clob, ':', INSTR( q.orch_state_clob, ':ProcessCorrelationId:' )+22 ) - INSTR( q.orch_state_clob, ':ProcessCorrelationId:' ) -22 ))
else ''
end as ProcessCorrelationId
from
PACE_MASTERDBO.ORCH_queue q
      where
      q.CREATE_DATE >= trunc(sysdate)
) queue_def
  where
     req_def.instance = queue_def.orch_req_def_instance and
     req_def.correlation_id = queue_def.ProcessCorrelationId and
     req_def.instance not in (select orch_instance from PACE_MASTERDBO.ORCH_REQUEST_PARAMS where PARAMETER_NAME in ('sn_stuck_workflow') and update_date>=trunc(sysdate))
order by req_def.INSTANCE desc

Result of this query is all necessary information about launched workflows (except already processed).

Result of the query is loaded to a file.

After that include check existing .spf profiles in ml2-0_cm_profiles folder.

New alerts processing

All processing will use methods from time utils include. Every workflow from profile will be looked for in query results file. For periodic alerts last workflow with name from profile will be checked through update_date and current time values difference and if difference will be greater than timeperiod – this workflow will be write down in current event array. For stuck workflow alerts type all processing workflows will be checked and if theirs processing time will be greater than timethreshold – it will be also write down in the current event array.

TSR generation

Every line in current event array will be parsed. With create_tsr.inc TSR with necessary information will be created. Type of the TSR message will be choosed from current event line time variables type.

<EagleML xmlns="http://www.eagleinvsys.com/2011/EagleML-2-0"  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="TaskStatusResponse" eaglemlVersion="2-0"  xsi:schemaLocation="http://www.eagleinvsys.com/2011/EagleML-2-0 eagleml-main-2-0.xsd" eaglemlType="TaskStatusResponse">
<header><messageId>B810J250HIKUOGBG</messageId>
 <sentBy>monitoring</sentBy>
 <sendTo>eas_distribution</sendTo>
<creationTimestamp></creationTimestamp>
</header>
<statusItem>
<taskIdentifier>
 <correlationId>84040C2ADCC6D46</correlationId>
 <businessTaskId>B810J250HCFFDWIC</businessTaskId>
 </taskIdentifier>
 <status>REPORTING</status>
 <severityCode>1</severityCode>
<reason>
 <reasonTypeEnum>INFO</reasonTypeEnum>
 <reasonCode>1</reasonCode>
 <description>==================================
PPFT3 PERIODIC ALERT: Workflow eagle_dkm_source_launcher didn't launch in time period 300 seconds. Last launch was in 20181009 091845 CorrId 84040C2ADCC6D46 AlertNotificationEmail:vmironov@eagleinvsys.com</description>
<reasonTag>PPFT3 PERIODIC ALERT</reasonTag>
</reason>
</statusItem>
</EagleML>

Eas_distribution changes

TSR parsing

For parsing the incoming TSR tsr_to_w_state.inc will be used. This include file transform incoming TSR with xslt translation to correlation Id and email address from error description (if TSR has in description AlertNotificationEmail) and with correlation ID and get_w_state.inc get task parameter AlertNotificationEmail (this parameter is unique!) . If this parameter is exist – it will be used as email address, if parameter doesn’t exist include will use address from description or default email.

Result of this stage – new email address (if TSR was for new types of alerts).

New xslt translation patterns

4 new patterns will be made:

1)PPFT3 PERIODIC ALERT

If error description contains keywords PPFT3 PERIODIC ALERT – this pattern will be used. This will make email with error description from TSR.

2)PPFT3 STUCK WORKFLOW ALERT

If error description contains keywords PPFT3 STUCK WORKFLOW ALERT – this pattern will be used. This will make email with error description from TSR.

3)PPFT3 ERROR

If error description contains keywords PPFT3 ERROR – this pattern will be used. This pattern will make email with all errors from all task, that it can get from incoming TSR.

4)PPFT3 WARNING

If error description contains keywords PPFT3 WARNING – this pattern will be used. This pattern will make blank email subject, and eas_distribution will not send emails with blank email subject

  • No labels