EQL Rest Requests Configuration

Starting with December 2020 release of MC2 the EQL Rest requests are being executed without running an EJM Task. The requests are executed in MC2 Load Balancer instance, without involving the EJM engine which is running in MC2 Worker instances. This results in the EQL Rest request not being reflected in the EJM Tasks window in Message Center Console.

The default configuration provided with MC2 will add the EQL Rest request execution information to the database so that it shows in Message Center Console in:

Tasks Status
File Statistics
Message Status if the request failed.

This can be controlled by changing the MC2 configuration file. The estar/tpe/dynamic/mc2/cfg/extractservice.yml file can be created to change the configuration for all instances of MC2 in a region. The configuration parameters should be in the extractservice-lb section. The default configuration is:

On this page

1 Increase the number of EDS executors
2 Change the Message Status Updates
3 Change the Tasks Status
4 Change the File Statistics status update
5 Execute EQL Requests Using The EJM Engine

---
spring:
  profiles: extractservice-lb
  application:
    name: extractservice-lb

eagle.eaglemlrestapi.services.EQL.directexecution: true
eagle.pyservice.executors: 4
eagle.eaglemlrestapi.services.EQL.messagedetails.level: errorsonly
eagle.eaglemlrestapi.services.EQL.tasks: true
eagle.eaglemlrestapi.services.EQL.filestatistics: true

The --- is important and has to be added as shown to the configuration file. Only one extractservice-lb section should be defined in the configuration file.

Increase the number of EDS executors

To increase the concurrency of request execution the number of python service executors can be increased by using the eagle.pyservice.executors parameter. By default MC2 starts four executors per instance. If the number of concurrent requests is big, this parameter can be adjusted. This will increase the memory and CPU usage. Having 32 Python Service Executors will lead to about 7GB of additional memory usage.

Change the Message Status Updates

The eagle.eaglemlrestapi.services.EQL.messagedetails.level controls what is shown in Message Status tab in Message Center Console. The information added to Message Center Console is registered for stream eagle_default_ruleservice. The eagle.eaglemlrestapi.services.EQL.messagedetails.level parameter can have the following values:

errorsonly - only errors will be shown in Message Center Console/Message Status
statusonly - each request will result in one record shown in Message Center Console/Message Status. The Incoming Message and Status Detail will be empty
allinformation - each request will result in one record shown in Message Center Console/Message Status. The Incoming Message and Status Detail will contain the information related to the request. The Outgoing Message will be empty.

Change the Tasks Status

If parameter eagle.eaglemlrestapi.services.EQL.tasks is set to false, the Tasks Status information will not show the execution of the EQL Rest Requests. This can improve the performance of the EQL REST requests execution if the database update is slow.

Change the File Statistics status update

If parameter eagle.eaglemlrestapi.services.EQL.filestatistics is set to false, the File Statistics information will not show the execution of the EQL Rest Requests. This can potentially improve the performance of the EQL REST requests execution if the database update is slow.

Execute EQL Requests Using The EJM Engine

Setting parameter eagle.eaglemlrestapi.services.EQL.directexecution to false will result in the EQL requests being executed on MC2 Workers using EJM. As a result the EQL Rest requests will be reflected in EJM Monitor and in Message Center Console EJM Tasks window. Currently, this can add overhead and decrease the EQL Rest requests performance under a big load/concurrency by a factor of 5.