Manage the EagleML / MC2 Installation

Owned by Pamela Totas (Deactivated)
Last updated: Jun 28, 2024
4 min read

This section describes the EagleML/MC32 installation process including the prerequisites and creating the director structure.

EagleML + EJM

While EagleML and EJM are version independent, your Eagle product suite must be at least version 12.1.5. The same installation package and steps are used for both EagleML and EJM. 
Java v1.6 or higher is required to be installed in the environment for correct Installer performance. The following prerequisites are required for the first time you are installing EagleML/EJM on an environment.  

For Windows OS Server Only

The following applications must exist in the preprocessors folders (example of paths: estar\dynamic\msgcenter\preprocessors and estar\servers\msgservice\preprocessors) :

  • base64.exe

  • pkzip25.exe

  • unzip.exe

  • tartool.exe

Similar commands are typically available in the default builds of Linux/Unix OS. This is why it is not necessary to copy to the preprocessors folders for Linux  servers

For Linux and Windows OS Servers

The eagle_ml-2-0_default_cm_w_setup stream will not actually exist on your environment if you are installing for the first time. Therefore, please do this particular step after you perform the actual install.
Configuration of FTP or SFTP is recommended. Configuration of SMTP is optional, but you need to configure this if you plan on leveraging email notifications.

Create the CMW Directory Structure

This step is necessary to create the required cmw directory as well as perform configuration tasks.

To Create the CMW Directory Structure:

  1. In your Eagle environment:

  2. Save control message below to EagleDBMLRequest.xml.

  3. Open up Message Center Console

  4. Locate the stream eagle_ml-2-0_default_cm_w_setup  and right click on it.

  5. Click Send Data

  6. Select the save control message file and click OK.

EagleDBMLRequest.xml
<EagleML xmlns="http://www.eagleinvsys.com/2011/EagleML-2-0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" eaglemlVersion="2-0" eaglemlType="ReferenceTransactionMessage" xsi:schemaLocation="http://www.eagleinvsys.com/2011/EagleML-2-0 eagleml-main-2-0.xsd" xsi:type="ReferenceTransactionMessage">  <header>   <messageId messageIdScheme="http://www.eagleinvsys.com/coding-scheme/message-id-2-0"> 1634B431F2296D98</messageId>   <sentBy>AUTOGENERATOR</sentBy>   <creationTimestamp>2012-02-20T14:54:27-05:00</creationTimestamp>  </header>  <referenceTransaction>   <header>    <identifier>     <versionedTransactionId>      <transactionId transactionIdScheme="http://www.eagleinvsys.com/coding-scheme/transaction-id-2-0"> 1634B431F2296D98</transactionId>      <version>1</version>     </versionedTransactionId>    </identifier>    <transactionDate>2012-02-20</transactionDate>    <action>ADD</action>    <objectType>EagleDBMLRequest</objectType>   </header>   <EagleDBMLRequest>    <REQUEST>     <MSG_TYPE>ORCH_PROCESS</MSG_TYPE>     <SUB_MSG_TYPE>INS</SUB_MSG_TYPE>     <INPUT_PAR>      <P n="orch_name">EJM Process</P>      <P n="orch_descr">Eagle Job Manager Process</P>      <P n="purge_flag">0</P>      <P n="update_source">MCADMIN</P>     </INPUT_PAR>    </REQUEST>   </EagleDBMLRequest>  </referenceTransaction>  </EagleML>

In order for the Date Rule calculations to work the Pace App Server must be running on same servers MC is running.

MC2

The Extract Service installation requires:

Eagle Application Services installation version 13.1.2 and later.

The latest Eagle ML Package, at least May 2017 Release.

Disk space, memory, CPU, file system (install and logging).

  • Disk space – the installer checks for 5GB of free disk space on dynamic, estar/tpe/ and data/msgcenter folders

  • Memory – we will be adding about 7GB memory footprint per server on top of what Eagle App Servers are using. If some load is shifted from MC to MC2 this can offset the need in memory increase.

  • CPU – depends on the usage. No specific requirements except what is required for Eagle App Server

  • File System – MC2 will be using data/msgcenter for shared data and dynamic/ for rules. MC2 Kafka Service will be writing local topic data to estar/tpe/servers/kafkaservice/kafka/kafka-logs (despite the name these are not logs – this is data). MC2 Kafka Service will write Apache Zookeeper data under estar/tpe/servers/kafkaservice/zookeeper/

Files updated (file list), version, components, and services installed and enabled, new events (any continuous streams or events that need to be enabled and running as part of deployment), new purge rules.

We do not have currently any continuous streams or events to be enabled or specific purge rules to be created.

We install:

  • A MC2 specific JRE in estar/tpe/java/

  • A MC2 specific Python deployment under estar/tpe/pypy and estar/tpe/python

  • MC2 Extract Service (MC2 Core) in estar/tpe/servers/eaglemc2s and configuration in estar/tpe/cfg/eaglemc2s/extractservice. Logs will be in eagle/logs/sarweb/eaglemc2s/extractservice/

  • MC2 Eagle Kafka Service in estar/tpe/servers/kafkaservice and configuration in estar/tpe/cfg/kafkaservice. Logs will be in eagle/logs/starweb/kafkaservice

  • MC2 Redis Server distribution in estar/tpe/servers/redis and configuration in estar/tpe/cfg/eaglemc2s/redis. The logs will be in eagle/logs/stdstreams/redis*.(stderr|stdout)

  • MC2 Python Rule Service in estar/tpe/dynamic/pyrules with the logs in eagle/logs/starweb/pyruleservice

  • MC2 Eagle Plugin (to implement the SOAP and REST interface thru standard Eagle IIS interfaces) in eagle/estar/tpe/servers/plugins/plugin_mc2.so and configuration in estar/tpe/cfg/plugin_mc2.ini. The logs will be eagle/logs/starweb/plugins/mc2/

  • The installer will be updating (or creating) eagle/eaglemgr/config/processguard_custom.xml and eagle/cfg/system.xml to register the new services.

Configuration details – how to scale, timeouts, what are the configuration parameters and where are they stored:

  1. We run one load balancer and two MC2 workers on each server. To scale additional servers can be added.

  2. The configuration template is in eagle/estar/tpe/cfg/eaglemc2s/extractservice/application.yml.template. This file has defaults for all parameters supported by MC2. To change a parameter value it can be redefined by creating a new application.yml file in the same folder and redefining only specific parameters.

  3. The timeouts are defined in the configuration files. We will define those in a separate document.

  4. The latest Extract Service rule package

  5. The maximum number of user processes limit on Linux should be set to at least 4096

           Login with a new SSH session to the Linux environment.

           Run the command ulimit –u and validate the value is less than 4096.

            If less than 4096:

  • Request the system administrator to increase the ulimit –u to 4096. Make sure to check the /etc/security/limits.conf has the correct limit for the user id under which Eagle Application Server is running.

  • Login with a new SSH session to the Linux environment

  • Run the command ulimit –u and validate the value is at least 4096

  • If the output is at least 4096:

  • Change the current working directory to eagle/eaglemgr and run the following commands in the specified order:

./restart emshell

./restart starengine

./restart starweb

 If Eagle PACE is installed and running, execute these additional commands:

./restart paceserver

./restart paceappserver

MC Reader process (see mcreader in processguard.xml) must be running on all servers MC2 is running.

In order for the Date Rule calculations to work the Pace App Server must be running on same servers MC2 is running.