API & Integrations - Data Collection Scripts for LogRhythm

This article covers Data Collection Scripts for LogRythm. Mimecast Data Collection Scripts for LogRhythm allow a LogRhythm administrator to download email and audit events from Mimecast ready for collection by the LogRhythm platform. Data is collected and parsed using LogRhythm Flat File collectors and MPE rules composed by LogRhythm.

System Requirements and Prerequisites

LogRhythm Server

• Mimecast MTA Log Source needs to be created first, the folder path should be updated before scheduling scripts siem_cleanup.py to run.
• Python v3.9.x installed on the server used to collect data from the Mimecast API.
• At least v7.1.433.0 of the LogRhythm KB - this release contains the Mimecast data parsing rules

Network

Data collection uses the Mimecast API. You must ensure that the server hosting the data collection scripts has outbound HTTPS access (TCP port 443) to the following hosts depending on the region where your Mimecast account is hosted:

Region	Hosts
EU	api.mimecast.com AND eu-api.mimecast.com
DE	api.mimecast.com AND de-api.mimecast.com
US	api.mimecast.com AND us-api.mimecast.com
USB	api.mimecast.com AND usb-api.mimecast.com
CA	api.mimecast.com AND ca-api.mimecast.com
ZA	api.mimecast.com AND za-api.mimecast.com
AU	api.mimecast.com AND au-api.mimecast.com
Offshore	api.mimecast.com AND jer-api.mimecast.com

Mimecast Permissions

See the table below for the endpoints used by the data collection scripts and the associated Mimecast administrator permissions required. For convenience, all permissions are included in the Basic Administrator role.

Endpoint	Permission Required
/api/login/discover-authentication	n/a
/api/audit/get-audit-events	Logs | Read
/api/audit/get-siem-logs	Tracking | Read

Configuring Mimecast

Warning: The data collection scripts require a Mimecast Administrator Authentication token. By default an authentication token expires after three days. This means that your scripts will stop collecting data from Mimecast after that time. For the best experience, you must create a new user and Authentication Profile defining a longer lived Authentication Token. The steps below describe this process.

The preparation required in the Mimecast Administration Console involves:

1. Creating a User
2. Adding the User to an Administrator Role
3. Creating a Group and adding the User to it
4. Creating an Authentication Profile
5. Creating an Application Setting
6. Enabling Logging

Creating a User

To create a user:

1. Navigate to Directories | Internal Directories.
2. Select the Internal Domain where you would like to create the user.
3. Click New Address.
4. Complete the New Address form. See the Managing User Email Addresses page for further details.

Keep a note of the password set, as you will use this when setting up the scripts.

5. Click Save and Exit.

Adding the User to an Administrator Role

To add the user to an administrator role:

1. Navigate to Account | Roles.
2. Right click Administrator Role (e.g. Basic Administrator).
3. Select the Add Users to Role menu item.
4. Browse for the User created in the "Creating a User" section.
5. Select the Tick Box to the left of the user.
6. Click Add Selected Users.

Creating a Group and Adding the User

To create a group and add the user to it:

1. Navigate to Directories | Profile Groups 
2. Create a Group. See the Managing Groups page for further details.

Give the group a descriptive name (e.g. LogRhythm Admin).

3. With the group selected, click on the Build button.
4. Click Add Email Addresses.
5. Type the name of the User created in the "Creating a User" section.
6. Click Save and Exit.

Creating an Authentication Profile

2-Step authentication must be disabled for this authentication profile. If you have issues, contact our Support team.

To create an authentication profile:

1. Navigate to Services | Application | Authentication Profiles
2. Click on the New Authentication Profile button. See the Configuring an Authentication Profile page for further details.
3. Type a Description for the profile.
4. Set the Authentication TTL setting to "Never Expires". This makes sure that when you create your authentication token, it will not expire and impact the data collection of the app.
5. Leave all other settings as their default.
6. Click Save and Exit.

Creating an Application Setting

To create an application setting:

1. Navigate to  Services | Applications.
2. Click on the New Application Settings button. See the Configuring Application Settings page for further details.
3. Type a Description.
4. Select the Group you created in the "Creating a Group and Adding the User" task.
5. Select the Authentication Profile created in the "Creating an Authentication Profile" task.
6. Leave all other settings as their default.
7. Click Save and Exit.

Enabling Logging

To enable logging on your account:

1. Navigate to Account | Account Settings.
2. Expand the Enhanced Logging section.
3. Select the types of logs you want to enable. The choices are:

• • Inbound: Logs for messages from external senders to internal recipients.
  • Outbound: Logs for messages from internal senders to external recipients.
  • Internal: Logs for messages between internal domains.

4. Click Save.

Once these settings have been saved, the Mimecast MTA starts logging data. Log files should become available for download up to 30 minutes after that.

Configuring Data Collection Scripts

api_setup.py

Before proceeding to configure the integration for LogRhythm, ensure that you have obtained an API key for your organization.

Step 1: Register Application Integration

1. Log in to the Mimecast Administration Console.
2. Navigate to Integrations | API and Platform Integrations.
3. On the Available Integrations tab, locate the LogRhythm card.
4. Click on the Generate Keys button.
5. Enter a Description.
6. Click Next.
7. Enter a name in the Technical Point of Contact field
8. Enter the technical point of contact's email address in the Email field.
9. To stay informed about changes that could impact the API integration, select the Opt-in checkbox. 
10. Click Next.
11. Review the Summary page and click Add.
12. A slide-out panel appears.
13. Copy the Application ID and Application Key into a notepad for use later in this guide.
14. You will need to wait for at least 30 minutes before you can obtain an Access and Secret key.

Step 2: Generating Access and Secret Keys.

1. Click on the Your Application Integrations tab.
2. Click on the newly registered LogRhythm application integration entry.
3. A slide out panel appears.
4. Click on the Create Keys button.
5. The Create Keys wizard is displayed.
6. Enter the Email Address of the dedicated administrator account.

You'll need to know the dedicated user's domain or cloud password for the next step.

7. Click Next.
8. Using the Type dropdown, select an authentication method (Cloud or Domain).
9. Enter the Password for the dedicated user in the Password field.
10. Click Next.
11. Copy the Access Key and Secret Key into a notepad for use later in this guide.

Step 3: Adding the user to an Administrator Role 

The user account created in the "Creating a User" section of this article needs an Administrator role with permissions to Gateway | Tracking | Read. For full details, see the Administrator Role Permissions article.

To add a user to an administrator role: 

1. Navigate to Account | Roles 
2. Right click on the desired administrator role. 
3. Click Add Users to Role.
4. Browse for the User created in the "Creating a User" section of this article.
5. Select the tick box to the left of the user. 
6. Click Add Selected Users. 

Step 4: Configure the Integration 

1. Download the Mimecast Data Collection Scripts for LogRhythm from here.
2. Extract the .ZIP File to the location on your LogRhythm server where you want the scripts to execute.
3. Open a Command Prompt.
4. Change the Directory to where Python is installed:

cd C:\Python39

4. Start the Setup Utility:

C:\Python39\python.exe "D:\Mimecast Data Collector for LogRhythm\api_setup.py"

5. Enter the Mimecast Data Directory. This is the full path to the directory where you would like to store the data download from Mimecast.
6. Enter the System Monitor Read Path. This is the full path to the directory where the LogRhythm System Monitor will need to be configured to read MTA logs from.
7. Enter the Log Archive Directory. This is the full path to the directory where MTA logs will be moved for deletion. Deletion of files from the folder is done after a set number of days. The number of days is asked for as part of this workflow. Files are only moved into this folder, after they have been read fully by LogRhythm System Monitor.
8. Enter the Log Source Id of the Log Source that  was created and configured to read Mimecast MTA logs.
9. Enter the Max number of files that should exist with the System Monitor Read Path specified in the 2nd step. This limits the number of files that can exist in the System Monitor read path.
10. Enter the Number of Days that the scripts execution logs and MTA logs should be retained for.
11. Enter the Application Id and Application Key.
12. Enter the Access Key and Secret Key.
13. Enter the Email Address for Mimecast administrator that the access and secret keys are associated with.
14. The Base URL is automatically discovered and configured automatically for you

You should see a message indicating the config has been saved successfully:

Setting base url for {email_address}
Base URL discovered: https://{base_url}
Retrieving customer account code to finalise config...
Config saved successfully.

Creating a Scheduled Task to Execute the Data Collectors

To create a scheduled task:

1. Open the Windows Task Scheduler on the server hosting the data collection scripts.
2. Click on the Actions | Create Task menu item.
3. Complete the General tab with the following information:

Field / Option	Setting
Name	Provide a name that describes the task (e.g. Mimecast MTA Log Collection).
Run Whether User is Logged On or Not	Select this option.

4. Complete the Triggers tab:

• • Click on the New button.
  • Configure the Schedule Settings to meet your requirements.

We recommend running the data collection scripts daily, every 30 minutes for an indefinite time period.

• • Click on the OK button.

5. Complete the Actions tab:

• • Click on the New button.
  • Complete the dialog as follows:

Field / Option	Setting
Action	Select "Start a Program" from the drop-down list
Program / Script	Specify the path to the python executable (e.g. C:\Python27\python.exe).
Add Arguments (Optional)	Specify the path to the Mimecast script (e.g. "C:\Mimecast Data Collector for LogRhythm\Mimecast Data Collector for Log- Rhythm\siem_log_collector.py").

6. Click on the OK button.
7. Optionally configure any Conditions or Settings you want to apply.
8. Click on the OK button to save the task.
9. Repeat this process for the audit_log_collector.py & siem_cleanup scripts.

Once complete, the scripts will execute as scheduled. MTA and Audit logs will then downloaded to subfolders within the Mimecast data directory. When the siem_cleanup script runs, it will move files from the MTA folder within the Mimecast data directory, to the System Monitor read path directory. The Log Source should be configured to read from this folder. Once System Monitor has read the files with this directory, logs that have been read are then moved to the archive folder. The archive folder is cleared of logs that are older than the number of days specified in the steps above.

Configuring LogRhythm

LogRhythm requires a LogRhythm System Monitor Agent to collect the logs. The file being collected must be viewable on the host, with the agent using a standard file name path such as /var/log/logfile.txt or C:\logs\logfile.txt.

To configure LogRhythm:

1. Start the LogRhythm Console.
2. Select the Deployment Manager button.
3. Select the System Monitors tab.
4. Either:

• • Double-click the System Monitor Agent that will be collecting the audit logs.
  • Right-click on the System Monitor Agent and select the Properties menu item.

5. Select the Agent Settings tab.
6. Right-click anywhere in the Log Sources List.
7. Select New from the context menu.
8. Select the Basic Configuration tab.
9. Select Flat-File Mimecast Audit from the Log Message Source Type box.
   
   
   
10. Select the Flat File Settings tab.
11. Populate the boxes on the Flat File Settings tab with the following information from our example: 

• • File Path: C:\SIEMLogs\Audit
  • Date Parsing Format: Mimecast Audit Logs*
  • Is Directory must be checked
  • Recursion Depth may be set to 1 or higher
  • Compression Type must be set to none
  • Click OK in the Log Message Source Properties window.
  • Click OK in the System Monitor Agent Properties window.

The File Path for Audit logs should point to the the Mimecast Data directory and the Audit subfolder.

12. Repeat this process for the Mimecast Email Log Source, but the File Path needs to be set to the System Monitor read path that was specified when api_setup.py was executed.

Optionally Install Dashboards

Sample dashboards can be downloaded here. These can be uploaded to your LogRhythm server using the LogRhythm web portal.

Troubleshooting

The data collection scripts provided by Mimecast output a log file of activity for troubleshooting purposes. These logs are written to logs directory in the directory where the scripts are executed from, for example: C: \Mimecast Data Collector for LogRhythm\log. Logs are kept for the number of days specified when api_setup.py was executed and saved the configuration.

These logs should be used to diagnose issues where data is not being populated in the data directory. Mimecast support will require these logs to assist you with any issues.

The logs will not provide any insight into LogRhythm system agents. Please consult LogRhythm documentation and / or support for issues relating to the Flat File - System Agents.