API 1.0 Documentation - Get SIEM Logs

This article contains information on using the /api/audit/get-siem-logs API to retrieve MTA log files, required headers and parameters, token-based log retrieval, response types, and sample code for downloading logs.

Overview

This API endpoint can be used to get MTA logs. Logs are available up to seven days from current date, and use tokens to bookmark progress in logs' downloaded.

Considerations

• Unlike the majority of our endpoints this function returns an application/octet-stream stream of log data.
• Log data is written on every Mimecast server where the application for each log type runs and is made available for you every 30 minutes throughout the day.
• Log data is stored by Mimecast for seven days only, however once downloaded you can keep the data for as long as you require.
• Only one log file is returned per request and Mimecast uses a token based system for you to request the next available log.
• Tokens are returned in the mc-siem-token response header.
• The data type and date of each log downloaded is indicated in the Content-Disposition response header.

Prerequisites

• You are Developer, using Mimecast APIs to develop integrations with Mimecast.
• You have Mimecast Administrator role, with at least the Gateway | Tracking | Read permission.
• The requested log type(s) have been enabled via Account | Account Settings | Enhanced Logging  in the Mimecast Administration Console. 

URI

To use this endpoint you send a POST request to: /api/audit/get-siem-logs

Request headers

The following request headers must be included in your request:

Field	Description
Authorization	Please see API 1.0 Documentation - Authorization for more information on building the Authorization header.
x-mc-req-id	A randomly generated GUID, for example, 8578FCFC-A305-4D9A-99CB-F4D5ECEFE297
x-mc-app-id	The Application ID provided with your Registered API Application.
x-mc-date	The current date and time in the following format, for example, Tue, 24 Nov 2015 12:50:11 UTC

Request body

{
  "data":[
    {
      "type": "String",
      "token": "String",
      "fileFormat": "String",
      "compress": true
    }
  ]
}

Data

Field	Type	Required	Description
type	String	Required	The type of log file to request. Currently the only type available is MTA. However Mimecast may add new log types in the future.
token	String	Optional	Used to request the next available log file. The token value is found in the "mc-siem-token" response header of the previous request. If not included the earliest available log will be returned by the API.
fileFormat	String	Optional	Used to determine output format. Must be one of - json for JSON output and key_value for Pipe Delimited. Default setting is key_value.
compress	Boolean	Optional	Used to determine if response will be a text-based output, or a .zip file stream containing multiple log files. Default value is false.

Response

Response with data when compress option is false

When there are log files available this endpoint returns an application/octet-stream stream of data, with the following example response headers:

Content-Type: application/octet-stream
Content-Disposition: attachment; filename="delivery_20160705162902600.log"
Pragma: private, no-cache
Cache-control: private, must-revalidate, no-store
mc-siem-token: eNo9jUELgjAYQP_Ldx60xTTyVi1a...
Transfer-Encoding: chunked
Date: Mon, 11 Jul 2016 09:55:50 +0100
Connection: Keep-Alive

Where:

Field	Description
Content-Disposition	Describes the type of data returned. The format of the filename key follows the type_dateTime pattern.
mc-siem-token	The token value that should be used to request the next available log file.

Response with data when compress option is true

When there are log files available, this endpoint returns an application/octet-stream stream of a ZIP archive file.

Content-Type: application/octet-stream
Content-Disposition: attachment; filename="8578FCFC-A305-4D9A-99CB-F4D5ECEFE297.zip"
Pragma: private, no-cache
Cache-control: private, must-revalidate, no-store
mc-siem-token: eNo9jUELgjAYQP_Ldx60xTTyVi1a...
Transfer-Encoding: chunked
Date: Mon, 11 Jul 2016 09:55:50 +0100
Connection: Keep-Alive

Where:

Field	Description
Content-Disposition	Describes the type of data returned. The format of the filename uses the GUID presented in request header x-mc-req-id.
mc-siem-token	The token value that should be used to request the next available log file.

Response when no new logs are available

When all logs have been downloaded the response will conform to the API's standard JSON payload with the isLastToken boolean value found in the meta section. E.g.:

200
OK
Response:
{
     "meta": {
          "isLastToken":Boolean,
          "status":Number
     },
     "data":[],
     "fail":[]
}

Sample code

Sample code is provided to demonstrate how to use the API, and is not representative of a production application. 
To use the sample code: complete the required variables as described, populate the desired values in the request body, and execute in your favorite IDE. 
Please see API 1.0 Documentation - API 1.0 Global Base URLs to find the correct base URL to use for your account.

HTTP

POST {base_url}/api/audit/get-siem-logs
Authorization: MC {accesskKey}:{Base64 encoded signed Data To Sign}
x-mc-date: {dateTime}
x-mc-req-id: {unique id}
x-mc-app-id: {applicationId}
Content-Type: application/json
Accept: application/json
 
{
  "data":[
    {
      "type": "MTA",
      "token": "value of previous mc-siem-token response header",
      "fileFormat": "JSON",
      "compress": true
    }
  ]
}

Python

import base64
import hashlib
import hmac
import uuid
import datetime
import requests
 
# Setup required variables
base_url = "https://xx-api.mimecast.com"
uri = "/api/audit/get-siem-logs"
url = base_url + uri
access_key = "YOUR ACCESS KEY"
secret_key = "YOUR SECRET KEY"
app_id = "YOUR APPLICATION ID"
app_key = "YOUR APPLICATION KEY"
 
# Generate request header values
request_id = str(uuid.uuid4())
hdr_date = datetime.datetime.utcnow().strftime("%a, %d %b %Y %H:%M:%S") + " UTC"
 
# DataToSign is used in hmac_sha1
dataToSign = ':'.join([hdr_date, request_id, uri, app_key])
 
# Create the HMAC SHA1 of the Base64 decoded secret key for the Authorization header
hmac_sha1 = hmac.new(base64.b64decode(secret_key), dataToSign.encode(), digestmod=hashlib.sha1).digest()
 
# Use the HMAC SHA1 value to sign the hdrDate + ":" requestId + ":" + URI + ":" + appkey
sig = base64.b64encode(hmac_sha1).rstrip()
 
# Create request headers
headers = {
    'Authorization': 'MC ' + access_key + ':' + sig.decode(),
    'x-mc-app-id': app_id,
    'x-mc-date': hdr_date,
    'x-mc-req-id': request_id,
    'Content-Type': 'application/json'
}
 
payload = {
    "data":[
        {
            'type': 'MTA',
            'token': 'value of previous mc-siem-token response header',
            'fileFormat': 'JSON',
            'compress': true
        }
    ]
}
 
r = requests.post(url=url, headers=headers, data=str(payload))
 
print(r.text)

C#

static void Main(string[] args)
        {
            //Setup required variables
            string baseUrl = "https://xx-api.mimecast.com";
            string uri = "/api/audit/get-siem-logs";
            string accessKey = "YOUR ACCESS KEY";
            string secretKey = "YOUR SECRET KEY";
            string appId = "YOUR APPLICATION ID";
            string appKey = "YOUR APPLICATION KEY";
 
            //Generate request header values
            string hdrDate = System.DateTime.Now.ToUniversalTime().ToString("R");
            string requestId = System.Guid.NewGuid().ToString();
 
            //Create the HMAC SHA1 of the Base64 decoded secret key for the Authorization header
            System.Security.Cryptography.HMAC h = new System.Security.Cryptography.HMACSHA1(System.Convert.FromBase64String(secretKey));
 
            //Use the HMAC SHA1 value to sign the hdrDate + ":" requestId + ":" + URI + ":" + appkey
            byte[] hash = h.ComputeHash(System.Text.Encoding.Default.GetBytes(hdrDate + ":" + requestId + ":" + uri + ":" + appKey));
 
            //Build the signature to be included in the Authorization header in your request
            string signature = "MC " + accessKey + ":" + System.Convert.ToBase64String(hash);
 
            //Build Request
            System.Net.HttpWebRequest request = (System.Net.HttpWebRequest)System.Net.WebRequest.Create(baseUrl + uri);
            request.Method = "POST";
            request.ContentType = "application/json";
 
            //Add Headers
            request.Headers[System.Net.HttpRequestHeader.Authorization] = signature;
            request.Headers.Add("x-mc-date", hdrDate);
            request.Headers.Add("x-mc-req-id", requestId);
            request.Headers.Add("x-mc-app-id", appId);
 
            //Add request body
            //Create and write data to stream
            string postData = @"{
                 ""data"":[
                        {
                            ""type"": ""MTA"",
                            ""token"": ""value of previous mc-siem-token response header"",
                            ""fileFormat"": """JSON""",
                            ""compress"": true
                        }
                    ]
                }";
 
            byte[] payload = System.Text.Encoding.UTF8.GetBytes(postData);
 
            System.IO.Stream stream = request.GetRequestStream();
            stream.Write(payload, 0, payload.Length);
            stream.Close();
 
            //Send Request
            System.Net.HttpWebResponse response = (System.Net.HttpWebResponse)request.GetResponse();
 
            //Output response to console
            System.IO.StreamReader reader = new System.IO.StreamReader(response.GetResponseStream());
            string responseBody = "";
            string temp = null;
            while ((temp = reader.ReadLine()) != null)
            {
                responseBody += temp;
            };
            System.Console.WriteLine(responseBody);
            System.Console.ReadLine();
        }

PowerShell

#Setup required variables
$baseUrl = "https://xx-api.mimecast.com"
$uri = "/api/audit/get-siem-logs"
$url = $baseUrl + $uri
$accessKey = "YOUR ACCESS KEY"
$secretKey = "YOUR SECRET KEY"
$appId = "YOUR APPLICATION ID"
$appKey = "YOUR APPLICATION KEY"
 
#Generate request header values
$hdrDate = (Get-Date).ToUniversalTime().ToString("ddd, dd MMM yyyy HH:mm:ss UTC")
$requestId = [guid]::NewGuid().guid
 
#Create the HMAC SHA1 of the Base64 decoded secret key for the Authorization header
$sha = New-Object System.Security.Cryptography.HMACSHA1
$sha.key = [Convert]::FromBase64String($secretKey)
$sig = $sha.ComputeHash([Text.Encoding]::UTF8.GetBytes($hdrDate + ":" + $requestId + ":" + $uri + ":" + $appKey))
$sig = [Convert]::ToBase64String($sig)
 
#Create Headers
$headers = @{"Authorization" = "MC " + $accessKey + ":" + $sig;
                "x-mc-date" = $hdrDate;
                "x-mc-app-id" = $appId;
                "x-mc-req-id" = $requestId;
                "Content-Type" = "application/json"}
 
#Create post body
$postBody = "{
                 ""data"":[
                        {
                            ""type"": ""MTA"",
                            ""token"": ""value of previous mc-siem-token response header""
                            ""fileFormat"": """JSON""",
                            ""compress"": true
                        }
                    ]
                }"
 
#Send Request
$response = Invoke-RestMethod -Method Post -Headers $headers -Body $postBody -Uri $url
 
#Print the response
$response

Java

public static void main(String[] args) throws java.io.IOException, java.security.NoSuchAlgorithmException, java.security.InvalidKeyException {
 
        //set up variables for request
        String baseUrl = "https://xx-api.mimecast.com";
        String uri = "/api/audit/get-siem-logs";
        String url = "https://" + baseUrl + uri;
        String accessKey = "YOUR ACCESS KEY";
        String secretKey = "YOUR SECRET KEY";
        String appId = "YOUR APPLICATION ID";
        String appKey = "YOUR APPLICATION KEY";
 
        //create URL object
        java.net.URL obj = new java.net.URL(url);
 
        // set guid for x-mc-req-id header
        String guid = java.util.UUID.randomUUID().toString();
 
        // set date for x-mc-date header
        java.text.SimpleDateFormat sdf = new java.text.SimpleDateFormat("EEE, d MMM yyyy HH:mm:ss z");
        sdf.setTimeZone(java.util.TimeZone.getTimeZone("UTC"));
        String date = sdf.format(new java.util.Date());
 
        //create signature for the Authorization header
        String dataToSign = date + ":" + guid + ":" + uri + ":" + appKey;
        String hmacSHA1 = "HmacSHA1";
        javax.crypto.spec.SecretKeySpec signingKey = new javax.crypto.spec.SecretKeySpec(org.apache.commons.codec.binary.Base64.decodeBase64(secretKey.getBytes()), hmacSHA1);
        javax.crypto.Mac mac = javax.crypto.Mac.getInstance(hmacSHA1);
        mac.init(signingKey);
        String sig = new String(org.apache.commons.codec.binary.Base64.encodeBase64(mac.doFinal(dataToSign.getBytes())));
 
        // create request object
        javax.net.ssl.HttpsURLConnection con = (javax.net.ssl.HttpsURLConnection) obj.openConnection();
 
        //set request type to POST
        con.setRequestMethod("POST");
        con.setDoOutput(true);
 
        //add reuqest headers
        con.setRequestProperty("Authorization", "MC " + accessKey + ":" + sig);
        con.setRequestProperty("x-mc-req-id", guid);
        con.setRequestProperty("x-mc-app-id", appId);
        con.setRequestProperty("x-mc-date", date);
        con.setRequestProperty("Content-Type", "application/json");
        con.setRequestProperty("Accept", "application/json");
 
        //Add post body to the request
        String postBody = "{\n" +
                "    \"data\":[\n" +
                "        {\n" +
                "            \"type\": \"MTA\",\n" +
                "            \"token\": \"value of previous mc-siem-token response header",\n" +
                "            \"fileFormat\": \"JSON\",\n" +
                "            \"compress\": true\n" +
                "        }\n" +
                "    ]\n" +
                "}";
        java.io.OutputStream os = con.getOutputStream();
        os.write(postBody.getBytes("UTF-8"));
        os.close();
 
        //process response
        java.io.BufferedReader in = new java.io.BufferedReader(
                new java.io.InputStreamReader(con.getInputStream()));
        String inputLine;
        StringBuffer response = new StringBuffer();
 
        while ((inputLine = in.readLine()) != null) {
            response.append(inputLine);
        }
        in.close();
 
        //return result
        java.lang.System.out.println(response.toString());
    }

See Also...

• API 1.0 Documentation
• API 1.0 Documentation - Understanding SIEM Logs
• API 1.0 Documentation - Downloading SIEM Logs