Since release 2.22, Unomaly supports fetching log data from the AWS service CloudWatch. This support was added by using the third party software fluentd, which is a general data collection framework that supports plugins. The fluentd component runs on the Unomaly instance in parallel with the Unomaly software. An input plugin for fluentd will be used to fetch log data from CloudWatch, and an output plugin for fluentd will be used to send this log data locally on the Unomaly instance to the Unomaly software.

With this setup, Unomaly supports fetching log data from CloudWatch in the same AWS account as the Unomaly instance. This setup also supports running the Unomaly instance in one AWS account while fetching log data from CloudWatch belonging to one or more different AWS account(s).


Instructions

The following instructions describe how to:

  • Configure the necessary permissions in AWS.
  • Enable and configure fluentd on the Unomaly instance.

Configure AWS permissions

Unomaly running on same AWS with Cloudwatch

If Unomaly is running in the same AWS account as the CloudWatch log data:

  1. Create a new IAM role by selecting the IAM service in the AWS console, then Roles in the left hand pane and Create role.

  2. Select “EC2” as the Trusted entity type.

  3. Attach the existing policy CloudWatchLogsReadOnlyAccess to this role.

  4. Name the role unomaly-ec2-cloudwatch-readonly-iam-ec2role (or a name that matches an established naming standard).

  5. Attach this IAM role to the EC2 instance running Unomaly.

Unomaly running on separate AWS from CloudWatch

If Unomaly is running in one AWS account while CloudWatch log data is located in one or more different AWS account(s), permissions needs to be configured in each AWS accounts.

For each AWS account hosting CloudWatch log data:

  1. Create a new IAM role by selecting the IAM service in the AWS console, then Roles in the left hand pane and Create role.

  2. Set the Trusted entity type to Another AWS account and enter the Account ID of the AWS account hosting the Unomaly instance.

  3. Attach the existing policy CloudWatchLogsReadOnlyAccess to this role.

  4. Name the role unomaly-account-cloudwatch-readonly-iam-role (or a name that matches an established naming standard).

For the AWS account hosting the Unomaly instance:

  1. Create a new IAM policy by selecting the IAM service in the AWS console, then Policies in the left hand pane and Create policy.

  2. Select Create Your Own Policy in the second step.

  3. Name the policy unomaly-ec2-assumerole-iam-policy (or a name that matches an established naming standard).

  4. Paste the text below into the textbox Policy Document:

    {
    "Version": "2012-10-17",
    "Statement": {
    "Effect": "Allow",
    "Action": "sts:AssumeRole",
    "Resource": "arn:aws:iam:: * :role/<unomaly-account-cloudwatch-readonly-iam-role>"
       }
    }
    

    The parameters * and <unomaly-account-cloudwatch-readonly-iam-role> are values specific to your account. In this example the AWS account ID is specified as a wildcard to allow the EC2 instance to AssumeRole to gather data from CloudWatch in multiple AWS accounts. For ease of setup, we recommend that the role name is shared between the AWS accounts if possible.

  5. Create a new IAM role by selecting the IAM service in the AWS console, then Roles in the left hand pane and Create role.

  6. Select “EC2” as the Trusted entity type.

  7. Attach the previously created policy “unomaly-ec2-assumerole-iam-policy” with this role.

  8. Name the role unomaly-ec2-assumerole-iam-ec2role (or a name that matches an established naming standard).

  9. Attach the previously created IAM role to the EC2 instance running Unomaly.

Enable fluentd on the Unomaly instance

Use the Unomaly console to install the latest version of the fluentd Unomaly output plugin that will send events ingested from CloudWatch to Unomaly:

  1. From the main SSH menu, enter choice ‘7’ to exit to system shell.

  2. Install the latest version of the fluentd output plugin for Unomaly:

    sudo cp /opt/unomaly/fluentd/fluentd-default/plugins/out_unomaly.rb /DATA/fluentd/plugins/
    
  3. Install the fluentd configuration file.

    sudo cp /opt/unomaly/fluentd/fluentd-default/etc/fluent.conf /DATA/fluentd/etc/
    

Configure fluentd to ingest CloudWatch log data

  1. Edit the fluentd configuration file /DATA/fluent/etc/fluent.conf and add the content of the relevant paragraphs below.

    If Unomaly should ingest log data from CloudWatch in the same AWS account:

    <source>
      @type cloudwatch_ingest
      tag cloudwatch
      region eu-west-1
      sts_enabled false
      #aws_logging_enabled false
      log_group_name_prefix /var/log/syslog
      log_stream_name_prefix foo
      # exclude log groups that start with a captial
      #log_group_exclude_regexp [^A-Z]{1}.*
      state_file_name /tmp/cloudwatch-source1.state
      interval 5
      # Time to wait between error conditions before retry
      api_interval 5
      error_interval 5
      # Number of events to fetch in any given iteration
      #limit_events 10000
      # Do not fetch events before this time (UNIX epoch, milliseconds)
      # Generate with date +%s000
      event_start_time 1510567048000
      # When true fetch the oldest logs first
      oldest_logs_first false
      # Fluentd may throw an exception if a blank event is emitted
      drop_blank_events true
      <parse>
        @type cloudwatch_ingest
        expression /^(?<message>.+)$/
        time_format %Y-%m-%d %H:%M:%S.%L
        event_time true
        # inject the group name into the record
        inject_group_name true
        # inject the stream name into the record
        inject_stream_name true
      </parse>
    </source>
    
    <match cloudwatch>
      @type unomaly
      host 127.0.0.1
      port 5514
      system_field log_stream_name / log_group_name
      message_field message
    </match>
    

    If Unomaly should ingest log data from CloudWatch in another AWS account:

    <source>
      @type cloudwatch_ingest
      tag cloudwatch
      region eu-west-1
      sts_enabled true
      sts_arn arn:aws:iam::<accountID>:role/unomaly-account-cloudwatch-readonly-iam-role
      sts_session_name fluentd
      #aws_logging_enabled false
      log_group_name_prefix system-logs
      log_stream_name_prefix foo
      # exclude log groups that start with a captial
      #log_group_exclude_regexp [^A-Z]{1}.*
      state_file_name /tmp/cloudwatch-source2.state
      interval 5
      # Time to wait between error conditions before retry
      api_interval 5
      error_interval 5
      # Number of events to fetch in any given iteration
      #limit_events 10000
      # Do not fetch events before this time (UNIX epoch, miliseconds)
      # Generate with date +%s000
      event_start_time 1510567048000
      # When true fetch the oldest logs first
      oldest_logs_first false
      # Fluentd may throw an exception if a blank event is emitted
      drop_blank_events true
      <parse>
        @type cloudwatch_ingest
        expression /^(?<message>.+)$/
        time_format %Y-%m-%d %H:%M:%S.%L
        event_time true
        # inject the group name into the record
        inject_group_name true
        # inject the stream name into the record
        inject_stream_name true
      </parse>
    </source>
    
    
    <match cloudwatch>
      @type unomaly
      host 127.0.0.1
      port 5514
      system_field log_stream_name / log_group_name
      message_field message
    </match>
    
  2. Restart fluentd to apply the configuration:

    $ unomaly restart fluentd
    

Verify the configuration

Tail the fluentd.log file to verify that the CloudWatch plugin is able to connect to AWS and ingest the log data:

$ tail -f /DATA/unomaly_logs/fluentd.log

Example outputs

Example output from the fluentd.log file when successfully ingesting log data from CloudWatch in the same AWS account:

[info]: #0 Starting fluentd-plugin-cloudwatch-ingest
[warn]: #0 api_interval is deprecated for error_interval
[info]: #0 Configured fluentd-plugin-cloudwatch-ingest
[info]: #0 starting fluentd worker pid=10 ppid=1 worker=0
[info]: #0 Started fluentd-plugin-cloudwatch-ingest
[info]: #0 Working in region eu-west-1
[info]: #0 Using local instance IAM role for authentication
[warn]: #0 No state file  Creating a new one.
[info]: #0 fluentd worker is now running worker=0
[info]: #0 Saved state to /tmp/cloudwatch-source1.state
[info]: #0 Obtaining exclusive lock on state file /tmp/cloudwatch-source1.state
[info]: #0 Loaded 0 groups from /tmp/cloudwatch-source1.state
[info]: #0 Found 1 log groups
[info]: #0 Found 1 streams for /var/log/syslog
[info]: #0 processing stream: i-086d2e3fceb81281f
[info]: #0 Saving state
[info]: #0 Saved state to /tmp/cloudwatch-source1.state
[info]: #0 2951 events processed.
[info]: #0 Pausing for 5.0
[info]: #0 Obtaining exclusive lock on state file /tmp/cloudwatch-source1.state
[info]: #0 Found 1 log groups
[info]: #0 Found 1 streams for /var/log/syslog
[info]: #0 processing stream: i-086d2e3fceb81281f
[info]: #0 Saving state
[info]: #0 Saved state to /tmp/cloudwatch-source1.state
[info]: #0 0 events processed.
[info]: #0 Pausing for 5.0
[info]: #0 Obtaining exclusive lock on state file /tmp/cloudwatch-source1.state
[info]: #0 Found 1 log groups
[info]: #0 Found 1 streams for /var/log/syslog
[info]: #0 processing stream: i-086d2e3fceb81281f
[info]: #0 Saving state
[info]: #0 Saved state to /tmp/cloudwatch-source1.state
[info]: #0 0 events processed.
[info]: #0 Pausing for 5.0

Example output from the fluentd.log file when successfully ingesting log data from CloudWatch in another AWS account:

[info]: #0 Starting fluentd-plugin-cloudwatch-ingest
[warn]: #0 api_interval is deprecated for error_interval
[info]: #0 Configured fluentd-plugin-cloudwatch-ingest
[info]: #0 starting fluentd worker pid=12 ppid=1 worker=0
[info]: #0 Started fluentd-plugin-cloudwatch-ingest
[info]: #0 Working in region eu-west-1
[info]: #0 Using STS for authentication with source account ARN:
arn:aws:iam::123456789012:role/unomaly-account-cloudwatch-readonly-iam-role, session name: fluentd
[info]: #0 fluentd worker is now running worker=0
[warn]: #0 No state file  Creating a new one.
[info]: #0 Saved state to /tmp/cloudwatch-source2.state
[info]: #0 Obtaining exclusive lock on state file /tmp/cloudwatch-source2.state
[info]: #0 Loaded 0 groups from /tmp/cloudwatch-source2.state
[info]: #0 Found 1 log groups
[info]: #0 Found 1 streams for system-logs
[info]: #0 processing stream: jumphost
[info]: #0 Saving state
[info]: #0 Saved state to /tmp/cloudwatch-source2.state
[info]: #0 3399 events processed.
[info]: #0 Pausing for 5.0
[info]: #0 Obtaining exclusive lock on state file /tmp/cloudwatch-source2.state
[info]: #0 Found 1 log groups
[info]: #0 Found 1 streams for system-logs
[info]: #0 processing stream: jumphost
[info]: #0 Saving state
[info]: #0 Saved state to /tmp/cloudwatch-source2.state
[info]: #0 0 events processed.
[info]: #0 Pausing for 5.0
[info]: #0 Obtaining exclusive lock on state file /tmp/cloudwatch-source2.state
[info]: #0 Found 1 log groups
[info]: #0 Found 1 streams for system-logs
[info]: #0 processing stream: jumphost
[info]: #0 Saving state
[info]: #0 Saved state to /tmp/cloudwatch-source2.state
[info]: #0 1 events processed.
[info]: #0 Pausing for 5.0