Fluentd

Fluentd is an open source data collector for unified logging layer. Fluentd allows you to unify data collection and consumption for a better use and understanding of data. It can read/write log streams and supports different formats/protocols for doing so, including customizable plug-ins.

The Unomaly plugin for Fluentd

You can use the Unomaly plugin for Fluentd to send Fluent records to the Unomaly ingestion API. You can find the Unomaly plugin for Fluentd here.

Sending data to Fluentd on Unomaly

As a fallback option for data ingestion, Unomaly also runs with fluentd pre-installed on the instance. Fluentd listens to port 24224, which is the default fluentd forwarder port. This means that the port needs to be accessible through the firewall, if there is one separating Unomaly from your Docker instances.

Configuration

The main fluent configuration file is located at /DATA/fluentd/etc/fluent.conf. Please do not make any changes to this file since they will be overwritten on upgrades. Instead add your configurations to /DATA/fluentd/etc/conf.d/ as separate files.

The full description of fluentd’s configuration is described at the following linkhttps://docs.fluentd.org/v0.12/articles/config-file.

Examples

Receive from other Fluentd or Docker

The following configuration accepts data forwarded from other fluentd instances, such as Docker’s fluentd logging-driver or standard fluentd installations, and ingests the log data to Unomaly using the Unomaly plugin for Fluentd.

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>
<filter **.docker.**>
  @type record_transformer
  <record>
    hostname "${tag_parts[2]}"
  </record>
</filter>
<match **.docker.**>
  @type unomaly
  host https://127.0.0.1
  flush_interval 1s
  source_key hostname
  message_key log
  accept_self_signed_certs true
</match>

Specifically in this case, the following statements are declared:

• The statement @type forward means that this plugin is mainly used to receive event logs from other Fluentd instances, the fluent-cat command, or client libraries. This is by far the most efficient way to retrieve the records.
• The statement port 24224 designates the port on which fluentd should listen for data.
• The statement bind 0.0.0.0 means that it will listen to any network interface.
• The statement filter **.docker.** means that messages are transformed
  • In this case we re-write the hostname in the <record>-block.
  • The tag_parts[2] means we’re matching against the second index of the tag in the sender’s message.
• The statement match **.docker.** means that messages from docker matched.
  • The statement @type unomaly means that fluentd will look for a file called out_unomaly.rb in /DATA/fluentd/plugins and pass log data to it. The out_unomaly.rb plugin will ingest data into Unomaly.

Receive standard syslog data

The following example receives standard syslogs and ingests them into Unomaly.

<source>
    @type syslog
    @label @mystream
    port 51400
    bind 0.0.0.0
    tag system
</source>
<label @mystream>
    <match system.**>
      @type unomaly
      host https://172.16.238.1
      flush_interval 1s
      source_key hostname
      message_key log
      accept_self_signed_certs true
    </match>
</label>

The label directive may be used if you for instance have several sources.

You may add the following to the source section to switch to TCP syslogs (by default, UDP is used):

    protocol_type tcp

If you are having problems receiving syslog messages it might be because there are different formats for syslogs. You can change format in the <source> section using either of:

   message_format auto
   message_format rfc3164
   message_format rfc5424

Installing custom plugins

You can write your own plugins or find existing ones for fluentd and save them into /DATA/fluentd/plugins:

• Make sure that they are registered with the name you use in the fluent.conf
• Make sure that the source-code registers them with the same name.
• Make sure that the file’s file-name matches and has the right type (such as out_ for output type plugins).

It is also possible to install fluentd plugins using unomaly-fluent-gem:

sudo unomaly-fluent-gem install <plugin name>

For a list of available commands:

sudo unomaly-fluent-gem --help

After installing new plugins you need to restart the fluentd service to be able to use them in a configuration:

unomaly restart fluentd

Activating configuration changes

1. After changing configuration or installing/changing plugins, you need to restart fluentd:

    unomaly restart fluentd
   

2. You can view the stdout/log of fluentd by running:

    unomaly logs fluentd
   

   Or, you can run the following for tail-mode viewing (scrolls in real-time).

    unomaly logs fluentd -f
   

Debugging tips

To see whether data comes into fluentd at all, you can use for example:

<match **>
    @type stdout
</match>

This will print the message on the stdout of the running fluentd process.

You can use for instance fluent-cat (a fluentd tool) or simply logger (a standard linux syslog tool) to produce log message input for fluentd.

1. Example of using fluent-cat:

    echo '{"message":"hello world"}' | fluent-cat --host 10.164.0.7 --port 24224 --format json hello
   

2. Example of using logger:

    logger --server 10.164.0.7 --port 51400 '<6>Jan 13 12:34:56 sunworld myapp[31802]: [info] test logline in rfc3164'