Fluentd

Fluentd is an open source data collector for unified logging layer. Fluentd allows you to unify data collection and consumption for a better use and understanding of data. It can read/write log streams and supports different formats/protocols for doing so, including customizable plug-ins.

Ingesting to Unomaly

Unomaly comes with fluentd pre-installed. It listens to port 24224, which is the default fluentd forwarder port. This means that the port needs to be accessible through the firewall, if there is one separating Unomaly from your Docker instances.

Configuration

The configuration file is stored at /DATA/fluentd/etc/fluent.conf. You may modify the contents to your needs.

The full description of fluentd’s configuration is described here.

Example: Receive from other Fluentd or Docker

The following configuration accepts data forwarded from other fluentd instances, such as Docker’s fluentd logging-driver or standard fluentd installations, and ingests the log data to Unomaly using the unomaly plugin.

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>
<filter **.docker.**>
  @type record_transformer
  <record>
    hostname "${tag_parts[2]}"
  </record>
</filter>
<match **.docker.**>
  @type unomaly
  host https://127.0.0.1
  flush_interval 1s
  source_key hostname
  message_key log
  accept_self_signed_certs true
</match>

Specifically in this case, the following statements are declared:

• The statement @type forward means that this plugin is mainly used to receive event logs from other Fluentd instances, the fluent-cat command, or client libraries. This is by far the most efficient way to retrieve the records.
• The statement port 24224 designates the port on which fluentd should listen for data.
• The statement bind 0.0.0.0 means that it will listen to any network interface.
• The statement filter **.docker.** means that messages are transformed
  • In this case we re-write the hostname in the <record>-block.
  • The tag_parts[2] means we’re matching against the second index of the tag in the sender’s message.
• The statement match **.docker.** means that messages from docker matched.
  • The statement @type unomaly means that fluentd will look for a file called out_unomaly.rb in /DATA/fluentd/plugins and pass log data to it. The out_unomaly.rb plugin will ingest data into Unomaly.

Example: Receive standard syslogs

The following example receives standard syslogs and ingests them into Unomaly.

<source>
    @type syslog
    @label @mystream
    port 51400
    bind 0.0.0.0
    tag system
</source>
<label @mystream>
    <match system.**>
      @type unomaly
      host https://172.16.238.1
      flush_interval 1s
      source_key hostname
      message_key log
      accept_self_signed_certs true
    </match>
</label>

The label directive may be used if you for instance have several sources.

You may add the following to the source section to switch to TCP syslogs (by default, UDP is used):

    protocol_type tcp

If you are having problems receiving syslog messages it might be because there are different formats for syslogs. You can change format in the <source> section using either of:

   message_format auto
   message_format rfc3164
   message_format rfc5424

Installing custom plugins

You can write your own plugins or find existing ones for fluentd and save them into /DATA/fluentd/plugins:

• Make sure that they are registered with the name you use in the fluent.conf
• Make sure that the source-code registers them with the same name.
• Make sure that the file’s file-name matches and has the right type (such as out_ for output type plugins).

Activating configuration changes

1. After changing configuration or installing/changing plugins, you need to restart fluentd:

    unomaly restart fluentd
   

2. You can view the stdout/log of fluentd by running:

    unomaly logs fluentd
   

   Or, you can run the following for tail-mode viewing (scrolls in real-time).

    unomaly logs fluentd -f
   

Debugging tips

To see whether data comes into fluentd at all, you can use for example:

<match **>
    @type stdout
</match>

This will print the message on the stdout of the running fluentd process.

You can use for instance fluent-cat (a fluentd tool) or simply logger (a standard linux syslog tool) to produce log message input for fluentd.

1. Example of using fluent-cat:

    echo '{"message":"hello world"}' | fluent-cat --host 10.164.0.7 --port 24224 --format json hello
   

2. Example of using logger:

    logger --server 10.164.0.7 --port 51400 '<6>Jan 13 12:34:56 sunworld myapp[31802]: [info] test logline in rfc3164'