Fluentd
Fluentd is an open source data collector for unified logging layer. Fluentd allows you to unify data collection and consumption for a better use and understanding of data. It can read/write log streams and supports different formats/protocols for doing so, including customizable plug-ins.
The Unomaly plugin for Fluentd
You can use the Unomaly plugin for Fluentd to send Fluent records to the Unomaly ingestion API. You can find the Unomaly plugin for Fluentd here.
Sending data to Fluentd on Unomaly
As a fallback option for data ingestion, Unomaly also runs with fluentd pre-installed on the instance. Fluentd listens to port 24224
, which is the default fluentd forwarder port. This means that the port needs to be accessible through the firewall, if there is one separating Unomaly from your Docker instances.
Configuration
The main fluent configuration file is located at /DATA/fluentd/etc/fluent.conf
. Please do not make any changes to this file since they will be overwritten on upgrades.
Instead add your configurations to /DATA/fluentd/etc/conf.d/
as separate files.
The full description of fluentd’s configuration is described at the following linkhttps://docs.fluentd.org/v0.12/articles/config-file.
Examples
Receive from other Fluentd or Docker
The following configuration accepts data forwarded from other fluentd instances, such as Docker’s fluentd logging-driver or standard fluentd installations, and ingests the log data to Unomaly using the Unomaly plugin for Fluentd.
<source>
@type forward
port 24224
bind 0.0.0.0
</source>
<filter **.docker.**>
@type record_transformer
<record>
hostname "${tag_parts[2]}"
</record>
</filter>
<match **.docker.**>
@type unomaly
host https://127.0.0.1
flush_interval 1s
source_key hostname
message_key log
accept_self_signed_certs true
</match>
Directive | Description |
---|---|
source |
Decides which interface fluentd should use to read data. |
filter |
Allows you to define the custom filters that modify event streams. |
match |
Tells fluentd what to do. |
Specifically in this case, the following statements are declared:
- The statement
@type forward
means that this plugin is mainly used to receive event logs from other Fluentd instances, the fluent-cat command, or client libraries. This is by far the most efficient way to retrieve the records. - The statement
port 24224
designates the port on which fluentd should listen for data. - The statement
bind 0.0.0.0
means that it will listen to any network interface. - The statement
filter **.docker.**
means that messages are transformed- In this case we re-write the hostname in the
<record>
-block. - The
tag_parts[2]
means we’re matching against the second index of thetag
in the sender’s message.
- In this case we re-write the hostname in the
- The statement
match **.docker.**
means that messages from docker matched.- The statement
@type unomaly
means that fluentd will look for a file calledout_unomaly.rb
in/DATA/fluentd/plugins
and pass log data to it. Theout_unomaly.rb
plugin will ingest data into Unomaly.
- The statement
Receive standard syslog data
The following example receives standard syslogs and ingests them into Unomaly.
<source>
@type syslog
@label @mystream
port 51400
bind 0.0.0.0
tag system
</source>
<label @mystream>
<match system.**>
@type unomaly
host https://172.16.238.1
flush_interval 1s
source_key hostname
message_key log
accept_self_signed_certs true
</match>
</label>
The label
directive may be used if you for instance have several sources.
You may add the following to the source section to switch to TCP syslogs (by default, UDP is used):
protocol_type tcp
If you are having problems receiving syslog messages it might be because there are different formats for syslogs. You can change format in the <source>
section using either of:
message_format auto
message_format rfc3164
message_format rfc5424
Installing custom plugins
You can write your own plugins or find existing ones for fluentd and save them into /DATA/fluentd/plugins
:
- Make sure that they are registered with the name you use in the
fluent.conf
- Make sure that the source-code registers them with the same name.
- Make sure that the file’s file-name matches and has the right type (such as
out_
for output type plugins).
It is also possible to install fluentd plugins using unomaly-fluent-gem
:
sudo unomaly-fluent-gem install <plugin name>
For a list of available commands:
sudo unomaly-fluent-gem --help
After installing new plugins you need to restart the fluentd service to be able to use them in a configuration:
unomaly restart fluentd
Activating configuration changes
-
After changing configuration or installing/changing plugins, you need to restart fluentd:
unomaly restart fluentd
-
You can view the stdout/log of fluentd by running:
unomaly logs fluentd
Or, you can run the following for tail-mode viewing (scrolls in real-time).
unomaly logs fluentd -f
Debugging tips
To see whether data comes into fluentd at all, you can use for example:
<match **>
@type stdout
</match>
This will print the message on the stdout of the running fluentd process.
You can use for instance fluent-cat
(a fluentd tool) or simply logger
(a standard linux syslog tool) to produce log message input for fluentd.
-
Example of using fluent-cat:
echo '{"message":"hello world"}' | fluent-cat --host 10.164.0.7 --port 24224 --format json hello
-
Example of using logger:
logger --server 10.164.0.7 --port 51400 '<6>Jan 13 12:34:56 sunworld myapp[31802]: [info] test logline in rfc3164'
Did this article help you?
Thank you for the feedback!