Getting data into Unomaly means configuring the data source to send the log data to the Unomaly instance. Unomaly analyzes this data without specific parsers or predefined knowledge of the data format or structure. Because some ingestion methods are better suited for certain types of log data, choosing the right protocol or integration to send your data can improve how Unomaly learns and detects anomalies.

Where is the data coming from?

The options that you have for sending data to Unomaly depend on where the data is coming from: directly from the host machines that generate the data, from log collectors or log servers, from cloud services, or from other applications and technologies. For just about any data source, you can use one of our plugins (Logstash, Graylog, or Fluentd) to send their data to Unomaly. But you can also send the data directly to the Unomaly ingestion API using HTTP.

Direct from host machines

Most operating systems have built-in support or services for collecting and forwarding their own data. For example, Unix-based systems, VMWare, and networking devices have pre-installed versions of syslog to forward data to Unomaly.

Data source Data input options
Unix and Linux Unix-based systems have built-in support for syslog (syslogd, rsyslog, or syslog-ng) which you can configure to forward all your local logs. See the guidelines for sending data from Unix and Linux. If you use Logstash, Graylog, or Fluentd to collect your logs, see how to forward their data in the next section.
VMWare ESX Forward logs from your ESXi hosts using the built-in vmsyslogd service. See the guidelines for sending data from VMWare ESX.
Windows Windows EventLogs require an agent to collect and forward to Unomaly. For example, you can use nxLog to forward the data to syslog on Unomaly. See guidelines for sending Windows EventLogs to Unomaly using nxLog.

From log collectors and log servers

There are many standard agents that specialize in collecting log data and forwarding them to another platform for processing. If you’re using Logstash, Graylog, or Fluentd to aggregate your logs, use one of Unomaly’s plugins to forward that data to a Unomaly instance.

Data source Data input options
Logstash Forward Logstash events to the Unomaly ingestion API for analysis. See the Logstash plugin.
Graylog Forward streaming logs from Graylog to the Unomaly ingestion API for analysis. See the Graylog plugin.
Fluentd Forward data from Fluentd to the Unomaly ingestion API for analysis. See the Fluentd plugin.
Syslog servers This is not the recommended option for forwarding structured data. Syslog

From cloud services

Data source Data input options
AWS Cloudtrail Send JSON formatted CloudTrail logs directly to Unomaly’s ingestion API. See our guidelines for AWS Cloudtrail.
AWS Cloudwatch We provide lambda functions to push AWS Cloudwatch logs directly to Unomaly’s ingestion API. See our GitHub repo for AWS Cloudwatch.

From other applications and technologies

We provide integrations and guidelines to get data from many of the applications, devices, and services that make up a typical IT infrastructure. If you already use other data technologies or log management tools to collect and process your data, you can use one of our integrations to forward that data to Unomaly for analysis. For example, Unomaly provides integrations to analyze data from Docker, Splunk, and others.

Learning new systems

When systems are correctly configured to send data, Unomaly will detect the new systems and display them in the web interface. New systems are put into training, which means that Unomaly is learning the baseline, or normal behavior of the system. After training is complete, Unomaly begins its anomaly detection.

During the learning process, Unomaly continuously analyzes the generated data, creates profiles for the system, and updates the learnings database. These profiles capture the parameters and frequencies of the events to summarize the normal behavior of each system. Read more about “How Unomaly detects anomalies”.

Troubleshooting data input issues

How can you tell when you have an issue with getting data into Unomaly?

  • The new system doesn’t show up in Unomaly. New systems are added in training, so check to make sure you have selected to “Show systems in training”. If you still don’t see the new systems, check that the communications settings are configured to receive data and that the services that handle ingestion and queuing of the data are running.
  • Unomaly receives the data but it does not look correct. This may indicate an issue with the tokenization of the events in the data.
  • The system is still in training after more than two weeks. The most common reason for this is that the system does not produce enough logs for Unomaly to learn its normal behavior. You may consider manually taking systems out of training. Another reason may be that there is an issue tokenizing the structures of the events.

See Troubleshooting data ingestion.

Next steps

While Unomaly learns your data, you can learn more about how to Organize systems into groups.