Splunk

Splunk is an aggregator, search engine and dashboarding solution for log data. This integration utilizes the real-time search function of the Splunk API, continuously receives data via that socket, and pushes the data towards the Unomaly analysis engine. This integration allows for plug-and-play real-time machine learning and anomaly detection by Unomaly on data sent to Splunk.

Install the Unomaly Splunk transport

1. Download the transport file: Unomaly Splunk Transport 0.1.2

2. Use SFTP to copy the package to the Unomaly instance.

3. Use SSH to connect to the Unomaly instance.

4. Find the uploaded file and run the following command to start the install process:

sudo sh unomaly-[transportname]-[version].run

5. Follow the installation instructions presented in the command line.

Configure the transport

Transport configuration files are located in /DATA/unomaly_transports/splunktransport.cfg-default. We don't recommend editing this file directly. Instead, make a copy of the file for your configuration changes.

1. Copy the file for your configuration changes:

cp /DATA/unomaly_transports/splunktransport.cfg-default /DATA/unomaly_transports/splunktransport.cfg

2. Customize the paramers of the configuration file to suit your Splunk environment:

[transport]
exporturl=https://[Splunk IP]:8089/servicesNS/admin/search/search/jobs/export
query=search index=main
username=admin
password=changeme

The query option may need to altered is Splunk is configured for multiple indices. The specified username may need to be reflected in the URL. The user also needs to have the privilege rtsearch to be able to perform these queries. This privilage is included in the default groups power, we recommend if you are specifying a separate user for this activity.

Verify the data

1. To verify that data is being extracted correctly from Splunk, you can start the transport in debug mode to output all the data to the terminal.

sudo docker run --net=host --rm -it unomaly/pyunomaly transportd --debug

The command should produce output similar to:

INFO: __init__ --- Loading transports from path /DATA/unomaly_transports (transportd.py:27)
INFO: __init__ --- Initializing SplunkTransport... (transportbase.py:33)
INFO: run --- Running with pid=16771... (transportd.py:55)
INFO: _new_conn --- Starting new HTTPS connection (1): 10.8.0.182 (connectionpool.py:635)
DEBUG: _make_request --- "POST /servicesNS/admin/search/search/jobs/export HTTP/1.1" 200 None (connectionpool.py:344)
[application data]
^CExiting
INFO: __del__ --- Stopping SplunkTransport... (transportbase.py:50)

2. To stop the service, press Control-C.

If everything is running, you will see your Splunk data being output after the initialization procedure. If issues with authentication or URL enpoints arise, they will be displayed:

DEBUG: _make_request --- "POST /servicesNS/admin/search/search/jobs/export HTTP/1.1" 401 53 (connectionpool.py:344)
ERROR: run --- Invalid response from server: {"messages":[{"type":"ERROR","text":"Unauthorized"}]} (splunktransport.py:59)
ERROR: run --- Transport SplunkTransport with pid=5716 terminated! Exitcode=0 (transportd.py:54)

The above error indicates wrong user/password combination. If the text entry in the log says "You do not have the permission to spawn real-time searches", this indicates that the configured user does not have the rtsearch privilege.

3. Restart transportd:

unomaly restart transportd

4. Verify in the Unomaly interface that the logs are being received properly.