Splunk App for Stream: How to configure a universal forwarder to monitor DNS and DHCP?

[phoenixdigital]

I'm having trouble getting the Splunk App for Stream working in a test environment with the following configuration:

1. Standalone Splunk Search Head and Indexer
2. Universal Forwarder (target for Splunk Stream)

I've tried a number of different configurations and even reinstalled Splunk from scratch, but still can't get this to work when following the documentation.

I tried using the deployment server, but was getting nowhere, so decided to manually configure the app on the Universal Forwarder, but still no luck.

The firewalls are off on both servers.

I am using the Splunk supplied SSL certs if that is an issue? Is there some flag that needs to be turned on to disable certificate verification? I used sslVerifyServerCert = false to be sure.

/opt/splunkforwarder/etc/apps/Splunk_TA_stream/local/inputs.conf

[streamfwd://streamfwd]
splunk_stream_app_location = https://xxx.xxx.xx.xx:8000/en-us/custom/splunk_app_stream/
stream_forwarder_id =
disabled = 0
sslVerifyServerCert = false

Then tested this with a curl command (from UF) as recommended on a previous Splunk Answers post.

[root@media local]# curl -k https://xxx.xxx.xx.xx:8000/en-us/custom/splunk_app_stream/ping                                                                                                                                                                                                                           
{"id": "appsmeta", "api_versions": {"ping": 1, "captureipaddresses": 1, "vocabularies": 1, "httpinputs": 1, "streams": 1, "streamforwardergroups": 1, "indexers": 1, "users": 1}, "dateLastUpdated": 1469655789359, "version": "6.6.0", "_key": "appsmeta", "_user": "nobody"}

My NICs are not named weirdly (match en pattern), so should work out of the box. Saw this in another Splunk Answers post

[root@media local]# ifconfig 
enp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.64.64  netmask 255.255.255.0  broadcast 192.168.64.255
        inet6 fe80::be5f:f4ff:fe89:3152  prefixlen 64  scopeid 0x20<link>
        ether bc:5f:f4:89:xx:xx txqueuelen 1000  (Ethernet)
        RX packets 2298867  bytes 1946057053 (1.8 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1590850  bytes 949878345 (905.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp4s1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether c8:3a:35:d9:xx:xx  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1  (Local Loopback)
        RX packets 365635  bytes 93720606 (89.3 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 365635  bytes 93720606 (89.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 52:54:00:26::xx:xx  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

The forwarder is successfully sending its data to the SH/IDX (well everything apart from Splunk Stream data)

[root@media local]# cat /opt/splunkforwarder/etc/apps/cfgd_fwd_to_idx/local/outputs.conf
[tcpout]
defaultGroup= indexers
indexAndForward= false

[tcpout:indexers]
server= xxx.xxx.xx.xx:9997
autoLB= true

I also have a configuration to ensure that _internal is also passed through to the IDX/SH. I can see data from the UF when I search on index=_internal

[root@media local]# cat /opt/splunkforwarder/etc/apps/cfgd_do_not_log_to_internal/local/outputs.conf 
# http://docs.splunk.com/Documentation/Splunk/latest/admin/outputsconf

[tcpout]

# If true, disables index filtering. Events for all indexes are then forwarded. Normally _internal indexes are not forwarded and are stored on the local Splunk instance
forwardedindex.filter.disable = true

# If set to true, data is indexed locally.
indexAndForward = false

I suppose from this point I'm not sure from the manual what I should do as I can't see this UF in the Splunk Stream interface on the Search Head?

I thought I should see it here at least shouldn't it?
http://imgur.com/a/86kRl

Where do I tell the UF that I want to monitor DNS and DHCP?

I've enabled the streams in "Configure Streams"
http://imgur.com/a/gvzn0

Creating a new group in Distributed Forwarder Management doesn't show my UF??
http://imgur.com/a/pXHEe

Any ideas or even a bit more detail in the documentation would be appreciated.

Edit: More information on Splunk Universal Forwarder server.

OS: Fedora 23
Kernel: 4.6.4-201.fc23.x86_64

Splunk running as root on all instances.

I am currently installing a standalone Splunk instance with Splunk App for Stream on this server to see if that will work purely for testing purposes.

[vshcherbakov_sp · Accepted solution]

What's the version of Stream app? Splunk app for Stream 6.6.0 has a bug where a symlink in Splunk_TA_stream package is not extracted correctly by Splunk. The workaround is to run sudo ./set_capturemode.sh agent from /etc/apps/Splunk_TA_stream directory. Only 64 bit linux is affected.

If that doesn't seem to be the problem you're experiencing, could you post the tail of <splunk_root>/var/log/splunk/streamfwd.log file?

[phoenixdigital]

Thats it thanks.

sudo ./set_capturemode.sh agent

This worked on both the standalone and Splunk Universal Forwarder instance.

[vshcherbakov_sp]

You're welcome! we're fixing this issue in the upcoming 6.6.1 maintenance release (should be available later this week or early next week)

[phoenixdigital]

Yep standalone fails as well 😞

[phoenixdigital]

Seeing this error also on standalone (which I also saw on distributed)
http://imgur.com/a/gVQLB

Starting to think maybe more recent versions of Linux are not supported or I am missing a config on my kernel (which is stock)?