Configuring Universal Forwarder to Splunk Enterprise

[jackjack]

Hi all,

I'm new to splunk so I hope I'm just missing a step or something. I've searched for a while and still am not sure what I'm doing wrong.

I have splunk enterprise running on one server. I have configured it to receive data via port 9997 through the "Forwarding and Receiving" settings page.

I have installed a Universal Forwarder on another server. I added a forward-server (side note: Can you pass in the group name via CLI or is it only editable in the outputs.conf file? I can't find the full options list) and verified it in the /etc/system/local/outputs.conf file. It is using the defaultGroup = default-autolb-group. I then added a monitor on /var/log. The commands:

./splunk add forward-server <host name or ip address>:<listening port>

./splunk add monitor /var/log

./splunk restart

This is where I'm confused. I created an indexer on the Enterprise named 'default-autolb-group' to capture the data but it does not populate this indexer. However, if I go to Apps > Search & Reporting, and filter by index=_internal, I see some info from the server where my universal forwarder is installed. The latest message was after a restart and is listing the cores, RAM, etc.

So data is coming through from the server but it's not going where I expect. What am I missing?

[richgalloway · Accepted solution]

It seems you've confused "index" and "indexer".  An indexer is a Splunk instance (a process on a server).  One sends data to an indexer by specifying it in outputs.conf.  Forwarders can automatically distribute data across a set of indexers, if you wish.  This set is called an "autolb group".  If you have a single indexer then you don't need to concern yourself with groups.

An index is where Splunk stores the data you send it.  It's analogous to a directory in your hard drive.  You tell Splunk which index to use in inputs.conf (not terribly intuitive, is it?).  You can have one inputs.conf file with many monitor statements in it and each monitored file/directory can be sent to a different index.  By default, data is sent to the index called "main".

Since you see data from the forwarder in _internal, we can presume monitored data is also getting indexed.  Search for index=main earliest=0 latest=+1y to find it.  This searches the main index over a wide range of time in case timestamps are incorrect.

[jackjack]

Thank you rich, that search was very helpful! I do see my data in there.

Now I'm having an issue where only certain log files are showed up. I tried using `sudo setfacl -R -m u:splunk:r /var/log` and have verified it with `getfacl /var/log/auth.log` (as I'm particularly interested in that file) but I'm still not seeing auth.log files come through. I will keep researching.

[jackjack]

I don't see an inputs.conf file on either server in the splunk /etc/system/local directory. I assumed one would be created on the forwarding server since I've added a few monitors. Has this file moved? Do I need to create it manually? 

[richgalloway]

The inputs.conf files can be in many places.  Use find or btool to find them.

find /opt -name inputs.conf

splunk btool --debug inputs list