I initially installed Splunk on a Windows 2008 R2 server and configured its splunktcp receiver to listen on port 9997 and in almost no time, the Windows systems, running SUF, appeared in Splunk Search. Over a 30 minute period, not once did I see anything in the logs from the Linux system itself. I reviewed the Management configuration options, and I just couldn't find anything that would prevent the indexer/database from showing/accepting Linux logs.
I reviewed tcpdump and the splunk logs. Absolutely nothing out of the ordinary from what I could see.
As a test, I removed splunk from the Windows 2008 system and installed it directly on the RHEL 5.6 system, and pointed all SUF to the Linux box. Still no change.
I'm baffled why every other log and indicator shows the RHEL box communicating, but the splunk database simply does not show any Linux log activity.
Searches for the IP, hostname, or even cron, reveal zero hits.
I have the most recent (within the last couple of months) of Server and the Universal Forwarder.
tcpdump you could see connections from the forwarders to the indexer? And with, say
netstat, you could see established connections.
Did you check that you have permissions to read the logs?
Most stuff under /var/log/ is root owned. If you installed Splunk as 'root', you should have no problem reading them, but if you've installed Splunk to run as 'splunk'...
Other interesting stuff to test;
To which index are you writing the linux logs? Check inputs.conf on the forwarder.
Does the index exist on the indexer? Check Manager -> indexes on the indexer
Do you have permissions to read this index? Check Manager -> Account settings -> roles -> your_role
Is this index searched by default? same as above.
Are the clocks in sync?
Have you searched for 'all time' and let it run?
Are your timestamps parsed correctly? Check for errors in Status -> Server Activity -> Splunkd activity overview. Bottom of page.
A lot of stuff in /var/log will be owned by root, so you'll have to make some changes, either;
a) configure logrotate to set the permissions on the files so that splunk can read them (rw----r--), but this will make those files readable to everyone.
b) configure the system so that the files have 'splunk' group membership (root/splunk) instead of (root/root) and set the file permissions to rw-r----- through logrotate or the logging application.
c) give the splunk account membership to the 'root' group. However, this could have implications outside the /var/log context.
I think that option B) would be best for most cases.
Please let us know how you progress.
but if you installed Splunk to run as splunk.....What? You didn't really elaborate on this. If you're running splunk as splunk, how can you index stuff in /var/log?
There is inherent difference between the setup of the Universal Forwarder in MS Windows and Linux. When you run the MSI installer on MS Windows you are prompted for the Indexer's identity and also for the type of default data that you want to monitor; i.e. Windows Event Logs for Application, Security, etc.
In Linux, however, you are not prompted for the identity of the Splunk Indexer or the data inputs.
Since you indicate that the Universal Forwarder, on the Linux system, is communicating with the Splunk Indexer, then can we assume that you've configured your outputs.conf? If so, have you defined the data that you would like to monitor in your inputs.conf?
If you have not done the latter, create the inputs.conf file under $SPLUNK_HOME/etc/apps/search/local and bounce the Forwarder. Here is an example of data that will get you started.
[monitor:///var/log/secure] disabled = false sourcetype = linux_secure [monitor:///var/log/messages] disabled = false sourcetype = syslog
I had this problem, I tried everything from several sources, the issue was resolved by getting the right permissions on the /var/log/secure file or files if logs are rolled locally. I had to open a ticket to splunk and send the diag file from the forwarding server. All they could tell me it was a permissions issue. As splunk did not have access to the files. I noticed other logs could be obtained fine from the /var/logs folder like update.log I could see that the permissions on that were -rw-r--r-- so I changed from -rw------- , the way to do this is chmod 644 as root, then you will have -rw-r--r-- I didn't even have to restart the forwarder, it instantly started working.