Hi. I've installed splunk on two machines; one is the server I want to coalesce data from various sources, and the other is one of the sources. The machine I want to monitor is a typical linux webserver running apache. I've installed the universal forwarder on my webserver. I configured it to talk to the main splunk machine, and I've told it to monitor /var/log. The results in the web interface are, frankly, astoundingly disappointing. I can see the webserver under hosts, but the sources are:
This seems like a pretty basic task. This is a project I'm doing on the side. I can't be a full time splunk administrator. I mainly want to monitor my web & system logs for a handful of different webservers. Eventually, if I'm feeling spunky, I may even add some windows webservers and database servers to the mix. Right now, though, I'd really like to be able to index something useful.
Is there a quick tutorial that might shed some light on this relatively trivial task?
mattlemay, your question wasn't originally clear that it wasn't picking up all the logs. It sounds as if you are unhappy that it picked up those logs you have listed.
Firstly have you got permissions to access those files if Splunk is not running as root? It shouldn't really be able to access the ones you've listed but always good to check just in-case.
Secondly could you post your inputs.conf from the forwarder please?
Thirdly if you are running syslog-ng or rsyslog you could always bypass the forwarder altogether and add a statement like;
to your syslog.conf (or rsyslog.conf or someother similar purmutation) in the /etc, /etc/syslog (or where-ever it may be!)
where the IP is your splunk server and configure your splunk instance to receive on port 514 (assuming it uses this as the default, which most do)
The forwarder is great for forwarding non-syslog files, load balancing across multiple forwarders and for integration with a deployment server for centralised management. If you have a one to one relationship however you might find it easier to use the syslog method. From memory I believe the weblogs will utilize this on the system but you may need to do a quick test first.
You're right that this is trivial and Splunk normally handles this folder fine - infact its usually the first place alot of people head when demo-ing to customers to show its ease of use and ability to recursively index a folder 🙂
We just need to figure out what is stopping it on your setup
I am trying hard not to be snarky about this. But I didn't configure it to ignore /var/log/syslog and /var/log/messages in favour of /var/log/dmesg and /var/log/rpmpkgs.2. Yes, I ultimately want it to index /var/log/apache2. The documentation strongly insinuates that when you tell it to monitor directories, it does so recursively, which is a reasonable thing for it to do. In fact, part of the documentation explicitly says that I don't want to configure it to monitor both /var/log and /var/log/apache2.
Thanks for the link, but it seems to link to documentation I've already seen. I feel like I've done what the documentation told me to do and it didn't work. Now I don't know if it's a problem on the forwarder or the indexer. Is it a data source problem or an index problem?
You configured Splunk to monitor (read: Index) all of /var/log and now you are disappointed that it did what you asked it to?
Generally you want to explicitly state what you want Splunk to do. For instance, if you want to monitor all of /var/log/apache then configure it to do that and it will.