Sorry but this is probably a stupid question. I have set up Splunk to be able to have centralized collection of all the event logs from my servers. Now that I have installed all the agents, I cannot seem to search all the machines' event logs. I put in host=MYSERVERNAME and there are several machines that do not return anything.
Does the agent need to have an app deployed to collect event logs?
@Frederik,
Let's say you have 5 machines that you want to collect xyz.log (which exists on all 5 hosts)
1. You need to have forwarders on all 5 hosts
2. On each of the 5 hosts, in inputs.conf under $SPLUNK_HOME/etc/system/local directory, host should be FQDN of that particular machine (eg, on host1, it would be host=host1) next is your monitor/batch (read inputs.conf from docs) should point to the actual log location on that host.
3. outputs.conf , since all these 5 hosts are sending to the same splunk (assumption) they can be pretty much be same on all 5 hosts. This is the typical process configuring a forwarder.
With the little information provided, we can only assume what might be wrong, example.
1. As one of the answers suggested, is forwarder service up and running on all the hosts?
2. What user is your splunk forwarder running as? Does that user have Read access to the log file you are trying to consume?
3. Have you checked the connectivity from all the data sources/hosts to Central splunk instance? telnet central splunk's ip 9997?
4. Is it NAT'd or probably firewall is blocking the communication? Completely different issue
What can be done is, go to the splunkd.log on the machines that are not forwarding logs (Located under $SPLUNK_HOME/var/log/ and do a tail -f splunkd.log) see any ERRORS and abnormal stuff? If yes, that would be the first step of your triage.
For more accurate answers, please provide more information and ERRORS if any from your splunkd.log. Hope this helps!
Thanks,
Raghav
Well written @Raghav2384
The "official" documentation at I can't find my data!
Can you verify that the Splunk forwarder is running on the host machines you installed it on? You can do this by going into Splunk_Home/bin
and run ./splunk status
Also, you will need to go into Splunk_Home/etc/system/local
and edited the outputs.conf
and make sure it's pointing to your indexer