Splunk Search

I have installed the Splunk forwarder, but why am I unable to search the event logs from all machines?

Frederik
New Member

Sorry but this is probably a stupid question. I have set up Splunk to be able to have centralized collection of all the event logs from my servers. Now that I have installed all the agents, I cannot seem to search all the machines' event logs. I put in host=MYSERVERNAME and there are several machines that do not return anything.

Does the agent need to have an app deployed to collect event logs?

0 Karma

Raghav2384
Motivator

@Frederik,

Let's say you have 5 machines that you want to collect xyz.log (which exists on all 5 hosts)
1. You need to have forwarders on all 5 hosts
2. On each of the 5 hosts, in inputs.conf under $SPLUNK_HOME/etc/system/local directory, host should be FQDN of that particular machine (eg, on host1, it would be host=host1) next is your monitor/batch (read inputs.conf from docs) should point to the actual log location on that host.
3. outputs.conf , since all these 5 hosts are sending to the same splunk (assumption) they can be pretty much be same on all 5 hosts. This is the typical process configuring a forwarder.

With the little information provided, we can only assume what might be wrong, example.
1. As one of the answers suggested, is forwarder service up and running on all the hosts?
2. What user is your splunk forwarder running as? Does that user have Read access to the log file you are trying to consume?
3. Have you checked the connectivity from all the data sources/hosts to Central splunk instance? telnet central splunk's ip 9997?
4. Is it NAT'd or probably firewall is blocking the communication? Completely different issue

What can be done is, go to the splunkd.log on the machines that are not forwarding logs (Located under $SPLUNK_HOME/var/log/ and do a tail -f splunkd.log) see any ERRORS and abnormal stuff? If yes, that would be the first step of your triage.

For more accurate answers, please provide more information and ERRORS if any from your splunkd.log. Hope this helps!

Thanks,
Raghav

skoelpin
SplunkTrust
SplunkTrust

Well written @Raghav2384

ddrillic
Ultra Champion

The "official" documentation at I can't find my data!

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Can you verify that the Splunk forwarder is running on the host machines you installed it on? You can do this by going into Splunk_Home/bin and run ./splunk status

Also, you will need to go into Splunk_Home/etc/system/local and edited the outputs.conf and make sure it's pointing to your indexer

0 Karma
Get Updates on the Splunk Community!

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability As businesses scale ...