Getting Data In

Disappearing data from forwarders when using deployment server

sf_user_199
Path Finder

I have a problem with some of my forwarders, and it has me pretty well stumped.

I have a pool of 10 app servers running light forwarders. The LF are reading & forwarding apache access log data. The LF all send their data to the same splunk instance that is functioning as an intermediate forwarder. That intermediate forwarder then sends the data to our only indexer, and does not do any indexing itself.

All 10 light forwarders were setup at the same time, and are controlled by the same deployment server. When watching the various splunk logs on the intermediate forwarder, it appears that the intermediate forwarder is receiving data from all 10 light forwarders.

However, the indexer is only indexing the data from 5 of the LF. The data just seems to disappear.

Any suggestions?

Tags (2)
0 Karma
1 Solution

sf_user_199
Path Finder

As it turns out, some of the light forwarders assigned themselves a host name that did not match the name of the host that the light forwarder was installed on. My assumption is that there is a problem on the host as a result of the server build process.

Removing the incorrect hostname from the .conf files & restarting the light forwarders fixed the issue.

View solution in original post

0 Karma

sf_user_199
Path Finder

As it turns out, some of the light forwarders assigned themselves a host name that did not match the name of the host that the light forwarder was installed on. My assumption is that there is a problem on the host as a result of the server build process.

Removing the incorrect hostname from the .conf files & restarting the light forwarders fixed the issue.

0 Karma

sf_user_199
Path Finder

I wish it was that simple. I checked the file input status page, and the Splunk LF's are able to read the file.

0 Karma

chris
Motivator

We once had a similar problem - until we found, that splunk had no access to the log files on one part of our servers

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Why Splunk Customers Should Attend Cisco Live 2026 Las Vegas

Why Splunk Customers Should Attend Cisco Live 2026 Las Vegas     Cisco Live 2026 is almost here, and this ...

What Is the Name of the USB Key Inserted by Bob Smith? (BOTS Hint, Not the Answer)

Hello Splunkers,   So you searched, “what is the name of the usb key inserted by bob smith?”  Not gonna lie… ...

Automating Threat Operations and Threat Hunting with Recorded Future

    Automating Threat Operations and Threat Hunting with Recorded Future June 29, 2026 | Register   Is your ...