Disappearing data from forwarders when using deployment server

[sf_user_199]

I have a problem with some of my forwarders, and it has me pretty well stumped.

I have a pool of 10 app servers running light forwarders. The LF are reading & forwarding apache access log data. The LF all send their data to the same splunk instance that is functioning as an intermediate forwarder. That intermediate forwarder then sends the data to our only indexer, and does not do any indexing itself.

All 10 light forwarders were setup at the same time, and are controlled by the same deployment server. When watching the various splunk logs on the intermediate forwarder, it appears that the intermediate forwarder is receiving data from all 10 light forwarders.

However, the indexer is only indexing the data from 5 of the LF. The data just seems to disappear.

Any suggestions?

[sf_user_199 · Accepted solution]

As it turns out, some of the light forwarders assigned themselves a host name that did not match the name of the host that the light forwarder was installed on. My assumption is that there is a problem on the host as a result of the server build process.

Removing the incorrect hostname from the .conf files & restarting the light forwarders fixed the issue.

[sf_user_199]

I wish it was that simple. I checked the file input status page, and the Splunk LF's are able to read the file.

[chris]

We once had a similar problem - until we found, that splunk had no access to the log files on one part of our servers