We have setup splunk in our environment, and we have logs coming in from different geographies (US/UK/Asia). The logs, all have different timestamps, but we have used a light forwarder to convert them all to current server time using ($SPLUNKHOME/etc/apps/search/local/props.conf):
DATETIME_CONFIG = CURRENT
Also the inputs.conf and outputs.conf are properly configured, and everything works fine.
But then after a few hours, i am unable to see any data coming from some of the machines (UK/Asia). I checked splunkd.log of light forwarder, there wasn't any ERROR in it.
I checked metrics.log of forwarder, it seems to be getting updated with each update in UK/Asia machines, but no data is going to the splunk receiver.
Checked splunkd.log at splunk receiver end, it contains this ERROR:
09-17-2012 08:05:20.470 -0400 ERROR SearchResults - Unable to write to file '/opt/splunk/etc/users/abcd/search/history/hostname.csv'. Retried 5 times, period=500 ms. error='No such file or directory'
but i don't think that is related to the issue in any way.
All clients and splunk receiver is on Linux, forwarder is on windows 2008.
Can someone please help on how to debug the issue and what could be causing it?
I have restored the system to a state (many times) where everything is working but then again the problem comes back.
... View more