recently our splunk instance has not been indexing our data. All licenses are OK and we are not exceeding our loads(we have other instances which are working with much higher loads).
At first some our data was being indexed, then nothing was. after a restart our data was being indexed again but a few hours later it stopped again and we have had nothing since.
We tried deleting old indexes and having multiple small log files rather than one large one yet nothing works and we are all out of ideas.
Has anybody any ideas?
Have you tried to search something like
'index=*' and removing the time constraints, then replacing them with with the 'All Time - Real Time' Drop Down from the all time menu? This will show you all logs presently being indexed by splunk. Perhaps the issue is that your data isn't being indexed in the manner you expect, or something is failing with timestamp extraction causing it not to be returned by your search string.