Hi daddyoh, as part of splunk startup you should see some events in splunkd.log as splunk loads up the particular input stanzas in question. I'd take a look at that, and ensure that you see those file monitors properly loaded.
Based on your comment, that's interesting that you see the initial two files showing up after adding a third, but not the third itself. I look into splunkd.log for that restart event when the two starting worked, but the third did not.
Past that, I think you covered most of the bases, that is
Making sure the files exist
have current events
sensible timestamp
user splunk is running as can read the files
Past that my other thoughts are potential delays from the forwarder to the indexer (indexer is under heavy load, latency in events being written to disk)
There could also be something strange going on with the timezone, potentially events are being written in the future, which would lead to some delay before you find them in search using the normal time range settings. You can override this by including something like "earliest=-4h latest=+4h" in the search string in order to search potential future events.
Please let me know if this helps to answer you question!
... View more
