Strangest thing. I have some Infoblox logs coming in from a Syslog-NG server where we have a UF installed. UF is successfully sending the Infoblox logs to Splunk BUT, I can only see those logs when doing an alltime-realtime search but can't see them anywhere when doing a historical alltime search even when logged in as admin. I can search other logs in the same index but just comes back with "0 events" and no errors in the job - just nothing. Can't find them via sourcetype, source or host.
Any ideas? I know the data is there but just can't see it on historical searches.
Thanks for the response! Logs are in UTC time it seems so a bit in the future but all time should show data anyway. Still, ran another search for 1 year in the future and 1 year in the past at the same time - still zero data returned with no errors.
Is the date/time in those syslog events far into the future or past? If they're in the future, you can try searching with earliest=now latest=+5y (for example). If they're too far into the past, Splunk is probably dropping them. Both of these scenarios are logged in the indexers' _internal index.