Tried with the default Admin and also created a brand new admin user, non works... I'm pretty clueless here TBH. ... View more

@PickleRick another update, perhaps it might be of help. I've installed a brand new server and confirmed that everything works as expected (including _internal index events). Then, I copied over my users, roles, and passwd file from the old server and it stopped working. Specifically: $SPLUNK_HOME/etc/users $SPLUNK_HOME/etc/system/local/authorize.conf $SPLUNK_HOME/etc/passwd Permissions on those files are set correctly, server was restarted. Any idea what might be the issue here? ... View more

@PickleRick an update. I've re-applied the permissions ownership on $SPLUNK_DB and restarted the server. It seems as the timestamp is now updated correctly but still no results when searching (including real-time). ... View more

The files in the original location ($SPLUNK_HOME/var/lib/splunk/_internaldb/) are not modified as well...  btool shows the following: /opt/splunk/etc/system/default/indexes.conf coldPath = $SPLUNK_DB/_internaldb/colddb homePath = $SPLUNK_DB/_internaldb/db thawedPath = $SPLUNK_DB/_internaldb/thaweddb   ... View more

Thanks again @PickleRick! Trying the real-time search yielded no results... My SPLUNK_DB is pointing to a different directory (/datadrive/splunk_db) and the files there were not modified for few days now. Probably since we first started the server... Any idea what might be the case? ... View more

Thanks for following up on this, @PickleRick! I've already verified that splunk is writing data to /opt/splunk/var/log/splunk/splunkd.log with correct times and ensured that the $SPLUNK_HOME/var/log/splunk folder is indeed monitored. Trying btool showed a whole lot of data but nothing seems out of the ordinary there... /opt/splunk/bin/splunk btool inputs list --debug   I forgot to mention that we have Splunk ES installed on the server, if it matters at all... The splunkd.log contains many errors, but I could not find anything directly connected to the issue in question. ... View more

Yes, I used the same approach as described in the provided link. I'm able to search other indexes just fine and see the results. ... View more

Copying the $SPLUNK_HOME folder to the new server  ... View more

Thanks richgalloway! So, can I safely assume that a correlation search is only related to SplunkES and simply negate other apps in my queries? I also wonder how the UI returns specifically "Correlation Searches"\"Saved Searches"\etc... when searching via the "content management" UI. Any idea how I can mimic this behaviour?  ... View more