I logged into Splunk today, and got the dreaded "Splunk must be restarted for changes to take effect" message. The thing is, neither I nor anyone on my team made any changes that require a restart. So now I'm combing through the logs, trying to figure out what the change was that Splunk is complaining about, but I'm having trouble determining where to look - I've mainly been looking at _audit events, but I'm not finding anything there. How do you all go about finding these sorts of changes?
Try looking in the _internal index, and search for "*restart*", or "*required a restart*".
I don't think you will see what you're looking for in _audit.
I looked at that, but it appears the only events I find with "restart" are logs associated with users seeing the "requires a restart" message - not with the event that caused the need for a restart. Any other ideas where it might be hiding?
Chances are you are looking for a single message.
Try searching for LocalAppsAdminHandler.
Is the message you're seeing "GET /services/messages/restart_required"?
If Splunk did not log an event that would require a restart, then once you find it you should submit a bug report.
If it is not in the splunk log, then I recommend searching the Splunk directory for file system changes that night.
You can also search the system's logs for activity pertaining to "*splunk*" over that timeframe.