8/2024: I get this message with Linux Splunk v9.3.0 It started appearing after I relocated $SPLUNK_DB and freed up the space under $SPLUNK_HOME/var/lib/splunk/ Update: The message stopped after splunkd re-created all the 2-byte index .dat files under the old location  $SPLUNK_HOME/var/lib/splunk/ Maybe I should have used a symbolic link to relocate the index DB instead of defining a new DB location in splunk-launch.conf ... View more

Well, I can't tell you if it would have solved your problem because I have no idea if it was the same problem. It had the same symptoms but maybe the underlying cause was different. It could have solved it if it was the same problem 🙂 ... View more

Correction... it doesn't seem to have anything to do with the upgrade but rather the use of (/opt/splunk/bin/splunk start|stop).  If either of these commands are issued, whatever Splunkd.service file exists is renamed and replaced with a new one. ... View more

I didn't resolve it.  It's still there, but my install has not stopped working either. ... View more

I spent too much of a day trying to figure out why 2 of 5 servers were not showing up in my Indexer. I tried removing then adding the forward-server information, restarting the forwarder over and over and even reinstalling the forwarder on each, but they just didn't show up. Using telnet I confirmed the connection was open and the Indexer was listening. In the forwarders' splunkd.log files I confirmed the connection was being made. Finally I happened to change my search string to "index=_internal host=*" and there they were, but there was only one source from each and it was $SPLUNK_HOME/var/log/splunk/splunkd.log. The other 3 servers that were working had many more sources. A little bit more searching and I found this command: $SPLUNK_HOME/bin/splunk list monitor Sure enough, only splunkd.log was being forwarded. So I ran: $SPLUNK_HOME/bin/splunk add monitor /var/log This was what the "working" servers showed in their "list monitor" results. When running "host=*" in the search app, there were the 2 servers that wouldn't work before. ... View more