Correction... it doesn't seem to have anything to do with the upgrade but rather the use of (/opt/splunk/bin/splunk start|stop). If either of these commands are issued, whatever Splunkd.service file exists is renamed and replaced with a new one.
... View more
I know this is an old thread but for anyone that is having the same problem, this might help. I had the same problem after upgrading to v8.2.x but after some tests I found the cause. Splunk Enterprise 8.2.x has some new integrated apps which are not part of the version I was upgrading from (7.2.x). In particular "Python Upgrade Readiness App" which comes in the version 1.0.0 but has the option to Update to a newer available version. This is what I found out: - If I do not update the "Python Upgrade Readiness" from version 1.0.0, I do not get any error message. I can also safely disabled the app and no error message appears after. - If I update "Python Upgrade Readiness" to the latest version (when this is written, latest version is 3.10), I get this error and even though I disable the App after upgrade. The error message still remains. The only way I found to get rid of the error message after updating the App was to downgrade back to version 1.0.0. To downgrade, simply replace the app folder "$SPLUNK_HOME/etc/apps/python_upgrade_readiness_app" with the 1.0.0 version (I got the old version from a fresh installed Splunk) and then restart Splunk. Voila, all errors gone! This was my solution, it does not have to be the same for others but hopefully helps some.
... View more
I spent too much of a day trying to figure out why 2 of 5 servers were not showing up in my Indexer. I tried removing then adding the forward-server information, restarting the forwarder over and over and even reinstalling the forwarder on each, but they just didn't show up. Using telnet I confirmed the connection was open and the Indexer was listening. In the forwarders' splunkd.log files I confirmed the connection was being made. Finally I happened to change my search string to "index=_internal host=*" and there they were, but there was only one source from each and it was $SPLUNK_HOME/var/log/splunk/splunkd.log. The other 3 servers that were working had many more sources. A little bit more searching and I found this command:
$SPLUNK_HOME/bin/splunk list monitor
Sure enough, only splunkd.log was being forwarded. So I ran:
$SPLUNK_HOME/bin/splunk add monitor /var/log
This was what the "working" servers showed in their "list monitor" results. When running "host=*" in the search app, there were the 2 servers that wouldn't work before.
... View more