Our Dev Splunk instance was recently upgraded from Splunk Enterprise 8.2.2.1 to 9.0.2. I am getting the following error on our primary Search Head from python.log on splunkd restart: ERROR config:149 - [HTTP 401] Client is not authenticated Traceback (most recent call last): File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/config.py", line 147, in getServerZoneInfoNoMem return times.getServerZoneinfo() File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/times.py", line 163, in getServerZoneinfo serverStatus, serverResp = splunk.rest.simpleRequest('/search/timeparser/tz', sessionKey=sessionKey) File "/opt/splunk/lib/python3.7/site-packages/splunk/rest/__init__.py", line 625, in simpleRequest raise splunk.AuthenticationFailed splunk.AuthenticationFailed: [HTTP 401] Client is not authenticated I did a re-scan of everything using the newest version of the Upgrade Readiness App (off Splunkbase). Some apps did have a Python warning. I verified currently installed versions of each app (with the exception of Splunk Enterprise package-included apps like Splunk Secure Gateway and Splunk RapidDiag) and the documentation states our installed versions are compatible with Enterprise 9.0. It does not appear that any installed apps are using a deprecated version of python. I also ran the following command and verified our python version as Python 3.7.11: splunk cmd python -V After combing over known issues for the 9.0 release and other Answers threads I’ve had no luck. I don’t know if this errors is meaningful so direction would be advised. Thank you! ... View more

I have the following stats search:     index=servers1 OR index=servers2 DBNAME=DATABASENAME source="/my/log/source/*" | stats list(Tablespace), list("Total MB"), list("Used MB"), list("Free MB"), list("Percent Free") by DBNAME     Which is supposed to produce the following results: DBNAME list(Tablespace) list(Total MB) list(Used MB) list(Free MB) list(Percent Free) DATABASENAME RED_DATA 165890 46350 119540 72   BLUE_DATA 2116 1016 1100 52   PINK_DATA 10 0 10 100   PURPLE_DATA 34 17 17 50   GREEN_DATA 6717 0 6717 100   ORANGE_DATA 51323 295 51028 99 …. Cont'd for 20+ rows           About 25-30% of the time (as both a live search and scheduled report) I get the following results instead. The list(Used MB) column will have extra garbage data which appears to be job search components: DBNAME list(Tablespace) list(Total MB) list(Used MB) list(Free MB) list(Percent Free) DATABASENAME RED_DATA 165890 46350 119540 72   BLUE_DATA 2116 1016 1100 52   PINK_DATA 10 0 10 100   PURPLE_DATA 34 17 17 50   GREEN_DATA 6717 0 6717 100   ORANGE_DATA 51323 295 51028 99       ion.command.search           4           duration.command.search.calcfields           0           duration.command.search.fieldalias           0           duration.command.search.filter           0           duration.command.search.index           2           duration.command.search.index.usec_1_8           0           duration.command.search.index.usec_8_64           0           duration.command.search.kv           0           duration.command.search.lookups     …. Cont'd for 20+ rows           I've adjusted the following limits.conf Search Head settings with no luck:     [stats] maxresultrows = 50000 maxvalues = 10000 maxvaluesize = 10000 list_maxsize = 10000   The search inspector also does not produce any notable error messages. Any ideas as to what is happening here and how I can solve it? ... View more

I'm trying to get auditd events into Splunk using the rlog.sh script from the Splunk Add-on for Unix and Linux. It isn't working. The audit logs are not being ingested. No errors are appearing in index=_internal for the host. It is successfully scheduled through the ExecProcessor component: 0400 INFO ExecProcessor [1975905 ExecProcessor] - New scheduled exec process: /opt/splunkforwarder/etc/apps/Splunk_TA_nix_l1_inputs/bin/rlog.sh To attempt to address the problem I have done the following: Had the host owner ensure dependent utilities are installed (listed in https://docs.splunk.com/Documentation/AddOns/released/UnixLinux/Requirements#Dependencies). Had the host owner change the log_group from root to splunk in /etc/audit/auditd.conf (suggested in https://community.splunk.com/t5/All-Apps-and-Add-ons/Can-t-get-rlog-sh-to-run/m-p/76143). When executing rlog in debug mode (./rlog.sh --debug) we get the following output: As splunk user: Blank output As root user: Expected output Additional details: This host was recently rebuilt. Before the rebuild the audit logs on this host were ingesting successfully through the Add-On. Other scripts through the Add-On are working on this host. This problem has not materialized on any of our other hosts utilizing the Add-On. Thanks in advance for your input! ... View more