Running version 9.3, the log-local.cfg doesn't seem to be applied. Even after a restart, Splunk is throwing >10 of these INFO lines per second. This message should probably be moved to the DEBUG category...    It is possible there's another issue with my instances, but this mess of logs is making it very hard to troubleshoot. `splunk set log-level TcpInputProc -level WARN`  does work Modifying log.cfg also works     ... View more

I have been receiving HTTP Events from an invalid token, and want to trace them back to the source. However, the HEC is behind an NGINX load-balancer, so I need to configure the HEC to use proxied_ip to find the original IP.  connection_host = [ip|dns|proxied_ip|none] * "proxied_ip" checks whether an X-Forwarded-For header was sent (presumably by a proxy server) and if so, sets the host to that value. Otherwise, the IP address of the system sending the data is used. * No default.  I would also like to apply it to every token, as all HEC ingest goes through the LB. However, it looks like this option is only available at a per-token level. HTTP Event Collector (HEC) - Local stanza for each token | inputs.conf  Nothing changed when I set it under [http]  Seems like this was implemented incorrectly... ... View more

If you are decommissioning indexers, they have to redistribute all the data on them to other peers in the cluster. If you try to take down several all at once, that process will likely break. Definitely decom them individually. Also, if you're using  ./splunk offline --enforce-counts DO NOT set maintenance mode The cluster cannot be in maintenance mode, because bucket fixup does not occur during maintenance mode. https://docs.splunk.com/Documentation/Splunk/9.3.0/Indexer/Takeapeeroffline#The_enforce-counts_offline_process ... View more

I get this too.  In Splunkd.log, we see the shutdown process, but then it just... doesn't shut down... until it times out. Looks like the shutdown process completes, but the HttpPubSubConnection keeps going. Shutdown [182482 Shutdown] - shutting down level="ShutdownLevel_Tailing" 08-15-2024 22:27:57.171 +0000 INFO Shutdown [182482 Shutdown] - shutting down name="TailingProcessor" 08-15-2024 22:27:57.171 +0000 INFO TailingProcessor [182482 Shutdown] - Will reconfigure input. 08-15-2024 22:27:57.171 +0000 INFO TailingProcessor [182482 Shutdown] - Calling addFromAnywhere in TailWatcher=0x7f4e53dfd a10. 08-15-2024 22:27:57.171 +0000 INFO TailingProcessor [182712 MainTailingThread] - Shutting down with TailingShutdownActor=0 x7f4e77429300 and TailWatcher=0x7f4e53dfda10. 08-15-2024 22:27:57.171 +0000 INFO TailingProcessor [182712 MainTailingThread] - Pausing TailReader module... 08-15-2024 22:27:57.171 +0000 INFO TailReader [182712 MainTailingThread] - State transitioning from 0 to 1 (pseudoPause). 08-15-2024 22:27:57.171 +0000 INFO TailReader [182712 MainTailingThread] - State transitioning from 0 to 1 (pseudoPause). 08-15-2024 22:27:57.171 +0000 INFO TailingProcessor [182712 MainTailingThread] - Removing TailWatcher from eventloop... 08-15-2024 22:27:57.176 +0000 INFO TailingProcessor [182712 MainTailingThread] - ...removed. 08-15-2024 22:27:57.176 +0000 INFO TailingProcessor [182712 MainTailingThread] - Eventloop terminated successfully. 08-15-2024 22:27:57.177 +0000 INFO TailingProcessor [182712 MainTailingThread] - Signaling shutdown complete. 08-15-2024 22:27:57.177 +0000 INFO TailReader [182712 MainTailingThread] - State transitioning from 1 to 2 (signalShutdown). 08-15-2024 22:27:57.177 +0000 INFO TailReader [182712 MainTailingThread] - Shutting down batch-reader 08-15-2024 22:27:57.177 +0000 INFO TailReader [182712 MainTailingThread] - State transitioning from 1 to 2 (signalShutdown). 08-15-2024 22:27:57.177 +0000 INFO Shutdown [182482 Shutdown] - shutting down level="ShutdownLevel_IdataDO_Collector" 08-15-2024 22:27:57.177 +0000 INFO Shutdown [182482 Shutdown] - shutting down name="IdataDO_Collector" 08-15-2024 22:27:57.178 +0000 INFO Shutdown [182482 Shutdown] - shutting down level="ShutdownLevel_PeerManager" 08-15-2024 22:27:57.178 +0000 INFO Shutdown [182482 Shutdown] - shutting down name="BundleStatusManager" 08-15-2024 22:27:57.178 +0000 INFO Shutdown [182482 Shutdown] - shutting down name="DistributedPeerManager" 08-15-2024 22:27:57.692 +0000 INFO TcpInputProc [182624 TcpPQReaderThread] - TcpInput queue shut down cleanly. 08-15-2024 22:27:57.692 +0000 INFO TcpInputProc [182624 TcpPQReaderThread] - Reader thread stopped. 08-15-2024 22:27:57.692 +0000 INFO TcpInputProc [182623 TcpListener] - TCP connection cleanup complete 08-15-2024 22:28:52.001 +0000 INFO HttpPubSubConnection .....  ... ... INFO IndexProcessor [199494 MainThread] - handleSignal : Disabling streaming searches. Splunk continues to write log lines from HttpPubSubConnection - Running phone.... after the Shutdown, nothing else shows up in the logs.  I re-ran "./splunk stop" in another session, and it finally logged one more line and actually stopped. ... View more

Do you have a link to those Splunk docs? ... View more

As I've learned recently, there's nothing stopping someone from defining a different Host value in an input, and causing that problematic situation much more quickly and broadly.  This seems like a huge oversight in functionality. ... View more

I strongly disagree, as the Forwarder Management in a Deployment Server does not allow you to search via Instance Name.  It makes it difficult to match data you see in Splunk to the actual Host sending it.  ... View more

Has this enhancement been added in 8.1.4, or 8.2.0?  ... View more