When you send data over HEC, especially for international character data, make sure to specify the charset encoding for the data set. That way the receiving end knows its charset encoding and how to decrypt. Oftentimes the encryption method is different than what you expect. Here's a good example for your reference: https://medium.com/@rysartem/sending-data-to-splunk-hec-in-a-right-way-4a84af3c44e2   ... View more

Here are my findings.   i) This has been there for a while even before the 8.2.10 upgrade - I checked some logs from old diags, it goes as far back with 8.1.x.   ii) In SHC environment, we recommend you to use source IP stickiness for any Load Balancers so that you log in to SH1 and launch a search on SH1, then gets updates through SH1.    However, it appears you requested search preview updates from another Search Head, for example, SH2 which would trigger and request SH1 for your search updates but, as it was then still running, SH1 would have put errors  as below;   ----- From SH1, IP Addr: 10.9.160.139 , error for the proxy request for a search that is still running -- 05-24-2023 13:33:49.894 -0700 ERROR SHCSlaveArtifactHandler [222579 TcpChannelThread] - Failed to trigger replication (artifact='1684960384.101_13E3A0F-27AE-49C5-9FB2-23862EDB224B') (err='event=SHPSlave::replicateArtifactTo invalid status=alive to be a source for replication') -----   iii) Logs proves the scenario :    1. The search is an adhoc with SID, "1684960384.101_13E3A0F-27AE-49C5-9FB2-23862EDB224B"  by "admin" and found in the search artifacts/dispatch dir;   $ ls -l $SPLUNK_HOME/var/run/splunk/dispatch/1684960384.101_13E3A0F-27AE-49C5-9FB2-23862EDB224B/ -rwxr-xr-x 1 support support    242 May 24 20:33 args.txt ... <SNIP> -rwxr-xr-x 1 support support 828809 May 24 20:34 search.log       2. From splunkd_ui_access.log : User, "admin" started the search with SID: 1684960384.101_13E3A0F-27AE-49C5-9FB2-23862EDB224B,   -- splunkd_ui_access.log -- 10.6.248.0 - admin  [24/May/2023:13:33:05.139 -0700] "GET /en-US/splunkd/__raw/servicesNS/admin/search/search/jobs/1684960384.101_13E3A0F-27AE-49C5-9FB2-23862EDB224B?output_mode=json&_=1684959918754 HTTP/1.1" 200 1147 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36" - 0dd2b80d72da3ed97e640b9101f0a698 5ms   3. Search string from SID:684960384.101_13E3A0F-27AE-49C5-9FB2-23862EDB224B -- search.log -- 05-24-2023 13:33:05.302 INFO  SearchParser [223948 RunDispatch] - PARSING: search index=apps sourcetype=api <SNIP>...ion  as ApplicationName | search  name!=default  pname!=zz-default | stats count by AppName pname | table AppName pname     4. 2s after the search started, a replicate request came in from 10.9.129.34 which is SH2 but immediately returned an error, 500 as it's not finished. This proxy request for the search results keeps coming in every second but always gets the error, 500 as it was not completed.    -- splunkd_access.log -- 10.9.129.34 - splunk-system-user [24/May/2023:13:33:07.016 -0700] "POST /services/shcluster/member/artifacts/1684960384.101_13E3A0F-27AE-49C5-9FB2-23862EDB224B/replicate?output_mode=json HTTP/1.1" 500 231 "-" "Splunk/8.2.10 (Linux 3.10.0-1160.49.1.el7.x86_64; arch=x86_64)" - 2ms   Search results preview request should hit the same SH as the search was launched but it appears to routed to different Search head.   Recommendation:  need to check with your LB or DNS team, such as whether LB has source IP Stickiness set or if you use DNS roundrobin between the multiple LBs  in front of SHC, that uses same FQDN or URL as this will lead the users to different LBs or Search heads.. ... View more

'splunk-MonitorNoHandle.exe' is designed to hold data when it's not able to send to UF, use unlimited memory and this symptom can happen when there are huge amount of data to forward to the indexers while the forwarding speed by UF is limited.  i) Check the queue status of parsing queue and tcpoutput queue to find which one is getting blocked first.    Parsing queue blocked firstly which means it receives over the capacity - This can happen when maxKBps is throttled to the default, 256 ,change this to 0, unlimited or something your environment allows.  - side effect it can bombard indexers if it's sending unlimited, huge amount of data.   ii) In splunkd log, find any messages showing difficulties in sending data to the next receiving ends: Even if the parsing pipeline can send more data by increasing maxKBps, if tcpoutput gets blocked then you will see the same issues again. Below are the example logs that UF has problem in connecting to the indexers ; 11-12-2021 11:00:11.365 -0500 WARN TcpOutputProc - Cooked connection to ip=172.22.1.218:9997 timed out 11-12-2021 11:01:48.391 -0500 WARN TcpOutputProc - Cooked connection to ip=172.22.1.218:9997 timed out 11-12-2021 11:12:28.757 -0500 WARN TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to output group ABC_indexers has been blocked for 500 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data. iii) Recommendations : iii-1) Parsing Queue being always full more often than the tcpoutput queue does - meaning MonitorNoHandle is sending data over the capacity that Parsing process can handle. This can happen when maxKBps is throttled to the default, 256 , then consider to increase the value according to your traffic size. http://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf  - side effect it can bombard indexers if it's sending unlimited, huge amount of data.   iii-2) How to set the memory limit used by the modInput, MonitorNoHandle; [inputproc] in limits.conf monitornohandle_max_heap_mb=5000 monitornohandle_max_driver_mem_mb=5000 ( 5000/5gb can be changed according to your environment) iii-3) If you find intermittent blockages on tcpoutput queue This can also contribute to the MonitorNoHandle's memory growth as the Parsing Pipeline can not send as much data as it receives - then MonitorNoHandle.exe has to hold the backlog data within own heap memory that can grow unexpectedly. Consider to implement asynchronous forwarding so that it can spread the data without pausing the data flow and that way MonitorNoHandle.exe may have less chances to hit the heap limit.  You can consult with our PS resources for implementation too. ... View more

You can use "splunk clean eventdata -index " command like below; $ splunk stop $ splunk clean eventdata -index "_internal" ... View more

Here's the steps to find the culprits. There are 17 crash logs found in the diag file. The crashing point from crash log files are all the same as below, ---- excerpts --- Libc abort message: splunkd: /opt/splunk/src/pipeline/indexer/JournalSlice.cpp:1780: bool ReadableJournalSliceDirectory::findEventTimeRange(st_time_t*, st_time_t*, bool): Assertion `tell() == pos' failed. ------------------ Which suggests that there must be corrupt bucket(s) or truncated but the crash log files don't have details of which bucket(s) being corrupt at all. To find the relevant logs in splunkd.log by matching the time, 14:45:24 and thread number such as, IndexerTPoolWorker-4 from the crash log file , I found the logs as below; - For crash-2022-09-07-14:45:24.log:    Crashing thread: IndexerTPoolWorker-4 Check the splunkd.log to find the logs created at 14:45:24 by IndexerTPoolWorker-4 ; splunkd.log: 09-07-2022 14:45:24.025 -0700 INFO HotBucketRoller [322700 IndexerTPoolWorker-4] - found hot bucket='/storage/tier1/20000_idx2/db/hot_v1_432' - For crash-2022-09-07-14:45:32.log: Crashing thread: IndexerTPoolWorker-7 Check splunkd.log to find the logs created at 14:45:32 by IndexerTPoolWorker-7 ; splunkd.log: 09-07-2022 14:45:32.238 -0700 INFO HotBucketRoller [322983 IndexerTPoolWorker-7] - found hot bucket='/storage/tier1/20000_idx2/db/hot_v1_432' - crash-2022-09-07-14:45:40.log: Crashing thread: IndexerTPoolWorker-6 splunkd.log: 09-07-2022 14:45:40.566 -0700 INFO HotBucketRoller [323220 IndexerTPoolWorker-6] - found hot bucket='/storage/tier1/20000_idx2/db/hot_v1_432' The crash log time and the thread number, IndexerTPoolWorker-# matches and the log messages pointing to the same bucket, '/storage/tier1/20000_idx2/db/hot_v1_432'. As it's suspected to have gone beyond the fsck repair we moved it out of the $SPLUNK_DB and then the indexer came up and running as the other peers.   This covers the specific situation of a corrupt bucket causing the indexer to crash and how to find the corrupt buckets. ... View more

There is a known issue where Indexer cluster with multi-sites for the versions prior to 8.1.8 and 8.2.4. The symptoms are exactly the same as above description - the search's working fine and finds the events in question. But with "| delete" it fails to find events that you want to delete due to no primary buckets that can be searched by delete command. - Fixed version: 8.1.8+ and 8.2.4+ , - Affected: versions in 8.1.0~8.1.7 or 8.2.0 ~ 8.2.3 - Workaround:  This may have some performance impact on SHC so you may want to use it when you hit the issue and revert it back. @cm, add the below and restart the CM. [clustering] in server.conf assign_primaries_to_all_sites = true ... View more

If you find some broken configs for the stanzas in savedsearches.conf, such as missing many fields or no 'search' field which will cause the KO to not show up in the search results on the GUI, you may hit a known issue for 8.2. Affected versions are 8.2.0 and 1. The fix is available from 8.2.2+. Another symptom is, those affected KO/searches may not run as it is scheduled to run. There is no workaround other than recreating the affected KO and consider to upgrade to 8.2.2+. ... View more

The issue started from the splunk-optimize process unable to access the tsidx files for optimization and eventually gave up data feeding and wait until optimizer catch up the backlogs - the log messages like,  -- splunkd.log The index processor has paused data flow. Too many tsidx files in idx=_metrics bucket="/opt/splunk/var/lib/splunk/_metrics/db/hot_v1_804" , waiting for the splunk-optimize indexing helper to catch up merging them. Ensure reasonable disk space is available, and that I/O write throughput is not compromised. -- Before this log there are a lot of error logs that complain about the access permissions(errno=1) for tsidx, ERROR SplunkOptimize [37311 MainThread] - (child_304072__SplunkOptimize) optimize finished: failed, see rc for more details, dir=/opt/splunk/var/lib/splunk/_metrics/db/hot_v1_70, rc=-4 (unsigned 252), errno=1   The graph for 'index=_internal "see rc for more details" | timechart span=1m count ' will match the timing of queue blockage issue, which means that's the cause of the symptoms. -- errno=1 (EPERM) why this happens? The customer confirmed that the indexers run antivirus scanner with $SPLUNK_HOME and $SPLUNK_DB excluded but it's found to have missed the sub directories. Due to this and the scanner algorithms the access to the tsidx files by Splunk-optimize is considered to be suspicious and gets blocked, which eventually paused the data flow, the port 9997 also gets closed.   ... View more