SplunkWeb users may experience different behaviors for the UI preferences that used to persist and show latest preferences by updating ui-prefs.conf on the fly. Now after upgrade to 9.0.5+ or 9.1.0+ its behavior changed and no longer uses ui-prefs.conf to remember the user's UI level preferences, but instead, uses the url in the request or localStorage/Web Storage. However the ui-prefs.conf is still used to set the initial values up - simply became read-only mode. Due to this change you will find the user level changes on the search page may not persist. Here are some examples you can experience by this changes:   - The search mode, fast/smart/verbose mode you set just now is discarded. - The chart type, line, bar or pie, you chose just now is not persisting and changed to one type whenever logging in back. - Some features, such as the time ranges you used to use in time range picker is reverted back after the browser cache clean-up. - Or you have to set your own search page preferences to each browser or different PCs you use.     Reason for the change: This change was decided to improve the slow UI loading time issue for a large deployment with thousands of users. Instead of using central location, ui-prefs.conf on the Search Head the load has been spread to the user level. Affected versions : 9.0.5+, 9.1.0+ What's the expected with the new behavior: With this change each parameter in the ui-prefs.conf has been implemented basically to remove POST method, either added into URL in GET request or stored in the localStorage of the user browser. If it’s stored in the URL as a parameter it lasts the length of the current workflow. ie. . Once you load the page again without the parameter in the URL, it is discarded. If it's in localStorage it will last until it gets deleted from the Web Storage. You can find more detailed conditions for different browsers here;  How the data in localStorage get deleted. How can I go back to the previous behavior although it's not recommended: If the customer would like to re-enable the old workflow temporarily (this will result in poorer performance for web page loads - the reason why this change was made), optimize_ui_prefs_performance in web-features.conf can be set to false as a temporary workaround. In web-features.conf:   [feature:ui_prefs_optimizations] optimize_ui_prefs_performance = false (true by default)       ... View more

The browser is supposed to check in and send keep alive/poll GETs to splunk server every 1 second while a search is running. However it's stopped at some point and comes back later, then the splunk server sends back the error 404 as it's already timed out.  Here's the logs: 1. auto-cancel logged in to search_messages.log ;   07-20-2023 13:37:45.264 -0400 ERROR SearchMessages - orig_component="" app="search" sid="1689874558.349256_AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEE" message_key="" message=Search auto-canceled 07-20-2023 13:46:45.343 -0400 ERROR SearchMessages - orig_component="" app="search" sid="1689874865.349389_AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEE" message_key="" message=Search auto-canceled 07-20-2023 14:00:45.323 -0400 ERROR SearchMessages - orig_component="" app="search" sid="1689875862.349793_AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEE" message_key="" message=Search auto-canceled     2.  splunkd_ui_access log for sid="1689875862.349793_AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEE" shows that the keep alive stopped and resumed after 1min in the graph.    There's a 1min gap between the last success of 200 and the error 404 in the graph below.    The browser took longer than 30 seconds (job_default_auto_cancel=30 for 8.2.9) at times to send keep alive pool to splunk server.   index=_internal source=*splunkd_ui_access.log "1689875862.349793_AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEE" |timechart span=1s count by status   Below has more details of the access log. Every second the browser sent KeepAlives then got http 200. But the last one received http 404 due to the timeout. 3. Recommendation:    i) In web.conf for the versions prior to 9.0 only. [settings] job_default_auto_cancel = 62 - 30 in ver 8.2.9 and increased to 62 in ver 9.0+. ii) In limits.conf [search] min_settings_period = 60       ... View more

The investigation steps to get to the bottom of the issue below; 1. Configuration - authentication.conf  and authorize.conf.  The same configs worked in another platform it surely is not a config issue. Make sure that the role config is set as that is in the authentication server.   As per the description in the question proves the configuration is correct in authentication.conf and authorize.conf - "There is no problem on another platform with the version 9.0.4, however, this one of the version 8.1.3 with the same configs fails to search the index." 2. Mapping of roles : In the search artifacts, args.txt in $SPLUNK_HOME/var/run/splunk/dispatch/<SID>   - Mapping looks correct as expected. "roleA and roleProblem".       --id=1689370508.2666921_AAAAAAAA-75CC-4A96-8BB5-AAAAAAAAAAAA --maxbuckets=300 --ttl=300 --maxout=500000 --maxtime=8640000 --lookups=1 --reduce_freq=10 --rf=* --user=badAccount --pro --roles=roleA:roleProblem       3. Search with DEBUG : To investigate the issue further, run the searches with DEBUG. One with admin and another with a bad user. When running searches make sure to use the smallest time ranges that return just a handful events and use a 'splunk_server' to collect a diag from an indexer too.     <The_search_being_used> splunk_server=IndexerA | noop log_appender=“searchprocessAppender;maxFileSize=100000000;maxBackupIndex=10" log_debug=* set_ttl=1h i) index=goodIndex OR index=badIndex splunk_server=IndexerA ii) index=goodIndex splunk_server=IndexerA iii) index=badIndex splunk_server=IndexerA       * This could be a corner case that may not happen that often. Here's what happened from indexer diag and SID/search.log :   i) The search logs have meaningful differences in terms of the "required indexes" as below;     i - (index=goodIndex OR index=badIndex) splunk_server=indexerA | noop log_debug=* set_ttl=1h Admin - SID: 1689370032.2666760_AAAAAAA-75CC-4A96-8BB5-AAAAAAAAAAAAA 07-14-2023 18:27:15.700 INFO DispatchCommandProcessor - Search requires the following indexes="[goodIndex,badIndex]" User "badAccount" - SID: 1689370104.2666778_AAAAAAA-75CC-4A96-8BB5-AAAAAAAAAAAAA 07-14-2023 18:28:26.631 INFO DispatchCommandProcessor - Search requires the following indexes="[goodIndex]" <<<<--- badIndex missing for badAccount ii - index=goodIndex splunk_server=indexerA | noop log_debug=* set_ttl=1h Admin - SID: 1689370231.2666860_AAAAAAA-75CC-4A96-8BB5-AAAAAAAAAAAAA 07-14-2023 18:30:35.769 INFO DispatchCommandProcessor - Search requires the following indexes="[goodIndex]" User "badAccount" - SID: 1689370287.2666877_AAAAAAA-75CC-4A96-8BB5-AAAAAAAAAAAAA 07-14-2023 18:31:29.105 INFO DispatchCommandProcessor - Search requires the following indexes="[goodIndex]" iii - index=badIndex splunk_server=indexerA | noop log_debug=* set_ttl=1h (Good Search) Admin - SID: 1689370349.2666882_AAAAAAA-75CC-4A96-8BB5-AAAAAAAAAAAAA 07-14-2023 18:32:31.004 INFO DispatchCommandProcessor - Search requires the following indexes="[badIndex]" (Bad Search) User "badAccount" - SID: 1689370411.2666889_AAAAAAA-75CC-4A96-8BB5-AAAAAAAAAAAAA 07-14-2023 18:33:32.492 INFO DispatchCommandProcessor - Search requires the following indexes="[]" <<<---- badIndex missing ============       ii) Search bundle information in search.log: Further looking into the logs, the bundle for the searches  appears to be a very old one, from May 12, 2023 3:36:39 PM GMT-07:00 PST ( 2 months old already Today is July 14) .  User "badAccount" has a log message as below;  '1683930999' at the end of the log is the epoch time of the search bundle.     07-14-2023 18:33:32.247 INFO dispatchRunner - search context: user="badAccount", app="search", bs-pathname="/opt/splunk/var/run/searchpeers/AAAAAAA-75CC-4A96-8BB5-AAAAAAAAAAAAA-1683930999"       iii) Search bundle information in indexerA diag:  This turned out to be the same in the bundle information of the IndexerA diag.     -----< Excerpts from indexer diag >---------- ********** search peer bundles dir listing ********** ls -alR "/opt/splunk/var/run/searchpeers" /opt/splunk/var/run/searchpeers: total 5927728 drwx------ 96 splunk splunk 16384 Jul 14 18:45 . drwx--x--- 5 splunk splunk 52 Dec 23 2020 .. drwx------ 7 splunk splunk 163 May 12 19:35 AAAAAAA-75CC-4A96-8BB5-AAAAAAAAAAAAA-1683930902 -rw------- 1 splunk splunk 10240 May 12 19:37 AAAAAAA-75CC-4A96-8BB5-AAAAAAAAAAAAA-1683930999.delta drwx------ 7 splunk splunk 163 May 12 19:37 AAAAAAA-75CC-4A96-8BB5-AAAAAAAAAAAAA-1683930999       iv) Looking into the bundles in the SHC Captain  : There are a couple of lookup csv files over 1.7GB and removing them fixed the issue. Here's what we can do to address the bundle size issue in general.     i) To reduce the size of bundles : - use replicationBlacklist https://docs.splunk.com/Documentation/Splunk/8.1.3/Admin/Distsearchconf - Find large files, such as lookup files : $ tar -tvf Bune_Name_file | grep "lookup" | sort -nk3 ii)Or to increase maxBundleSize if there's no more file to exclude @all SH, [replicationSettings] in distsearch.conf maxBundleSize=2048 >> change [httpServer] in server.conf max_content_length = 2147483648 (2gb in bytes) >> Change @All indexers for the cluster [httpServer] in server.conf max_content_length = 2147483648 (2gb in bytes) >> Change       ... View more

  i) What's the WARN message? This is a new one added to support containerized host to the versions above 9.0.3. If it's confirmed to be hosted by a container it will use different library calls to detect system configurations/capacities.. Basically we query /sys/fs/cgroup/memory/ and /sys/fs/cgroup/cpu,cpuacct and we read /proc/1/cgroup to see if we are running inside of a container. Firstly, validates that the splunkd is running inside the container and once it validates then its reads the cpu/ram from the cgroup rather than reading from the traditional syscalls. - ii) Why it is happening after upgrade: - With the version 8.2.4 it's not there yet. It  just starts to happen after the 9.0.4.1 upgrade as it's just added to the versions above 9.0.3.   iii) What the log message tells us: - The user, splunk is not allowed to access "/proc/1/cgroup", which has even set to proper permissions  #ls -ld /proc/1 dr-xr-xr-x 9 root root 0 Apr 12 00:45 /proc/1 # ls -l /proc/1/cgroup -r--r--r-- 1 root root 0 Jun 15 17:37 /proc/1/cgroup   - Other group users should be able to access and execute commands in "/proc/1" with "r-w" but the splunk account fails to access it and puts the WARN message:   iv) The reason for the access failure: The proc file system was created with gid and hidepid=2 as below - only accessible by the account in the group 14001, that splunk account doesn't belong to; ----- mount options for proc -- $ findmnt /proc proc on /proc type proc (rw,relatime,gid=14001,hidepid=2) ------   - hidepid=0 :By default, the hidepid option has the value zero (0). This means that every user can see all data. - hidepid=1: When setting it to 1, the directories entries in /proc will remain visible, but not accessible. - hidepid=2: With value 2 they are hidden altogether. Reference: https://linux-audit.com/linux-system-hardening-adding-hidepid-to-proc/  v) Resolutions: i) add splunk uid to the gid 14001,  $usermod -aG 14001 splunk Or   ii) remove the gid and hidepid. $ mount -o remount,gid=0,hidepid=0 /proc         FYI, the redhat doc recommends hidepid should not be used with systemd in rhel 7+ due to some reason mentioned in the link, https://access.redhat.com/solutions/6704531 . ... View more

When you send data over HEC, especially for international character data, make sure to specify the charset encoding for the data set. That way the receiving end knows its charset encoding and how to decrypt. Oftentimes the encryption method is different than what you expect. Here's a good example for your reference: https://medium.com/@rysartem/sending-data-to-splunk-hec-in-a-right-way-4a84af3c44e2   ... View more

Here are my findings.   i) This has been there for a while even before the 8.2.10 upgrade - I checked some logs from old diags, it goes as far back with 8.1.x.   ii) In SHC environment, we recommend you to use source IP stickiness for any Load Balancers so that you log in to SH1 and launch a search on SH1, then gets updates through SH1.    However, it appears you requested search preview updates from another Search Head, for example, SH2 which would trigger and request SH1 for your search updates but, as it was then still running, SH1 would have put errors  as below;   ----- From SH1, IP Addr: 10.9.160.139 , error for the proxy request for a search that is still running -- 05-24-2023 13:33:49.894 -0700 ERROR SHCSlaveArtifactHandler [222579 TcpChannelThread] - Failed to trigger replication (artifact='1684960384.101_13E3A0F-27AE-49C5-9FB2-23862EDB224B') (err='event=SHPSlave::replicateArtifactTo invalid status=alive to be a source for replication') -----   iii) Logs proves the scenario :    1. The search is an adhoc with SID, "1684960384.101_13E3A0F-27AE-49C5-9FB2-23862EDB224B"  by "admin" and found in the search artifacts/dispatch dir;   $ ls -l $SPLUNK_HOME/var/run/splunk/dispatch/1684960384.101_13E3A0F-27AE-49C5-9FB2-23862EDB224B/ -rwxr-xr-x 1 support support    242 May 24 20:33 args.txt ... <SNIP> -rwxr-xr-x 1 support support 828809 May 24 20:34 search.log       2. From splunkd_ui_access.log : User, "admin" started the search with SID: 1684960384.101_13E3A0F-27AE-49C5-9FB2-23862EDB224B,   -- splunkd_ui_access.log -- 10.6.248.0 - admin  [24/May/2023:13:33:05.139 -0700] "GET /en-US/splunkd/__raw/servicesNS/admin/search/search/jobs/1684960384.101_13E3A0F-27AE-49C5-9FB2-23862EDB224B?output_mode=json&_=1684959918754 HTTP/1.1" 200 1147 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36" - 0dd2b80d72da3ed97e640b9101f0a698 5ms   3. Search string from SID:684960384.101_13E3A0F-27AE-49C5-9FB2-23862EDB224B -- search.log -- 05-24-2023 13:33:05.302 INFO  SearchParser [223948 RunDispatch] - PARSING: search index=apps sourcetype=api <SNIP>...ion  as ApplicationName | search  name!=default  pname!=zz-default | stats count by AppName pname | table AppName pname     4. 2s after the search started, a replicate request came in from 10.9.129.34 which is SH2 but immediately returned an error, 500 as it's not finished. This proxy request for the search results keeps coming in every second but always gets the error, 500 as it was not completed.    -- splunkd_access.log -- 10.9.129.34 - splunk-system-user [24/May/2023:13:33:07.016 -0700] "POST /services/shcluster/member/artifacts/1684960384.101_13E3A0F-27AE-49C5-9FB2-23862EDB224B/replicate?output_mode=json HTTP/1.1" 500 231 "-" "Splunk/8.2.10 (Linux 3.10.0-1160.49.1.el7.x86_64; arch=x86_64)" - 2ms   Search results preview request should hit the same SH as the search was launched but it appears to routed to different Search head.   Recommendation:  need to check with your LB or DNS team, such as whether LB has source IP Stickiness set or if you use DNS roundrobin between the multiple LBs  in front of SHC, that uses same FQDN or URL as this will lead the users to different LBs or Search heads.. ... View more

'splunk-MonitorNoHandle.exe' is designed to hold data when it's not able to send to UF, use unlimited memory and this symptom can happen when there are huge amount of data to forward to the indexers while the forwarding speed by UF is limited.  i) Check the queue status of parsing queue and tcpoutput queue to find which one is getting blocked first.    Parsing queue blocked firstly which means it receives over the capacity - This can happen when maxKBps is throttled to the default, 256 ,change this to 0, unlimited or something your environment allows.  - side effect it can bombard indexers if it's sending unlimited, huge amount of data.   ii) In splunkd log, find any messages showing difficulties in sending data to the next receiving ends: Even if the parsing pipeline can send more data by increasing maxKBps, if tcpoutput gets blocked then you will see the same issues again. Below are the example logs that UF has problem in connecting to the indexers ; 11-12-2021 11:00:11.365 -0500 WARN TcpOutputProc - Cooked connection to ip=172.22.1.218:9997 timed out 11-12-2021 11:01:48.391 -0500 WARN TcpOutputProc - Cooked connection to ip=172.22.1.218:9997 timed out 11-12-2021 11:12:28.757 -0500 WARN TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to output group ABC_indexers has been blocked for 500 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data. iii) Recommendations : iii-1) Parsing Queue being always full more often than the tcpoutput queue does - meaning MonitorNoHandle is sending data over the capacity that Parsing process can handle. This can happen when maxKBps is throttled to the default, 256 , then consider to increase the value according to your traffic size. http://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf  - side effect it can bombard indexers if it's sending unlimited, huge amount of data.   iii-2) How to set the memory limit used by the modInput, MonitorNoHandle; [inputproc] in limits.conf monitornohandle_max_heap_mb=5000 monitornohandle_max_driver_mem_mb=5000 ( 5000/5gb can be changed according to your environment) iii-3) If you find intermittent blockages on tcpoutput queue This can also contribute to the MonitorNoHandle's memory growth as the Parsing Pipeline can not send as much data as it receives - then MonitorNoHandle.exe has to hold the backlog data within own heap memory that can grow unexpectedly. Consider to implement asynchronous forwarding so that it can spread the data without pausing the data flow and that way MonitorNoHandle.exe may have less chances to hit the heap limit.  You can consult with our PS resources for implementation too. ... View more

You can use "splunk clean eventdata -index " command like below; $ splunk stop $ splunk clean eventdata -index "_internal" ... View more

Here's the steps to find the culprits. There are 17 crash logs found in the diag file. The crashing point from crash log files are all the same as below, ---- excerpts --- Libc abort message: splunkd: /opt/splunk/src/pipeline/indexer/JournalSlice.cpp:1780: bool ReadableJournalSliceDirectory::findEventTimeRange(st_time_t*, st_time_t*, bool): Assertion `tell() == pos' failed. ------------------ Which suggests that there must be corrupt bucket(s) or truncated but the crash log files don't have details of which bucket(s) being corrupt at all. To find the relevant logs in splunkd.log by matching the time, 14:45:24 and thread number such as, IndexerTPoolWorker-4 from the crash log file , I found the logs as below; - For crash-2022-09-07-14:45:24.log:    Crashing thread: IndexerTPoolWorker-4 Check the splunkd.log to find the logs created at 14:45:24 by IndexerTPoolWorker-4 ; splunkd.log: 09-07-2022 14:45:24.025 -0700 INFO HotBucketRoller [322700 IndexerTPoolWorker-4] - found hot bucket='/storage/tier1/20000_idx2/db/hot_v1_432' - For crash-2022-09-07-14:45:32.log: Crashing thread: IndexerTPoolWorker-7 Check splunkd.log to find the logs created at 14:45:32 by IndexerTPoolWorker-7 ; splunkd.log: 09-07-2022 14:45:32.238 -0700 INFO HotBucketRoller [322983 IndexerTPoolWorker-7] - found hot bucket='/storage/tier1/20000_idx2/db/hot_v1_432' - crash-2022-09-07-14:45:40.log: Crashing thread: IndexerTPoolWorker-6 splunkd.log: 09-07-2022 14:45:40.566 -0700 INFO HotBucketRoller [323220 IndexerTPoolWorker-6] - found hot bucket='/storage/tier1/20000_idx2/db/hot_v1_432' The crash log time and the thread number, IndexerTPoolWorker-# matches and the log messages pointing to the same bucket, '/storage/tier1/20000_idx2/db/hot_v1_432'. As it's suspected to have gone beyond the fsck repair we moved it out of the $SPLUNK_DB and then the indexer came up and running as the other peers.   This covers the specific situation of a corrupt bucket causing the indexer to crash and how to find the corrupt buckets. ... View more

There is a known issue where Indexer cluster with multi-sites for the versions prior to 8.1.8 and 8.2.4. The symptoms are exactly the same as above description - the search's working fine and finds the events in question. But with "| delete" it fails to find events that you want to delete due to no primary buckets that can be searched by delete command. - Fixed version: 8.1.8+ and 8.2.4+ , - Affected: versions in 8.1.0~8.1.7 or 8.2.0 ~ 8.2.3 - Workaround:  This may have some performance impact on SHC so you may want to use it when you hit the issue and revert it back. @Anonymous, add the below and restart the CM. [clustering] in server.conf assign_primaries_to_all_sites = true ... View more