Hi, I have two clustered indexers which are now constantly generating crash logs in /splunk/var/log/splunk every few minutes and is unable to figure out the cause from the crash log or the error in splunkd.log. Would anyone here be able to shed some light on this? Splunkd Error: WARN SearchProcessRunner [19356 PreforkedSearchesManager-0] - preforked process=0/38 status=killed, signum=6, signame="Aborted", coredump=1, uptime_sec=37.282768, stime_sec=19.850199, max_rss_kb=472688, vm_minor=902282, vm_major=37, fs_r_count=608, fs_w_count=50856, sched_vol=3413, sched_invol=10923 Contents of one of the crash.log: [build b6436b649711] 2023-11-02 11:39:40 Received fatal signal 6 (Aborted) on PID 23624. Cause: Signal sent by PID 23624 running under UID 1001. Crashing thread: BucketSummaryActorThread Registers: RIP: [0x00007F0D7E2DA387] gsignal + 55 (libc.so.6 + 0x36387) RDI: [0x0000000000005C48] RSI: [0x00000000000059CC] RBP: [0x0000000000000BE7] RSP: [0x00007F0CF85F2268] RAX: [0x0000000000000000] RBX: [0x0000562A9ADF7598] RCX: [0xFFFFFFFFFFFFFFFF] RDX: [0x0000000000000006] R8: [0x00007F0CF85FF700] R9: [0x00007F0D7E2F12CD] R10: [0x0000000000000008] R11: [0x0000000000000206] R12: [0x0000562A9AC0E070] R13: [0x0000562A9AF9CFB0] R14: [0x00007F0CF85F2420] R15: [0x00007F0CF806F260] EFL: [0x0000000000000206] TRAPNO: [0x0000000000000000] ERR: [0x0000000000000000] CSGSFS: [0x0000000000000033] OLDMASK: [0x0000000000000000] Regards, Zijian ... View more

Hi, One of our three clustered indexers is having search errors and high CPU fluctuations for splunkd main process after an improper reboot as follows: In splunk web search: remote search process failed on peer Search results might be incomplete: the search process on the peer:[Affected indexer] ended prematurely. Check the peer log, such as $SPLUNK_HOME/var/log/splunk/splunkd.log and as well as the search.log for the particular search. [Affected indexer] Search process did not exit cleanly, exit_code=111, description="exited with error: Application does not exist: Splunk_SA_CIM". Please look in search.log for this peer in the Job Inspector for more info. In splunkd.log of affected indexer: WARN SearchProcessRunner [31756 PreforkedSearchesManager-0] - preforked process=0/101006 with search=0/127584 exited with code=111 ERROR SearchProcessRunner [31756 PreforkedSearchesManager-0] - preforked search=0/127584 on process=0/101006 caught exception: used=1, bundle=7471316304185390773, workload_pool=, generation=11, age=7.418, runtime=7.203, search_started_ago=7.204, search_ended_ago=0.000 ERROR SearchProcessRunner [31756 PreforkedSearchesManager-0] - preforked process=0/101006 with search=0/127584 and cmd=splunkd\x00search\x00--id=remote_SH-ES_scheduler__splunkadmin__SplunkEnterpriseSecuritySuite__RMD5852d4ed30e6a890b_at_1698892200_90939\ x00--maxbuckets=0\x00--ttl=60\x00--maxout=0\x00--maxtime=0\x00--lookups=1\x00--streaming\x00--sidtype=normal\x00--outCsv=true\x00--acceptSrsLevel=1\ died on exception (exit_code=111): Application does not exist: SplunkEnterpriseSecuritySuite   WARN PeriodicReapingTimeout [30157 DispatchReaper] - Spent 10650ms reaping search artifacts in /splunk/var/run/splunk/dispatch WARN DispatchReaper [30157 DispatchReaper] - The number of search artifacts in the dispatch directory is higher than recommended (count=6608, warning threshold=5000) and could have an impact on search performance. Remove excess search artifacts using the "splunk clean-dispatch" CLI command, and review artifact retention policies in limits.conf and savedsearches.conf. You can also raise this warning threshold in limits.conf / dispatch_dir_warning_size.   WARN DispatchManager [13827 TcpChannelThread] - quota enforcement for user=splunk_user1, sid=soc_user1_c29jX2Njb191c2VyMQ__SplunkEnterpriseSecuritySuite__RMD57f02abc0263583b0_1697962710.21728, elapsed_ms=23865, cache_size=1591 took longer than 15 seconds. Poor search start performance will be observed. Consider removing some old search job artifacts.   Regards, Zijian ... View more

Hi all, I found that searches in my unix index returns events only up to the past two months for a significant number of sourcetypes (bash_history, audit, secure, sudo logs). Shouldn't the events be retained according to the retention period set using 'frozenTimePeriodInSecs'? We set the period to 365 days.   Regards, Zijian   ... View more

Hi, I am using Splunk Enterprise Version 9 where the new index _configtracker is able to show changes made to configuration files. However, it is hard to identify the changes made to a correlation search in savedsearches.conf at a glance or use the data.changes{}.properties{}.new_value field as it contains multiple values. Furthermore, the change is spread over two events where one shows data.changes{}.properties{}.new_value (post-change field.jpg) and the other shows data.changes{}.properties{}.old_value (empty values) How can I compare all the multiple values under the field and return the property that is being changed? I am guessing I can link the two events using the "new_checksum" and "old_checksum". I removed most of the fields to make it easier to read and changed the content of the SPL to <Search content> to mask some information. Pre-change raw details: {"datetime":"06-21-2022 16:29:41.119 +0800","log_level":"INFO ","component":"ConfigChange","data":{"path":"/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/savedsearches.conf","action":"update","modtime":"Tue Jun 21 16:29:41 2022","epoch_time":"1655800181","new_checksum":"0x621552b3fcbdfc9e","old_checksum":"0x95c4bf5f0b449f9","changes":[{"stanza":"Endpoint - Linux/MS - Server Reboot/Shutdown - Rule","properties":[{"name":"action.correlationsearch.annotations","new_value":"","old_value":"{}"},{"name":"realtime_schedule","new_value":"","old_value":"0"}, {"name":"search","new_value":"","old_value":"(<Search content>"}]}]}} Post-change raw details: {"datetime":"06-21-2022 16:29:41.642 +0800","log_level":"INFO ","component":"ConfigChange","data":{"path":"/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/savedsearches.conf","action":"update","modtime":"Tue Jun 21 16:29:41 2022","epoch_time":"1655800181","new_checksum":"0xf5867665b8a15f4","old_checksum":"0x621552b3fcbdfc9e","changes":[{"stanza":"Endpoint - Linux/MS - Server Reboot/Shutdown - Rule","properties":[{"name":"action.correlationsearch.annotations","new_value":"{}","old_value":""},{"name":"realtime_schedule","new_value":"0","old_value":""}, {"name":"search","new_value":"(<Search content>","old_value":""}]}]}}   Regards, Zijian ... View more