As of this writing, the filter implementation only support "&(AND)" operator and no OR. This poses a limit in various use-cases. According to the doc, https://docs.splunk.com/Documentation/AddOns/released/ServiceNow/Configureinputs which gives us hints that it accepts different key names in the filter. { Enter filters, in key-value pairs for indexing selected data from the table. For example, key1=value1&key2=value2. By default, there is no filter. } You may want to share your ideas and concerns via https://ideas.splunk.com . Hope it helps. ... View more

If you find no crash logs even if the splunkd crash log messages are found in splunkd.log such as below; 12-08-2020 14:08:20.238 -0400 ERROR ProcessRunner - helper process seems to have died (child killed by signal 9: Killed)! 12-08-2020 21:08:56.398 -0400 INFO ServerConfig - My GUID is F428C1-CB1F-4A95-85B5-6DD86B It must have been killed by other signals that the applications can not handle properly - mostly sent by kernel like SIGKILL. i) if you use initd and OOM killer enabled. Then check /var/log/messages, search for "OOM" or "out of memory" to see if it kills splunkd process. - If you use systemd the Splunk Service will restart right after the crash so there will not be much noticeable outage. Then check which processes are using more memory compared to the others at the time of crash - Use Monitoring Console for this. Make sure that THP is disabled, https://docs.splunk.com/Documentation/Splunk/8.0.3/ReleaseNotes/SplunkandTHP Check if, at the time of crash, any heavy searches ran hogging more than usual memories using Monitoring Console. If then, you may want to implement the memory tracker for search processes to prevent the service outage. https://docs.splunk.com/Documentation/Splunk/8.0.3/Admin/limitsconf#Memory_tracker After memory_tracker is enable you can find how many searches are affected by the memory tracker, search for "Forcefully terminated search process" . https://docs.splunk.com/Documentation/Splunk/8.0.3/Search/Limitsearchprocessmemoryusage ii) If you use systemd with splunk version prior to 7.2.2 or some of 7.2.X version the splunkd process could get killed way before it reaches the maximum memory configured. Check the systemd unit file for the parameter, MemoryLimit is accidentally set to 100G while you have a lot more allowed. Then configure it to the reasonable size .. maybe 90% of the max mem. iii) Check the ulimit for, like open files accidentally it came up with 4096 which is system default value and suffers lack of FD. 04-17-2020 23:15:19.073 -0700 INFO ulimit - Limit: open files: 4096 files Also check this out too - https://docs.splunk.com/Documentation/Splunk/8.0.3/Troubleshooting/ulimitErrors There could be various splunkd log messages caused by the lack of FD, below could be one of them - "Too many open file"; 04-05-2019 11:50:06.415 +1000 WARN SelfPipe - TcpChannelThread: about to throw a SelfPipeException: can't create selfpipe: *Too many open files*** iv) Also find the recommended hardware spec in the doc and try to add enough resources as per your daily usage of the server. https://docs.splunk.com/Documentation/Splunk/8.0.3/Capacity/Referencehardware v) If you find any crash logs in $SPLUNK_HOME/var/log/splunk then open a Splunk Support ticket along with a diag attachment for further analysis. ... View more

Typically this can happen due to the precedence between the apps & add-ons. As the different apps/add-ons are installed on SHC SH and the standalone SH the first thing is to try to find the differences. Use find . -name "props.conf" -o -name "transforms.conf" | xargs grep "aws:cloudtrail" this will tell us what files have configurations for the sourcetype in question. This time, no other "aws:cloudtrail" related configuration in the standalone search head. However, in the SHC there is another props.conf in SAI, which has many other aws:XYZ sourcetypes, also it has a suspicious one, i.e, kv_mode = none - this is basically disabling automatic key-value field extractions. [aws:cloudtrail] in splunkappinfrastructure/default/props.conf KV_MODE = none The props.conf in SAI has higher precedence than what is defined in splunkappaws, "kvmode=json" . https://docs.splunk.com/Documentation/Splunk/8.0.3/Admin/Wheretofindtheconfigurationfiles#Howappdirectorynamesaffectprecedence " In the app/user context, precedence is determined instead by reverse-lexicographical order. " To solve this problem, comment out the kv_mode in SAI. ... View more

It turned out to be a firewall issue and to get to the point, here's what I did ; i) Configuration : Check if there's any configuration missing -> All good according to the docs. ii) Certs : Use the same certs and configuration from A for B so that it can eliminate the mistakes during the cert creation or configs on forwarder B. -> still gets error. iii) Certs: Use default certs and see if it is still the same -> still the same issue happens. This tells us it could be an issue caused by outside splunk entities. iv) splunk cmd openssl sclient -connect C:9997 -cert   from A and B to see what happens.    HF A - OK HF B still fails. v) Network entities sitting between B and C could break sessions: Capture tcp packets to get some hints for sockerror=104, to understand why the other end sends tcp reset (104)    Capture it from both end as network entity in between is suspicious.  v-1)On B,  tcpdump -i any -ta  -s0 host C and port 9997 -w output.pcap  v-2)On C, tcpdump -i any -ta  -s0 host B and port 9997 -w output.pcap Findings from the step v) v-1) pcap from HF B shows the intermediary HF C is disconnecting SSL in the middle of handshake. C sends to B a RST which is expected as the splunkd log message says so. v-2) pcap from HF C says differently - HF B sends a RST to HF C Judging by v-1) & v-2) outcomes there must be a network entity that breaks the session by sending RSTs to both ends. This has been fixed with the help of network team by fixing a firewall configuration. ... View more

Here's what you need to do to achieve what you want, [root]$visudo "to add the below further to the what you have done earlier" splunk ALL=(root) NOPASSWD: /usr/bin/systemctl restart Splunkd.service splunk ALL=(root) NOPASSWD: /usr/bin/systemctl stop Splunkd.service splunk ALL=(root) NOPASSWD: /usr/bin/systemctl start Splunkd.service splunk ALL=(root) NOPASSWD: /usr/bin/systemctl status Splunkd.service splunk ALL=(root) NOPASSWD: /opt/splunk/bin/splunk restart splunk ALL=(root) NOPASSWD: /opt/splunk/bin/splunk stop splunk ALL=(root) NOPASSWD: /opt/splunk/bin/splunk start Then, "sudo ./splunk start" or "sudo /opt/splunk/bin/splunk start" will work without requesting password. And still "sudo splunk start" will needs authentication. To make "sudo splunk start" work passwordless then add /opt/splunk/bin to secure_path. visudo & locate "secure_path", then add "/opt/splunk/bin" to the end. i.e that would look like as below; Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/opt/splunk/bin Save the change, confirm it has been configured Okay by "$ visudo -c " Use sudo command to splunk start/stop/restart, such as $sudo splunk start If you have concerns about the Security by using secure_path you better use systemctl or just use /opt/splunk/bin/splunk command with full path. ... View more

This appears to be caused by Kernel bugs as this has been reproduced constantly with Splunk version 7.2.4.2 and Linux, 3.10.0-957.12.1.el7.x8664. Now with the latest version as of this writing, 3.10.0-1062.4.1.el7.x8664 it works fine as described in the splunk doc. It's been always reproduced with Ubuntu 16.04.1 but not with Ubuntu 16.04.5 - whoever sees this issue it'd be worth while to upgrade the OS kernel. ... View more

As per our doc - I agree it's not very clear, the srchIndexesDefault is used when there is no index listed in the search but when you use srchFilter with index, like "index!=main" which is eventually added into the search, " index!=main ". Because of this replacement the srchIndexesDefault will not be considered during the search time. Here's excerpts from doc srchIndexesDefault = semicolon-separated list A list of indexes to search when no index is specified. To work around it, adjust srchIndexesAllowed so that it doesn't include the indexes you do not want for the roles, or change your searches to explicitly specify the index(es) that a search should look at. ... View more