In any clustered environment, Splunk or otherwise, you must have an odd number of cluster members in order to prevent split-brain situations. ... View more

Verify your ulimit settings are correct for the splunk daemon. If you are running Splunk via systemd, make sure the memory limit is set correctly within your unit file within the [Service] stanza /etc/systemd/system/multi-user.target.wants/Splunkd.service [Service] ... MemoryLimit=100G ... ... View more

Yes, you most likely have one or more pass4SymmKey values set incorrectly. Ensure these values are set correctly within /opt/splunk/etc/system/local/server.conf On the master: [general] serverName = fqdn_of_your_master pass4SymmKey = cluster_wide_password_shared_across_all_nodes [clustering] mode = master replication_factor = int_value search_factor = int_value pass4SymmKey = password_unique_to_this_index_cluster # Note, this is NOT the same p/w as under [general] cluster_label = a_unique_label_for_your_index_cluster # Note, indexers only [indexer_discovery] pass4SymmKey = password_unique_to_indexer_discovery # assuming you use this On each indexer: [general] serverName = fqdn_of_your_indexer pass4SymmKey = cluster_wide_password_shared_across_all_nodes [clustering] mode = slave pass4SymmKey = password_unique_to_this_index_cluster master_uri = full_uri_and_port_of_your_master # e.g. https://mymasternode.com:8089 Then cycle the master, and each indexer one by one. ... View more

Your ulimits are not set correctly, or are using the system defaults. As a result, splunkd is likely using more memory than allowed or available, so the kernel kills the process in order to protect itself. ... View more

Your regex is causing catastrophic backtracking. You need to check it with different types of examples, and more of them. By adding just two spaces to the example you pasted I was able to cause it to fail. Splunk (thankfully) protects itself from catastrophic backtracking with a limit of 100K. ... View more

Is your forwarder running as the splunk user? Files and directories under /var/log/ are typically owned by root. Try changing the user on your forwarder, or change the ownerships of your directory under /var/log/ ... View more

Below is the unit file generated by Splunk 7.2.6 #This unit file replaces the traditional start-up script for systemd #configurations, and is used when enabling boot-start for Splunk on #systemd-based Linux distributions. [Unit] Description=Systemd service file for Splunk, generated by 'splunk enable boot-start' After=network.target [Service] Type=simple Restart=always ExecStart=/opt/splunk/bin/splunk _internal_launch_under_systemd LimitNOFILE=65536 SuccessExitStatus=51 52 RestartPreventExitStatus=51 RestartForceExitStatus=52 User=splunk Delegate=true MemoryLimit=100G CPUShares=1024 KillMode=mixed KillSignal=SIGINT TimeoutStopSec=10min PermissionsStartOnly=true ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/%n" ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/%n" [Install] WantedBy=multi-user.target ... View more

I've encountered this previously, especially on v7.2.x. One thing you need to change in your unit file is the type. Set Type=simple in your [Service] stanza, instead of forking. Also, check the SPLUNK_SERVER_NAME setting in /opt/splunk/etc/splunk-launch.conf # SPLUNK_OS_USER #SPLUNK_SERVER_NAME=Splunkd SPLUNK_OS_USER=splunk If the value is set there, comment it out and cycle Splunk. The setting there overrides what is in server.conf and causes issues. Also in 7.2.x, if you run splunk enable boot-start it will generate a properly formed systemd unit file. Be sure to follow that up with enable and start. /opt/splunk/bin/splunk enable boot-start -user root -systemd-managed 1 systemctl enable Splunkd.service systemctl restart Splunkd.service ... View more

Is the file and directory owned by splunk? ... View more

You have an errant pipe in your search between main and dedup: ERROR SearchParser - Missing a search command before '|'. Error at position '2086' of search query '| tstats count AS count sum(Web_Access_Event.bytes...{snipped} {errorcontext = main | |dedup user}'. ... View more

Your file does not have an extension. Splunk interprets files with no extension as binary files and will not ingest them. ... View more

Sure, glad to help! ... View more

It appears you have two issues going on here. First, it is not necessary to pass in "auth" to restart splunk, and is actually invalid as you can see from your output. Next, you have a syntax error in your outputs.conf that you should check using btool. splunk btool check --debug If you still have problems with that file after correcting those issues I would suggest you also add crcSalt to your inputs.conf for the directory you are monitoring. crcSalt = <SOURCE> ... View more

This should return what you're looking for: index=_audit user=admin action="login attempt" ... View more

Did you cycle Splunk after modifying limits.conf? ... View more

Not without being absolutely certain that disk is not in use. How did you come to end up with /dev/sdb3 ? Was that partition created after you presented the new disk? The size just doesn't match up. Can you provide output from the pvs command as well? ... View more

Replace your opening and closing quotation marks with single ticks to define the contents as a literal string. curl -ku admin:admin https://192.168.1.4:8089/servicesNS/admin/search/search/jobs/export -d search='search index=text|rex field=_raw \"ApplicationRegistry-(?<text>.*)\" max_match=0 |table source,sourcetype,text' -d output_mode=json Then it should work for you. ... View more

Glad to help, and I fought the same issue myself previously, and only on reboots. The underlying problem is that Splunk is running under init.d on a systemd system and limits are applied differently than the older init.d. The sequence in which limits are read and applied by the kernel and process are out of sync on reboot so it falls back to the OS defaults. You can solve it by creating a systemd unit file for splunk as it should technically be configured, but placing the limits in the startup script solved the issue for me. Also, I would consider the cat /proc/splunk_pid/limits method as the definitive source of truth for what limits have been applied to the process. Hope this helps. ... View more

Glad to help. Yes I have used it successfully. And yes to your scenario. If LDAP1 is down the LDAP2 is tried next, assuming it is next in your comma separated list. The order of appearance is the order of precedence. ... View more

There are a couple of methods to verify the ulimits took effect. Check ulimits as splunk user su - splunk ulimit -a Check via PID: ps -ef |grep -i splunk (copy any of the PID's in the output) cat /proc/splunk_pid/limits Worth noting for you or your admin, setting ulimits via /etc/security/limits.conf is generally considered deprecated on RHEL/Centos 7.x (or any systemd based OS). The preferred method is via conf files located at /etc/security/limits.d/ When the OS boots up, /etc/security/limits.conf is read first, then each file under /etc/security/limits.d/ is read sequentially and can/will override any previous files (with 99 being the highest). Meaning, any limits set in /etc/security/limits.d/99-mylimits.conf will override all previous settings. I suspect something similar happened in your case. ... View more

I suspected that may be the case. Your ulimit configurations were not honored and reverted back to the OS defaults upon reboot (expected behavior). Depending on your OS flavor and version there a number of methods to resolve this. You can create a splunk specific config by creating a file at /etc/security/limits.d/ and name the file with a number higher than what exists there now, 90-splunk.conf e.g. Or you add the limits directly to the start function in the init.d script as such: cat /etc/init.d/splunk splunk_start() { ulimit -Sn 64000 ulimit -Hn 100000 ulimit -Su 8192 ulimit -Hu 16000 echo Starting Splunk... "/opt/splunk/bin/splunk" start --no-prompt --answer-yes RETVAL=$? [ $RETVAL -eq 0 ] && touch /var/lock/subsys/splunk In either case you'll need to cycle splunk in order to pick up the "new" limits. You can also set them via systemd, but depending on your version of Splunk this can be a pain. I prefer to just drop them in the init.d script, it's proven the most reliable. ... View more

If you are standing up a new cluster on new hardware, and want to transfer your indexes over it is pretty straightforward, if you have an identical number of indexers. If not, it becomes a bit more complicated. If you do though, here are the high level steps: On new cluster: Deploy your existing indexes.conf but set the index(es) to disabled = 1 On old cluster: Roll all hot buckets to warm Disable the index From old cluster indexers to new cluster indexes: rsysnc -az /path/to/your/data new_indexer_name:/path/to/your/data/ Repeat that for each path defined (hot,warm,cold,etc), for each index, and for each indexer (old_indexer02 to new_indexer02 e.g.). On the new cluster: Verify/update user:group ownerships on the new data paths (chown -RP splunk:splunk /path/to/your/data/ e.g.). Enable the index Splunk may do some housekeeping, so to speak, once you enable the index. But after that you should be good. ... View more

"No route to host" generally means that DNS can't resolve or a firewall is blocking access. Check your DNS configuration at /etc/resolv.conf Test DNS resolution with nslookup: nslookup google.com Check for firewall rules that may be blocking you: iptables -L ... View more

Did this help you get the disk scanned in? ... View more