It's likely a permissions issue. Have you tailed the log files while trying to access the health check? tail -100f $SPLUNK_HOME/var/log/splunk/splunkd.log ... View more

Check the user:group permissions on your deploymentclient.conf file. Also, if you defined your targetUri by hostname (not ip addr) then make certain that hostname resolves in DNS (or in /etc/hosts). nslookup your.host.name.com ... View more

How long was the captain down? If you were running on two clustered search heads only for any period of time it is possible that the cluster encountered a split-brain situation. With only two nodes, a quorum can't be reached so neither is able to (quickly) decide/vote on who should be captain. ... View more

Yes, you can handle this scenario directly in Splunk. In order to configure multiple LDAP servers, you would define them in authentication.conf. Under the [authentication] stanza, set the authSettings parameter to a comma separated list of your LDAP servers. The order in which they appear is the order of query precedence. Also worth noting, Splunk will consider a user authenticated upon the first successful LDAP auth. Meaning, if the user successfully authenticates against LDAP server one, then LDAP server two will never be queried. This can be potentially problematic if you have users/groups configured differently on different LDAP servers. Further documentation: https://docs.splunk.com/Documentation/Splunk/7.2.6/Admin/Authenticationconf ... View more

When you added parallelIngestionPipelines to server.conf on the forwarders, did you also update the indexers? The default value is 1, so increasing the value on the forwarders without increasing it on the indexers will gain you no performance increase. Also, have you checked the ulimit settings for the Splunk user and/or daemon? If not, you may want to check those, especially the open files limit. The OS default is generally 1024, which is way too low for Splunk. ... View more

If you had a local version of props.conf and removed it, then you'll likely need to cycle your search head or SHC. Then re-test. ... View more

If you set case_sensitive_match = false in the [default] stanza of $SPLUNK_HOME/etc/system/local/transforms.conf then yes, it will become global. The system local directory has the highest precedence and will override settings encountered elsewhere (app default, app local, etc). ... View more

Be sure that you don't have a local version of props.conf. If you do, it will take precedence over the version you are pushing out and override any settings there. ... View more

The pic of your output looks like the raw event from search bar results to me. Unless I'm still missing something, the data you are seeing is stored in the field _raw. If you want to add the timestamp as you're seeing in the pic just add _time as well. index=your_index_name |table _time,_raw ... View more

Still a bit confused then. You're ingesting formatted JSON, and want to display it as formatted JSON? Or you want to display it without the tags? ... View more

If I read your question correctly, you want to display the raw data. It's a bit odd, but easily accomplished: index=your_index_name | table _raw ... View more

Glad to help. ... View more

Sounds like you need an isolated development environment or more strict control of your users/roles. ... View more

Nothing bad will happen, it is just not a graceful way to bring down a process. So I was adding my disclaimer, just in case 🙂 ... View more

You have indexed more than 500MB of data. When that happens Splunk will disable searching. You'll need a new license, or a reset license. ... View more

Why not just clone your report, disable notifications, and use that as your working copy. Then after validation, move your changes up to the published version. ... View more

Find the PID of your splunk daemon: ps -ef |grep -i splunk Then kill that process: kill -9 pid_of_splunk_daemon Wait a few seconds then re-run the ps command to verify it comes back up: ps -ef |grep -i splunk Note that if you really want to test this, it should be on a DEV or non-critical box. ... View more

It should come preconfigured to do so, but you can add Restart=always to the splunk unit file. Example: [Unit] Description=Systemd service file for Splunk, generated by 'splunk enable boot-start' After=network.target [Service] Type=simple Restart=always Then reload the config: systemctl daemon-reload ... View more

Both commands apply only to configuring Splunk to start after a reboot, not a crash. ... View more

The dot notation you're using doesn't seem quite right. Specify the DM without it, and verify acceleration is working by limiting your search to only the indexed data: | tstats summariesonly=t count from datamodel=Web If your datamodel is tied to an app, be sure you are searching within that context, and your user has adequate permissions to both. ... View more

This is older but looks like it never got a solid answer. The easiest solution is to perform a simple port check. I don't use F5, but the haproxy equivalent is such: balance source server shca101 :8000 weight 1 maxconn 2000 check port 8000 ssl verify none server shca102 :8000 weight 1 maxconn 2000 check port 8000 ssl verify none server shca103 :8000 weight 1 maxconn 2000 check port 8000 ssl verify none Before sending traffic to any of the search heads the load balancer verifies that port 8000 is active. Assuming your web UI is using the default port 8000 of course. ... View more

As I suspected, sdb is already in use. Though it does not have a physical volume created, it is part of the centos VG with a 443 home LV created on it and mapped to /home. Just a heads up, the lack of a physical volume (PV) could be problematic, and your system may not survive a reboot. You may want to test that. In any case, I do not see a new, unused 160GB disk that you described. You likely need to scan the SCSI bus. In order to do that you can run the following commands: echo "- - -" > /sys/class/scsi_host/host0/scan echo "- - -" > /sys/class/scsi_host/host1/scan echo "- - -" > /sys/class/scsi_host/host2/scan Generally on VM's you will have 3 scsi hosts (host0,host1,host2) but you may have more or fewer. After you run those commands you should see a new disk come in as /dev/sdc with no partition. Find it by re-running the fdisk command: fdisk -l |grep sd ... View more