The sequence seems to be that the captain delegates the search to one search head, but suffers the error indicated by "status=delegated_remote_error" in the first log message in the original post. Then, understandably, it delegates the search to a different search head. But the first search head also runs the search. I have seen the same problem, and when I looked at splunkd.log on the search head that the captain said "status=delegated_remote_error" for, I saw these errors: 06-29-2016 10:04:01.014 -0400 WARN ISplunkDispatch - sid:scheduler__nobody__prelert__RMD57403f7e9934e6f78_at_1467208920_3438_6408AC6F-431C-41D6-B4EE-7BE2A978D5D6 Gave up waiting for the captain to establish a common bundle version across all search peers; using most recent bundles on all peers instead 06-29-2016 10:04:55.643 -0400 WARN SHPMasterHTTPProxy - Low Level http request failure err=failed method=POST path=/services/shcluster/captain/jobs/scheduler__nobody__prelert__RMD57403f7e9934e6f78_at_1467208920_3438_6408AC6F-431C-41D6-B4EE-7BE2A978D5D6/report_job_completion captain=xsplunkm3d:8089 rc=0 actual_response_code=500 expected_response_code=200 status_line=Internal Server Error error="<response>\n <messages>\n <msg type="ERROR">\n In handler 'shclustercaptainjobs': job completion failed peer=6408AC6F-431C-41D6-B4EE-7BE2A978D5D6 sid=scheduler__nobody__prelert__RMD57403f7e9934e6f78_at_1467208920_3438_6408AC6F-431C-41D6-B4EE-7BE2A978D5D6 (reason: event=SHPMaster::handleDelegatedJobCompletion peer=6408AC6F-431C-41D6-B4EE-7BE2A978D5D6 job=scheduler__nobody__prelert__RMD57403f7e9934e6f78_at_1467208920_3438_6408AC6F-431C-41D6-B4EE-7BE2A978D5D6 did not exist in map)</msg>\n </messages>\n</response>\n" 06-29-2016 10:04:55.643 -0400 WARN SHPSlave - event=SHPSlave::delegatedJobCompletion Failed to notify captain of job completion job=scheduler__nobody__prelert__RMD57403f7e9934e6f78_at_1467208920_3438_6408AC6F-431C-41D6-B4EE-7BE2A978D5D6. reason=failed method=POST path=/services/shcluster/captain/jobs/scheduler__nobody__prelert__RMD57403f7e9934e6f78_at_1467208920_3438_6408AC6F-431C-41D6-B4EE-7BE2A978D5D6/report_job_completion captain=xsplunkm3d:8089 rc=0 actual_response_code=500 expected_response_code=200 status_line=Internal Server Error error="<response>\n <messages>\n <msg type="ERROR">\n In handler 'shclustercaptainjobs': job completion failed peer=6408AC6F-431C-41D6-B4EE-7BE2A978D5D6 sid=scheduler__nobody__prelert__RMD57403f7e9934e6f78_at_1467208920_3438_6408AC6F-431C-41D6-B4EE-7BE2A978D5D6 (reason: event=SHPMaster::handleDelegatedJobCompletion peer=6408AC6F-431C-41D6-B4EE-7BE2A978D5D6 job=scheduler__nobody__prelert__RMD57403f7e9934e6f78_at_1467208920_3438_6408AC6F-431C-41D6-B4EE-7BE2A978D5D6 did not exist in map)</msg>\n </messages>\n</response>\n" Does this give any clues to someone who knows the inner workings of this to say what's going wrong? ... View more

Not in all cases, as I have seen this problem with Splunk 6.3.3, and the bug you refer to is claimed to be fixed in 6.3.0 ... View more

You can also put the single text file containing all the individual node licenses in the $SPLUNK_HOME/etc/shcluster/apps/prelert/lic directory on your SHC deployer - the file must be called "license". Then when you push from the deployer the overall license file will immediately be on every search head in the cluster. ... View more

I had the same errors as in the original post after upgrading to Splunk 6.4, and regenerating the certificates using the splunk createssl command fixed it. ... View more

I can confirm that no special actions are required to get Splunk 6.3.3 to find its dynamic libraries on OS X 10.11 (El Capitan). I can see you've used the @executable_path and @loader_path linker variables to make it relocatable too. Many thanks for fixing this in an update rather than making everyone wait for the next major release. There is just one tiny glitch, which is that if you install on a Mac that doesn't have the Apple Developer Tools installed then part way through the installation you get a popup that says, "The "DeRez" command requires the command line developer tools. Do you want to install them now?" However, if you click "Not Now" on this popup then everything still appears to work fine. If you install on a Mac that has the Apple Developer Tools installed then everything is slick. ... View more

A couple of extra points to add to this answer are: It only works if you have the Apple developer tools installed on your Mac. Without the developer tools install_name_tool and otool just print a message saying you need to install the developer tools: xcode-select: error: no developer tools were found at '/Applications/Xcode.app', and no install could be requested (perhaps no UI is present), please install manually from 'developer.apple.com'. You can make it even more generic by replacing $splunk_home/lib/$f with "@rpath/$f" on the install_name_tool command line, plus adding a call to install_name_tool -add_rpath "@loader_path/../lib" just once for each file. In other words, change the script at the end of the answer to: !/bin/sh splunk_home="/Applications/splunk" for lib in $splunk_home/bin/splunkd $(ls $splunk_home/lib/*.dylib); do sudo install_name_tool -add_rpath "@loader_path/../lib" $lib for file in $(otool -L $lib | grep "/Users" | cut -d " " -f 1); do f=$(echo $file | rev | cut -d / -f 1 | rev) sudo install_name_tool -change $file "@rpath/$f" $lib done done Using @rpath and @loader_path means that the correct directories will be searched even if Splunk is relocated. This is the professional way to do it and I would very much hope that the Splunk dev team will be incorporating something like this into their build process in the near future! ... View more

Is any effort being made to fix the "concurrency or cpu / memory related issues that can come up"? We have done some investigation and it seems to be that there is a race condition somewhere in splunkd that means there is a chance that it tells SplunkJS that there are no running searches related to a dashboard when in fact there are. Then SplunkJS never asks again about the searches that it thought were running after being told there are no running searches related to the dashboard. I can totally believe this is a concurrency issue, as we only ever see the problem when there are multiple dashboard panels. Also, it is related to timing in some way as adding a custom search command to the searches on the dashboard makes the problem far more likely to manifest itself. And finally, it's much worse on Windows than other platforms. We find that when you have a dashboard with several panels and searches containing custom search commands on Windows some of the panels always freeze up "waiting for data". It's not really a great response to just disown the bug because it's due to "concurrency or cpu / memory related issues". ... View more

The setting works correctly in Splunk 6.2 but not in Splunk 6.3.0. It seems that any attempt to increase the limit above 1000 fails. For example, this also doesn't work in Splunk 6.3.0: <option name="charting.data.count">50000</option> You still get the message and the chart is truncated. Also, whilst I agree that you cannot fit more than about 1000 points on a screen (or maybe 1700 on a standard 1920x1080 monitor taking into account borders), if a limit of 1000 is going to be enforced it should be done by skipping points evenly through the range rather than truncating. And if this change was a deliberate design for Splunk 6.3 rather than a mistake then the documentation should have been updated. ... View more

This is a regression between Splunk 6.2 and Splunk 6.3. Is there already a bug raised for it in Splunk's bug tracking system? Or is it worth reporting to get it fixed? ... View more

Do you have custom search commands in your base search? We have seen a problem intermittently of panels hanging with the message "Waiting for data" and suspect that it happens when the base search is slow to get going. We moved all our custom search commands out of the base search and into post processes (even though they're then duplicated in all the post processes) and it avoids the hang. Having said that, maybe this problem of panels hanging intermittently is different to the problem of the original post, as the original post makes it sound like the hanging happened 100% of the time. ... View more

Thanks for the answer but I'm not convinced it's as simple as you make out. The replication_host setting in the [kvstore] stanza of server.conf seems to tell the instance which local IP address to accept connections on. However, another piece of information is needed to form the KV store replica set, namely which remote IP address to attempt to connect to. In Splunk 6.2 I cannot see any config setting to tell an arbitrary Splunk instance that it should communicate with some other arbitrary Splunk instance for KV store replication. (Obviously for search head pooling or clustering each search head knows the IP addresses of the others in the pool/cluster and can try to join the replica set consisting of the KV stores on all of these.) Also, the documentation for replication_host in server.conf (http://docs.splunk.com/Documentation/Splunk/6.2.3/Admin/Serverconf) states, "This setting has no effect on a single Splunk instance." I have done some testing trying to get two independent search heads on the same subnet to replicate their KV stores and they never attempt to talk to each other on port 8191. This is not due to firewalls: I can telnet from each of the search heads to port 8191 on the other one and get a TCP connection. So, as far as I can see, the ability to do what I want is tantalisingly close - it just needs one extra config setting to specify a remote IP address to connect to for KV store replication - but not possible in 6.2. Maybe such a setting has been added to the development version of Splunk that you have access to but I don't... ... View more

Check out dgladkikh's answer here: http://answers.splunk.com/answers/231043/how-to-configure-and-distribute-kvstore-in-a-splun.html#answer-231053 The replication_host needs to be set to a name that can be resolved by your DNS from the other members of the replica set. ... View more

I can confirm that the KV store can be made to work in a search head pooling environment using these steps. However, it's disappointing that the main KV store documentation doesn't make clearer that manually setting the replication_host setting in the [kvstore] stanza is a requirement when using search head pooling. This crucial piece of information is somewhat hidden in the monospaced text of server.conf.spec. Another thing is that if you run in a search head pooling environment without setting replication_host then mongod.log fills up with messages like: 2015-03-17T11:27:29.454Z [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) logged once per second. ... View more

It will depend how exactly JSON parsing is configured. You can set INDEXED_EXTRACTIONS=json in props.conf and then the JSON fields will be extracted on the indexer, or even earlier on a heavy forwarder if you are using one. So if you are using INDEXED_EXTRACTIONS=json then I think you need to change the limit on all indexers and heavy forwarders. In this case data that is already in your indexes will not benefit. On the other hand, you can also do JSON parsing at search time using KV_MODE=json in props.conf, the spath command or the spath() eval function. If you're doing any of these then you'd want the setting on the search head. It's conceivable you could also need the increased cutoff on the indexers as well because I don't see why JSON parsing couldn't be delegated to the indexers in a distributed environment. Obviously for JSON parsing at search time any changed cutoff will apply to previously indexed data as well as data indexed after the change. ... View more

The problem with dots in field names is also covered by this question. ... View more

Your problem is particularly tricky because your keys are IP addresses, and hence contain dots. I had a similar problem to you and managed to solve my case, but when I tried to apply the same solution to your data it didn't work, and the extra complication is the dots within the field names. For the benefit of anyone else who stumbles across this question with keys that do not contain dots, here is search that demonstrates the solution: | stats count | eval _raw="{ \"key1\":{ \"name\" : \"server1\" }, \"key2\":{ \"name\" : \"server2\" } }" | eval extract_key="key2" | spath | eval desired_name=spath(_raw, extract_key.".name") (Obviously instead of the stats count and eval _raw you'd be searching an index to get your _raw data.) The magic is that the spath() eval function can accept a variable for its second argument. By contrast the spath command cannot accept a variable for the path - it treats unquoted paths as literals. However, changing "key1" and "key2" to IP addresses in the above example breaks it. The problem is that spath uses dots as fieldname separators so when passing an IP address to the spath eval function it looks for a top level object called "10" with a nested object called "10" with a nested object called "1", etc. To summarise, because dots are separators it's impossible to search for field names containing dots with spath, but it IS possible to "dereference" a field as long as no fields on the path to it contain dots. ... View more

The reason those fields are not parsed automatically is probably due to the extraction_cutoff limit in limits.conf, which defaults to 5000 (search for extraction_cutoff in the limits.conf spec. Your example JSON is over 7000 characters long, and the 3 fields that you've said are not automatically extracted are near the end. As for why it works with the spath command, it might be that you're not using spath in "extract-all" mode. Apparently the cutoff only applies to "extract-all" mode. ... View more