The sequence seems to be that the captain delegates the search to one search head, but suffers the error indicated by "status=delegated_remote_error" in the first log message in the original post. Then, understandably, it delegates the search to a different search head. But the first search head also runs the search. I have seen the same problem, and when I looked at splunkd.log on the search head that the captain said "status=delegated_remote_error" for, I saw these errors: 06-29-2016 10:04:01.014 -0400 WARN ISplunkDispatch - sid:scheduler__nobody__prelert__RMD57403f7e9934e6f78_at_1467208920_3438_6408AC6F-431C-41D6-B4EE-7BE2A978D5D6 Gave up waiting for the captain to establish a common bundle version across all search peers; using most recent bundles on all peers instead 06-29-2016 10:04:55.643 -0400 WARN SHPMasterHTTPProxy - Low Level http request failure err=failed method=POST path=/services/shcluster/captain/jobs/scheduler__nobody__prelert__RMD57403f7e9934e6f78_at_1467208920_3438_6408AC6F-431C-41D6-B4EE-7BE2A978D5D6/report_job_completion captain=xsplunkm3d:8089 rc=0 actual_response_code=500 expected_response_code=200 status_line=Internal Server Error error="<response>\n <messages>\n <msg type="ERROR">\n In handler 'shclustercaptainjobs': job completion failed peer=6408AC6F-431C-41D6-B4EE-7BE2A978D5D6 sid=scheduler__nobody__prelert__RMD57403f7e9934e6f78_at_1467208920_3438_6408AC6F-431C-41D6-B4EE-7BE2A978D5D6 (reason: event=SHPMaster::handleDelegatedJobCompletion peer=6408AC6F-431C-41D6-B4EE-7BE2A978D5D6 job=scheduler__nobody__prelert__RMD57403f7e9934e6f78_at_1467208920_3438_6408AC6F-431C-41D6-B4EE-7BE2A978D5D6 did not exist in map)</msg>\n </messages>\n</response>\n" 06-29-2016 10:04:55.643 -0400 WARN SHPSlave - event=SHPSlave::delegatedJobCompletion Failed to notify captain of job completion job=scheduler__nobody__prelert__RMD57403f7e9934e6f78_at_1467208920_3438_6408AC6F-431C-41D6-B4EE-7BE2A978D5D6. reason=failed method=POST path=/services/shcluster/captain/jobs/scheduler__nobody__prelert__RMD57403f7e9934e6f78_at_1467208920_3438_6408AC6F-431C-41D6-B4EE-7BE2A978D5D6/report_job_completion captain=xsplunkm3d:8089 rc=0 actual_response_code=500 expected_response_code=200 status_line=Internal Server Error error="<response>\n <messages>\n <msg type="ERROR">\n In handler 'shclustercaptainjobs': job completion failed peer=6408AC6F-431C-41D6-B4EE-7BE2A978D5D6 sid=scheduler__nobody__prelert__RMD57403f7e9934e6f78_at_1467208920_3438_6408AC6F-431C-41D6-B4EE-7BE2A978D5D6 (reason: event=SHPMaster::handleDelegatedJobCompletion peer=6408AC6F-431C-41D6-B4EE-7BE2A978D5D6 job=scheduler__nobody__prelert__RMD57403f7e9934e6f78_at_1467208920_3438_6408AC6F-431C-41D6-B4EE-7BE2A978D5D6 did not exist in map)</msg>\n </messages>\n</response>\n" Does this give any clues to someone who knows the inner workings of this to say what's going wrong? ... View more

Not in all cases, as I have seen this problem with Splunk 6.3.3, and the bug you refer to is claimed to be fixed in 6.3.0 ... View more

Is any effort being made to fix the "concurrency or cpu / memory related issues that can come up"? We have done some investigation and it seems to be that there is a race condition somewhere in splunkd that means there is a chance that it tells SplunkJS that there are no running searches related to a dashboard when in fact there are. Then SplunkJS never asks again about the searches that it thought were running after being told there are no running searches related to the dashboard. I can totally believe this is a concurrency issue, as we only ever see the problem when there are multiple dashboard panels. Also, it is related to timing in some way as adding a custom search command to the searches on the dashboard makes the problem far more likely to manifest itself. And finally, it's much worse on Windows than other platforms. We find that when you have a dashboard with several panels and searches containing custom search commands on Windows some of the panels always freeze up "waiting for data". It's not really a great response to just disown the bug because it's due to "concurrency or cpu / memory related issues". ... View more

The setting works correctly in Splunk 6.2 but not in Splunk 6.3.0. It seems that any attempt to increase the limit above 1000 fails. For example, this also doesn't work in Splunk 6.3.0: <option name="charting.data.count">50000</option> You still get the message and the chart is truncated. Also, whilst I agree that you cannot fit more than about 1000 points on a screen (or maybe 1700 on a standard 1920x1080 monitor taking into account borders), if a limit of 1000 is going to be enforced it should be done by skipping points evenly through the range rather than truncating. And if this change was a deliberate design for Splunk 6.3 rather than a mistake then the documentation should have been updated. ... View more

This is a regression between Splunk 6.2 and Splunk 6.3. Is there already a bug raised for it in Splunk's bug tracking system? Or is it worth reporting to get it fixed? ... View more

Thanks for the answer but I'm not convinced it's as simple as you make out. The replication_host setting in the [kvstore] stanza of server.conf seems to tell the instance which local IP address to accept connections on. However, another piece of information is needed to form the KV store replica set, namely which remote IP address to attempt to connect to. In Splunk 6.2 I cannot see any config setting to tell an arbitrary Splunk instance that it should communicate with some other arbitrary Splunk instance for KV store replication. (Obviously for search head pooling or clustering each search head knows the IP addresses of the others in the pool/cluster and can try to join the replica set consisting of the KV stores on all of these.) Also, the documentation for replication_host in server.conf (http://docs.splunk.com/Documentation/Splunk/6.2.3/Admin/Serverconf) states, "This setting has no effect on a single Splunk instance." I have done some testing trying to get two independent search heads on the same subnet to replicate their KV stores and they never attempt to talk to each other on port 8191. This is not due to firewalls: I can telnet from each of the search heads to port 8191 on the other one and get a TCP connection. So, as far as I can see, the ability to do what I want is tantalisingly close - it just needs one extra config setting to specify a remote IP address to connect to for KV store replication - but not possible in 6.2. Maybe such a setting has been added to the development version of Splunk that you have access to but I don't... ... View more

Check out dgladkikh's answer here: http://answers.splunk.com/answers/231043/how-to-configure-and-distribute-kvstore-in-a-splun.html#answer-231053 The replication_host needs to be set to a name that can be resolved by your DNS from the other members of the replica set. ... View more

It will depend how exactly JSON parsing is configured. You can set INDEXED_EXTRACTIONS=json in props.conf and then the JSON fields will be extracted on the indexer, or even earlier on a heavy forwarder if you are using one. So if you are using INDEXED_EXTRACTIONS=json then I think you need to change the limit on all indexers and heavy forwarders. In this case data that is already in your indexes will not benefit. On the other hand, you can also do JSON parsing at search time using KV_MODE=json in props.conf, the spath command or the spath() eval function. If you're doing any of these then you'd want the setting on the search head. It's conceivable you could also need the increased cutoff on the indexers as well because I don't see why JSON parsing couldn't be delegated to the indexers in a distributed environment. Obviously for JSON parsing at search time any changed cutoff will apply to previously indexed data as well as data indexed after the change. ... View more

The problem with dots in field names is also covered by this question. ... View more

Your problem is particularly tricky because your keys are IP addresses, and hence contain dots. I had a similar problem to you and managed to solve my case, but when I tried to apply the same solution to your data it didn't work, and the extra complication is the dots within the field names. For the benefit of anyone else who stumbles across this question with keys that do not contain dots, here is search that demonstrates the solution: | stats count | eval _raw="{ \"key1\":{ \"name\" : \"server1\" }, \"key2\":{ \"name\" : \"server2\" } }" | eval extract_key="key2" | spath | eval desired_name=spath(_raw, extract_key.".name") (Obviously instead of the stats count and eval _raw you'd be searching an index to get your _raw data.) The magic is that the spath() eval function can accept a variable for its second argument. By contrast the spath command cannot accept a variable for the path - it treats unquoted paths as literals. However, changing "key1" and "key2" to IP addresses in the above example breaks it. The problem is that spath uses dots as fieldname separators so when passing an IP address to the spath eval function it looks for a top level object called "10" with a nested object called "10" with a nested object called "1", etc. To summarise, because dots are separators it's impossible to search for field names containing dots with spath, but it IS possible to "dereference" a field as long as no fields on the path to it contain dots. ... View more

The reason those fields are not parsed automatically is probably due to the extraction_cutoff limit in limits.conf, which defaults to 5000 (search for extraction_cutoff in the limits.conf spec. Your example JSON is over 7000 characters long, and the 3 fields that you've said are not automatically extracted are near the end. As for why it works with the spath command, it might be that you're not using spath in "extract-all" mode. Apparently the cutoff only applies to "extract-all" mode. ... View more