You can search API based access to the search heads using the following search: index=_internal sourcetype=splunkd_access host= NOT ( user="splunk-system-user" OR user="-") NOT clientip="127.0.0.1" NOT clientip IN() GUI traffic all comes from localhost on the search heads, and any other splunk related traffic (deployer, cluster-master, dmc) will use the splunk-system-user username. Anything with a username of “-“ is unauthenticated and will be blocked or part of the login flow. The only edge case is when search heads are clustered and they proxy requests for jobs that have already been run. The job requests themselves have an “isProxyRequest” query string field that we can filter on, but you may also need to filter out anything coming from IPs of the other search heads in the cluster. Everything else will be authenticated API traffic. You can then analyze the URLs to see if they are running searches or taking other administrative actions. Requests with a POST method are taking action against the API, like running a search or editing a config If you want to see the actual searches that are being run, you can pull out the search ids from the URLs and search for them in the audit logs. index=_internal sourcetype=splunkd_access host=<search_head_hosts> NOT ( user="splunk-system-user" OR user="-") NOT clientip="127.0.0.1" NOT clientip IN(<shc_ips>) | rex field=uri_path "search/jobs/(?<search_id>[^/\?]+)" | eval search_id="'"+coalesce(search_id,id)+"'" | stats count by search_id | eval isAPIsearch="True" | outputlookup api_searches.csv and index=_audit action=search (info=granted search=*) OR info=completed | lookup api_searches.csv search_id | search isAPIsearch="True" | stats earliest(_time) AS _time, values(user) AS user, values(host) AS search_head, values(search) AS search, values(event_count) AS event_count, values(result_count) AS result_count, values(total_run_time) AS total_run_time by search_id Hope that helps. ... View more

The docs describe the new last_conf_replication state field in the shcluster-status command: "The last_conf_replication field indicates when the member last pulled a set of configurations from the captain" http://docs.splunk.com/Documentation/Splunk/6.5.2/DistSearch/ViewSHCstatus#Show_cluster_status So, it makes sense that the master will not have this status field. Not sure why all of the members would be stuck in a "Pending" state after an upgrade though. Maybe there is just a lot of config information to pull down? Are you seeing any other SHC related log messages that might indicate a problem? ... View more

Under the covers the export button calls the /en-US/api/search/jobs/ /results?isDownload=true endpoint. I’ve included an example of the full call below. Also see the docs on the job results endpoint. http://127.0.0.1:8000/en-US/api/search/jobs/1395156402.45/results?isDownload=true&timeFormat=%25FT%25T.%25Q%25%3Az&maxLines=0&count=0&filename=results.csv&outputMode=csv&spl_ctrl-limit=unlimited&spl_ctrl-count=1000 You would have to write some JS code to get the search_id from the search manager and then call the API endpoint to get the file. I don’t have any sample code, but hopefully that will get you started. ... View more