When user try to update kvstore using command like | inputlookup [backupdata] | output This errors out with error below Error : Could not write to collection [kvstore] error occurred during the last operation ('createCollection', domain: '5', code: '10334'): exception: BSONObj size: 1751347969 (0x68637301) is invalid. Size must be between 0 and 16793600(16MB) First element: d_time: ?type=101. Preliminary research indicates this could be a database related error. 3-08-2017 08:39:21.322 ERROR KVStorageProvider - An error occurred during the last operation ('createCollection', domain: '5', code: '10334'): exception: BSONObj size: 1751347969 (0x68637301) is invalid. Size must be between 0 and 16793600(16MB) First element: d_time: ?type=101 03-08-2017 08:39:21.322 ERROR outputcsv - sid:1488991161.995 Could not write to collection 'vulnerability_management': An error occurred during the last operation ('createCollection', domain: '5', code: '10334'): exception: BSONObj size: 1751347969 (0x68637301) is invalid. Size must be between 0 and 16793600(16MB) First element: d_time: ?type=101. ... View more

I have Cluster Master with 2 Indexers. These are on version 6.3.5 We noticed that on one of the indexers and for an index “winevent”, we hit the following errors: 1-11-2017 09:31:54.947 -0500 ERROR TcpInputProc - event=replicationData status=failed err="Could not open file for bid= winevent ~127~051261C5-3A44-451D-B5EA-48DEC971466A err="Cannot find config for idx=ads_main" (The operation completed successfully.)" 1-11-2017 08:25:25.289 -0500 WARN IndexerService - Received event for unconfigured/disabled/deleted index= winevent with source="source::WinEventLog:Security" host="host::SDRT0086" sourcetype="sourcetype::WinEventLog:Security". So far received events from 1 missing index(es). 01-11-2017 08:22:20.879 -0500 ERROR DatabaseDirectoryManager - idx= winevent bucket=db_1483621671_1483545612_420_051261C5-3A44-451D-B5EA-48DEC971466A Detected directory manually copied into its database, causing id conflicts [path1='\\gofisilon01.nac.ppg.com\Archive\Splunk\Idx2\ winevent \colddb\rb_1483621671_1483545612_420_051261C5-3A44-451D-B5EA-48DEC971466A' path2='\\gofisilon01.nac.ppg.com\Archive\Splunk\Idx2\ winevent \colddb\db_1483621671_1483545612_420_051261C5-3A44-451D-B5EA-48DEC971466A']. Due to these duplicate buckets the index “winevent” got marked as “Disabled? What can be done for this issue? ... View more

Looking at scheduler.log from a search head cluster member, the scheduler.log has a status column that can have following values: delegated_remote delegated_remote_completion delegarte_remote_error skipped success It will be useful to get information about each of these values in context of Search Head Clustering. index=case_356897 host=* sourcetype=*sched* source="*scheduler.log" | timechart span=6month count by status ... View more

The link provided (https://answers.splunk.com/answers/230771/search-head-cluster-how-to-manage-new-roles-betwee-1.html) does not appear to address the issue with permissions in *.meta files not being replicated. To further elaborate, permissions changes made in the "Manage Apps" page are not being propagated through the Search Head Cluster. However, permissions changes made on individual Knowledge Objects are propagating successfully. The documentation (http://docs.splunk.com/Documentation/Splunk/6.2.7/DistSearch/HowconfrepoworksinSHC) does not appear to suggest that only Knowledge Objects permissions changes are replicated, and not permissions from the "Manage Apps" page. It does, however, mention that the "cluster also replicates permissions stored in *.meta files". ... View more

Issue : We don't see run async query using Ruby SDK against a Splunk 6.4 search head cluster via a BIG-IP load balancer. Here is the sample Code. require 'splunk-sdk-ruby' # How to get to the Splunk server. Edit this to match your # own Splunk install. config = { :scheme => :https, :host => "localtest.com", :port => 8089, :username => "splunk.sensu", :password => "redacted" } # First open a connection to Splunk. service = Splunk::connect(config) # For longer running jobs, you don't want to wait until the job finishes, as # create_oneshot in 3_blocking_searches.rb does. In this case, use the # create_search method of Service. Instead of returning a stream, it creates # an asynchronous job on the server and returns a Job object referencing it. job = service.create_search("search index=_internal | head 1", :earliest_time => "-1d", :latest_time => "now") # Before you can do anything with a Job, you must wait for it to be ready. # Before it is, you cannot do anything with it, even read its state. while !job.is_ready?() sleep(0.1) end # More typically you will want to wait until the job is done and its events # ready to retrieve. For that, use the is_done? method instead. Note that a # job is always ready before it's done. while !job.is_done?() sleep(0.1) end # If you want the transformed results (equivalent to what create_oneshot would # return), call the results method on the Job. If you want the untransformed # results, call events. You can optionally pass an offset and total count, # which are useful to get hunks of large sets of results. stream = job.results(:count => 1, :offset => 0) # Or: stream = job.events(:count => 3, :offset => 0) results = Splunk::ResultsReader.new(stream) results.each do |result| puts result["_raw"] end # If you want to run a real time search, it must be asynchronous, and it is # never done, so neither results or events will work. Instead, you must call # preview (which takes the same arguments as the other two). rt_job = service.create_search("search index=_internal | head 1", :earliest_time => "rt-1h", :latest_time => "rt") while !rt_job.is_ready?() sleep(0.1) end stream = rt_job.preview() results = Splunk::ResultsReader.new(stream) results.each do |result| puts result["_raw"] end ... View more

Indexer cluster with 30 peers Search head cluster of 5 instances connected to the indexer cluster Configured a schedule search that runs every 15min Noticed that job inspector has some searches that just says: | [earliest time=1/1/70 12:00:00 AM, latest time] with dispatched time at: 1/1/70 12:00:00 AM, and status Running 0% The SHC members are on Splunk version 6.4.1 on Solaris. ... View more

In Search Head Cluster environment Unable to share Tag created from Event Actions>Action . On the other hand Tag created by navigating to Setting>Tags can be shared without issue. ... View more