Deployment Architecture

Data durability - Search factor is not met?

Gh0st_rid3r
Explorer

Hi,
We have a splunk architecture of 2 search head,2 indexers,1 management server.These are all installed on RHEL7. After patching the OS, We are seeing an error on the management node.

Health status of splunkd - red

Data durability  - Searchfactor is not met.

  • 06-18-2021 04:34:14.490 -0400 INFO CMMaster - updateSummaries did not find bid=bit9~683~3D4C1480-1250-4989-BFE2-F6E36EE3F2ED
  • 06-18-2021 04:33:04.118 -0400 INFO CMMaster - event=commitGenerationFailure pendingGen=99 requesterReason=service failureReason='event=checkDirtyBuckets first unmet bid=os~988~3D4C1480-1250-4989-BFE2-F6E36EE3F2ED'

Multiple messages like above.

Data searchable - 

    • All data is not searchable. Please ensure all the buckets have primary copies
  • 06-18-2021 04:34:55.677 -0400 INFO CMMaster - event=addTarget bid=bit9~683~3D4C1480-1250-4989-BFE2-F6E36EE3F2ED peer=C175D91B-8801-4DAE-8C4B-344E4475F9A9 peer_name= <peer name>  status=StreamingTarget searchable=no mask=0

There is message on our search heads as well.

The number of search artifacts in the dispatch directory is higher than recommended (count=7391, warning threshold=5000) and could have an impact on search performance.

Splunk Version:8.1.3

We have been having the splunkhot buckets reach to 90% utilization and trying to figure out the solution. But after patching,things seem to go bit worse. Please help me guys :/.

Labels (1)
0 Karma

ccsfdave
Builder

I am experiencing the same thing you describe here.   It's been over a year since this post.  Do you still remember the solution?

Thanks!

Dave

0 Karma

ccsfdave
Builder

For us the problem ended up being a fat finger issue when setting up networks.

The subnet mask on one of the indexers was incorrect.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

how you did update? And which Splunk version you have?

There is a bug on 8.1.3+ which can cause this kind of behavior.

Can you reboot CM and see if those ask vanished (temporary) from fi list?

r. Ismo

0 Karma

simplyt83
Observer

Good day,

The cluster master looks healthy but I have the data durability error where the root cause is search factor not met.

My splunk version is 8.2.9

 

0 Karma
Get Updates on the Splunk Community!

Exporting Splunk Apps

Join us on Monday, October 21 at 11 am PT | 2 pm ET!With the app export functionality, app developers and ...

Cisco Use Cases, ITSI Best Practices, and More New Articles from Splunk Lantern

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Build Your First SPL2 App!

Watch the recording now!.Do you want to SPL™, too? SPL2, Splunk's next-generation data search and preparation ...