Deployment Architecture

Data durability - Search factor is not met?

Gh0st_rid3r
Explorer

Hi,
We have a splunk architecture of 2 search head,2 indexers,1 management server.These are all installed on RHEL7. After patching the OS, We are seeing an error on the management node.

Health status of splunkd - red

Data durability  - Searchfactor is not met.

  • 06-18-2021 04:34:14.490 -0400 INFO CMMaster - updateSummaries did not find bid=bit9~683~3D4C1480-1250-4989-BFE2-F6E36EE3F2ED
  • 06-18-2021 04:33:04.118 -0400 INFO CMMaster - event=commitGenerationFailure pendingGen=99 requesterReason=service failureReason='event=checkDirtyBuckets first unmet bid=os~988~3D4C1480-1250-4989-BFE2-F6E36EE3F2ED'

Multiple messages like above.

Data searchable - 

    • All data is not searchable. Please ensure all the buckets have primary copies
  • 06-18-2021 04:34:55.677 -0400 INFO CMMaster - event=addTarget bid=bit9~683~3D4C1480-1250-4989-BFE2-F6E36EE3F2ED peer=C175D91B-8801-4DAE-8C4B-344E4475F9A9 peer_name= <peer name>  status=StreamingTarget searchable=no mask=0

There is message on our search heads as well.

The number of search artifacts in the dispatch directory is higher than recommended (count=7391, warning threshold=5000) and could have an impact on search performance.

Splunk Version:8.1.3

We have been having the splunkhot buckets reach to 90% utilization and trying to figure out the solution. But after patching,things seem to go bit worse. Please help me guys :/.

Labels (1)
0 Karma

ccsfdave
Builder

I am experiencing the same thing you describe here.   It's been over a year since this post.  Do you still remember the solution?

Thanks!

Dave

0 Karma

ccsfdave
Builder

For us the problem ended up being a fat finger issue when setting up networks.

The subnet mask on one of the indexers was incorrect.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

how you did update? And which Splunk version you have?

There is a bug on 8.1.3+ which can cause this kind of behavior.

Can you reboot CM and see if those ask vanished (temporary) from fi list?

r. Ismo

0 Karma

simplyt83
Observer

Good day,

The cluster master looks healthy but I have the data durability error where the root cause is search factor not met.

My splunk version is 8.2.9

 

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...