Deployment Architecture

Data durability - Search factor is not met?

Gh0st_rid3r
Explorer

Hi,
We have a splunk architecture of 2 search head,2 indexers,1 management server.These are all installed on RHEL7. After patching the OS, We are seeing an error on the management node.

Health status of splunkd - red

Data durability  - Searchfactor is not met.

  • 06-18-2021 04:34:14.490 -0400 INFO CMMaster - updateSummaries did not find bid=bit9~683~3D4C1480-1250-4989-BFE2-F6E36EE3F2ED
  • 06-18-2021 04:33:04.118 -0400 INFO CMMaster - event=commitGenerationFailure pendingGen=99 requesterReason=service failureReason='event=checkDirtyBuckets first unmet bid=os~988~3D4C1480-1250-4989-BFE2-F6E36EE3F2ED'

Multiple messages like above.

Data searchable - 

    • All data is not searchable. Please ensure all the buckets have primary copies
  • 06-18-2021 04:34:55.677 -0400 INFO CMMaster - event=addTarget bid=bit9~683~3D4C1480-1250-4989-BFE2-F6E36EE3F2ED peer=C175D91B-8801-4DAE-8C4B-344E4475F9A9 peer_name= <peer name>  status=StreamingTarget searchable=no mask=0

There is message on our search heads as well.

The number of search artifacts in the dispatch directory is higher than recommended (count=7391, warning threshold=5000) and could have an impact on search performance.

Splunk Version:8.1.3

We have been having the splunkhot buckets reach to 90% utilization and trying to figure out the solution. But after patching,things seem to go bit worse. Please help me guys :/.

Labels (1)
0 Karma

ccsfdave
Builder

I am experiencing the same thing you describe here.   It's been over a year since this post.  Do you still remember the solution?

Thanks!

Dave

0 Karma

ccsfdave
Builder

For us the problem ended up being a fat finger issue when setting up networks.

The subnet mask on one of the indexers was incorrect.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

how you did update? And which Splunk version you have?

There is a bug on 8.1.3+ which can cause this kind of behavior.

Can you reboot CM and see if those ask vanished (temporary) from fi list?

r. Ismo

0 Karma

simplyt83
Observer

Good day,

The cluster master looks healthy but I have the data durability error where the root cause is search factor not met.

My splunk version is 8.2.9

 

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...