Deployment Architecture

Data durability - Search factor is not met

Gh0st_rid3r
Explorer

Hi,
We have a splunk architecture of 2 search head,2 indexers,1 management server.These are all installed on RHEL7. After patching the OS, We are seeing an error on the management node.

Health status of splunkd - red

Data durability  - Searchfactor is not met.

  • 06-18-2021 04:34:14.490 -0400 INFO CMMaster - updateSummaries did not find bid=bit9~683~3D4C1480-1250-4989-BFE2-F6E36EE3F2ED
  • 06-18-2021 04:33:04.118 -0400 INFO CMMaster - event=commitGenerationFailure pendingGen=99 requesterReason=service failureReason='event=checkDirtyBuckets first unmet bid=os~988~3D4C1480-1250-4989-BFE2-F6E36EE3F2ED'

Multiple messages like above.

Data searchable - 

    • All data is not searchable. Please ensure all the buckets have primary copies
  • 06-18-2021 04:34:55.677 -0400 INFO CMMaster - event=addTarget bid=bit9~683~3D4C1480-1250-4989-BFE2-F6E36EE3F2ED peer=C175D91B-8801-4DAE-8C4B-344E4475F9A9 peer_name= <peer name>  status=StreamingTarget searchable=no mask=0

There is message on our search heads as well.

The number of search artifacts in the dispatch directory is higher than recommended (count=7391, warning threshold=5000) and could have an impact on search performance.

Splunk Version:8.1.3

We have been having the splunkhot buckets reach to 90% utilization and trying to figure out the solution. But after patching,things seem to go bit worse. Please help me guys :/.

Labels (1)
0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

how you did update? And which Splunk version you have?

There is a bug on 8.1.3+ which can cause this kind of behavior.

Can you reboot CM and see if those ask vanished (temporary) from fi list?

r. Ismo

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!