Why am I receiving "ERROR S2SFileReceiver - event=statSize replicationType=eJournalReplication...status=failed" in splunkd on my indexer cluster?

[mwdbhyat]

Hi,

I am receiving the above error ERROR S2SFileReceiver - event=statSize replicationType=eJournalReplication...status=failed

Running in a clustered environment. There are always only 5 events, once a day at different times. The error is identical to this:

ERROR S2SFileReceiver - event=statSize replicationType=eJournalReplication bid=test~26~89C0FF94-5EB0-410A-9B4D-0E17DBD7FB78 path=/opt/splunk/var/lib/splunk/test/db/26_89C0FF94-5EB0-410A-9B4D-0E17DBD7FB78/rawdata/journal.gz status=failed

Any thoughts?

Thanks

[anand_singh17]

This is an issue of bucket corruption. Your bucket movement will also not happen and you will see, your disk utilization going very high.

To resolve this issue, you need to,
1. delete the bucket creating issue (either with zero size, or did not complete properly)
2. check the bucket movement (observation, if approx retention - 90days (24hrs observation required))
3. check all equivalent number of buckets based on your replication factor(very important)

What causes this issue:
1. Properly not restarting or non-graceful shutdown of indexers.
2. network disruption such as huge flip flops (very rare, but one of possible reason).

[anand_singh17]

It resolved our issue.

[Masa · Accepted solution]

Usually you see more errors around that message. Basically hot bucket replication(streaming replication) was failed for some reason, such as network packet drop or splunk got frozen as high CPU usage, etc.

As long as you do not have any other messages, Splunk automatically recover from this state and source bucket will be replicated to a peers/indexers properly.