Deployment Architecture

Why am I receiving "ERROR S2SFileReceiver - event=statSize replicationType=eJournalReplication...status=failed" in splunkd on my indexer cluster?

mwdbhyat
Builder

Hi,

I am receiving the above error ERROR S2SFileReceiver - event=statSize replicationType=eJournalReplication...status=failed

Running in a clustered environment. There are always only 5 events, once a day at different times. The error is identical to this:

ERROR S2SFileReceiver - event=statSize replicationType=eJournalReplication bid=test~26~89C0FF94-5EB0-410A-9B4D-0E17DBD7FB78 path=/opt/splunk/var/lib/splunk/test/db/26_89C0FF94-5EB0-410A-9B4D-0E17DBD7FB78/rawdata/journal.gz status=failed

Any thoughts?

Thanks

0 Karma
1 Solution

Masa
Splunk Employee
Splunk Employee

Usually you see more errors around that message. Basically hot bucket replication(streaming replication) was failed for some reason, such as network packet drop or splunk got frozen as high CPU usage, etc.

As long as you do not have any other messages, Splunk automatically recover from this state and source bucket will be replicated to a peers/indexers properly.

View solution in original post

0 Karma

anand_singh17
Path Finder

This is an issue of bucket corruption. Your bucket movement will also not happen and you will see, your disk utilization going very high.

To resolve this issue, you need to,
1. delete the bucket creating issue (either with zero size, or did not complete properly)
2. check the bucket movement (observation, if approx retention - 90days (24hrs observation required))
3. check all equivalent number of buckets based on your replication factor(very important)

What causes this issue:
1. Properly not restarting or non-graceful shutdown of indexers.
2. network disruption such as huge flip flops (very rare, but one of possible reason).

0 Karma

anand_singh17
Path Finder

It resolved our issue.

0 Karma

Masa
Splunk Employee
Splunk Employee

Usually you see more errors around that message. Basically hot bucket replication(streaming replication) was failed for some reason, such as network packet drop or splunk got frozen as high CPU usage, etc.

As long as you do not have any other messages, Splunk automatically recover from this state and source bucket will be replicated to a peers/indexers properly.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...