Deployment Architecture

Why am I receiving "ERROR S2SFileReceiver - event=statSize replicationType=eJournalReplication...status=failed" in splunkd on my indexer cluster?

mwdbhyat
Builder

Hi,

I am receiving the above error ERROR S2SFileReceiver - event=statSize replicationType=eJournalReplication...status=failed

Running in a clustered environment. There are always only 5 events, once a day at different times. The error is identical to this:

ERROR S2SFileReceiver - event=statSize replicationType=eJournalReplication bid=test~26~89C0FF94-5EB0-410A-9B4D-0E17DBD7FB78 path=/opt/splunk/var/lib/splunk/test/db/26_89C0FF94-5EB0-410A-9B4D-0E17DBD7FB78/rawdata/journal.gz status=failed

Any thoughts?

Thanks

0 Karma
1 Solution

Masa
Splunk Employee
Splunk Employee

Usually you see more errors around that message. Basically hot bucket replication(streaming replication) was failed for some reason, such as network packet drop or splunk got frozen as high CPU usage, etc.

As long as you do not have any other messages, Splunk automatically recover from this state and source bucket will be replicated to a peers/indexers properly.

View solution in original post

0 Karma

anand_singh17
Path Finder

This is an issue of bucket corruption. Your bucket movement will also not happen and you will see, your disk utilization going very high.

To resolve this issue, you need to,
1. delete the bucket creating issue (either with zero size, or did not complete properly)
2. check the bucket movement (observation, if approx retention - 90days (24hrs observation required))
3. check all equivalent number of buckets based on your replication factor(very important)

What causes this issue:
1. Properly not restarting or non-graceful shutdown of indexers.
2. network disruption such as huge flip flops (very rare, but one of possible reason).

0 Karma

anand_singh17
Path Finder

It resolved our issue.

0 Karma

Masa
Splunk Employee
Splunk Employee

Usually you see more errors around that message. Basically hot bucket replication(streaming replication) was failed for some reason, such as network packet drop or splunk got frozen as high CPU usage, etc.

As long as you do not have any other messages, Splunk automatically recover from this state and source bucket will be replicated to a peers/indexers properly.

0 Karma
Get Updates on the Splunk Community!

Security Highlights: September 2022 Newsletter

 September 2022 The Splunk App for Fraud Analytics (SFA) is now Splunk SupportedUse your existing Splunk ...

Platform Highlights | September 2022 Newsletter

 September 2022 What’s New in 9.0 and How to UpgradeGet a walk through of what is new Splunk Enterprise 9.0 ...

Observability Highlights | September 2022 Newsletter

 September 2022 Splunk Observability SuiteAccess to "Classic" SignalFx Interface Will be Removed on Sept 30, ...