Getting Data In

Indexer stops receiving data from forwarders

jhupka
Path Finder

This is less of a question and more of a record on Splunk Answers of an issue we ran into.

Symptoms:
You are on Red Hat 6.6, 7.0, or 7.1

The Indexer stops receiving data from Forwarders, but looks to be up and running fine otherwise. In a sense it seems stuck for our TCP input port on splunkd. For example, the Indexer still participates as a peer for any searches, and the Indexer also looks to continue indexing any data generated locally - internal Splunk logs or scripted inputs running on that Indexer. Essentially the TCP input port is bad, but the splunkd admin port is fine.

When running the following search, any Indexers having this problem would show up has having 1 or 0 Distinct Hosts since they weren't able to receive anything from Forwarders:

index=_internal sourcetype=splunkd earliest=-15m | stats count as event_count, dc(host) as distinct_hosts by splunk_server

Also, we would notice a lot of TCP connections sitting in a CLOSE_WAIT or SYN_RECV state when running netstat -an on the Indexer.

Workaround:

Use pstack to dump a list of processes associated with splunkd. This will magically fix things without having to restart the Indexer:

pstack `head -1 $SPLUNK_HOME/var/run/splunk/splunkd.pid`

Solution:

There is a bug in Red Hat that causes things to go awry:

https://access.redhat.com/solutions/1386323

In our case, the solution was to patch from kernel 2.6.32-504.8.1 to 2.6.32-504.16.2

Environment Details:
- Splunk: 6.2.5 (build 272645)
- OS: Red Hat Linux 2.6.32-504.8.1.el6.x86_64 #1 SMP Fri Dec 19 12:09:25 EST 2014 x86_64 x86_64 x86_64 GNU/Linux
- HW: Cisco UCS C240

Tags (1)
1 Solution

jhupka
Path Finder

Solution:

There is a bug in Red Hat that causes things to go awry:

https://access.redhat.com/solutions/1386323

In our case, the solution was to patch from kernel 2.6.32-504.8.1 to 2.6.32-504.16.2

View solution in original post

jhupka
Path Finder

Solution:

There is a bug in Red Hat that causes things to go awry:

https://access.redhat.com/solutions/1386323

In our case, the solution was to patch from kernel 2.6.32-504.8.1 to 2.6.32-504.16.2

muebel
SplunkTrust
SplunkTrust

nice work!

0 Karma
Get Updates on the Splunk Community!

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...