Getting Data In

No data from indexes in cloud search head

mysicksi
Path Finder

Hi everyone,

I could really use some input from you all. I am using Splunk cloud in my environment, with a deployment server on-prem for universal forwarders. Two days ago, I stopped receiving data in six indexes. The data retrieved from the indexes originates from a Syslog server.

Steps I have taken so far:
Verify logs are currently being created in Syslog from the sources
Verify Syslog server can still reach deployment server via ping
Verify Splunkd is running on the Syslog server
Verify deployment server has received a recent phone home from the Syslog server
Verify data from other universal forwarders is searchable on the Search head

0 Karma
1 Solution

mysicksi
Path Finder

I'm adding both steps that I performed with Splunk support to correct this issue.

  1. Remediated ERROR message seen in /opt/splunk/etc/apps/Splunk_TA_nix/bin/interfaces.sh by adding adding "grep -v virbr0" to the line below.

Error seen:

04-11-2020 01:00:34.329 -0400 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/interfaces.sh" cat: /sys/class/net/virbr0/duplex: Invalid argument
04-11-2020 01:00:34.329 -0400 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/interfaces.sh" cat: /sys/class/net/virbr0/speed: Invalid argument
04-11-2020 01:00:34.345 -0400 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/interfaces.sh" cat: /sys/class/net/virbr0-nic/duplex: Invalid argument
04-11-2020 01:00:34.352 -0400 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/interfaces.sh" cat: /sys/class/net/virbr0-nic/speed: Invalid argument

Argument added:

 # Customizing the command to support customer's requirement
         CMD_LIST_UP_INTERFACES="eval ip -s a | tee $TEE_DEST|grep 'state UP' | grep mtu | grep -Ev lo | tee -a $TEE_DEST | cut -d':' -f2 | tee -a $TEE_DEST | cut -d '@' -f 1 | tee -a $TEE_DEST | sort -u | tee -a $TEE_DEST" | grep -v virbr0
  1. INFO message seen in Splunkd.log and remediated.

    04-23-2020 13:05:39.200 -0400 INFO ThruputProcessor - Current data throughput (514 kb/s) has reached maxKBps. As a result, data forwarding may be throttled. Consider increasing the value of maxKBps in limits.conf.

Recommendation is to change maxKbps = 0 in [thruput] in limits.conf file on the forwarder.

View solution in original post

0 Karma

mysicksi
Path Finder

I'm adding both steps that I performed with Splunk support to correct this issue.

  1. Remediated ERROR message seen in /opt/splunk/etc/apps/Splunk_TA_nix/bin/interfaces.sh by adding adding "grep -v virbr0" to the line below.

Error seen:

04-11-2020 01:00:34.329 -0400 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/interfaces.sh" cat: /sys/class/net/virbr0/duplex: Invalid argument
04-11-2020 01:00:34.329 -0400 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/interfaces.sh" cat: /sys/class/net/virbr0/speed: Invalid argument
04-11-2020 01:00:34.345 -0400 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/interfaces.sh" cat: /sys/class/net/virbr0-nic/duplex: Invalid argument
04-11-2020 01:00:34.352 -0400 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/interfaces.sh" cat: /sys/class/net/virbr0-nic/speed: Invalid argument

Argument added:

 # Customizing the command to support customer's requirement
         CMD_LIST_UP_INTERFACES="eval ip -s a | tee $TEE_DEST|grep 'state UP' | grep mtu | grep -Ev lo | tee -a $TEE_DEST | cut -d':' -f2 | tee -a $TEE_DEST | cut -d '@' -f 1 | tee -a $TEE_DEST | sort -u | tee -a $TEE_DEST" | grep -v virbr0
  1. INFO message seen in Splunkd.log and remediated.

    04-23-2020 13:05:39.200 -0400 INFO ThruputProcessor - Current data throughput (514 kb/s) has reached maxKBps. As a result, data forwarding may be throttled. Consider increasing the value of maxKBps in limits.conf.

Recommendation is to change maxKbps = 0 in [thruput] in limits.conf file on the forwarder.

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Verify the syslog server can still reach the cloud indexers using ping and telnet/curl/wget/traceroute. Be sure to test using port 9997.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

mysicksi
Path Finder

So I ended up restarting the Splunk forwarder on the server and it did the trick. Is there anywhere I can view logs of why this occurred for a root cause?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Splunkd.log on the forward probably is the best place.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

mysicksi
Path Finder

Thank you, looking through them now.

What response should I see when trying the\ose commands. I currently see this for curl and wget.
curl: (56) Recv failure: Connection reset by peer

HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers.
Retrying.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Connection reset by peer indicates the connection is reaching the destination, but then is denied by the application (probably because indexers don't expect HTTP).

---
If this reply helps you, an upvote would be appreciated.
0 Karma

mysicksi
Path Finder

Hi Rich, thank you for your help. Looking in Splunkd.log helped point us in the right direction. I included the steps we took to correct the issue.

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!