No data from indexes in cloud search head

[mysicksi]

Hi everyone,

I could really use some input from you all. I am using Splunk cloud in my environment, with a deployment server on-prem for universal forwarders. Two days ago, I stopped receiving data in six indexes. The data retrieved from the indexes originates from a Syslog server.

Steps I have taken so far:
Verify logs are currently being created in Syslog from the sources
Verify Syslog server can still reach deployment server via ping
Verify Splunkd is running on the Syslog server
Verify deployment server has received a recent phone home from the Syslog server
Verify data from other universal forwarders is searchable on the Search head

[mysicksi · Accepted solution]

I'm adding both steps that I performed with Splunk support to correct this issue.

1. Remediated ERROR message seen in /opt/splunk/etc/apps/Splunk_TA_nix/bin/interfaces.sh by adding adding "grep -v virbr0" to the line below.

Error seen:

04-11-2020 01:00:34.329 -0400 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/interfaces.sh" cat: /sys/class/net/virbr0/duplex: Invalid argument
04-11-2020 01:00:34.329 -0400 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/interfaces.sh" cat: /sys/class/net/virbr0/speed: Invalid argument
04-11-2020 01:00:34.345 -0400 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/interfaces.sh" cat: /sys/class/net/virbr0-nic/duplex: Invalid argument
04-11-2020 01:00:34.352 -0400 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/interfaces.sh" cat: /sys/class/net/virbr0-nic/speed: Invalid argument

Argument added:

 # Customizing the command to support customer's requirement
         CMD_LIST_UP_INTERFACES="eval ip -s a | tee $TEE_DEST|grep 'state UP' | grep mtu | grep -Ev lo | tee -a $TEE_DEST | cut -d':' -f2 | tee -a $TEE_DEST | cut -d '@' -f 1 | tee -a $TEE_DEST | sort -u | tee -a $TEE_DEST" | grep -v virbr0

1. INFO message seen in Splunkd.log and remediated.

   04-23-2020 13:05:39.200 -0400 INFO ThruputProcessor - Current data throughput (514 kb/s) has reached maxKBps. As a result, data forwarding may be throttled. Consider increasing the value of maxKBps in limits.conf.

Recommendation is to change maxKbps = 0 in [thruput] in limits.conf file on the forwarder.

[richgalloway]

Verify the syslog server can still reach the cloud indexers using ping and telnet/curl/wget/traceroute. Be sure to test using port 9997.

[mysicksi]

So I ended up restarting the Splunk forwarder on the server and it did the trick. Is there anywhere I can view logs of why this occurred for a root cause?

[richgalloway]

Splunkd.log on the forward probably is the best place.

[mysicksi]

Thank you, looking through them now.

What response should I see when trying the\ose commands. I currently see this for curl and wget.
curl: (56) Recv failure: Connection reset by peer

HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers.
Retrying.

[richgalloway]

Connection reset by peer indicates the connection is reaching the destination, but then is denied by the application (probably because indexers don't expect HTTP).

[mysicksi]

Hi Rich, thank you for your help. Looking in Splunkd.log helped point us in the right direction. I included the steps we took to correct the issue.