topic Re: No data from indexes in cloud search head in Getting Data In

No data from indexes in cloud search head

mysicksi — Wed, 15 Apr 2020 17:20:36 GMT

Hi everyone,

I could really use some input from you all. I am using Splunk cloud in my environment, with a deployment server on-prem for universal forwarders. Two days ago, I stopped receiving data in six indexes. The data retrieved from the indexes originates from a Syslog server.

Steps I have taken so far:
Verify logs are currently being created in Syslog from the sources
Verify Syslog server can still reach deployment server via ping
Verify Splunkd is running on the Syslog server
Verify deployment server has received a recent phone home from the Syslog server
Verify data from other universal forwarders is searchable on the Search head

Re: No data from indexes in cloud search head

richgalloway — Wed, 15 Apr 2020 18:29:45 GMT

Verify the syslog server can still reach the cloud indexers using ping and telnet/curl/wget/traceroute. Be sure to test using port 9997.

Re: No data from indexes in cloud search head

mysicksi — Wed, 15 Apr 2020 19:52:54 GMT

So I ended up restarting the Splunk forwarder on the server and it did the trick. Is there anywhere I can view logs of why this occurred for a root cause?

Re: No data from indexes in cloud search head

richgalloway — Wed, 15 Apr 2020 20:07:25 GMT

Splunkd.log on the forward probably is the best place.

Re: No data from indexes in cloud search head

mysicksi — Thu, 16 Apr 2020 16:34:55 GMT

Thank you, looking through them now.

What response should I see when trying the\ose commands. I currently see this for curl and wget.
curl: (56) Recv failure: Connection reset by peer

HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers.
Retrying.

Re: No data from indexes in cloud search head

richgalloway — Fri, 17 Apr 2020 12:02:42 GMT

Connection reset by peer indicates the connection is reaching the destination, but then is denied by the application (probably because indexers don't expect HTTP).

Re: No data from indexes in cloud search head

mysicksi — Wed, 30 Sep 2020 05:17:59 GMT

I'm adding both steps that I performed with Splunk support to correct this issue.

Remediated ERROR message seen in /opt/splunk/etc/apps/Splunk_TA_nix/bin/interfaces.sh by adding adding "grep -v virbr0" to the line below.

Error seen:

04-11-2020 01:00:34.329 -0400 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/interfaces.sh" cat: /sys/class/net/virbr0/duplex: Invalid argument
04-11-2020 01:00:34.329 -0400 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/interfaces.sh" cat: /sys/class/net/virbr0/speed: Invalid argument
04-11-2020 01:00:34.345 -0400 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/interfaces.sh" cat: /sys/class/net/virbr0-nic/duplex: Invalid argument
04-11-2020 01:00:34.352 -0400 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/interfaces.sh" cat: /sys/class/net/virbr0-nic/speed: Invalid argument

Argument added:

 # Customizing the command to support customer's requirement
         CMD_LIST_UP_INTERFACES="eval ip -s a | tee $TEE_DEST|grep 'state UP' | grep mtu | grep -Ev lo | tee -a $TEE_DEST | cut -d':' -f2 | tee -a $TEE_DEST | cut -d '@' -f 1 | tee -a $TEE_DEST | sort -u | tee -a $TEE_DEST" | grep -v virbr0

INFO message seen in Splunkd.log and remediated.

04-23-2020 13:05:39.200 -0400 INFO ThruputProcessor - Current data throughput (514 kb/s) has reached maxKBps. As a result, data forwarding may be throttled. Consider increasing the value of maxKBps in limits.conf.

Recommendation is to change maxKbps = 0 in [thruput] in limits.conf file on the forwarder.

Re: No data from indexes in cloud search head

mysicksi — Mon, 04 May 2020 16:49:06 GMT

Hi Rich, thank you for your help. Looking in Splunkd.log helped point us in the right direction. I included the steps we took to correct the issue.