Getting Data In

Why is my Linux forwarder not sending data to the indexer?

kpavan
Path Finder

Hi,

One of my Linux Forwarder not sending data to indexer. Could you please assist me what is wrong in my configuration.

Forwarder Conf:

inputs.conf
~~~~~~~~~
[default]
index = _Notifier
host = hostname.com

[monitor:///path/log/*]
disabled = false
followTail = 0
sourcetype = _Alerts

[monitor:///path/log/*]
disabled = false
followTail = 0
sourcetype = _Alerts

Outputs.conf:
~~~~~~~~~~~
[tcpout]
autoLB = true
compressed = true
defaultGroup = groupname
useACK=true

[tcpout:groupname]
server = splunk.hostname.com:9997

server.conf
[general]
serverName = xxxxx

[sslConfig]
sslKeysfilePassword = xxxxxx

[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder

[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free

On indexer

inputs.conf
~~~~~~~~~
[default]
host = xxxxxx

[splunktcp://9997]
disabled = 0

server.conf
~~~~~~~~~
[general]
serverName = xxxxxxx

[sslConfig]
sslKeysfilePassword = xxxxxxx

[lmpool:auto_generated_pool_download-trial]
description = auto_generated_pool_download-trial
quota = MAX
slaves = *
stack_id = download-trial

[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder

[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free

Tags (2)
0 Karma

brod_geico
Path Finder

Few things can you verify it from UF to Indexer server.

Check the port connections as well firewall rules most of the time TCP error/Cooked data refer to these kind of issues.
Telnet to port and IP. etc
Try to troubleshoot problem from Network layer to Splunk you will find it.

0 Karma

jrodman
Splunk Employee
Splunk Employee

I think this use of underscores is probably not a good idea. Index called _Notifier and sourcetype called _Alerts. If you don't want roles having access to the index, you can simply set up the roles that way. As for sourcetype with underscore the goal is not clear, so I can't comment.

This may actually be the cause of the data not being forwarded. There are some funny rules about _indexes being forwarded expressed in etc/system/default/outputs.conf because underscore indexes are considered to be internal to Splunk, and we want to avoid unnecessarily flooding user indexers with data that is not useful.

Incidentally,

  • The server.conf contents are not relevant.
  • You should probably remove one of the duplicated stanzas in inputs.conf
0 Karma

cbright
Explorer

Did you ever get an answer on resolving this as I am seeing the same thing on one of my Linux forwarders?

0 Karma

kpavan
Path Finder

Hi,

I have changed index now, but getting error as below, could you please suggest me what can be done to fix this issue

On forwarders splunkd.log 10-13-2014 05:33:39.728 -0700 INFO TcpOutputProc - Connected to idx=x.x.x.x:9997 using ACK. 10-13-2014 05:33:39.732 -0700 WARN TimeoutHeap - Detected system time adjusted backwards by 1ms. 10-13-2014 05:33:54.765 -0700 WARN TimeoutHeap - Detected system time adjusted backwards by 1ms. 10-13-2014 05:33:58.057 -0700 WARN TimeoutHeap - Detected system time adjusted backwards by 1ms. 10-13-2014 05:34:09.746 -0700 WARN TcpOutputProc - Cooked connection to ip=x.x.x.x:9997 timed out on indexer metric.log getting connectionType=cooked.

the command ouput /opt/splunkforwarder/bin/splunk list forward-server Active forwards: indexername:9997 Configured but inactive forwards: None

Thanks!

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.