topic Re: main splunker doesn't collect forwarders in Getting Data In

main splunker doesn't collect forwarders

levisik — Thu, 05 Jan 2012 12:02:22 GMT

Hi.

I have two linux boxes - one is full splunk with web interface.

Second one is a test one, which is supposed to sent all performance, logs etc to main splunk server.

I tried Universal forwarder and SplunkLightForwarder.

I can't see any data in my main splunk server - checked for tcp listening on 6996 port.

Here is configuration from Universal and light forwarder:

for uniwersal: /opt/splunkforwarder/etc/system/local/outputs.conf

for Light : /opt/splunk/etc/apps/SplunkLightForwarder/local/outputs.conf

[tcpout]
defaultGroup=my_indexers
heartbeatFrequency=15
indexAndForward=true

[tcpout:my_indexers]
server=10.251.1.132:6996

[syslog]
defaultGroup = my_indexers

[syslog:my_indexers]
server=10.251.1.132:6996

priority=37

Is this configuration enough for example to pass root access logs ?
Do I have to run splunk client as root ?

If configuration is fine, how I can force main splunk to get and show messages and alerts ??

Thanks in advance.

Re: main splunker doesn't collect forwarders

Ayn — Thu, 05 Jan 2012 13:28:09 GMT

I don't see any mention of your configured inputs? Out of the box Splunk doesn't monitor anything at all, so you will have to tell it which logs or other inputs it should monitor.

Re: main splunker doesn't collect forwarders

LCM — Thu, 05 Jan 2012 14:50:32 GMT

Check if the connection is established. On the "sending" system type: splunk list forward-server What does it say? As long as you don't see a connection here, it won't work. Further you may check also the $SPLUNK_HOME/var/log/splunkd.log

Re: main splunker doesn't collect forwarders

levisik — Mon, 16 Jan 2012 10:01:59 GMT

This is some strange stuff...

Configuration looks like that

[tcpout]
defaultGroup = 10.251.1.132_8668
disabled = false

[tcpout:10.251.1.132_8668]
server = 10.251.1.132:8668

[tcpout-server://10.251.1.132:8668]

output from list servers looks like that:

[splunkuser@CentOS01 bin]$ ./splunk list forward-server
Active forwards:
        None
Configured but inactive forwards:
        10.251.1.132:8668

in log I can find following problem:

01-16-2012 09:51:57.332 +0100 WARN  DeploymentClient - Unable to send handshake message to deployment server. Error status is: not_connected

I have checked netstat command in revciever host:

[root@CentOS02 ~]# netstat -a |grep 8668
tcp        0      0 *:8668                      *:*                         LISTEN
tcp        0      0 10.251.1.132:8668           10.251.1.131:35020          ESTABLISHED

telnet from forwarder to reciever:

[splunkuser@CentOS01 bin]$ telnet 10.251.1.132 8668
Trying 10.251.1.132...
Connected to 10.251.1.132.
Escape character is '^]'.
^]
telnet> quit
Connection closed.
[splunkuser@CentOS01 bin]$

I don't know what is the problem here... :((

Re: main splunker doesn't collect forwarders

LCM — Mon, 16 Jan 2012 12:25:57 GMT

tail the splunkd.log on the "receiving" server while starting Splunk on the "sending" system as well . . . What does it tell you there?
There is no established connection yet! Did you tell Splunk to listen on port 8668 on the receiving server?