topic Re: starting splunk stops syslogd in Getting Data In

starting splunk stops syslogd

maires — Fri, 01 Apr 2011 01:25:33 GMT

I have looked around for an answer to this so I feel confident that no one will be shouting for me to use the search feature.

I have been setting up several servers to all report via syslogd to a central reporting server. Everything works great and I confirmed this by tail -f /var/log/syslog (or whichever is your default log file that everything gets funneled into)

When I start splunk... ./splunk start i notice that the log files stop scrolling. I get no data into these log files until I kill the splunkd process. I have confirmed this several times by starting splunk and then later killing the splunkd process the moment i stop it my logs start scolling again recording all the data just like they should.

The only lead I have right now is something about splunk having its own syslogd server built in... Is there a config file I have to edit in splunk that maybe tells the incoming log messages where to go like my current /etc/rsyslog.d/50-default.conf?

Or am I missing something simple here?

~Matt

Re: starting splunk stops syslogd

dwaddle — Fri, 01 Apr 2011 01:36:16 GMT

Splunk can support syslog natively (that is, listen on UDP port 514) - but I don't think that is enabled "out of the box" In fact I'm pretty sure it isn't.

You can run something like this

splunk cmd btool --debug inputs list | grep "\[.*\]"

To see what inputs are currently defined. Here are some from my test box:

system     [SSL]
system     [batch:///opt/splunk/var/spool/splunk]
system     [default]
system     [fschange:/opt/splunk/etc]
sample_app [monitor:///opt/splunk/etc/apps/sample_app/logs]
system     [monitor:///opt/splunk/etc/splunk.version]
system     [monitor:///opt/splunk/var/log/splunk]
system     [script]
system     [splunktcp]
launcher   [tcp://9876]
system     [udp:2514]

That last one, udp:2514 is defining a syslog-style UDP input on port 2514. I would predict based on your explanation that you have one of these on port 514. If so, you can run something like:

find /opt/splunk/etc -name inputs.conf | xargs egrep -l "udp:514"

To see which file it is in.

Once you get your immediate issue resolved, how you proceed is kinda up to you. You can continue to let a syslogd be responsible for the network input, and tell Splunk to monitor the files it writes. Alternately, you can let Splunk directly ingest the network input -- but then you have no flat files to refer to. In my own installation, we let syslog handle the UDP input, and then turn Splunk loose on the flat files.

Re: starting splunk stops syslogd

maires — Mon, 04 Apr 2011 19:20:46 GMT

Thanks for the quick response. I was unable to get the commands to work or even the find however you set me on the right track.

Manager » Data inputs » UDP

Under the following area I was able to disable it listening to the UDP port. This fixed the problem.

Re: starting splunk stops syslogd

dwaddle — Mon, 04 Apr 2011 20:18:18 GMT

If you would, please click the checkbox next to the answer so the answers system treats it as "answered". Thanks.

Re: starting splunk stops syslogd

maires — Fri, 08 Apr 2011 20:28:50 GMT

Got it checked for ya.... sorry new here.