starting splunk stops syslogd

[maires]

I have looked around for an answer to this so I feel confident that no one will be shouting for me to use the search feature.

I have been setting up several servers to all report via syslogd to a central reporting server. Everything works great and I confirmed this by tail -f /var/log/syslog (or whichever is your default log file that everything gets funneled into)

When I start splunk... ./splunk start i notice that the log files stop scrolling. I get no data into these log files until I kill the splunkd process. I have confirmed this several times by starting splunk and then later killing the splunkd process the moment i stop it my logs start scolling again recording all the data just like they should.

The only lead I have right now is something about splunk having its own syslogd server built in... Is there a config file I have to edit in splunk that maybe tells the incoming log messages where to go like my current /etc/rsyslog.d/50-default.conf?

Or am I missing something simple here?

~Matt

[dwaddle · Accepted solution]

Splunk can support syslog natively (that is, listen on UDP port 514) - but I don't think that is enabled "out of the box" In fact I'm pretty sure it isn't.

You can run something like this

splunk cmd btool --debug inputs list | grep "\[.*\]"

To see what inputs are currently defined. Here are some from my test box:

system     [SSL]
system     [batch:///opt/splunk/var/spool/splunk]
system     [default]
system     [fschange:/opt/splunk/etc]
sample_app [monitor:///opt/splunk/etc/apps/sample_app/logs]
system     [monitor:///opt/splunk/etc/splunk.version]
system     [monitor:///opt/splunk/var/log/splunk]
system     [script]
system     [splunktcp]
launcher   [tcp://9876]
system     [udp:2514]

That last one, udp:2514 is defining a syslog-style UDP input on port 2514. I would predict based on your explanation that you have one of these on port 514. If so, you can run something like:

find /opt/splunk/etc -name inputs.conf | xargs egrep -l "udp:514"

To see which file it is in.

Once you get your immediate issue resolved, how you proceed is kinda up to you. You can continue to let a syslogd be responsible for the network input, and tell Splunk to monitor the files it writes. Alternately, you can let Splunk directly ingest the network input -- but then you have no flat files to refer to. In my own installation, we let syslog handle the UDP input, and then turn Splunk loose on the flat files.

[maires]

Got it checked for ya.... sorry new here.

[dwaddle]

If you would, please click the checkbox next to the answer so the answers system treats it as "answered". Thanks.

[maires]

Thanks for the quick response. I was unable to get the commands to work or even the find however you set me on the right track.

Manager » Data inputs » UDP

Under the following area I was able to disable it listening to the UDP port. This fixed the problem.