Getting Data In

starting splunk stops syslogd

maires
New Member

I have looked around for an answer to this so I feel confident that no one will be shouting for me to use the search feature.

I have been setting up several servers to all report via syslogd to a central reporting server. Everything works great and I confirmed this by tail -f /var/log/syslog (or whichever is your default log file that everything gets funneled into)

When I start splunk... ./splunk start i notice that the log files stop scrolling. I get no data into these log files until I kill the splunkd process. I have confirmed this several times by starting splunk and then later killing the splunkd process the moment i stop it my logs start scolling again recording all the data just like they should.

The only lead I have right now is something about splunk having its own syslogd server built in... Is there a config file I have to edit in splunk that maybe tells the incoming log messages where to go like my current /etc/rsyslog.d/50-default.conf?

Or am I missing something simple here?

~Matt

Tags (2)
0 Karma
1 Solution

dwaddle
SplunkTrust
SplunkTrust

Splunk can support syslog natively (that is, listen on UDP port 514) - but I don't think that is enabled "out of the box" In fact I'm pretty sure it isn't.

You can run something like this

splunk cmd btool --debug inputs list | grep "\[.*\]"

To see what inputs are currently defined. Here are some from my test box:

system     [SSL]
system     [batch:///opt/splunk/var/spool/splunk]
system     [default]
system     [fschange:/opt/splunk/etc]
sample_app [monitor:///opt/splunk/etc/apps/sample_app/logs]
system     [monitor:///opt/splunk/etc/splunk.version]
system     [monitor:///opt/splunk/var/log/splunk]
system     [script]
system     [splunktcp]
launcher   [tcp://9876]
system     [udp:2514]

That last one, udp:2514 is defining a syslog-style UDP input on port 2514. I would predict based on your explanation that you have one of these on port 514. If so, you can run something like:

find /opt/splunk/etc -name inputs.conf | xargs egrep -l "udp:514"

To see which file it is in.

Once you get your immediate issue resolved, how you proceed is kinda up to you. You can continue to let a syslogd be responsible for the network input, and tell Splunk to monitor the files it writes. Alternately, you can let Splunk directly ingest the network input -- but then you have no flat files to refer to. In my own installation, we let syslog handle the UDP input, and then turn Splunk loose on the flat files.

View solution in original post

dwaddle
SplunkTrust
SplunkTrust

Splunk can support syslog natively (that is, listen on UDP port 514) - but I don't think that is enabled "out of the box" In fact I'm pretty sure it isn't.

You can run something like this

splunk cmd btool --debug inputs list | grep "\[.*\]"

To see what inputs are currently defined. Here are some from my test box:

system     [SSL]
system     [batch:///opt/splunk/var/spool/splunk]
system     [default]
system     [fschange:/opt/splunk/etc]
sample_app [monitor:///opt/splunk/etc/apps/sample_app/logs]
system     [monitor:///opt/splunk/etc/splunk.version]
system     [monitor:///opt/splunk/var/log/splunk]
system     [script]
system     [splunktcp]
launcher   [tcp://9876]
system     [udp:2514]

That last one, udp:2514 is defining a syslog-style UDP input on port 2514. I would predict based on your explanation that you have one of these on port 514. If so, you can run something like:

find /opt/splunk/etc -name inputs.conf | xargs egrep -l "udp:514"

To see which file it is in.

Once you get your immediate issue resolved, how you proceed is kinda up to you. You can continue to let a syslogd be responsible for the network input, and tell Splunk to monitor the files it writes. Alternately, you can let Splunk directly ingest the network input -- but then you have no flat files to refer to. In my own installation, we let syslog handle the UDP input, and then turn Splunk loose on the flat files.

maires
New Member

Got it checked for ya.... sorry new here.

0 Karma

dwaddle
SplunkTrust
SplunkTrust

If you would, please click the checkbox next to the answer so the answers system treats it as "answered". Thanks.

0 Karma

maires
New Member

Thanks for the quick response. I was unable to get the commands to work or even the find however you set me on the right track.

Manager » Data inputs » UDP

Under the following area I was able to disable it listening to the UDP port. This fixed the problem.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...