Getting Data In
Highlighted

rsyslog server with UF not sending events to Splunk

Builder

Hi. At Splunk's recommendation, I have a centralized syslog server (using rsyslog) that writes to /logs/hostname/year/month/day/file.log
This works fine.

However, I cannot get the Universal Forwarder to send the events to the Splunk Indexer. I added my stanza to /opt/splunkforwarder/etc/system/local/inputs.conf. When that didn't work, I created an app and put the stanza into /opt/splunkforwarder/etc/apps/syslog/local/inputs.conf
Didn't work.
Here is my stanza:

[monitor:///logs/*]
disabled = false
host_segment = 2
index = main
sourcetype = syslog

That looks straightforward to me.
I checked the Splunk logs on the Indexer and there's no sign that it's ever receiving these events.
In the UF logs I see that it has added a watch to /logs:

INFO  TailingProcessor - Parsing configuration stanza: monitor:///logs/*.
INFO  TailingProcessor - Adding watch on path: /logs.

I have verified that the port is open between the UF and the Indexer.

Indexer is running 7.2.4 and UF is running 7.1.2.
Am I missing something?

Thank you in advance!

0 Karma
Highlighted

Re: rsyslog server with UF not sending events to Splunk

Builder

Have you verified the user account running Splunk can access the syslog files?

By default, I believe only root can read them. You may need to change the permission, and update your rsyslog configuration to write the files so that Splunk can read them.

View solution in original post

Highlighted

Re: rsyslog server with UF not sending events to Splunk

Builder

Thank you for your response. I thought of that and, yes, I have verified that the splunk user can access the files. The dirs are 755 and the files are 644. I also changed to the splunk user and manually verified.

0 Karma
Highlighted

Re: rsyslog server with UF not sending events to Splunk

Builder

I think I see the issue:
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf

Note concerning wildcards and monitor:
* You can use wildcards to specify your input path for monitored inputs. Use
"..." for recursive directory matching and "*" for wildcard matching in a
single directory segment.

Try the following for you input:
[monitor:///logs/.../*]

Highlighted

Re: rsyslog server with UF not sending events to Splunk

Builder

You are amazing, my friend. That solved this issue. Thanks!

0 Karma
Highlighted

Re: rsyslog server with UF not sending events to Splunk

Builder

I think I see the issue:
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf

Note concerning wildcards and monitor:
You can use wildcards to specify your input path for monitored inputs. Use
"..." for recursive directory matching and "" for wildcard matching in a
single directory segment.

Try the following for you input:

 [monitor:///logs/.../*]
Highlighted

Re: rsyslog server with UF not sending events to Splunk

Ultra Champion

Absolutely @solarboyz1 and @Branden can see it via the message you posted earlier -

INFO  TailingProcessor - Adding watch on path: /logs.
0 Karma
Highlighted

Re: rsyslog server with UF not sending events to Splunk

Builder

Posted it as a separate answer for points and other users who might find it useful...but might not drill into the notes of the first answer.

0 Karma