rsyslog server with UF not sending events to Splunk

[Branden]

Hi. At Splunk's recommendation, I have a centralized syslog server (using rsyslog) that writes to /logs/hostname/year/month/day/file.log
This works fine.

However, I cannot get the Universal Forwarder to send the events to the Splunk Indexer. I added my stanza to /opt/splunkforwarder/etc/system/local/inputs.conf. When that didn't work, I created an app and put the stanza into /opt/splunkforwarder/etc/apps/syslog/local/inputs.conf
Didn't work.
Here is my stanza:

[monitor:///logs/*]
disabled = false
host_segment = 2
index = main
sourcetype = syslog

That looks straightforward to me.
I checked the Splunk logs on the Indexer and there's no sign that it's ever receiving these events.
In the UF logs I see that it has added a watch to /logs:

INFO  TailingProcessor - Parsing configuration stanza: monitor:///logs/*.
INFO  TailingProcessor - Adding watch on path: /logs.

I have verified that the port is open between the UF and the Indexer.

Indexer is running 7.2.4 and UF is running 7.1.2.
Am I missing something?

Thank you in advance!

[solarboyz1]

I think I see the issue:
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf

Note concerning wildcards and monitor:
You can use wildcards to specify your input path for monitored inputs. Use
"..." for recursive directory matching and "" for wildcard matching in a
single directory segment.

Try the following for you input:

 [monitor:///logs/.../*]

[ddrillic]

Absolutely @solarboyz1 and @Branden can see it via the message you posted earlier -

INFO  TailingProcessor - Adding watch on path: /logs.

[solarboyz1]

Posted it as a separate answer for points and other users who might find it useful...but might not drill into the notes of the first answer.

[solarboyz1 · Accepted solution]

Have you verified the user account running Splunk can access the syslog files?

By default, I believe only root can read them. You may need to change the permission, and update your rsyslog configuration to write the files so that Splunk can read them.

[Branden]

Thank you for your response. I thought of that and, yes, I have verified that the splunk user can access the files. The dirs are 755 and the files are 644. I also changed to the splunk user and manually verified.

[solarboyz1]

I think I see the issue:
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf

Note concerning wildcards and monitor:
* You can use wildcards to specify your input path for monitored inputs. Use
"..." for recursive directory matching and "*" for wildcard matching in a
single directory segment.

Try the following for you input:
[monitor:///logs/.../*]

[Branden]

You are amazing, my friend. That solved this issue. Thanks!