Splunk Inputs Help

[JoshuaJJ]

Good morning, 

So I am trying to monitor all files within this directory /var/log/syslog/<IP>

Directory structure: 

/var/log/syslog/<IP>/2024/01 | 02 | 03 | 04 | 05 | 06 | 07/secure | cron | messages 
Hope this makes sense there are multiple subdirectories, the end goal is to monitor secure, cron, and messages 

I wrote this stanza within inputs.conf and the configuration did take on the Universal Forwarder

[monitor:///var/log/syslog/192.168.1.1/.../secure]

disabled = false

host_segment = 4 

index = insght



[monitor:///var/log/syslog/192.168.1.1/.../cron]

disabled = false

host_segment = 4 

index = insght



[monitor:///var/log/syslog/192.168.1.1/.../messages]

disabled = false

host_segment = 4 

index = insght

I have also tried this to capture all subdirs/files

[monitor:///var/log/syslog/192.168.1.1]

disabled = false

host_segment = 4 

recursive = true

index = insght

Also within _internal I get this message: 

INFO TaillingProcess [#### MainTailingThread] - Parsing configuration stanza: monitor:///var/log/syslog/<IP>  

Which seems to hang there with no other messages logged for the particular stanza(s)

IP Address used is notional, thanks for the help! 

[gcusello]

Hi @JoshuaJJ ,

at first: you're running Splunk as root or asplunk user?

if splunk user, has this user the grants to read these files?

then please try this:

[monitor:///var/log/syslog/*/*/*/]
disabled = false
host_segment = 4 
index = insght
whitelist=secure|cron|message

Ciao.

Giuseppe

[JoshuaJJ]

Thanks for your speedy reply!  @gcusello 

Splunk is running as root and is monitoring other files within /var/log for example /var/log/audit/audit.log 

There is a specific IP I want to monitor /var/log/syslog/192.168.1.1 and all subdirectories and files under it. My thought was this should work 

[monitor:///var/log/syslog/192.168.1.1]

disabled = false 

recursive = true 

index = insight 

The index does exist and Splunk is running as root. 

[gcusello]

Hi @JoshuaJJ ,

if in the /var/log/syslog/192.168.1.1 folder you have only those three files add * at the end of the monitor stanza,

[monitor:///var/log/syslog/192.168.1.1/*]
disabled = false 
recursive = true 
index = insight 

otherwise use the whitelist option I hinted.

[monitor:///var/log/syslog/192.168.1.1/*/*/]
disabled = false
host_segment = 4 
index = insght
whitelist=secure|cron|message

Ciao.

Giuseppe

[PickleRick]

First step in debugging such stuff is to run two commands

splunk list monitor

and

splunk list inputstatus

But as far as I remember Splunk has problems with monitor inputs overlapping the same directories.

You could instead just monitor whole directory with a whitelist of all four types of files and then dynamically rewrite sourcetype on ingest depending on the file path included in the source field.

But yes, it can cause issues with multiple significantly different sourcetypes (especially if they differ in timestamp format/placement).

[JoshuaJJ · Accepted solution]

Figured it out thanks to you all! @PickleRick  @gcusello 

Here is what I did. 

First I ran the    ./splunk list monitor command to see what was being monitored. It was as you said Splunk does not like multiple overlapping input stanzas. This is what my inputs configuration looks like. I had to remove the /var/log/audit stanzas 

inputs.conf 

[monitor:///var/log]

disabled = false

blacklist = syslog 

[monitor:///var/log/syslog/10.10.10.10/.../*]

disabled = false 

host = loginsight01

index = insight 

[monitor:///var/log/syslog/10.10.10.11/.../*]

disabled = false 

host = loginsight02

index = insight 

[monitor:///var/log/syslog/10.10.10.12/.../*]

disabled = false 

host = loginsight03

index = insight 

Thank you all for the help!