Getting Data In

Splunk Inputs Help

JoshuaJJ
Path Finder

Good morning, 

So I am trying to monitor all files within this directory /var/log/syslog/<IP>

Directory structure: 

/var/log/syslog/<IP>/2024/01 | 02 | 03 | 04 | 05 | 06 | 07/secure | cron | messages 
Hope this makes sense there are multiple subdirectories, the end goal is to monitor secure, cron, and messages 

I wrote this stanza within inputs.conf and the configuration did take on the Universal Forwarder

[monitor:///var/log/syslog/192.168.1.1/.../secure]

disabled = false

host_segment = 4 

index = insght



[monitor:///var/log/syslog/192.168.1.1/.../cron]

disabled = false

host_segment = 4 

index = insght



[monitor:///var/log/syslog/192.168.1.1/.../messages]

disabled = false

host_segment = 4 

index = insght

 

I have also tried this to capture all subdirs/files

[monitor:///var/log/syslog/192.168.1.1]

disabled = false

host_segment = 4 

recursive = true

index = insght

 

Also within _internal I get this message: 

INFO TaillingProcess [#### MainTailingThread] - Parsing configuration stanza: monitor:///var/log/syslog/<IP>  

Which seems to hang there with no other messages logged for the particular stanza(s)

 

IP Address used is notional, thanks for the help! 

Labels (3)
0 Karma
1 Solution

JoshuaJJ
Path Finder

Figured it out thanks to you all! @PickleRick  @gcusello 

Here is what I did. 

First I ran the    ./splunk list monitor command to see what was being monitored. It was as you said Splunk does not like multiple overlapping input stanzas. This is what my inputs configuration looks like. I had to remove the /var/log/audit stanzas 

 

inputs.conf 

[monitor:///var/log]

disabled = false

blacklist = syslog 

 

[monitor:///var/log/syslog/10.10.10.10/.../*]

disabled = false 

host = loginsight01

index = insight 

 

[monitor:///var/log/syslog/10.10.10.11/.../*]

disabled = false 

host = loginsight02

index = insight 

 

[monitor:///var/log/syslog/10.10.10.12/.../*]

disabled = false 

host = loginsight03

index = insight 

 

Thank you all for the help! 

 

 

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @JoshuaJJ ,

at first: you're running Splunk as root or asplunk user?

if splunk user, has this user the grants to read these files?

then please try this:

[monitor:///var/log/syslog/*/*/*/]
disabled = false
host_segment = 4 
index = insght
whitelist=secure|cron|message

Ciao.

Giuseppe

0 Karma

JoshuaJJ
Path Finder

Thanks for your speedy reply!  @gcusello 

Splunk is running as root and is monitoring other files within /var/log for example /var/log/audit/audit.log 

There is a specific IP I want to monitor /var/log/syslog/192.168.1.1 and all subdirectories and files under it. My thought was this should work 

[monitor:///var/log/syslog/192.168.1.1]

disabled = false 

recursive = true 

index = insight 

 

The index does exist and Splunk is running as root. 

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @JoshuaJJ ,

if in the /var/log/syslog/192.168.1.1 folder you have only those three files add * at the end of the monitor stanza,

[monitor:///var/log/syslog/192.168.1.1/*]
disabled = false 
recursive = true 
index = insight 

otherwise use the whitelist option I hinted.

[monitor:///var/log/syslog/192.168.1.1/*/*/]
disabled = false
host_segment = 4 
index = insght
whitelist=secure|cron|message

Ciao.

Giuseppe

PickleRick
SplunkTrust
SplunkTrust

First step in debugging such stuff is to run two commands

splunk list monitor

and

splunk list inputstatus

But as far as I remember Splunk has problems with monitor inputs overlapping the same directories.

You could instead just monitor whole directory with a whitelist of all four types of files and then dynamically rewrite sourcetype on ingest depending on the file path included in the source field.

But yes, it can cause issues with multiple significantly different sourcetypes (especially if they differ in timestamp format/placement).

JoshuaJJ
Path Finder

Figured it out thanks to you all! @PickleRick  @gcusello 

Here is what I did. 

First I ran the    ./splunk list monitor command to see what was being monitored. It was as you said Splunk does not like multiple overlapping input stanzas. This is what my inputs configuration looks like. I had to remove the /var/log/audit stanzas 

 

inputs.conf 

[monitor:///var/log]

disabled = false

blacklist = syslog 

 

[monitor:///var/log/syslog/10.10.10.10/.../*]

disabled = false 

host = loginsight01

index = insight 

 

[monitor:///var/log/syslog/10.10.10.11/.../*]

disabled = false 

host = loginsight02

index = insight 

 

[monitor:///var/log/syslog/10.10.10.12/.../*]

disabled = false 

host = loginsight03

index = insight 

 

Thank you all for the help! 

 

 

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...