Getting Data In

Splunk Inputs Help

JoshuaJJ
Path Finder

Good morning, 

So I am trying to monitor all files within this directory /var/log/syslog/<IP>

Directory structure: 

/var/log/syslog/<IP>/2024/01 | 02 | 03 | 04 | 05 | 06 | 07/secure | cron | messages 
Hope this makes sense there are multiple subdirectories, the end goal is to monitor secure, cron, and messages 

I wrote this stanza within inputs.conf and the configuration did take on the Universal Forwarder

[monitor:///var/log/syslog/192.168.1.1/.../secure]

disabled = false

host_segment = 4 

index = insght



[monitor:///var/log/syslog/192.168.1.1/.../cron]

disabled = false

host_segment = 4 

index = insght



[monitor:///var/log/syslog/192.168.1.1/.../messages]

disabled = false

host_segment = 4 

index = insght

 

I have also tried this to capture all subdirs/files

[monitor:///var/log/syslog/192.168.1.1]

disabled = false

host_segment = 4 

recursive = true

index = insght

 

Also within _internal I get this message: 

INFO TaillingProcess [#### MainTailingThread] - Parsing configuration stanza: monitor:///var/log/syslog/<IP>  

Which seems to hang there with no other messages logged for the particular stanza(s)

 

IP Address used is notional, thanks for the help! 

Labels (3)
0 Karma
1 Solution

JoshuaJJ
Path Finder

Figured it out thanks to you all! @PickleRick  @gcusello 

Here is what I did. 

First I ran the    ./splunk list monitor command to see what was being monitored. It was as you said Splunk does not like multiple overlapping input stanzas. This is what my inputs configuration looks like. I had to remove the /var/log/audit stanzas 

 

inputs.conf 

[monitor:///var/log]

disabled = false

blacklist = syslog 

 

[monitor:///var/log/syslog/10.10.10.10/.../*]

disabled = false 

host = loginsight01

index = insight 

 

[monitor:///var/log/syslog/10.10.10.11/.../*]

disabled = false 

host = loginsight02

index = insight 

 

[monitor:///var/log/syslog/10.10.10.12/.../*]

disabled = false 

host = loginsight03

index = insight 

 

Thank you all for the help! 

 

 

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @JoshuaJJ ,

at first: you're running Splunk as root or asplunk user?

if splunk user, has this user the grants to read these files?

then please try this:

[monitor:///var/log/syslog/*/*/*/]
disabled = false
host_segment = 4 
index = insght
whitelist=secure|cron|message

Ciao.

Giuseppe

0 Karma

JoshuaJJ
Path Finder

Thanks for your speedy reply!  @gcusello 

Splunk is running as root and is monitoring other files within /var/log for example /var/log/audit/audit.log 

There is a specific IP I want to monitor /var/log/syslog/192.168.1.1 and all subdirectories and files under it. My thought was this should work 

[monitor:///var/log/syslog/192.168.1.1]

disabled = false 

recursive = true 

index = insight 

 

The index does exist and Splunk is running as root. 

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @JoshuaJJ ,

if in the /var/log/syslog/192.168.1.1 folder you have only those three files add * at the end of the monitor stanza,

[monitor:///var/log/syslog/192.168.1.1/*]
disabled = false 
recursive = true 
index = insight 

otherwise use the whitelist option I hinted.

[monitor:///var/log/syslog/192.168.1.1/*/*/]
disabled = false
host_segment = 4 
index = insght
whitelist=secure|cron|message

Ciao.

Giuseppe

PickleRick
SplunkTrust
SplunkTrust

First step in debugging such stuff is to run two commands

splunk list monitor

and

splunk list inputstatus

But as far as I remember Splunk has problems with monitor inputs overlapping the same directories.

You could instead just monitor whole directory with a whitelist of all four types of files and then dynamically rewrite sourcetype on ingest depending on the file path included in the source field.

But yes, it can cause issues with multiple significantly different sourcetypes (especially if they differ in timestamp format/placement).

JoshuaJJ
Path Finder

Figured it out thanks to you all! @PickleRick  @gcusello 

Here is what I did. 

First I ran the    ./splunk list monitor command to see what was being monitored. It was as you said Splunk does not like multiple overlapping input stanzas. This is what my inputs configuration looks like. I had to remove the /var/log/audit stanzas 

 

inputs.conf 

[monitor:///var/log]

disabled = false

blacklist = syslog 

 

[monitor:///var/log/syslog/10.10.10.10/.../*]

disabled = false 

host = loginsight01

index = insight 

 

[monitor:///var/log/syslog/10.10.10.11/.../*]

disabled = false 

host = loginsight02

index = insight 

 

[monitor:///var/log/syslog/10.10.10.12/.../*]

disabled = false 

host = loginsight03

index = insight 

 

Thank you all for the help! 

 

 

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In December, the Splunk Threat Research Team had 1 release of new security content via the Enterprise Security ...

Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?

(This is the first of a series of 2 blogs). Splunk Enterprise Security is a fantastic tool that offers robust ...

Index This | What are the 12 Days of Splunk-mas?

December 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...