topic Re: Splunk Inputs Help in Getting Data In

Splunk Inputs Help

JoshuaJJ — Wed, 31 Jul 2024 17:19:17 GMT

Good morning,

So I am trying to monitor all files within this directory /var/log/syslog/<IP>

Directory structure: /var/log/syslog/<IP>/2024/01 | 02 | 03 | 04 | 05 | 06 | 07/secure | cron | messages Hope this makes sense there are multiple subdirectories, the end goal is to monitor secure, cron, and messages I wrote this stanza within inputs.conf and the configuration did take on the Universal Forwarder [monitor:///var/log/syslog/192.168.1.1/.../secure] disabled = false host_segment = 4 index = insght [monitor:///var/log/syslog/192.168.1.1/.../cron] disabled = false host_segment = 4 index = insght [monitor:///var/log/syslog/192.168.1.1/.../messages] disabled = false host_segment = 4 index = insght

I have also tried this to capture all subdirs/files

[monitor:///var/log/syslog/192.168.1.1] disabled = false host_segment = 4 recursive = true index = insght

Also within _internal I get this message:

INFO TaillingProcess [#### MainTailingThread] - Parsing configuration stanza: monitor:///var/log/syslog/<IP>

Which seems to hang there with no other messages logged for the particular stanza(s)

IP Address used is notional, thanks for the help!

Re: Splunk Inputs Help

gcusello — Wed, 31 Jul 2024 16:07:47 GMT

Hi @JoshuaJJ ,

at first: you're running Splunk as root or asplunk user?

if splunk user, has this user the grants to read these files?

then please try this:

[monitor:///var/log/syslog/*/*/*/] disabled = false host_segment = 4 index = insght whitelist=secure|cron|message

Ciao.

Giuseppe

Re: Splunk Inputs Help

JoshuaJJ — Wed, 31 Jul 2024 17:20:04 GMT

Thanks for your speedy reply! @gcusello

Splunk is running as root and is monitoring other files within /var/log for example /var/log/audit/audit.log

There is a specific IP I want to monitor /var/log/syslog/192.168.1.1 and all subdirectories and files under it. My thought was this should work

[monitor:///var/log/syslog/192.168.1.1] disabled = false recursive = true index = insight

The index does exist and Splunk is running as root.

Re: Splunk Inputs Help

PickleRick — Wed, 31 Jul 2024 17:21:24 GMT

First step in debugging such stuff is to run two commands

splunk list monitor

and

splunk list inputstatus

But as far as I remember Splunk has problems with monitor inputs overlapping the same directories.

You could instead just monitor whole directory with a whitelist of all four types of files and then dynamically rewrite sourcetype on ingest depending on the file path included in the source field.

But yes, it can cause issues with multiple significantly different sourcetypes (especially if they differ in timestamp format/placement).

Re: Splunk Inputs Help

gcusello — Thu, 01 Aug 2024 06:47:12 GMT

Hi @JoshuaJJ ,

if in the /var/log/syslog/192.168.1.1 folder you have only those three files add * at the end of the monitor stanza,

[monitor:///var/log/syslog/192.168.1.1/*] disabled = false recursive = true index = insight

otherwise use the whitelist option I hinted.

[monitor:///var/log/syslog/192.168.1.1/*/*/] disabled = false host_segment = 4 index = insght whitelist=secure|cron|message

Ciao.

Giuseppe

Re: Splunk Inputs Help

JoshuaJJ — Thu, 01 Aug 2024 17:21:38 GMT

Figured it out thanks to you all! @PickleRick @gcusello

Here is what I did.

First I ran the ./splunk list monitor command to see what was being monitored. It was as you said Splunk does not like multiple overlapping input stanzas. This is what my inputs configuration looks like. I had to remove the /var/log/audit stanzas

inputs.conf

[monitor:///var/log]

disabled = false

blacklist = syslog

[monitor:///var/log/syslog/10.10.10.10/.../*]

disabled = false

host = loginsight01

index = insight

[monitor:///var/log/syslog/10.10.10.11/.../*]

disabled = false

host = loginsight02

index = insight

[monitor:///var/log/syslog/10.10.10.12/.../*]

disabled = false

host = loginsight03

index = insight

Thank you all for the help!