Getting Data In

Need help with inputs.conf

tkw03
Communicator

Hello

I have some directories that I need to monitor. Using updated inputs for the TA_nix app I am adding syslog/linux:audit data is specific paths. It mostly works as expected BUT I had a few outliers.

Heres the basic directory structure:
/var/log is standard BUT the messages coming from other hosts goes to a path
/var/log/remote
in this path is the 2 types of logs: syslog and linux:audit as well as .bz2 which we never want indexed from any path.

/var/log/remote/202/02/<environment_name>/messages/<files>
/var/log/remote/202/02/<environment_name>/audisp/<files>

within each one of these is an archive directory as well, it contains files being written to and .bz2 which we never want indexed from any path.

/var/log/remote/202/02/<environment_name>/messages/archive/<files>
/var/log/remote/202/02/<environment_name>/audisp/archive/<files>

So the inputs I created looks like this:

[monitor:///var/log]
whitelist=(\.log|log$|messages|secure|auth|mesg$|cron$|acpid$|\.out)
blacklist=(lastlog|anaconda\.syslog|\.bz2$|audisp|\_audisp.log|\audisp.log\-)
index=nix_os
disabled = 0


[monitor:///var/log/remote/*]
whitelist=(messages|\_messages\.log|_messages\.log\-)
blacklist=(\.bz2$|audisp|\_audisp.log|\audisp.log\-)
index=nix_os
sourcetype = syslog
disabled = 0
recursive=true


[monitor:///var/log/remote/*]
whitelist=(audisp|\_audisp.log|\audisp.log\-)
blacklist=(\.bz2$|\_messages\.log|_messages\.log\-)
index=nix_os
sourcetype = linux:audit
disabled = 0
recursive=true

What I have found is that there are files with the sourcetype set as the filename, which it should be either syslog or linux:audit since the path is:
/var/log/remote/2020/02/corp/messages/archive/hostname.domain.com_messages.log-20200206

got the sourcetype set to the file name:
hostname.domain.com_messages.log-20200206

Also these did not index:
/var/log/remote/2020/02/corp2/audisp/archive/:

<ip-hidden>_messages_audisp.log-20200204
<ip-hidden>_messages_audisp.log-20200205 
<ip-hidden>_messages_audisp.log-20200206 

Can anyone tell me:

1.Why did the messages file

hostname1234.domain.com_messages.log-20200206

get the sourcetype set to the file name (some are set to "too-small" as well)
sourcetype=hostname1234.domain.com_messages or sourcetype=hostname1234.domain.com_messages-too_small

  1. Why didnt the /audisp directory and the corresponding files index? For example:

    /var/log/remote/2020/02/corp2/audisp/archive/<ip-hidden>_messages_audisp.log-20200204
    

Thanks for you assistance

0 Karma
1 Solution

nickhills
Ultra Champion

You can't have (as per your original post) two monitors on the same path:

[monitor:///var/log/remote/*]
 whitelist=(messages|\_messages\.log|_messages\.log\-)
 blacklist=(\.bz2$)
 index=nix_os
 sourcetype = syslog
 disabled = 0
 recursive=true

 [monitor:///var/log/remote/*]
 whitelist=(audisp|\_audisp.log|\audisp.log\-)
 blacklist=(\.bz2$)
 index=nix_os
 sourcetype = linux:audit
 disabled = 0
 recursive=true

Only one of these will ever take effect - the first one I think, which is why your audit logs were not correctly picked up.

Your latest comment has the correct approach:

[monitor:///var/log/remote/.../messages]
whitelist=(archive|_messages.log|_messages.log-)
blacklist=(.bz2$)
index=nix_os
sourcetype = syslog
disabled = 0
recursive=true

[monitor:///var/log/remote/.../audisp]
whitelist=(archive|_audisp.log|\audisp.log-)
blacklist=(.bz2$)
index=nix_os
sourcetype = linux:audit
disabled = 0
recursive=true

You probably want to associate a hostname to these logs too - is corp/corp2 the hostname?
in which case you can add host_segment = 6 to each monitor to associate them with the hostname from the 6th segment in the filename

If my comment helps, please give it a thumbs up!

View solution in original post

0 Karma

nickhills
Ultra Champion

You can't have (as per your original post) two monitors on the same path:

[monitor:///var/log/remote/*]
 whitelist=(messages|\_messages\.log|_messages\.log\-)
 blacklist=(\.bz2$)
 index=nix_os
 sourcetype = syslog
 disabled = 0
 recursive=true

 [monitor:///var/log/remote/*]
 whitelist=(audisp|\_audisp.log|\audisp.log\-)
 blacklist=(\.bz2$)
 index=nix_os
 sourcetype = linux:audit
 disabled = 0
 recursive=true

Only one of these will ever take effect - the first one I think, which is why your audit logs were not correctly picked up.

Your latest comment has the correct approach:

[monitor:///var/log/remote/.../messages]
whitelist=(archive|_messages.log|_messages.log-)
blacklist=(.bz2$)
index=nix_os
sourcetype = syslog
disabled = 0
recursive=true

[monitor:///var/log/remote/.../audisp]
whitelist=(archive|_audisp.log|\audisp.log-)
blacklist=(.bz2$)
index=nix_os
sourcetype = linux:audit
disabled = 0
recursive=true

You probably want to associate a hostname to these logs too - is corp/corp2 the hostname?
in which case you can add host_segment = 6 to each monitor to associate them with the hostname from the 6th segment in the filename

If my comment helps, please give it a thumbs up!
0 Karma

tkw03
Communicator

Thanks for the response! My issues now are that:

small files in the /messages path get either the filename as the sourcetype OR they get "too_small" added to the sourcetype.

All of the linux:audit logs got indexed with the vmstat source/sourcetype.

Any ideas on how to fix/make correct for future data?

0 Karma

tkw03
Communicator

Updated my inputs to this, just in acse something was blacklisting for some reason, still not getting the audisp files and still getting too_small sourcetype and sourcetype=filename on the syslog files:

[monitor:///var/log]
whitelist=(\.log|log$|messages|secure|auth|mesg$|cron$|acpid$|\.out)
blacklist=(lastlog|anaconda\.syslog|\.bz2$)
index=nix_os
disabled = 0


[monitor:///var/log/remote/*]
whitelist=(messages|\_messages\.log|_messages\.log\-)
blacklist=(\.bz2$)
index=nix_os
sourcetype = syslog
disabled = 0
recursive=true


[monitor:///var/log/remote/*]
whitelist=(audisp|\_audisp.log|\audisp.log\-)
blacklist=(\.bz2$)
index=nix_os
sourcetype = linux:audit
disabled = 0
recursive=true
0 Karma

tkw03
Communicator

I update the inputs again:
[monitor:///var/log]
whitelist=(.log|log$|messages|secure|auth|mesg$|cron$|acpid$|.out)
blacklist=(lastlog|anaconda.syslog|.bz2$)
index=nix_os
disabled = 0

[monitor:///var/log/remote/.../messages]
whitelist=(archive|_messages.log|_messages.log-)
blacklist=(.bz2$)
index=nix_os
sourcetype = syslog
disabled = 0
recursive=true

[monitor:///var/log/remote/.../audisp]
whitelist=(archive|_audisp.log|\audisp.log-)
blacklist=(.bz2$)
index=nix_os
sourcetype = linux:audit
disabled = 0
recursive=true

0 Karma

tkw03
Communicator

Found a BUNCH of my audit data in the vmstat sourcetype. How would this happen?

0 Karma
Get Updates on the Splunk Community!

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...

Introducing Ingest Actions: Filter, Mask, Route, Repeat

WATCH NOW Ingest Actions (IA) is the best new way to easily filter, mask and route your data in Splunk® ...