topic Why is the sourcetype set as filename in Getting Data In

Why is the sourcetype set as filename

tkw03 — Wed, 19 Feb 2020 16:05:54 GMT

Hello

I have some syslog data collected and forwarded to a custom path:

/var/log/remote/2020/<month>/messages/<filename>

This data, for most logs got the correct sourcetype = syslog

 inputs.conf:
[monitor:///var/log/remote/.../messages]
whitelist=(archive|\_messages\.log|_messages\.log\-)
blacklist=(\.bz2$)
index=nix_os
sourcetype = syslog
disabled = 0
recursive=true
crcSalt=SOURCE1

props.conf

[source::.../var/log/remote/.../messages*]
    sourcetype = syslog

I have unfortunately seen an issue where if the file is below a certain size it gets the filename set as the sourcetype

filename:
hostname.env.ext.company.com_messages.log

path to filename:
/var/log/remote/2020/02/env/messages/hostname.env.ext.company.com_messages.log

sourcetype set as:
hostname.env.ext.company.com_messages

Why would the sourcetype get created as the filename?

Thanks for the help!

Re: Why is the sourcetype set as filename

manjunathmeti — Wed, 19 Feb 2020 17:01:54 GMT

Remove attribute sourcetype = syslog in stanza [source::.../var/log/remote/.../messages*] in props.conf and check.

Update:

This looks like default splunk behaviour. Try with following configurations.

props.conf:

 [source::.../var/log/remote/.../*messages.log(.\d+)?]
 TRANSFORMS-change_sourcetype_syslogs = change_sourcetype_syslogs

 [source::.../var/log/remote/.../*audisp.log(.\d+)?]]
 TRANSFORMS-change_sourcetype_audit = change_sourcetype_audit

transforms.conf:

[change_sourcetype_syslogs]
REGEX = .*
FORMAT = sourcetype::syslogs
DEST_KEY = MetaData:Sourcetype

[change_sourcetype_audit]
REGEX = .*
FORMAT = sourcetype::linux:audit
DEST_KEY = MetaData:Sourcetype

Re: Why is the sourcetype set as filename

tkw03 — Tue, 03 Mar 2020 14:29:56 GMT

I did change my props and inputs a bit:

    Inputs:
    [monitor:///var/log/remote/.../*messages.log(.\d+)?]
    whitelist=(archive|\_messages\.log|_messages\.log\-)
    blacklist=(\.bz2$)
    index=nix_os
    sourcetype = syslog
    disabled = 0
    recursive=true
    crcSalt = <SOURCE>


    [monitor:///var/log/remote/.../*audisp.log(.\d+)?]]
    whitelist=(archive|\_audisp.log|\audisp.log\-)
    blacklist=(\.bz2$)
    index=nix_os
    sourcetype = linux:audit
    disabled = 0
    recursive=true
    crcSalt = <SOURCE>





    Props:
    [source::.../var/log/remote/.../*messages.log(.\d+)?]
    sourcetype = syslog

    [source::.../var/log/remote/.../*audisp.log(.\d+)?]]
    sourcetype = linux:audit

This has almost worked but I still get the too_small sourcetypes as well as the filename as sourcetype

Re: Why is the sourcetype set as filename

manjunathmeti — Tue, 03 Mar 2020 15:09:31 GMT

I've updated my answer. Please check.

Re: Why is the sourcetype set as filename

tkw03 — Tue, 03 Mar 2020 18:48:09 GMT

That didn't seem to work, I still get some that are filename as sourcetype and too_small on some as well

Re: Why is the sourcetype set as filename

tkw03 — Thu, 05 Mar 2020 17:29:31 GMT

Got it working.

I separated out the inputs like this:

[monitor:///var/log/remote/.../messages/]
whitelist=(messages.log)
blacklist=(\.bz2$)
index=nix_os
sourcetype = syslog
disabled = 0
recursive=true
crcSalt = <SOURCE>


[monitor:///var/log/remote/.../messages/archive/]
whitelist=(messages.log)
blacklist=(\.bz2$)
index=nix_os
sourcetype = syslog
disabled = 0
recursive=true
crcSalt = <SOURCE>


[monitor:///var/log/remote/.../audisp/]
whitelist=(audisp.log)
blacklist=(\.bz2$)
index=nix_os
sourcetype = linux:audit
disabled = 0
recursive=true
crcSalt = <SOURCE>


[monitor:///var/log/remote/.../audisp/archive/]
whitelist=(audisp.log)
blacklist=(\.bz2$)
index=nix_os
sourcetype = linux:audit
disabled = 0
recursive=true
crcSalt = <SOURCE>

along with the props and transforms above.

Thanks again