Getting Data In

Why is the sourcetype set as filename

tkw03
Communicator

Hello

I have some syslog data collected and forwarded to a custom path:

/var/log/remote/2020/<month>/messages/<filename>

This data, for most logs got the correct sourcetype = syslog

 inputs.conf:
[monitor:///var/log/remote/.../messages]
whitelist=(archive|\_messages\.log|_messages\.log\-)
blacklist=(\.bz2$)
index=nix_os
sourcetype = syslog
disabled = 0
recursive=true
crcSalt=SOURCE1

props.conf

[source::.../var/log/remote/.../messages*]
    sourcetype = syslog

I have unfortunately seen an issue where if the file is below a certain size it gets the filename set as the sourcetype

filename:
hostname.env.ext.company.com_messages.log

path to filename:
/var/log/remote/2020/02/env/messages/hostname.env.ext.company.com_messages.log

sourcetype set as:
hostname.env.ext.company.com_messages

Why would the sourcetype get created as the filename?

Thanks for the help!

0 Karma
1 Solution

manjunathmeti
SplunkTrust
SplunkTrust

Remove attribute sourcetype = syslog in stanza [source::.../var/log/remote/.../messages*] in props.conf and check.

Update:

This looks like default splunk behaviour. Try with following configurations.

props.conf:

 [source::.../var/log/remote/.../*messages.log(.\d+)?]
 TRANSFORMS-change_sourcetype_syslogs = change_sourcetype_syslogs

 [source::.../var/log/remote/.../*audisp.log(.\d+)?]]
 TRANSFORMS-change_sourcetype_audit = change_sourcetype_audit

transforms.conf:

[change_sourcetype_syslogs]
REGEX = .*
FORMAT = sourcetype::syslogs
DEST_KEY = MetaData:Sourcetype

[change_sourcetype_audit]
REGEX = .*
FORMAT = sourcetype::linux:audit
DEST_KEY = MetaData:Sourcetype

View solution in original post

0 Karma

manjunathmeti
SplunkTrust
SplunkTrust

Remove attribute sourcetype = syslog in stanza [source::.../var/log/remote/.../messages*] in props.conf and check.

Update:

This looks like default splunk behaviour. Try with following configurations.

props.conf:

 [source::.../var/log/remote/.../*messages.log(.\d+)?]
 TRANSFORMS-change_sourcetype_syslogs = change_sourcetype_syslogs

 [source::.../var/log/remote/.../*audisp.log(.\d+)?]]
 TRANSFORMS-change_sourcetype_audit = change_sourcetype_audit

transforms.conf:

[change_sourcetype_syslogs]
REGEX = .*
FORMAT = sourcetype::syslogs
DEST_KEY = MetaData:Sourcetype

[change_sourcetype_audit]
REGEX = .*
FORMAT = sourcetype::linux:audit
DEST_KEY = MetaData:Sourcetype
0 Karma

tkw03
Communicator

That didn't seem to work, I still get some that are filename as sourcetype and too_small on some as well

0 Karma

tkw03
Communicator

Got it working.

I separated out the inputs like this:

[monitor:///var/log/remote/.../messages/]
whitelist=(messages.log)
blacklist=(\.bz2$)
index=nix_os
sourcetype = syslog
disabled = 0
recursive=true
crcSalt = <SOURCE>


[monitor:///var/log/remote/.../messages/archive/]
whitelist=(messages.log)
blacklist=(\.bz2$)
index=nix_os
sourcetype = syslog
disabled = 0
recursive=true
crcSalt = <SOURCE>


[monitor:///var/log/remote/.../audisp/]
whitelist=(audisp.log)
blacklist=(\.bz2$)
index=nix_os
sourcetype = linux:audit
disabled = 0
recursive=true
crcSalt = <SOURCE>


[monitor:///var/log/remote/.../audisp/archive/]
whitelist=(audisp.log)
blacklist=(\.bz2$)
index=nix_os
sourcetype = linux:audit
disabled = 0
recursive=true
crcSalt = <SOURCE>

along with the props and transforms above.

Thanks again

0 Karma

tkw03
Communicator

I did change my props and inputs a bit:

    Inputs:
    [monitor:///var/log/remote/.../*messages.log(.\d+)?]
    whitelist=(archive|\_messages\.log|_messages\.log\-)
    blacklist=(\.bz2$)
    index=nix_os
    sourcetype = syslog
    disabled = 0
    recursive=true
    crcSalt = <SOURCE>


    [monitor:///var/log/remote/.../*audisp.log(.\d+)?]]
    whitelist=(archive|\_audisp.log|\audisp.log\-)
    blacklist=(\.bz2$)
    index=nix_os
    sourcetype = linux:audit
    disabled = 0
    recursive=true
    crcSalt = <SOURCE>





    Props:
    [source::.../var/log/remote/.../*messages.log(.\d+)?]
    sourcetype = syslog

    [source::.../var/log/remote/.../*audisp.log(.\d+)?]]
    sourcetype = linux:audit

This has almost worked but I still get the too_small sourcetypes as well as the filename as sourcetype

0 Karma

manjunathmeti
SplunkTrust
SplunkTrust

I've updated my answer. Please check.

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...