Getting Data In

Why is the sourcetype set as filename

tkw03
Communicator

Hello

I have some syslog data collected and forwarded to a custom path:

/var/log/remote/2020/<month>/messages/<filename>

This data, for most logs got the correct sourcetype = syslog

 inputs.conf:
[monitor:///var/log/remote/.../messages]
whitelist=(archive|\_messages\.log|_messages\.log\-)
blacklist=(\.bz2$)
index=nix_os
sourcetype = syslog
disabled = 0
recursive=true
crcSalt=SOURCE1

props.conf

[source::.../var/log/remote/.../messages*]
    sourcetype = syslog

I have unfortunately seen an issue where if the file is below a certain size it gets the filename set as the sourcetype

filename:
hostname.env.ext.company.com_messages.log

path to filename:
/var/log/remote/2020/02/env/messages/hostname.env.ext.company.com_messages.log

sourcetype set as:
hostname.env.ext.company.com_messages

Why would the sourcetype get created as the filename?

Thanks for the help!

0 Karma
1 Solution

manjunathmeti
SplunkTrust
SplunkTrust

Remove attribute sourcetype = syslog in stanza [source::.../var/log/remote/.../messages*] in props.conf and check.

Update:

This looks like default splunk behaviour. Try with following configurations.

props.conf:

 [source::.../var/log/remote/.../*messages.log(.\d+)?]
 TRANSFORMS-change_sourcetype_syslogs = change_sourcetype_syslogs

 [source::.../var/log/remote/.../*audisp.log(.\d+)?]]
 TRANSFORMS-change_sourcetype_audit = change_sourcetype_audit

transforms.conf:

[change_sourcetype_syslogs]
REGEX = .*
FORMAT = sourcetype::syslogs
DEST_KEY = MetaData:Sourcetype

[change_sourcetype_audit]
REGEX = .*
FORMAT = sourcetype::linux:audit
DEST_KEY = MetaData:Sourcetype

View solution in original post

0 Karma

manjunathmeti
SplunkTrust
SplunkTrust

Remove attribute sourcetype = syslog in stanza [source::.../var/log/remote/.../messages*] in props.conf and check.

Update:

This looks like default splunk behaviour. Try with following configurations.

props.conf:

 [source::.../var/log/remote/.../*messages.log(.\d+)?]
 TRANSFORMS-change_sourcetype_syslogs = change_sourcetype_syslogs

 [source::.../var/log/remote/.../*audisp.log(.\d+)?]]
 TRANSFORMS-change_sourcetype_audit = change_sourcetype_audit

transforms.conf:

[change_sourcetype_syslogs]
REGEX = .*
FORMAT = sourcetype::syslogs
DEST_KEY = MetaData:Sourcetype

[change_sourcetype_audit]
REGEX = .*
FORMAT = sourcetype::linux:audit
DEST_KEY = MetaData:Sourcetype
0 Karma

tkw03
Communicator

That didn't seem to work, I still get some that are filename as sourcetype and too_small on some as well

0 Karma

tkw03
Communicator

Got it working.

I separated out the inputs like this:

[monitor:///var/log/remote/.../messages/]
whitelist=(messages.log)
blacklist=(\.bz2$)
index=nix_os
sourcetype = syslog
disabled = 0
recursive=true
crcSalt = <SOURCE>


[monitor:///var/log/remote/.../messages/archive/]
whitelist=(messages.log)
blacklist=(\.bz2$)
index=nix_os
sourcetype = syslog
disabled = 0
recursive=true
crcSalt = <SOURCE>


[monitor:///var/log/remote/.../audisp/]
whitelist=(audisp.log)
blacklist=(\.bz2$)
index=nix_os
sourcetype = linux:audit
disabled = 0
recursive=true
crcSalt = <SOURCE>


[monitor:///var/log/remote/.../audisp/archive/]
whitelist=(audisp.log)
blacklist=(\.bz2$)
index=nix_os
sourcetype = linux:audit
disabled = 0
recursive=true
crcSalt = <SOURCE>

along with the props and transforms above.

Thanks again

0 Karma

tkw03
Communicator

I did change my props and inputs a bit:

    Inputs:
    [monitor:///var/log/remote/.../*messages.log(.\d+)?]
    whitelist=(archive|\_messages\.log|_messages\.log\-)
    blacklist=(\.bz2$)
    index=nix_os
    sourcetype = syslog
    disabled = 0
    recursive=true
    crcSalt = <SOURCE>


    [monitor:///var/log/remote/.../*audisp.log(.\d+)?]]
    whitelist=(archive|\_audisp.log|\audisp.log\-)
    blacklist=(\.bz2$)
    index=nix_os
    sourcetype = linux:audit
    disabled = 0
    recursive=true
    crcSalt = <SOURCE>





    Props:
    [source::.../var/log/remote/.../*messages.log(.\d+)?]
    sourcetype = syslog

    [source::.../var/log/remote/.../*audisp.log(.\d+)?]]
    sourcetype = linux:audit

This has almost worked but I still get the too_small sourcetypes as well as the filename as sourcetype

0 Karma

manjunathmeti
SplunkTrust
SplunkTrust

I've updated my answer. Please check.

0 Karma
Get Updates on the Splunk Community!

Announcing the 1st Round Champion’s Tribute Winners of the Great Resilience Quest

We are happy to announce the 20 lucky questers who are selected to be the first round of Champion's Tribute ...

We’ve Got Education Validation!

Are you feeling it? All the career-boosting benefits of up-skilling with Splunk? It’s not just a feeling, it's ...

What’s New in Splunk Cloud Platform 9.1.2308?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2308! Analysts can ...