Getting Data In

Why is the sourcetype set as filename

tkw03
Communicator

Hello

I have some syslog data collected and forwarded to a custom path:

/var/log/remote/2020/<month>/messages/<filename>

This data, for most logs got the correct sourcetype = syslog

 inputs.conf:
[monitor:///var/log/remote/.../messages]
whitelist=(archive|\_messages\.log|_messages\.log\-)
blacklist=(\.bz2$)
index=nix_os
sourcetype = syslog
disabled = 0
recursive=true
crcSalt=SOURCE1

props.conf

[source::.../var/log/remote/.../messages*]
    sourcetype = syslog

I have unfortunately seen an issue where if the file is below a certain size it gets the filename set as the sourcetype

filename:
hostname.env.ext.company.com_messages.log

path to filename:
/var/log/remote/2020/02/env/messages/hostname.env.ext.company.com_messages.log

sourcetype set as:
hostname.env.ext.company.com_messages

Why would the sourcetype get created as the filename?

Thanks for the help!

0 Karma
1 Solution

manjunathmeti
SplunkTrust
SplunkTrust

Remove attribute sourcetype = syslog in stanza [source::.../var/log/remote/.../messages*] in props.conf and check.

Update:

This looks like default splunk behaviour. Try with following configurations.

props.conf:

 [source::.../var/log/remote/.../*messages.log(.\d+)?]
 TRANSFORMS-change_sourcetype_syslogs = change_sourcetype_syslogs

 [source::.../var/log/remote/.../*audisp.log(.\d+)?]]
 TRANSFORMS-change_sourcetype_audit = change_sourcetype_audit

transforms.conf:

[change_sourcetype_syslogs]
REGEX = .*
FORMAT = sourcetype::syslogs
DEST_KEY = MetaData:Sourcetype

[change_sourcetype_audit]
REGEX = .*
FORMAT = sourcetype::linux:audit
DEST_KEY = MetaData:Sourcetype

View solution in original post

0 Karma

manjunathmeti
SplunkTrust
SplunkTrust

Remove attribute sourcetype = syslog in stanza [source::.../var/log/remote/.../messages*] in props.conf and check.

Update:

This looks like default splunk behaviour. Try with following configurations.

props.conf:

 [source::.../var/log/remote/.../*messages.log(.\d+)?]
 TRANSFORMS-change_sourcetype_syslogs = change_sourcetype_syslogs

 [source::.../var/log/remote/.../*audisp.log(.\d+)?]]
 TRANSFORMS-change_sourcetype_audit = change_sourcetype_audit

transforms.conf:

[change_sourcetype_syslogs]
REGEX = .*
FORMAT = sourcetype::syslogs
DEST_KEY = MetaData:Sourcetype

[change_sourcetype_audit]
REGEX = .*
FORMAT = sourcetype::linux:audit
DEST_KEY = MetaData:Sourcetype
0 Karma

tkw03
Communicator

That didn't seem to work, I still get some that are filename as sourcetype and too_small on some as well

0 Karma

tkw03
Communicator

Got it working.

I separated out the inputs like this:

[monitor:///var/log/remote/.../messages/]
whitelist=(messages.log)
blacklist=(\.bz2$)
index=nix_os
sourcetype = syslog
disabled = 0
recursive=true
crcSalt = <SOURCE>


[monitor:///var/log/remote/.../messages/archive/]
whitelist=(messages.log)
blacklist=(\.bz2$)
index=nix_os
sourcetype = syslog
disabled = 0
recursive=true
crcSalt = <SOURCE>


[monitor:///var/log/remote/.../audisp/]
whitelist=(audisp.log)
blacklist=(\.bz2$)
index=nix_os
sourcetype = linux:audit
disabled = 0
recursive=true
crcSalt = <SOURCE>


[monitor:///var/log/remote/.../audisp/archive/]
whitelist=(audisp.log)
blacklist=(\.bz2$)
index=nix_os
sourcetype = linux:audit
disabled = 0
recursive=true
crcSalt = <SOURCE>

along with the props and transforms above.

Thanks again

0 Karma

tkw03
Communicator

I did change my props and inputs a bit:

    Inputs:
    [monitor:///var/log/remote/.../*messages.log(.\d+)?]
    whitelist=(archive|\_messages\.log|_messages\.log\-)
    blacklist=(\.bz2$)
    index=nix_os
    sourcetype = syslog
    disabled = 0
    recursive=true
    crcSalt = <SOURCE>


    [monitor:///var/log/remote/.../*audisp.log(.\d+)?]]
    whitelist=(archive|\_audisp.log|\audisp.log\-)
    blacklist=(\.bz2$)
    index=nix_os
    sourcetype = linux:audit
    disabled = 0
    recursive=true
    crcSalt = <SOURCE>





    Props:
    [source::.../var/log/remote/.../*messages.log(.\d+)?]
    sourcetype = syslog

    [source::.../var/log/remote/.../*audisp.log(.\d+)?]]
    sourcetype = linux:audit

This has almost worked but I still get the too_small sourcetypes as well as the filename as sourcetype

0 Karma

manjunathmeti
SplunkTrust
SplunkTrust

I've updated my answer. Please check.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...