Hello
I have some syslog data collected and forwarded to a custom path:
/var/log/remote/2020/<month>/messages/<filename>
This data, for most logs got the correct sourcetype = syslog
inputs.conf:
[monitor:///var/log/remote/.../messages]
whitelist=(archive|\_messages\.log|_messages\.log\-)
blacklist=(\.bz2$)
index=nix_os
sourcetype = syslog
disabled = 0
recursive=true
crcSalt=SOURCE1
props.conf
[source::.../var/log/remote/.../messages*]
sourcetype = syslog
I have unfortunately seen an issue where if the file is below a certain size it gets the filename set as the sourcetype
filename:
hostname.env.ext.company.com_messages.log
path to filename:
/var/log/remote/2020/02/env/messages/hostname.env.ext.company.com_messages.log
sourcetype set as:
hostname.env.ext.company.com_messages
Why would the sourcetype get created as the filename?
Thanks for the help!
Remove attribute sourcetype = syslog in stanza [source::.../var/log/remote/.../messages*] in props.conf and check.
This looks like default splunk behaviour. Try with following configurations.
props.conf:
[source::.../var/log/remote/.../*messages.log(.\d+)?]
TRANSFORMS-change_sourcetype_syslogs = change_sourcetype_syslogs
[source::.../var/log/remote/.../*audisp.log(.\d+)?]]
TRANSFORMS-change_sourcetype_audit = change_sourcetype_audit
transforms.conf:
[change_sourcetype_syslogs]
REGEX = .*
FORMAT = sourcetype::syslogs
DEST_KEY = MetaData:Sourcetype
[change_sourcetype_audit]
REGEX = .*
FORMAT = sourcetype::linux:audit
DEST_KEY = MetaData:Sourcetype
Remove attribute sourcetype = syslog in stanza [source::.../var/log/remote/.../messages*] in props.conf and check.
This looks like default splunk behaviour. Try with following configurations.
props.conf:
[source::.../var/log/remote/.../*messages.log(.\d+)?]
TRANSFORMS-change_sourcetype_syslogs = change_sourcetype_syslogs
[source::.../var/log/remote/.../*audisp.log(.\d+)?]]
TRANSFORMS-change_sourcetype_audit = change_sourcetype_audit
transforms.conf:
[change_sourcetype_syslogs]
REGEX = .*
FORMAT = sourcetype::syslogs
DEST_KEY = MetaData:Sourcetype
[change_sourcetype_audit]
REGEX = .*
FORMAT = sourcetype::linux:audit
DEST_KEY = MetaData:Sourcetype
That didn't seem to work, I still get some that are filename as sourcetype and too_small on some as well
Got it working.
I separated out the inputs like this:
[monitor:///var/log/remote/.../messages/]
whitelist=(messages.log)
blacklist=(\.bz2$)
index=nix_os
sourcetype = syslog
disabled = 0
recursive=true
crcSalt = <SOURCE>
[monitor:///var/log/remote/.../messages/archive/]
whitelist=(messages.log)
blacklist=(\.bz2$)
index=nix_os
sourcetype = syslog
disabled = 0
recursive=true
crcSalt = <SOURCE>
[monitor:///var/log/remote/.../audisp/]
whitelist=(audisp.log)
blacklist=(\.bz2$)
index=nix_os
sourcetype = linux:audit
disabled = 0
recursive=true
crcSalt = <SOURCE>
[monitor:///var/log/remote/.../audisp/archive/]
whitelist=(audisp.log)
blacklist=(\.bz2$)
index=nix_os
sourcetype = linux:audit
disabled = 0
recursive=true
crcSalt = <SOURCE>
along with the props and transforms above.
Thanks again
I did change my props and inputs a bit:
Inputs:
[monitor:///var/log/remote/.../*messages.log(.\d+)?]
whitelist=(archive|\_messages\.log|_messages\.log\-)
blacklist=(\.bz2$)
index=nix_os
sourcetype = syslog
disabled = 0
recursive=true
crcSalt = <SOURCE>
[monitor:///var/log/remote/.../*audisp.log(.\d+)?]]
whitelist=(archive|\_audisp.log|\audisp.log\-)
blacklist=(\.bz2$)
index=nix_os
sourcetype = linux:audit
disabled = 0
recursive=true
crcSalt = <SOURCE>
Props:
[source::.../var/log/remote/.../*messages.log(.\d+)?]
sourcetype = syslog
[source::.../var/log/remote/.../*audisp.log(.\d+)?]]
sourcetype = linux:audit
This has almost worked but I still get the too_small sourcetypes as well as the filename as sourcetype
I've updated my answer. Please check.