I have remote servers dropping logs to a syslog server where I have a Splunk forwarder configured to push it to Splunk.
Remote servers are collecting logs at /var/log/rsyslog//syslog
In input.conf, I am using host_segment = 4 to override host field to remote server's hostname. It's working fine for most of the remote servers.
I have a special case where my remote server is creating a log directory with IP instead of hostname as /var/log/rsyslog//syslog.
I was assuming inputs.conf would still extract IP and use it to override host field, but it didn't work as expected. Somehow, my Splunk forwarder is not doing that, and it's using the default host, which is Splunk forwarder's hostname. It's not able to override hostname with .
I also tried with transforms.conf and props.conf. that didn't work as well. following are my configs.
Monitor everything from rsyslog directory [monitor:///var/log/rsyslog/*/syslog] disabled = false followTail=0 host_segment = 4 blacklist = \.(gz)$ sourcetype = syslog source=/var/log/syslog
[syslog] DEST_KEY = MetaData:Host REGEX = \s(\w*)$ FORMAT = host::$1
[source::../syslog] TRANSFORMS-hostname = syslog
well, you don't need to explicitly specify source=/var/log/syslog in your inputs, your monitor path with be the source in this case.(eg.: source=/var/log/rsyslog/(ip or hostname)/syslog)
since sourcetype=syslog is a pre-trained sourceytpe within Splunk, you might have to check if any configs are overriding your configs..
./splunk btool props list --debug <sourcetype> ./splunk btool props list --debug syslog
checkout this splunk blog for reference...
When using sourcetype syslog, you get hostname extraction from the logs 'for free' based on below config in etc/system/default/props.conf
[syslog] TRANSFORMS = syslog-host
What I guess is happening:
source=/var/log/syslogactually breaks the host_segment setting (not sure about this, but it would explain the rest of the behavior)
What exactly is the idea behind that transforms you have defined? The way you wrote this, it takes the last word of the event (since you do not define a SOURCE_KEY, it will work on _raw) and put that in the host field. It is probably a good thing you made a mistake in how you wrote
[source::../syslog] (looks like you missed 1 . ), otherwise that transforms would have made quite a mess.
So to solve it:
TRANSFORMS-hostname = syslog(or explain us what you want to do with that, so we can help you write it properly)
TRANSFORMS =to your props.conf under
[syslog]to disable the default syslog host extraction
source=/var/log/syslog(I expect the host_segment will then start functioning correctly)
Okay, You are right. Thanks for this but not done yet 🙂 My syslog location in splunk forwarder is /var/log/rsyslog/%year%/%month%/%date%/%host%/syslog which I don't want to get on splunk as source. I want to override it to /var/log/syslog so I can have common source for all hosts.
How do I override source?
I would highly recommend keeping all that detail, as it can be very useful when troubleshooting problems with your syslog data to know which exact file the data came from. If you really still want to do it, I see you have a separate question raised on that already, so I'll reply to that.