I'm trying to create on my network a "syslog" server that would act as a hub to concentrate logs from various sources, then filter them (like force a hostname, change sourcetype, drop some events...) and eventually forward them to my 2 dedicated indexers into different indexes.
problem #1: many sources ("remote_servers") are not able to select a port to send their logs to. Hence I'm having multiple sources dropping their logs on UDP:514 and I must differentiate those sources (different indexes, different sourcetype/host settings).
problem #2: some pieces of hardware wont send proper syslog data (no hostname for example), so I have to rely only on the IP of incoming connection to dispatch their logs.
problem #3: I can't use a modern rsyslogd. I've tried version 5.x as included in RHEL 6, but it's not supporting sophisticated actions (I think) I need to use rsyslogd as dispatcher.
What would be the best way to achieve this?
I've tried with rsyslogd 5.x and Universal Forwarder, and failed. I'm trying right now with Heavy Forwarder. I've created inputs like this:
[udp://192.168.120.18:514] source=aruba host=192.168.120.18 index=rezo [udp://192.168.181.253:514] source=lb-bv-prod host=192.168.181.253 index=rezo_1m
But I'm not even sure it can really work (still have to test with real steam of data). Is it a proper way to achieve my goal?
We are using iptables to redirect incoming UPD 514 packets to other ports, so we can have different stanzas in the inputs.conf in Splunk. Each stanza has its own index and source type so we can deliver the events as needed. For small numbers of hosts we use source based rules, if we have larger numbers, we add a virtual IP to our host and use destination based rules.
user@host ~]$ sudo /sbin/iptables -t nat -L PREROUTING Chain PREROUTING (policy ACCEPT) target prot opt source destination REDIRECT udp -- anywhere virtualhost_a udp dpt:syslog redir ports 60018 REDIRECT udp -- anywhere virtualhost_b udp dpt:syslog redir ports 60017 REDIRECT udp -- sourcenet_a/31 anywhere udp dpt:syslog redir ports 60034 REDIRECT udp -- sourcehost_b anywhere udp dpt:syslog redir ports 60001 REDIRECT udp -- sourcehost_c anywhere udp dpt:syslog redir ports 60001 ...
and then in Inputs.conf:
[udp://60018] index=aaa sourcetype = aaa_src connection_host = dns
Thank you Fritz. I'm not confident enough with my iptables ability to explore this path, though I thought about it and even tried once.
I've tested the setup described in my 1st post:
Heavy forwarder with inputs.conf reading:
[udp://192.168.120.18:514] source=foo host=192.168.120.18 index=rezo [udp://192.168.12.5:514] source=bar host=192.168.12.5 index=sstm [udp://514] index=catchall
I've sent bogus syslog events with logger command from 192.168.120.18: they landed into index "rezo". Same test from 192.168.12.5: landed into index "sstm". Then a third test from a non-listed host was captured in index "catchall".
So it works great, I just hope it won't turn out to be unsupported. I'm quite happy with Heavy forwarder capabilities, and with Splunk in general, but I could have gained a lot of time if documentation had clearly stated "you can use same destination port in different inputs.conf stanzas".
This kind of things is quite un-natural for most sysadmins ("can't use a port that is already in use").