Getting Data In
Highlighted

best way to concentrate logs, filter, then forward them to different indexes

Explorer

I'm trying to create on my network a "syslog" server that would act as a hub to concentrate logs from various sources, then filter them (like force a hostname, change sourcetype, drop some events...) and eventually forward them to my 2 dedicated indexers into different indexes.

problem #1: many sources ("remote_servers") are not able to select a port to send their logs to. Hence I'm having multiple sources dropping their logs on UDP:514 and I must differentiate those sources (different indexes, different sourcetype/host settings).

problem #2: some pieces of hardware wont send proper syslog data (no hostname for example), so I have to rely only on the IP of incoming connection to dispatch their logs.

problem #3: I can't use a modern rsyslogd. I've tried version 5.x as included in RHEL 6, but it's not supporting sophisticated actions (I think) I need to use rsyslogd as dispatcher.

What would be the best way to achieve this?

I've tried with rsyslogd 5.x and Universal Forwarder, and failed. I'm trying right now with Heavy Forwarder. I've created inputs like this:

[udp://192.168.120.18:514]
source=aruba
host=192.168.120.18
index=rezo

[udp://192.168.181.253:514]
source=lb-bv-prod
host=192.168.181.253
index=rezo_1m

But I'm not even sure it can really work (still have to test with real steam of data). Is it a proper way to achieve my goal?

Highlighted

Re: best way to concentrate logs, filter, then forward them to different indexes

Contributor

We are using iptables to redirect incoming UPD 514 packets to other ports, so we can have different stanzas in the inputs.conf in Splunk. Each stanza has its own index and source type so we can deliver the events as needed. For small numbers of hosts we use source based rules, if we have larger numbers, we add a virtual IP to our host and use destination based rules.

user@host  ~]$  sudo /sbin/iptables -t nat -L PREROUTING
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
REDIRECT   udp  --  anywhere             virtualhost_a  udp dpt:syslog redir ports 60018
REDIRECT   udp  --  anywhere             virtualhost_b  udp dpt:syslog redir ports 60017
REDIRECT   udp  --  sourcenet_a/31       anywhere       udp dpt:syslog redir ports 60034
REDIRECT   udp  --  sourcehost_b         anywhere       udp dpt:syslog redir ports 60001
REDIRECT   udp  --  sourcehost_c         anywhere       udp dpt:syslog redir ports 60001
...

and then in Inputs.conf:

[udp://60018]
index=aaa
sourcetype = aaa_src
connection_host = dns
Highlighted

Re: best way to concentrate logs, filter, then forward them to different indexes

Explorer

Thank you Fritz. I'm not confident enough with my iptables ability to explore this path, though I thought about it and even tried once.
I've tested the setup described in my 1st post:

Heavy forwarder with inputs.conf reading:

[udp://192.168.120.18:514]
source=foo
host=192.168.120.18
index=rezo

[udp://192.168.12.5:514]
source=bar
host=192.168.12.5
index=sstm

[udp://514]
index=catchall

I've sent bogus syslog events with logger command from 192.168.120.18: they landed into index "rezo". Same test from 192.168.12.5: landed into index "sstm". Then a third test from a non-listed host was captured in index "catchall".

So it works great, I just hope it won't turn out to be unsupported. I'm quite happy with Heavy forwarder capabilities, and with Splunk in general, but I could have gained a lot of time if documentation had clearly stated "you can use same destination port in different inputs.conf stanzas".
This kind of things is quite un-natural for most sysadmins ("can't use a port that is already in use").

View solution in original post

0 Karma