While you can send data outbound from SW via syslog, it's not the best or most efficient method. You'd basically be crafting either
1) a single alert that sends a syslog for each data point you want exported
or
2) multiple alerts that send syslogs for specific types of events/data collections
(remember, alerts are queries, so you'd be banging the database with queries on a near constant basis, obstructing the ability of the database to do the thing you want - which is to collect monitoring data.)
SolarWinds has a full REST-ful API (it's on Github, look for "SolarWinds Orion API" and you'll find it) which would let craft a script that exported the data you wanted, when you wanted, with all appropriate security checks, throttling, etc in place. There's a whole forum dedicated to building this kind of stuff on THWACK.com.
If you are still committed to the syslog method, I'd just want to add a couple of other thoughts:
syslog-ng is awesome. But if you're group is not comfortable with nix, it might be hard to set up and manage. In that case I'd recommend Kiwi Syslog. No, not because I'm a shill for SolarWinds, but because it's the only *other tool that does "transparent forwarding" of syslogs - meaning that the original source IP is retained when you forward. That's critical in this situation since you don't want splunk to think that it was the solarwinds box sending all those syslogs.
if you have syslog coming to splunk from a number of sources, it's going to get chattier than any single machine can handle. I don't care what vendor we're talking about. To mitigate that, you need to build a "trap and syslog filtration system". that looks like this:
a load balancer set to do UDP round robin forwarding to a pool of devices
the devices are all running either syslog-ng or kiwi syslog (for the reasons cited above) and the rulesets synchronized across all pool members
the rules will send relevant events to solarwinds, splunk, etc
the rules can ALSO send events to the audit team, network team, etc if they have their own systems they need to maintain. That reduces the number of un-necessary (but expensive) transactions you are throwing at your splunk server.
FINALLY:
I'm going to take off my SolarWinds hat and put on my "I've been doing monitoring for 20 years" hat - remember that monitoring is fundamentally a heterogeneous pass time. There's no one-size-fits-all solution (hell, there's not even a one-size-fits-most solution!). While you want to avoid dealing with 27 tools that do the same thing, you are going to have a mix of primary and secondary sources of truth; of cheap (sorry, "economical") systems that do the dog's work of monitoring and pricey solutions that provide laser-pinpoint insight for a specific condition. Just make sure you're designing for the correct mix.
Hope this helps.
- Leon Adato
... View more