Splunk Enterprise Security
Highlighted

How do I get Solarwinds data into Splunk with Syslog?

Explorer

I would like retrieve data from Solarwinds when events trigger via Syslog. I know you can use the Solarwinds Splunk App but I would like to use Syslog instead.

0 Karma
Highlighted

Re: How do I get Solarwinds data into Splunk with Syslog?

SplunkTrust
SplunkTrust

Best and easiest path -

1) Build a syslog server (I suggest syslog-ng). Instructions abound on the internet for this. For this purpose, it shouldn't have to be huge - a smallish linux VM would be fine.

2) Make solarwinds do syslog, sending it to that server. (If SW can't syslog, this is a problem we can't fix).

3) Then install the Splunk universal forwarder on that machine, have it read/parse the files and send them in.

4) (and more steps mostly not specific to this task, but important, like don't forget to logrotate them and so on.)

Happy Splunking!
-Rich

View solution in original post

Highlighted

Re: How do I get Solarwinds data into Splunk with Syslog?

New Member

While you can send data outbound from SW via syslog, it's not the best or most efficient method. You'd basically be crafting either
1) a single alert that sends a syslog for each data point you want exported
or
2) multiple alerts that send syslogs for specific types of events/data collections

(remember, alerts are queries, so you'd be banging the database with queries on a near constant basis, obstructing the ability of the database to do the thing you want - which is to collect monitoring data.)

SolarWinds has a full REST-ful API (it's on Github, look for "SolarWinds Orion API" and you'll find it) which would let craft a script that exported the data you wanted, when you wanted, with all appropriate security checks, throttling, etc in place. There's a whole forum dedicated to building this kind of stuff on THWACK.com.

If you are still committed to the syslog method, I'd just want to add a couple of other thoughts:

  • syslog-ng is awesome. But if you're group is not comfortable with nix, it might be hard to set up and manage. In that case I'd recommend Kiwi Syslog. No, not because I'm a shill for SolarWinds, but because it's the only *other tool that does "transparent forwarding" of syslogs - meaning that the original source IP is retained when you forward. That's critical in this situation since you don't want splunk to think that it was the solarwinds box sending all those syslogs.
  • if you have syslog coming to splunk from a number of sources, it's going to get chattier than any single machine can handle. I don't care what vendor we're talking about. To mitigate that, you need to build a "trap and syslog filtration system". that looks like this:
  1. a load balancer set to do UDP round robin forwarding to a pool of devices
  2. the devices are all running either syslog-ng or kiwi syslog (for the reasons cited above) and the rulesets synchronized across all pool members
  3. the rules will send relevant events to solarwinds, splunk, etc
  4. the rules can ALSO send events to the audit team, network team, etc if they have their own systems they need to maintain. That reduces the number of un-necessary (but expensive) transactions you are throwing at your splunk server.

FINALLY:
I'm going to take off my SolarWinds hat and put on my "I've been doing monitoring for 20 years" hat - remember that monitoring is fundamentally a heterogeneous pass time. There's no one-size-fits-all solution (hell, there's not even a one-size-fits-most solution!). While you want to avoid dealing with 27 tools that do the same thing, you are going to have a mix of primary and secondary sources of truth; of cheap (sorry, "economical") systems that do the dog's work of monitoring and pricey solutions that provide laser-pinpoint insight for a specific condition. Just make sure you're designing for the correct mix.

Hope this helps.
- Leon Adato

0 Karma