Archive

Advice on Using Splunk

I am evaluating Splunk to see if it is a good fit for an application we need to build. I'd appreciate thoughts on whether a/ it is a good fit or not and b/ if it is how best to build it with splunk.

We are automating a report that displays information about transactions occurring in some of our systems. It displays information such as:

  • Volume of transactions in the last day, month, FY
  • Peak volume / hr in the last day
  • Average / median etc transaction durations in the last day etc plus peaks
  • Static data about objectives for the above (e.g. target max volume per day etc)
  • Comparisons of actual results vs objectives (e.g. number transactions that exceed the objective for duration)

The results are to be displayed in a grid, where each cell is a different splunk search. From my reading of the splunk doco it seems the two main options for this are:

  • using the SingleValue module. I've played around with the CSS a bit to get it to look more grid'ish. Is this a sensible approach? Am I likely to need a custom mako template at some point?
  • using the REST API. This gives us full control of the UI but what do we lose? What things are harder to do now?

Users will all be looking at the same values so caching will be important. It seems that saved searches with a schedule achieves this? Is that the best approach?

This data exists in several databases. So we need batch jobs to export into files suitable for splunk to monitor. Is there any code / tools that help with this task?

Any help appreciated.

Tags (1)
0 Karma
1 Solution

SplunkTrust
SplunkTrust

This sounds like a pretty good candidate for a Splunk app. As far as the user interface requirements most of this stuff is pretty straightforward. The only thing here that's a little advanced, is to display the target thresholds as a static element within the same graph as the live data. I'm not saying it's hard and many apps are doing this sort of thing now, but it stands out as less trivial than the other stuff you're listing here.

scheduling saved searches is a tried-and-true approach. Scheduling a real-time search is another approach that can be better on balance, although that's not always true. Indeed though if you have more than a couple users you'll want to do one or the other to prevent lots of ad-hoc searches from being kicked off when people load dashboards. I cant find a good docs link about scheduling real-time searches and then using them on dashboards but it's pretty mainstream functionality at this point.

As far as getting data in from databases, it's also been done a lot. The method I'm familiar with is to use what Splunk calls a "scripted input". You connect via ODBC or JDBC or whatever from your python script and it returns rows that then get indexed. Again I can't seem to find a good link or a good example, but with some persistence (and maybe another question up here on answers), im sure you can shake one out of the tree because there's a lot of people out there doing it.

general scripted input docs:
http://docs.splunk.com/Documentation/Splunk/latest/Developer/ScriptSetup

One other nitty gritty point -- instead of using the SingleValue module I recommend installing the Sideview Utils app, and checking out its HTML module. It's a much more powerful module but on balance it's a little easier to use than SingleValue. And trying to customize SingleValue's appearance isn't a lot of fun. And there are a lot of other benefits as well.

As the author of Sideview Utils I'm a little biased, but I don't think anyone who's tried it both ways would disagree -- custom app development is much easier with the modules from Sideview Utils than without them.

View solution in original post

0 Karma

SplunkTrust
SplunkTrust

This sounds like a pretty good candidate for a Splunk app. As far as the user interface requirements most of this stuff is pretty straightforward. The only thing here that's a little advanced, is to display the target thresholds as a static element within the same graph as the live data. I'm not saying it's hard and many apps are doing this sort of thing now, but it stands out as less trivial than the other stuff you're listing here.

scheduling saved searches is a tried-and-true approach. Scheduling a real-time search is another approach that can be better on balance, although that's not always true. Indeed though if you have more than a couple users you'll want to do one or the other to prevent lots of ad-hoc searches from being kicked off when people load dashboards. I cant find a good docs link about scheduling real-time searches and then using them on dashboards but it's pretty mainstream functionality at this point.

As far as getting data in from databases, it's also been done a lot. The method I'm familiar with is to use what Splunk calls a "scripted input". You connect via ODBC or JDBC or whatever from your python script and it returns rows that then get indexed. Again I can't seem to find a good link or a good example, but with some persistence (and maybe another question up here on answers), im sure you can shake one out of the tree because there's a lot of people out there doing it.

general scripted input docs:
http://docs.splunk.com/Documentation/Splunk/latest/Developer/ScriptSetup

One other nitty gritty point -- instead of using the SingleValue module I recommend installing the Sideview Utils app, and checking out its HTML module. It's a much more powerful module but on balance it's a little easier to use than SingleValue. And trying to customize SingleValue's appearance isn't a lot of fun. And there are a lot of other benefits as well.

As the author of Sideview Utils I'm a little biased, but I don't think anyone who's tried it both ways would disagree -- custom app development is much easier with the modules from Sideview Utils than without them.

View solution in original post

0 Karma

Thanks a lot for your help. I like the look of the HTML module. Gives me the confidence that we can get all the control over the UI that we'd like

0 Karma