topic Re: Too complex for splunk? in Splunk Enterprise

Too complex for splunk?

renems — Tue, 18 Aug 2015 14:21:21 GMT

Is this possible in splunk?

For a while now, I see at a lot of splunk customers having the same troubles they're going through:

a lot of complexity on a very low level (per host: do I have enough resources? Are my deamons running? Are my certificates still valid? Is the responsetime of my application performing well? etc).
large server farms, applications that live on multiple hosts, relations that exist between multiple clusters etc.
To say something of any of those larger entities, it's necessary to know the status of all those details of the seperate hosts.

I'd like to build a traffic light, that shows the status on the highest level. Connecting the dots, really. When it goes red, you would be able to click down to a level deeper. Let's say it's on department level, and you can see which department has issues. Clicking on that department, you jump to a page showing all applications of that department. One application is red, and clicking on it shows all the servers associated with it. Finally we go to the specific server, and see what's actually wrong. Any thoughts how to work this out with splunk?

Re: Too complex for splunk?

Gilberto_Castil — Tue, 18 Aug 2015 14:42:58 GMT

At a very elemental level, Splunk can deal with your wish list if the appropriate data is available. The approach would require the mapping of data onto a common framework that can be used to aggregate health and performance indicators. Splunk's Common Information Model allows for the disambiguation of data for common analysis and refactoring. In more technical terms, a search can produce a notable event. Once you have a collection, notable events can be summarized mathematically for comparisons against thresholds. The combination of thresholds and active, aggregated monitoring produces KPIs. If you can map multiple KPIs to reflect an entity, then you can create state. Grouping and mapping all of these onto logical entities creates a high level service view.

There is a project called Splunk IT Service Intelligence which provides you with the next generation service monitoring based on a data-driven approach. You can read about it here. The assets are currently hosted in Splunkbase.

To finally answer your question: No; it is not too complex for Splunk.

Re: Too complex for splunk?

jensonthottian — Tue, 18 Aug 2015 15:19:12 GMT

We created a composite drilldown dashboard that had traffic lights to show the RAG status of each component.

How we acheived it:

Use custom css and images for traffic lights.
Use a flag CSV for representing the RAG status of a component: for e.g. Component A traffic light (Red Amber or Green) depends on 4 conditions. so for all 4 conditions keep individual search string which will individually update the traffic light color based on the flag csv file which had been updated regularly using these conditions.

Re: Too complex for splunk?

esix_splunk — Tue, 18 Aug 2015 15:37:06 GMT

As mentioned, Splunk ITSi will provide this functionality. Stay tuned to .conf this year for more information!

Re: Too complex for splunk?

harish0557 — Tue, 18 Aug 2015 16:59:07 GMT

Was trying to do this, the above steps should be able to get our job done.

Re: Too complex for splunk?

aljohnson_splun — Tue, 18 Aug 2015 17:14:08 GMT

this should be an answer

Re: Too complex for splunk?

millern4 — Tue, 18 Aug 2015 17:27:19 GMT

Taking a peek at the documentation, this looks very similar to another app - Enterprise Security with the correlation searches, notables, and so on. Will be interesting to see this presented at .conf

Re: Too complex for splunk?

renems — Tue, 18 Aug 2015 19:24:36 GMT

Looks very promising, Pablo! I'll dive into the docs and see if it fits my needs.
You think ITSI would also support multiple drilldown levels? (I tried to explain this in the last bit, do you understand what I mean?)

Re: Too complex for splunk?

bmacias84 — Tue, 18 Aug 2015 22:43:07 GMT

All of that can be done but some of the navigation and mapping are not built in to simple XML and require the use of the Web Framework.

Re: Too complex for splunk?

nasamajh09 — Tue, 22 Mar 2016 01:08:50 GMT

Could you please explain in detail...I am very new to Splunk and I have a similar requirements.Thanks in advance.

Re: Too complex for splunk?

MuS — Tue, 22 Mar 2016 01:27:23 GMT

First create searches that will provide the needed information or results to be able to determine the status of the component or service. Next take a look at this App https://splunkbase.splunk.com/app/1923/#/overview and use it together with your search.

Try something simple and easy to start with, like Apache status codes. If there are more than 10 x 404 error codes within one minute set the ample to yellow, else it's green.

cheers, MuS