Archive

What's the right way to think about working with splunk dashboards?

Explorer

Let me preface this question with the "nothing useful is coming up via google" question. In short, I have about 4-5 source of log data that other departments ingests into Enterprise Splunk. I have no control of how this data is ingested - it's not my data.

I also have about 3-4 external metadata stores which describe my universe (i.e. bounding things to a subset of hosts; adding additional descriptions about device usage etc).

To "get what I want" - it seems that my dashboards queries need to be extremely complex every time. Is this normal?

An example dashboard would be:
Over the last 15 minutes, show me all time and temperature events for my systems broken down by enhancedTag75 and region.
EnhancedTag75 is an additional piece of metadata that joins on the hostname.
On a dashboard, region would be a dropdown based on all region values in the data set to allow further bucketing.

My system is defined as the 150-bucket subset of about 10,000 buckets which correlate to about 100,000 machines.

What do normal people do in these situations? If I can clean up this question to make it more precise, please let me know.

Alternatively, I'm thinking of not using spunk for the dashboarding and exporting the data to a middle tier service (streambase) which does the enrichment for presentation at a web-gui layer (datawatch or some such product).

Many thanks in advance!

Tags (1)
0 Karma
1 Solution

SplunkTrust
SplunkTrust

Splunk has the lookup mechanism for enrichment data: http://docs.splunk.com/Documentation/Splunk/6.4.0/Knowledge/Usefieldlookupstoaddinformationtoyoureve...

Once an automatic lookup is defined for a sourcetype, source, or host, its enriched fields magically appear in results and can be used for filtering or reporting. Using the lookup table itself you can dynamically fill dropdowns in dashboards as well.

View solution in original post

SplunkTrust
SplunkTrust

Splunk has the lookup mechanism for enrichment data: http://docs.splunk.com/Documentation/Splunk/6.4.0/Knowledge/Usefieldlookupstoaddinformationtoyoureve...

Once an automatic lookup is defined for a sourcetype, source, or host, its enriched fields magically appear in results and can be used for filtering or reporting. Using the lookup table itself you can dynamically fill dropdowns in dashboards as well.

View solution in original post

Path Finder

Not that unusual. A few ideas.

Check out search macros - this could help if the searches are large, complex and similar. You can define a couple of macros that help with your metadata looksups and use them:
http://docs.splunk.com/Documentation/Splunk/6.2.5/Search/Usesearchmacros

Next, you can define one large crazy search at the top of the dashboard and control the reporting in sub-searches:
http://docs.splunk.com/Documentation/Splunk/6.2.5/AdvancedDev/PostProcess

I don't see why exporting the data to another system would help in the case. I'm not sure exactly what you are doing, but having used Oracle CEP and Streambase CEP, I don't see that as making it any easier to manage the data/dashboards/reports.

0 Karma

Explorer

Thank you for the note vis-a-vis Streambase ! Is Splunk smart-enough to somehow optimize for these "always enrich" sort of metadata queries? I'm concerned that with enough enrichment I might be asked to fund my own hardware...

0 Karma