Are there set guidelines for Splunk search best practices, and are there any other resources on this topic?

[ianbruton]

I am not sure exactly how to ask this question, so I will try to just dive right in.

Background:
I work for a company that has a lot of environments for different customers. The hosts in these environments are all feeding their logs Splunk via a forwarder installed on each host. We have started to utilize Splunk more and more over the last few months by setting up alerts and dashboards and such, which is putting more load on the Splunk infrastructure.

Issue:
I wanted to see if there was any set of guidelines for how you we should be using Splunk. Is there a right way and a wrong way to write a search, e.g. Are there methods that we should avoid using because they are inefficient and you can get the same results with a search that has been thought out more?

Getting down to brass tacks, it looks like more and more of our monitoring is going to be handled by Splunk and I don't want it to become this big bloated monster. I want to try and see if we can streamline what we are already doing before we add more checks (and more importantly reliance) onto the system.

I have been going through some of the posts that are already on here and some of the submissions on this page: http://wiki.splunk.com/Community:More_best_practices_and_processes, but I just thought it would be a good idea to do it here too.

Any help or insight would be greatly appreciated, even a link to another knowledge base would be great.

[ddrillic]

Very complex topic ; -)

It requires a lot of planning otherwise, you end up having a mishmash very quickly. Most software products leave for us the best practices which is really unfortunate.

One major thing is to come up with a plan for the sourcetypes, which is really important, otherwise we end up with different schemes for this important logical entity.

Let me think please about other points...

[ianbruton]

Yeah I had a feeling that it was going to be a large topic, given that I couldn't think of a way to encapsulate it in a straight forward question. Thank you for your insight though, this is all great information that I can go back to my team with and we can work on creating a plan to move forward with any changes that we need. And if you can think of any other important areas to look out for , please feel free to let me know 🙂

[ppablo · Accepted solution]

Hi @ianbruton

I'm not sure which Answers posts you've had the chance to check out, but these are some that I've found helpful in learning more about search optimization.

How to compare fields over multiple sourcetypes without 'join', 'append' or use of subsearches?
https://answers.splunk.com/answers/129424/how-to-compare-fields-over-multiple-sourcetypes-without-jo...

How do optimizations for field-based searches work?
https://answers.splunk.com/answers/172275/how-do-optimizations-for-field-based-searches-work.html

What is more efficient for performance: Eventtypes, lookups or calculated fields?
https://answers.splunk.com/answers/149115/what-is-more-efficient-for-performance-eventtypes-lookups-...

Why does a simple Splunk search such as index=abc take a long time to complete?
https://answers.splunk.com/answers/225289/why-does-a-simple-splunk-search-such-as-indexabc-t.html

There are some apps in Splunkbase you might want to consider trying out to see how effective your configurations are for optimizing searches and overall health in your environment.

Knowledge Object Explorer
https://splunkbase.splunk.com/app/2871/

Data Curator
https://splunkbase.splunk.com/app/1848/

Some users use this site for some inspiration when crafting up searches to see different and more efficient ways of getting the same results.
http://gosplunk.com/

I'm by no means a search expert, but I'm sure there are many others in the community that can chime in here with their 2 cents. I just spend a lot of time on Answers 🙂

[ianbruton]

Wow, that's a great amount of info for me to go through! Thanks for taking the time to provide it all to me! I will certainly have a deep dive in there and see if there is anything that we can do better.

Hopefully some other people will be able to chime in as you say.

Thanks again!

[ppablo]

No problem, I hope you get some good value out it!