Splunk Search

How to improve the performance of our Splunk search query?

dhavamanis
Builder

We have indexed access logs into index="mpsapp", When we do a stats search or filter any records for these data for a particular month, it's extremely slow (took more than 1.5 hours for first query). Can you please tell us how to optimize this query?

Sample Splunk Queries :

  1. index="mpsapp" | stats count by response_code (filtered time range only for October month, we have extracted the filed response_code in search node)
  2. source="mps" date_year=2014 date_month=october response_code!=200
  3. date_year=2014 date_month=november site="fandango-web" path="%2F"

Total Events for the month of October is : 355,925,951 events (10/1/14 12:00:00.000 AM to 11/1/14 12:00:00.000 AM)
Splunk Version : 6.1.4

Is there any configuration level optimization required to speed-up the query response? Please share your suggestions.

Tags (3)
1 Solution

MuS
SplunkTrust
SplunkTrust

Hi dhavamanis,

try to use index and any of the metadata http://docs.splunk.com/Documentation/Splunk/6.2.0/SearchReference/Metadata fields ( host, source or sourcetype) in your searches. Try to filter the fields as tight as possible in your base search or use fields http://docs.splunk.com/Documentation/Splunk/6.2.0/SearchReference/Fields#Examples as @aljohnson_splunk mentioned.

Do not use NOT searches, rather search for events that you want and need.

Also, take a look at this great Answer of @jrodman about How do optimizations for field-based searches work http://answers.splunk.com/answers/172275/how-do-optimizations-for-field-based-searches-work.html#ans...

hope this helps to understand the possibilities to speed up your searches...

cheers, MuS

View solution in original post

preactivity
Path Finder

You can improve the performance by 10 X times by using Splunk meta data fields. I can help you in that please contact me in fiverr or Email (hurdlej1@gmail.com)

https://www.fiverr.com/s2/affc9b7a8a
https://www.fiverr.com/s2/608e8ed73f?utm_source=CopyLink_Mobile

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi dhavamanis,

try to use index and any of the metadata http://docs.splunk.com/Documentation/Splunk/6.2.0/SearchReference/Metadata fields ( host, source or sourcetype) in your searches. Try to filter the fields as tight as possible in your base search or use fields http://docs.splunk.com/Documentation/Splunk/6.2.0/SearchReference/Fields#Examples as @aljohnson_splunk mentioned.

Do not use NOT searches, rather search for events that you want and need.

Also, take a look at this great Answer of @jrodman about How do optimizations for field-based searches work http://answers.splunk.com/answers/172275/how-do-optimizations-for-field-based-searches-work.html#ans...

hope this helps to understand the possibilities to speed up your searches...

cheers, MuS

dhavamanis
Builder

Thanks!. We have accelerated the report and added to Dashboard. Using the summary index and its fetching results very fast.

0 Karma

puneethgowda
Communicator

We have done the following things after doing R & D.

1.Changed date range from real time to today.
2.Set dashboard refresh time to every 5 minutes.
3.Summary indexing
4.Report acceleration
5.Scheduled this search every 5 minutes so it will save in the cache.
6.Search query optimization.
7.Auto restart splunk daily at 2:00 AM UTC so that memory will be released.
8.Set high priority to this dashboard.
7.Set high priority to this scheduled search.
8.Run stats tables first then start charts.
9.Changed the delimer of raw data from text files method to new way which will reduce the time while converting raw data to fields of delimiting proccess.
10.Reduce the number of indexes and source type

After all this my dashboards loading time reduced from 3 minutes to less than 10 seconds.

Super fast

aljohnson_splun
Splunk Employee
Splunk Employee

Do you ever use the fields command to only retrieve the relevant fields ?

0 Karma

dhavamanis
Builder

we haven't tried any fields command. Can you please give me some sample for fields command.

0 Karma

aljohnson_splun
Splunk Employee
Splunk Employee
 sourcetype=access_combined

Job Inspector: This search has completed and has returned 1,000 results by scanning 177,830 events in 14.393 seconds.

sourcetype=access_combined | fields clientip bytes action

Job Inpector: This search has completed and has returned 1,000 results by scanning 177,835 events in 3.508 seconds.

It's similar to using fast mode. Are you using fast mode?
Set search mode to adjust your search experience

Performance recommendations
How search types affect performance

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...