Best Practices for Finding Data Across All Splunk Logs and Indexes

[Ben]

Hello,

As a SOC analyst, what are the best practices for writing SPL queries to quickly find specific data (such as an IP address, a string, or a keyword) across all logs and indexes?

I understand that it's generally recommended to narrow down searches and avoid using `index=*`, but sometimes I don't know exactly where the data is indexed (i.e., which index, sourcetype, or field name).

Any advice would be greatly appreciated.

Thanks in advance!

[livehybrid]

There is a number of things to consider here, such as maintaining a data dictionary of where certain types of data are stored, etc. 

However - One of the first things I always try and do to optimise a search is use TERM("value") in the SPL.

See screenshots below of simple example across 30 days for an IP on a local instance. This returned the data in half the time when using TERM("some.ip.address")

Before

After:

Please let me know how you get on and consider accepting this answer or adding karma this answer if it has helped.
Regards

Will

[gcusello]

Hi @Ben ,

Googleing you can find many best practices for searches, e.g.:

https://www.splunk.com/en_us/blog/customers/splunk-clara-fication-search-best-practices.html

https://lantern.splunk.com/Splunk_Platform/Product_Tips/Searching_and_Reporting/Optimizing_search

https://community.splunk.com/t5/Splunk-Search/How-to-achieve-Optimized-Search-time-in-Splunk/m-p/292...

https://conf.splunk.com/files/2016/slides/best-practices-and-better-practices-for-users.pdf

https://docs.splunk.com/Documentation/Splunk/9.4.0/Search/Quicktipsforoptimization

In general, the first rule is limiting the time used in your searches, avoid "All Time" or monts or days.

If you need to understand which are the indexes to use, you could use a first search to understand the indexes to use and a search limited to only the interesting indexs.

Then, use Fast Mode instead Verbose Mode.

If you have to search using as filter only one the index time fields (index, source, sourcetype and host), you can also use | tstats that's faster then a normal search.

Ciao.

Giuseppe