Splunk Search

Searches taking long?

suhprano
Path Finder

I'm not sure if I am storing the streams properly in order for the search to process optimally. How do I know if the extracted fields are indexed? OR should I be indexing extracted fields per sourcetype to make the search go faster?

It takes me 23 minutes to search through a source returning only 13k events, scanning over 7 million events for the past 48hr time frame. Is that about right in regards to standard performance and speed?

Also, there are 6 other log sources monitored and stored in the same main index, with different sourcetypes and field extractions. Currently my main index holds about 140GB, should this search be taking this long?

Events indexed 3,849,887,346

UPDATED:

Main query - very long, 20+ min.

source="/mnt/journal_logs/current.log" earliest=-48h [search index=cs_logs earliest=-24h |dedup id|fields id] |convert ctime(_time) as GMTTime|geoip clientip|table id,accountid, clientip,score, clientip_countryname, GMTTime |dedup clientip consecutive=true sortby id

Subsearch - very fast, returns 2K events

index=cs_logs earliest=-24h |dedup wowid |fields wowid
Tags (2)
0 Karma

fox
Path Finder

sounds like a long time, however it is hard to compare as it all depends on your hardware and any concurrent processes running...

tips for speed: always explicitly name the index first and then any other known filters such as sourcetype and source. then reduce the to only the fields that you need (...|fields + ....) this reduces the work load for following piped operations. Then i would build the search up incrementally to see if any one operation is the beast.

If you still need to improve - you may want to consider adding a metadata extracted field at index time...

0 Karma

netwrkr
Communicator

"It takes me 23 minutes to search through a source returning only 13k events, scanning over 7 million events for the past 48hr time frame. Is that about right in regards to standard performance and speed?"

Heck no, that's 'grep' speed (or slower). Lets see your query.

0 Karma

netwrkr
Communicator

hrm. I'm not a search guru but I'm thinking ~13K queries to geoip could be causing some of your pain. I'm still using version 3 of Splunk but found that by changing the geoip script to use memory cache it significantly sped up my searches. In the old geoip script I did

gi = GeoIP.open(gi_db, GeoIP.GEOIP_MEMORY_CACHE)

Which makes it load the entire geoip DB once.

You don't mention what version of Splunk you're using but I found with version 3, if I separate the indexes by year it also made searches significantly faster. I don't think this is an issue with v4 though.

HTH

0 Karma

suhprano
Path Finder

I'm testing if this search is faster by moving the index off the main, which has almost 4 billion events for the past week of logs collected...

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...