topic Re: How to track slow-running field extractions? in Splunk Search

How to track slow-running field extractions?

stanwin — Wed, 10 Feb 2016 17:33:31 GMT

Hi All

I have had a really bad Field extractor bogging down my system (discovered it from search.log on indexer) , tuning it made my search upto 18x faster for that app 😐 It slowed down no only searches but data model acceleration/pivots etc obviosly

I was interested in generating a auditing report on all slow running Field extractor's which would help boost the system by quite a bit ( though not all searches may be so bad to give such a performance boost)

But I dont see above log for quite a long timeline which means its not logged ( debug needed?) or criteria is different.

splunk version is 6.2.3 Build 264376

Re: How to track slow-running field extractions?

martin_mueller — Thu, 11 Feb 2016 03:48:12 GMT

I've converted this to a question so it doesn't get lost in a year-old topic.

Re: How to track slow-running field extractions?

stanwin — Thu, 29 Dec 2016 10:39:15 GMT

bump!

Any analytics to run to get worst performing field extractions..

Re: How to track slow-running field extractions?

martin_mueller — Fri, 30 Dec 2016 17:52:20 GMT

I've converted this to a comment because it's not an answer.

Re: How to track slow-running field extractions?

rjthibod — Fri, 30 Dec 2016 19:36:33 GMT

This would be very nice to have.

Re: How to track slow-running field extractions?

koshyk — Sun, 01 Jan 2017 09:05:16 GMT

I don't know any specific apps which might do, other than trying enabling debug mode and testing it specifically.

What I would do is:
- Isolate the problematic app only and remove all other apps (good place to try-out is your DEV system).
- Enable some sample data using eventgen or copy from prod and Index it with the problematic app. (approx 1 million events)
- Within the same app (or a different app) in "local" directory, create props.conf,transforms.conf,eventtypes.conf,tags.conf (or if you have clue about the problematic .conf file, just use that file only)
- copy all the extracts and put the value of all keys as "HELLO-test" (or some hardcoded variable)
- Now run the speed test to see if it gives you the 18x speed. Ideally it should give , otherwise the problem is somewhere else (eg problem in index time extractions)
- If the above results comes quickly, that means definitely it is a search time extraction regex.
- Now split the extractions: eg. copy half of the key-value extractions into the "local" conf files and run the speed test. If speed reduces, then it is one within your block you just copied.
- Repeat this process until you identify the extraction

Another option is, to use online "regex101.com" website and use the "debugger" enabled mode and check number of iterations and time taken for each regex.