I have had a really bad Field extractor bogging down my system (discovered it from search.log on indexer) , tuning it made my search upto 18x faster for that app 😐 It slowed down no only searches but data model acceleration/pivots etc obviosly
I was interested in generating a auditing report on all slow running Field extractor's which would help boost the system by quite a bit ( though not all searches may be so bad to give such a performance boost)
But I dont see above log for quite a long timeline which means its not logged ( debug needed?) or criteria is different.
splunk version is 6.2.3 Build 264376
I don't know any specific apps which might do, other than trying enabling debug mode and testing it specifically.
What I would do is:
- Isolate the problematic app only and remove all other apps (good place to try-out is your DEV system).
- Enable some sample data using eventgen or copy from prod and Index it with the problematic app. (approx 1 million events)
- Within the same app (or a different app) in "local" directory, create props.conf,transforms.conf,eventtypes.conf,tags.conf (or if you have clue about the problematic .conf file, just use that file only)
- copy all the extracts and put the value of all keys as "HELLO-test" (or some hardcoded variable)
- Now run the speed test to see if it gives you the 18x speed. Ideally it should give , otherwise the problem is somewhere else (eg problem in index time extractions)
- If the above results comes quickly, that means definitely it is a search time extraction regex.
- Now split the extractions: eg. copy half of the key-value extractions into the "local" conf files and run the speed test. If speed reduces, then it is one within your block you just copied.
- Repeat this process until you identify the extraction
Another option is, to use online "regex101.com" website and use the "debugger" enabled mode and check number of iterations and time taken for each regex.