Splunk query leveraging TA-user-agents runs extremely long

[beetlegeuse]

I have version 1.76 of the TA-user-agents app installed on my search head for use with searching against web access logs; to prepare for this, I created a field extraction called "http_user_agent" to extract a string similar to this:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.3 Safari/605.1.15

When I run the following search over a 15 minute timeframe in Fast Mode...

index=foo source=*access* sourcetype=bar 
| lookup user_agents http_user_agent 
| search ua_family="Safari" ua_os_family="Mac OS X" 
| eval browserOS = ua_family . ":" . ua_os_family 
| timechart count by browserOS limit=0

...I find that it takes what seems to be a VERY long time to complete (446 seconds to search through 418,000 events). If I just run the base search without the "lookup", "search", "eval" and "timechart", it takes 3.3 seconds to execute against the same number of events.

Is this expected behavior for the TA, or is my search not optimized correctly? 

[PickleRick]

It probably is an expected performance. See the Job Inspect details to see which step is most time-consuming and which produces which number of results.

One easy thing for possible improvement is limiting your search scope. For now you're doing lookup on every single event from your timerange. It could speed up your search a little if you can limit your number of events early - if you're expecting the word "Safari" in your ecents, filter early - look for Safari in the first step.

But it might simply be that it's the aggregation at the end that takes most of the time. In that case you can't do much.