Why is there a big difference in performance of searches run admin vs splunk system users?

[anupkpal]

I have been investigating into searches for both admin user and splunk system user. Searched conducted by System User takes very long time. Searches done by system user are typically bucket copy trigger, copy buckets, summarize etc. Can someone explain why there is such a big difference:

Search Activity by User (2)
User Search Count Median Runtime 90th Percentile Runtime Cumulative Runtime Last Search
1 admin 201 0.17s 0.46s 5h 32min 13.32s 2018-01-31 09:55:39
2 splunk-system-user 150 3.58s 10.00s 13min 42.47s 2018-01-31 09:47:14

Common Search Commands (first there are for Splunk System user)
Command Count Average Runtime Max Runtime
1 summarize 100 3.86s 20.25s
2 bucket 25 9.56s 26.89s
3 copybuckets 25 7.92s 21.65s
4 kv 4 0.06s 0.12s
5 metadata 2 1.23s 1.41s
6 search 2 1.23s 1.41s

Appreciate your response.

Thanks,

Anup Pal
Solution Engineer,
SwiftStack Inc.

[lguinn2]

It looks like you are not comparing the same searches. The splunk-system-user account is used internally by Splunk to accomplish a lot of background work, so it is doing a lot of searches (and possibly alerts) that are more complex than most user searches.

Ad-hoc searches run by users, typically using the Splunk GUI, have a higher priority than background or scheduled searches. They are also typically less complex.

If this doesn't answer the question, then can you post a specific search that was run by both users, and give the execution time statistics for both searches?