Any ideas what is causing my delay in a search using the collect and mcollect command?

[denys_k]

Hello,

I have a simple collect query that looks the like the following:

| makeresults
| eval test=1
| collect index=collect_test

For some reason, i'm getting collected data with enormous delay (in average ~25 seconds, however i had few events with delay about 1500 seconds)

To find out the delay value, I am using the next query:

index=collect_test
| eval delay = _indextime - _time
| table _time test delay

This issue is reproducible with the mcollect command on metrics index.

My environment is just a full Splunk instance on a Windows server with a few forwarders that are sending data to it

Any ideas what might cause this behaviour?

[woodcock]

@tom_frotscher is correct. When you use | collect, the data is written to the Search Head's disk, and then read and stored where the index is defined. If you used the GUI to create the summary index, it got created on the Search Head, which is wrong: you should make sure that every index exists only on the Indexers. It is exceedingly rare that the speed of the Search Head's disk is given any consideration at all and it is quite common for a Search Head to have very much slower IOPS than we ensure that Indexers have and by default, you will make 2 writes to the Search Head. First, make sure that the summary index is on the Indexers. That should make a big difference.

[tom_frotscher]

I had the same issue in a project. In my case the problem was performance.
Behind the scene, the results of your search are written to disc and then read again to be indexed. If the delay is high, the machine you are running on might be fully utilizied or has slow IO performance and is just not able to quickly index your data. you should expirience the delay also for other data and indexes.