Knowledge Management

Any ideas what is causing my delay in a search using the collect and mcollect command?

denys_k
Explorer

Hello,

I have a simple collect query that looks the like the following:

| makeresults
| eval test=1
| collect index=collect_test

For some reason, i'm getting collected data with enormous delay (in average ~25 seconds, however i had few events with delay about 1500 seconds)

To find out the delay value, I am using the next query:

index=collect_test
| eval delay = _indextime - _time
| table _time test delay

This issue is reproducible with the mcollect command on metrics index.

My environment is just a full Splunk instance on a Windows server with a few forwarders that are sending data to it

Any ideas what might cause this behaviour?

0 Karma

woodcock
Esteemed Legend

@tom_frotscher is correct. When you use | collect, the data is written to the Search Head's disk, and then read and stored where the index is defined. If you used the GUI to create the summary index, it got created on the Search Head, which is wrong: you should make sure that every index exists only on the Indexers. It is exceedingly rare that the speed of the Search Head's disk is given any consideration at all and it is quite common for a Search Head to have very much slower IOPS than we ensure that Indexers have and by default, you will make 2 writes to the Search Head. First, make sure that the summary index is on the Indexers. That should make a big difference.

0 Karma

tom_frotscher
Builder

I had the same issue in a project. In my case the problem was performance.
Behind the scene, the results of your search are written to disc and then read again to be indexed. If the delay is high, the machine you are running on might be fully utilizied or has slow IO performance and is just not able to quickly index your data. you should expirience the delay also for other data and indexes.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...