Knowledge Management

Any ideas what is causing my delay in a search using the collect and mcollect command?

denys_k
Explorer

Hello,

I have a simple collect query that looks the like the following:

| makeresults
| eval test=1
| collect index=collect_test

For some reason, i'm getting collected data with enormous delay (in average ~25 seconds, however i had few events with delay about 1500 seconds)

To find out the delay value, I am using the next query:

index=collect_test
| eval delay = _indextime - _time
| table _time test delay

This issue is reproducible with the mcollect command on metrics index.

My environment is just a full Splunk instance on a Windows server with a few forwarders that are sending data to it

Any ideas what might cause this behaviour?

0 Karma

woodcock
Esteemed Legend

@tom_frotscher is correct. When you use | collect, the data is written to the Search Head's disk, and then read and stored where the index is defined. If you used the GUI to create the summary index, it got created on the Search Head, which is wrong: you should make sure that every index exists only on the Indexers. It is exceedingly rare that the speed of the Search Head's disk is given any consideration at all and it is quite common for a Search Head to have very much slower IOPS than we ensure that Indexers have and by default, you will make 2 writes to the Search Head. First, make sure that the summary index is on the Indexers. That should make a big difference.

0 Karma

tom_frotscher
Builder

I had the same issue in a project. In my case the problem was performance.
Behind the scene, the results of your search are written to disc and then read again to be indexed. If the delay is high, the machine you are running on might be fully utilizied or has slow IO performance and is just not able to quickly index your data. you should expirience the delay also for other data and indexes.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...