topic Re: Search & timechart: faster and slower when using fields command? in Splunk Search

Search & timechart: faster and slower when using fields command?

orinciog — Wed, 28 Nov 2018 09:09:55 GMT

Hello!

I have an index with more than 25 million events (and there are going to be more). There is a saved search that inserts into an auxiliary summary index with some events based on a custom lookup (big index=domains, summary index=infected domains). I tried to make a timechart (with the count of all the events) with an overlay chart where the line represents only the infected ones.

However, for the last 24h (with about 15M events), the timechart commands takes forever. I observed the following behaviour:

This search...

index="myindex"
| fields ip
| timechart span=10m count(_raw)

...is much more faster than when used in a timechart panel (in the search there are batches of 100.000+ events at a time). However,

index="myindex"
| timechart span=10m count(_raw)

...is much faster when creating the timechart and very slow (batches of 10.000 events) when used with search.

Why is that and what's the best method to create a time chart when having a lot of events?
Does is affect other chart types too? If yes, what's the best practice?

Thanks!

Re: Search & timechart: faster and slower when using fields command?

FrankVl — Wed, 28 Nov 2018 09:53:32 GMT

Not entirely sure what exactly you are saying with which is faster when. But consider these things:

When running as a dashboard, it typically runs in fast mode, so no field extractions. So much faster than when running it as a normal search (which typically uses smart mode). Using the fields command, you limit the extractions performed, making the search in smart mode much faster.

In general, this type of simple counting is best done using tstats:

| tstats count where index=myindex by _time span=10m | timechart span=10m sum(count)

Or combined into one search for both indexes:

| tstats count where index=domain OR index="infected domains" by _time,index span=10m | timechart span=10m sum(count) by index

Re: Search & timechart: faster and slower when using fields command?

skalliger — Wed, 28 Nov 2018 10:13:06 GMT

Because timechart is a transforming command. See Splunk's command types
.

What you probably want to use, is sitimechart.

Skalli

Re: Search & timechart: faster and slower when using fields command?

orinciog — Wed, 28 Nov 2018 10:29:02 GMT

That's exactly what I needed!
The results in the timechart were instant.
Thanks!

Re: Search & timechart: faster and slower when using fields command?

orinciog — Wed, 28 Nov 2018 10:37:11 GMT

@FrankVl is there a way to assign custom name to each of the series (instead of the name of the index)?