Splunk Search
Highlighted

Search & timechart: faster and slower when using fields command?

New Member

Hello!

I have an index with more than 25 million events (and there are going to be more). There is a saved search that inserts into an auxiliary summary index with some events based on a custom lookup (big index=domains, summary index=infected domains). I tried to make a timechart (with the count of all the events) with an overlay chart where the line represents only the infected ones.

However, for the last 24h (with about 15M events), the timechart commands takes forever. I observed the following behaviour:

This search...

index="myindex"
| fields ip
| timechart span=10m count(_raw)

...is much more faster than when used in a timechart panel (in the search there are batches of 100.000+ events at a time). However,

index="myindex"
| timechart span=10m count(_raw)

...is much faster when creating the timechart and very slow (batches of 10.000 events) when used with search.

  1. Why is that and what's the best method to create a time chart when having a lot of events?
  2. Does is affect other chart types too? If yes, what's the best practice?

Thanks!

0 Karma
Highlighted

Re: Search & timechart: faster and slower when using fields command?

Ultra Champion

Not entirely sure what exactly you are saying with which is faster when. But consider these things:

When running as a dashboard, it typically runs in fast mode, so no field extractions. So much faster than when running it as a normal search (which typically uses smart mode). Using the fields command, you limit the extractions performed, making the search in smart mode much faster.

In general, this type of simple counting is best done using tstats:

| tstats count where index=myindex by _time span=10m | timechart span=10m sum(count)

Or combined into one search for both indexes:

| tstats count where index=domain OR index="infected domains" by _time,index span=10m | timechart span=10m sum(count) by index

View solution in original post

0 Karma
Highlighted

Re: Search & timechart: faster and slower when using fields command?

New Member

That's exactly what I needed!
The results in the timechart were instant.
Thanks!

0 Karma
Highlighted

Re: Search & timechart: faster and slower when using fields command?

New Member

@FrankVl is there a way to assign custom name to each of the series (instead of the name of the index)?

0 Karma
Highlighted

Re: Search & timechart: faster and slower when using fields command?

SplunkTrust
SplunkTrust

Because timechart is a transforming command. See Splunk's command types
.

What you probably want to use, is sitimechart.

Skalli

0 Karma