Search & timechart: faster and slower when using fields command?

[orinciog]

Hello!

I have an index with more than 25 million events (and there are going to be more). There is a saved search that inserts into an auxiliary summary index with some events based on a custom lookup (big index=domains, summary index=infected domains). I tried to make a timechart (with the count of all the events) with an overlay chart where the line represents only the infected ones.

However, for the last 24h (with about 15M events), the timechart commands takes forever. I observed the following behaviour:

This search...

index="myindex"
| fields ip
| timechart span=10m count(_raw)

...is much more faster than when used in a timechart panel (in the search there are batches of 100.000+ events at a time). However,

index="myindex"
| timechart span=10m count(_raw)

...is much faster when creating the timechart and very slow (batches of 10.000 events) when used with search.

1. Why is that and what's the best method to create a time chart when having a lot of events?
2. Does is affect other chart types too? If yes, what's the best practice?

Thanks!

[skalliger]

Because timechart is a transforming command. See Splunk's command types
.

What you probably want to use, is sitimechart.

Skalli

[FrankVl · Accepted solution]

Not entirely sure what exactly you are saying with which is faster when. But consider these things:

When running as a dashboard, it typically runs in fast mode, so no field extractions. So much faster than when running it as a normal search (which typically uses smart mode). Using the fields command, you limit the extractions performed, making the search in smart mode much faster.

In general, this type of simple counting is best done using tstats:

| tstats count where index=myindex by _time span=10m | timechart span=10m sum(count)

Or combined into one search for both indexes:

| tstats count where index=domain OR index="infected domains" by _time,index span=10m | timechart span=10m sum(count) by index

[orinciog]

@FrankVl is there a way to assign custom name to each of the series (instead of the name of the index)?

[orinciog]

That's exactly what I needed!
The results in the timechart were instant.
Thanks!