Chaining from a timechart

[JohnGregg]

All,

I know from the docs that the base of a chained search should be a transforming command, such as stats or timechart.  However I cannot figure out how to make timechart work.  I want to have 2 visualizations on my dashboard: one that is a timechart (visualized as lines) and another that shows the totals for each line or bar over the entire range.  (It would also be ok to integrate the total into the timechart/linechart but I haven't been able to figure that out, either.)

Here is the timechart that I'm trying to use as my base:

...| timechart count by kubernetes.pod_name

This works as expected.  Now I want to show a table of the totals for each pod.  IOW, if the counts were 100, 120, and 200 for 3 time periods for a single pod, I want one row that shows 420 for that pod:

| stats sum(count) by kubernetes.pod_name

This does not produce any output.  I think the issue might be kubernetes.pod_name, but I don't know what the output fields of timechart are called.

I have also tried using just the initial search as the base and having 2 chained searches-- one for timechart and one for stats-- but I run in to the event limit.

How can I chain from timechart?

thanks

[JohnGregg · Accepted solution]

I ended up using the stats "trick" mentioned by bowesmana here:

https://community.splunk.com/t5/Splunk-Search/help-on-base-search-event-limit/m-p/574058/highlight/t...

Specifically I added this to my base search:

stats count by _time,kubernetes.pod_name

[bowesmana]

Glad to see the old posts still have value 😊

Just a minor addition to consider - timechart will create buckets of _time depending on the time period you are looking at, so if you are searching a week, it will give you 1 day buckets and if you search 24h it will give you 30 minute buckets.

When you do stats by _time if you have lots of data per second, then your stats will generate a lot of results, so if you do use the stats technique mentioned, it's worth considering a 

| bin _time span=X

to define the bucket. If you don't know in advance what bucket size you want, but know what your minimum would be, e.g. 1h, then add in span=1h simply to reduce the volume. So if you had 1 million events per hour, if you don't do the bin, you would exceed the base search result set, whereas with span=1h you would get a single result per hour.

[richgalloway]

Run the base search interactively and notice the fields produced.  You should see the timechart fields don't match those expected by stats.

I suggest changing the base search to stop immediately before the timechart command.  Then have two chained searches - one for timechart and one for stats.