Somehow combine stats data over timechart?

[dang]

I am attempting to report on server connection data which is polled at regular intervals (i.e. every 10 minutes). The data is presented to Splunk like this:

1300727510,Servers,server01,LockedCount=3
1300727510,Servers,server01,UnusedCountBelow100=8
1300727510,Servers,server01,UnusedCountAbove100=5
1300727510,Servers,server01,UnusedCountAbove500=0
1300727510,Servers,server01,UnusedCountAbove1000=99
1300727510,Servers,server01,TotalAmount=112
1300727510,Servers,server01,BlockedCount=0
1300727510,Servers,server02,LockedCount=5
1300727510,Servers,server02,UnusedCountBelow100=10
1300727510,Servers,server02,UnusedCountAbove100=2
1300727510,Servers,server02,UnusedCountAbove500=16
1300727510,Servers,server02,UnusedCountAbove1000=81
1300727510,Servers,server02,TotalAmount=109
1300727510,Servers,server02,BlockedCount=0
1300727510,Servers,server03,LockedCount=3
1300727510,Servers,server03,UnusedCountBelow100=9
1300727510,Servers,server03,UnusedCountAbove100=2
1300727510,Servers,server03,UnusedCountAbove500=1
1300727510,Servers,server03,UnusedCountAbove1000=99
1300727510,Servers,server03,TotalAmount=111
1300727510,Servers,server03,BlockedCount=0

Note that the events all have the same timestamp (the first column) for a given polling period. If I use a search like this:

index="monitoring" Servers 
  | timechart span=10m sum(UnusedCountBelow100) as DB<100sec sum(TAUnusedCountAbove100) as DB<500sec 

I get the data timecharted, but it appears It I'm only getting data for the first server, when I really want a sum of all of the servers for that given period of time. I can get a total sum of the values if I replace the timechart command with stats, like this:

index="monitoring" Servers 
  | stats sum(UnusedCountBelow100) as DB<100sec sum(TAUnusedCountAbove100) as DB<500sec

But, I can't seem to figure out how to get the data summed like the stats command over a timechart. Can anyone help me out with either the syntax, or a better way to accomplish this search?

[hazekamp]

With timechart/stats you can "split-by". In this case i think you want to split by "server".

index="monitoring" Servers | timechart span=10m sum(UnusedCountBelow100) as DB<100sec sum(TAUnusedCountAbove100) as DB<500sec by server

Update: Reading more closely it looks like you don't want to split, and that timechart might not be working as expected. You may want to submit a ticket on this, but you could also try stats followed by timechart:

index="monitoring" Servers | stats sum(UnusedCountBelow100) as DB<100sec, sum(TAUnusedCountAbove100) as DB<500sec by _time | timechart span=10m sum(DB<100sec) as DB<100sec, sum(DB<500sec) as DB<500sec

Also, in your sample above, if that is a single event I would recommend turning off line merging such that each line is a single event. "SHOULD_LINEMERGE = false" in props.conf

[dang]

It appears stats followed by timechart is yielding the same results as just using timechart, or by using the "by _time" clause in your recommendation above.

Another way of looking at this problem is that I'm looking to add up the "TotalAmount" values and get 332 from the data above (along with the other fields mentioned in the search, but I left this out to make it simpler)

I'll pass along the recommendation regarding props.conf to our splunk admins.