topic Re: Unable to use results of Streamstats in Splunk Search

Unable to use results of Streamstats

Jason — Wed, 06 Oct 2010 21:19:31 GMT

I'm dealing with some web logs, and have generated statistics on how long a certain user stayed on a certain page by using the streamstats command below:

search ... | streamstats current=t global=f window=2 range(_time) as Dur by User | eval Duration=if(isnull(Dur), 0, if(Dur>1800, 0, Dur)) | stats count by _time, User, Page, Duration | fields - count

This shows Duration, the amount of time a particular User spent on a particular Page. (The eval ignores times over 30 minutes; they are assumed to be different web sessions).

Now I am trying to do more things with Duration, such as sum it up per page, or make a total amount of time all users spent on all pages. But I am running into the same problem - I can't seem to use the Duration field!

search ... | streamstats current=t global=f window=2 range(_time) as Dur by User | eval Duration=if(isnull(Dur), 0, if(Dur>1800, 0, Dur)) | stats count sum(Duration) by Page

Gives an error, saying Specified field(s) missing from results: Duration

And when I try to sum up all Durations using eventstats so I can make a percentage calculation later,

search ... | streamstats current=t global=f window=2 range(_time) as Dur by User | eval Duration=if(isnull(Dur), 0, if(Dur>1800, 0, Dur)) | eventstats sum(Duration) as AllDuration

The AllDuration field doesn't even show up. What is going wrong here? I thought streamstats (especially followed by an eval) would definitely create a usable field like any other.

Behavior seen on both 4.1.5/Linux64 and 4.1.5/Windows32.

Re: Unable to use results of Streamstats

Jason — Wed, 06 Oct 2010 21:21:10 GMT

I tried adding [Duration] INDEXED_VALUE = false to my app's fields.conf, but this didn't work.

Re: Unable to use results of Streamstats

Lowell — Thu, 07 Oct 2010 01:16:11 GMT

Weird. I'm running 4.1.5 on Linux 32 bit, and I tried a similar search with no issues. This search worked fine: sourcetype=*ftpd* | streamstats current=t global=f window=2 range(_time) as Dur by pid | eval Duration=if(isnull(Dur), 0, if(Dur>1800, 0, Dur)) | stats count by Duration

Re: Unable to use results of Streamstats

Jason — Thu, 07 Oct 2010 01:23:05 GMT

Yes, stats count by Duration, fields is the only thing that works. stats sum(Duration) by fields fails, as does | eventstats sum(Duration) as Total | stats count by Total, fields.

Re: Unable to use results of Streamstats

Jason — Thu, 07 Oct 2010 04:52:35 GMT

Turns out it was weird because I was running stuff on a summary index that had been populated by sistats.

I thought you had to populate a summary index with sistats, but it turns out that's only if you plan to do the exact stats query when looking at the summary index. My workaround was to use the fields Duration, fields.. to kick out some prsrvd_* fields that were messing with the functionality of stats.