Splunk Search
Highlighted

Bucket search command discarding extra details?

Path Finder

We've got log events that read like the following:

Mar 14 12:26:38 mailsrv.example.com MM: [Jilter Processor 21 - 
  Async Jilter Worker 30 - 127.0.0.1:47850-o29CQWO2002696] INFO user.log  
  - mtaqid=o29CQWO2002696, engine=virscan-olympus, 
  from=<postmaster@example.com>, recipients=<pbmserv@example.com>, 
  relay=[92.54.189.138], size=40236, virus_name=W32/Mydoom.o@MM, 
  virus_state=infected, filename=text.pif

The goal is to write a search to a time-bucketed set of results (suitable for passing to a time chart, or for use in a summary index search) to count distinct mtaqid fields versus the logging host and the virus_name.

The following search returns zero results. Further experimentation shows that after the bucket command, virus_name is null.

eventtype=smimm_virus virus_state!="clean" 
| bin _time span=5m 
| stats distinct_count(qid) by _time, virus_name, host

Is there a description somewhere of what bucket keeps or throws away when grouping log lines? When preparing such a search for sistats or other summary index use, what is the best way to perform this bucketing, without losing this detail?

Tags (1)
Highlighted

Re: Bucket search command discarding extra details?

Motivator

Using sistats will include the correct field and summary information when attempting to insert into a summary index. It seems as though your search might not be the ideal search. I would try removing each distinct sort "by" to verify that there are existing values for virus_name.

0 Karma
Highlighted

Re: Bucket search command discarding extra details?

Path Finder

When I indicated that I had performed "further experimentation", I meant that I had discarded various group bys, removed bucket, etc.

Compare the results of:
eventtype=smimmvirus virusstate!=clean | table time, qid, virusstate, virus_name

with

eventtype=smimmvirus virusstate!=clean | bucket time span=5m | table _time, qid, virusstate, virus_name.

On 4.1.0 (Linux), the bucketed search has non-null virus names (for every row).

On 4.1.2 (Mac), the non-bucketed search has non-null virus names, but the bucketed search has a blank column there....

Regression issue?

0 Karma
Highlighted

Re: Bucket search command discarding extra details?

Path Finder

I should note that putting the | sistats before the | bucket seems to work even in 4.1.2.

0 Karma
Highlighted

Re: Bucket search command discarding extra details?

Legend

bucket basically should not be modifying any field except the one you are bucketing.

0 Karma
Highlighted

Re: Bucket search command discarding extra details?

Path Finder

It is, in 4.1.2 only, it seems. (More below.)

0 Karma
Highlighted

Re: Bucket search command discarding extra details?

Path Finder

I can't seem to reproduce this any longer. I'll write it off as a transient thing.

View solution in original post

0 Karma
Highlighted

Re: Bucket search command discarding extra details?

Legend

If you were relying on the Field Picker to tell you if a field was extracted, then starting with 4.1, it's not that reliable for this as it will automatically suppress some fields unless they're explicitly used in some other part of the search.

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.