Events field is blank for first time query

[foxwu]

Hi,

I got one weird problem that when I run query in splunk, there're events found, but the Event log field is always blank.

However, the problem will be fixed by below steps:

1. in the Splunk search result, go to All Fields at left rail
2. change Coverage: 1% more to All fields
3. click Deselect All
4. click Select All Within Filter

Then the problem is fixed, I can see the Event logs. Even I change from All fields back to Coverage: 1% more, I can still see Events logs.

But after I close the browser tab and go to Splunk and search again, the problem still exists. So the problem is, why Coverage: 1% more will have problem to me for the first time query?

Anyone has idea about this? Thanks.

[yuanliu]

This is a great description of your reproduction method.  More clarifications are needed, though.  This "Event log field", it must be a custom field specific to the data source you are looking at.  Is this correct? (Splunk can be used for all kinds of data.  "Event", or "Event log" is not an inherent field name.)  I also assume that the behavior is observed in "Smart Mode" or "Verbose Mode".

After trying your sequence with index=_internal, I have some educated guess what you are trying to solve.  Rest assured that "Event" or "Event log" field in your data is not "always blank".  But this specific field is indeed populated in fewer than 1% of total events in your search period.  In other words, the field always exists in events that have populated it.  But Splunk won't list the field name in the left-hand side, either "Selected Fields" or "Interesting Fields".

This is normal behavior that saves analysts from being overwhelmed by too many fields.  I suggest the following exercise to familiarize yourself with Splunk's search UI.  Instead of "Select All Within Filter", just scroll to the field of interest, say "Event".  I predict that you will see "Event Coverage" < 1%.  Click the corresponding checkbox, then return to search window.  Now you will see "Event" under "Selected Fields".

Indeed Splunk doesn't save this in user preference.  You do not even need to restart browser.  Just click "! Smart Mode", change to "Fast Mode", then change back to "Smart Mode".  Your selection is gone.

So, what to do?  As said above, nothing is lost.  If you search for events in which this field is populated, those events will still show.  Try this:

<your source specs> Event=*

Now Splunk will automatically show "Event" field under "Interesting Fields". (Because when you do this, this field has 100% event coverage.)

If you perform other calculations with this field, even as its event coverage is <1%, Splunk will still get it.  Try this:

<your source specs>
| stats count(Event) as EventCount count

It will still display how many events has Event field populated; you can see how this population compares with your event space.  You can also try this exercise:

<your source specs>
| eval does_Event_exist = "yes, yes, yes!! " . Event
| stats values(Event) as EventValue values(does_Event_exist) as does_Event_exist count(does_Event_exist) as calculatedCount count(Event) as EventCount

You should get calculatedCount and EventCount equal to each other even though you are counting different fields; field EventValue should show you all values in field Event during the search period; and field does_Event_exist should show you "yes, yes, yes!! " prepended to each value of EventValue.

Hope this helps.