Does 'lookup' have to be used before stats commands?

[Lowell]

Is there some reason why using the lookup command doesn't seem to be working properly after stats?

The search I'm trying to use, is like this:

sourcetype=access_combined | stats count by my_field | lookup my_lookup my_field OUTPUT out1

However out1 is never populated. If I move the lookup before the stats command (which now means I have to group by this value, and protect against null values), then the search works. Here is an example:

sourcetype=access_combined | lookup my_lookup my_field OUTPUT out1 | eval out1=coalesce(out1, "default") | stats count by my_field, out1

Is there some fundamental concept I'm missing here, or is this a bug?

Solution:

Thanks to jrodman for pointing me in the right direction on this one.

My problem is because my lookup is date effective, that is to say, I've defined time_field in my transforms.conf entry, like so:

[my_lookup]
filename = my_lookup_file.csv
time_field = my_time_field
time_format = %s

So because of my stats command dropped the _time field, splunk was unable to available to complete the lookup. Therefore, since I don't need to be strict about date-effective lookups in my search, I can use the following to get the results I'm looking for:

sourcetype=access_combined | stats count, min(_time) as _time by my_field | lookup my_lookup my_field OUTPUT out1

[jrodman · Accepted solution]

There's no fixed requirement for when lookup should be invoked. There are two possibilities here.

• One is that your lookup is keyed to some fields that aren't available post-stats.
• Another is that the lookup operator presumes some fields which aren't available post-stats.

Perhaps you should provide a bit more about how the lookup is defined?

[jrodman]

I'm a bit murky on exactly when we should complain about stuff and when we shouldn't. With all the things that can get defined by various parties, we can't complain about everything by default. However, in this case it sure seems like you're explicitly running the lookup command, and it should probably provide feedback. I'll file some sort of defect tomorrow, hopefully. It wouldn't hurt to report it to support as a "please pass this along".

[Lowell]

Thanks. I figured out my issue. I was doing a date-effective lookup and I stats was stripping out _time. I've updated the post. I am somewhat surprised that lookup didn't complain about missing fields like I've seen it do in other cases, but I suppose there is no single "right way" to handle the situation of a missing _time field, and it certainly makes sense why lookup failed to give me what I was looking for.