topic Re: Temporal Input lookups? in Splunk Search

Temporal Input lookups?

bread555 — Tue, 27 Nov 2012 19:03:48 GMT

We have been able to successfully use inputlookup with lookup files we have created. However, our lookup files have timestamps that are not being picked up by splunk despite annotating it in the transforms.conf. We are not currently using automatic lookups. Here is an example of our setup:

The below is our "lookup" csv file

test.csv
    date,interesting_data
    2012-11-09 09:32:34,things that occurred

and then this is in our transforms.conf

 transforms.conf
    [test_lookup_file]
    filename = test.csv
    time_field = date
    time_format = %Y-%m-%d %H:%M:%s

We can invoke the inputlookup and return all of the data,but its time is exactly when we did the search IE:

|inputlookup test_lookup_file |  _time

returns nothing

|inputlookup test_lookup_file | table date interesting_data

returns all of the data in the lookup file. My assumption is that I am either A: not understanding exactly how temporal lookup files work or B: have a really stupid mistake somewhere. Any help would be greatly appreciated. Please note, that my data has been sanitized an reviewed, so any typos that exist above are my own fault and not what is currently in the system

Re: Temporal Input lookups?

lguinn2 — Tue, 27 Nov 2012 19:19:38 GMT

The way you have set it up, the time is not bounded, so everything matches. You probably need to add max_offset_secs to your transforms.conf stanza

max_offset_secs = <integer>
* For temporal lookups, this is the maximum time (in seconds) that the event timestamp can be 
  later than the lookup entry time for a match to occur.
* Default is 2000000000 (no maximum, effectively).

I copied the above from the Admin manual on transforms.conf.

Re: Temporal Input lookups?

bread555 — Tue, 27 Nov 2012 20:48:27 GMT

So, I had tried this before, and I think because I am just trying to pull back the results in splunk raw rather than correlate these offset settings wouldn't apply.

One of my goals would be to do graphs/statistics on the lookup file WITHOUT injesting it or correlating it. The reason being, that lookup file will be changing frequently (and may even have data that is deleted). Thus, I don't want to injest it.