Splunk Search

Kicking off lookup at index time?

Motivator

Is it possible to kick off a scripted lookup (to query a database) at index time?

Use case: client has a database that maps the current user ID assigned to a particular IP address. Older mappings are overwritten. For a sourcetype of data coming into Splunk containing IPs, enrich the data with the user ID assigned at that particular point in time. (Because of the current nature of the database, a search time lookup would bring back current data, not data accurate to the event time.)

The client knows it would cause a lot of traffic to their database server, and is considering not doing it, but is interested to know if it can be done.

0 Karma
1 Solution

Splunk Employee
Splunk Employee

No, lookups are search time only. If your lookup mapping changes over time, your best bet is a time based lookup where you add to the lookup table and each row has the timestamp that it takes effect.

View solution in original post

Explorer

You could populate a summary index with the data from the lookup at the time of the event was created using a real time search with output to a summary index. Then do your reporting from the summary index. If the data in the lookup changes less frequently real time may not be required. For instance if the lookup table gets updated onece per day you could use a scheduled search to build the summary prior to the lookup table being updated. Goal is the populate the summary index before the event orphans from the lookup table.

Splunk Employee
Splunk Employee

No, lookups are search time only. If your lookup mapping changes over time, your best bet is a time based lookup where you add to the lookup table and each row has the timestamp that it takes effect.

View solution in original post

State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!