Generate lookup tables from searches with guarantee of unique entries

[mzorzi]

what is the most efficient way to achieve this.

I run search #1 that populates the lookup table file with data.

Then search #2 will search for values a specific field in the lookup table and only reports events that are NOT a match for anything already in the lookup table.

Finally I append the results of the second search to the same lookup table. So in the end my lookup file will now have 1 list of unique entries combined from 2 different searches.

Is that possible? Otherwise , what would be the most efficient way?

[kristian_kolb]

Well, starting from this:

http://blogs.splunk.com/2011/01/11/maintaining-state-of-the-union/

You could probably achieve something similar to your wishes. I have created a search (no access to it at the moment - could post later) which - in pseudo search language - works like this for maintaining a list of userid's;

sourcetype=xxx userid=* NOT [search |inputlookup userid_file | fields + userid] | fields + userid | outputlookup append=t userid_file

OR this (don't remember)

sourcetype=xxx userid=* | fields + userid | inputlookup append=t userid_file | dedup userid | outputlookup userid_file

---

EDIT: several small fixes.

---

Good luck
/Kristian

[kml_uvce]

please explain it with some data..