topic Re: Does outputlookup append or overwrite? in Splunk Search

Does outputlookup append or overwrite?

hulahoop — Fri, 18 Jun 2010 03:19:43 GMT

Does the outputlookup command overwrite or append to the existing specified lookup file? The documentation does not clarify: http://www.splunk.com/base/Documentation/latest/SearchReference/Outputlookup.

Re: Does outputlookup append or overwrite?

jwestberg — Fri, 18 Jun 2010 03:36:30 GMT

It will overwrite.

Because of this, the |outputlookup command is well suited to being used in scheduled saved searches, keeping a lookup table up to date with each run.

Re: Does outputlookup append or overwrite?

gkanapathy — Fri, 18 Jun 2010 04:07:10 GMT

It will overwrite. If you want to append, you should first do an ... | inputlookup append=true myoldfile, and then probably some kind of dedup depending on the specifics of the lookup, then the outputlookup myoldfile, e.g.,

stats count by host,hostip | fields - count | inputlookup append=true hostiplookup | dedup host | outputlookup hostiplookup

Re: Does outputlookup append or overwrite?

gkanapathy — Fri, 18 Jun 2010 04:08:30 GMT

technically i guess this prepends, not appends, but that's more probably what you want anyway, especially if you're constructing a time-based lookup.

Re: Does outputlookup append or overwrite?

dadi — Sun, 30 Dec 2012 21:22:17 GMT

I'm not sure if you are aware of this issue(splunk 5), but when I've outputlookup with append=true, I wasn't managed to write more than 1198 new records. This solution make the append=true to be unnecessary and work around this bug. Thanks!

Re: Does outputlookup append or overwrite?

bhawkins1 — Tue, 08 Nov 2016 16:13:01 GMT

In splunk 6.x the above did not work until I change | inputlookup x to append [| inputlookup x]. To clarify, this is useful for cases where you want to append data to the csv file without making duplicate "keys". Without the extra dedup, splunk will basically just open the file in append mode ( 'a' ) or write mode ( 'w' ).

Re: Does outputlookup append or overwrite?

spunk311z — Mon, 17 Feb 2020 15:35:57 GMT

in my use cases it is better (safer) to export the results as csv (by hand, via the splunk results gui, top right download button) and then use the AWESOME Lookup editor splunk app to manually copy / paste the data i want to append. (i open the csv in excel to copy fields)

granted this is not automated, but it keeps me from making errors or accidentally overwriting prior data in the lookup.

app: https://splunkbase.splunk.com/app/1724/

Re: Does outputlookup append or overwrite?

inventsekar — Thu, 08 Oct 2020 01:44:22 GMT

i was searching for the similar issue and after reading this solution i assumed the current Splunk version also works similar to this solution. Then i was reading the docs and learnt the new options available, so i thought to update other readers who may face similar situation like myself.

Until Splunk 6.5, the scheduled reports never had "Write to a CSV lookup file" option. Ref:

https://docs.splunk.com/Documentation/Splunk/6.5.0/Report/Schedulereports

From Splunk 6.6, the scheduled reports are having a "Write to a CSV lookup file" option. Ref:

https://docs.splunk.com/Documentation/Splunk/6.6.0/Report/Schedulereports#Define_a_Write_to_a_CSV_Lookup_File_action

so, if we are using Splunk 6.6 or newer versions, then, simply with scheduled reports, we can "append" the lookup files. thanks.

(PS - i have given around 500+ karma points so far, received badge for that, if an answer helped you, a karma point would be nice!. we all should start "Learn, Give Back, Have Fun")

Re: Does outputlookup append or overwrite?

inventsekar — Mon, 23 Nov 2020 07:34:35 GMT

Are you guys sure of this method with tstats?

| tstats count where (index=test* earliest="2/1/2020:00:00:00" latest="3/1/2020:00:00:00") BY _time span=1d 
| inputlookup append=true testlookup.cvs 
| outputlookup testlookup.csv