Splunk Search

Does outputlookup append or overwrite?

hulahoop
Splunk Employee
Splunk Employee

Does the outputlookup command overwrite or append to the existing specified lookup file? The documentation does not clarify: http://www.splunk.com/base/Documentation/latest/SearchReference/Outputlookup.

Tags (2)
1 Solution

gkanapathy
Splunk Employee
Splunk Employee

It will overwrite. If you want to append, you should first do an ... | inputlookup append=true myoldfile, and then probably some kind of dedup depending on the specifics of the lookup, then the outputlookup myoldfile, e.g.,

stats count by host,hostip | fields - count | inputlookup append=true hostiplookup | dedup host | outputlookup hostiplookup

View solution in original post

inventsekar
Super Champion

i was searching for the similar issue and after reading this solution i assumed the current Splunk version also works similar to this solution. Then i was reading the docs and learnt the new options available, so i thought to update other readers who may face similar situation like myself.

 

Until Splunk 6.5, the scheduled reports never had "Write to a CSV lookup file" option. Ref:

https://docs.splunk.com/Documentation/Splunk/6.5.0/Report/Schedulereports

From Splunk 6.6, the scheduled reports are having a "Write to a CSV lookup file" option. Ref: 

https://docs.splunk.com/Documentation/Splunk/6.6.0/Report/Schedulereports#Define_a_Write_to_a_CSV_Lo...

so, if we are using Splunk 6.6 or newer versions, then, simply with scheduled reports, we can "append" the lookup files. thanks. 

 

(PS - i have given around 500+ karma points so far, received badge for that, if an answer helped you, a karma point would be nice!. we all should start "Learn, Give Back, Have Fun")

spunk311z
Path Finder

in my use cases it is better (safer) to export the results as csv (by hand, via the splunk results gui, top right download button) and then use the AWESOME Lookup editor splunk app to manually copy / paste the data i want to append. (i open the csv in excel to copy fields)

granted this is not automated, but it keeps me from making errors or accidentally overwriting prior data in the lookup.

app: https://splunkbase.splunk.com/app/1724/

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

It will overwrite. If you want to append, you should first do an ... | inputlookup append=true myoldfile, and then probably some kind of dedup depending on the specifics of the lookup, then the outputlookup myoldfile, e.g.,

stats count by host,hostip | fields - count | inputlookup append=true hostiplookup | dedup host | outputlookup hostiplookup

View solution in original post

inventsekar
Super Champion

Are you guys sure of this method with tstats? 

| tstats count where (index=test* earliest="2/1/2020:00:00:00" latest="3/1/2020:00:00:00") BY _time span=1d 
| inputlookup append=true testlookup.cvs
| outputlookup testlookup.csv
0 Karma

bhawkins1
Communicator

In splunk 6.x the above did not work until I change | inputlookup x to append [| inputlookup x]. To clarify, this is useful for cases where you want to append data to the csv file without making duplicate "keys". Without the extra dedup, splunk will basically just open the file in append mode ( 'a' ) or write mode ( 'w' ).

dadi
Path Finder

I'm not sure if you are aware of this issue(splunk 5), but when I've outputlookup with append=true, I wasn't managed to write more than 1198 new records. This solution make the append=true to be unnecessary and work around this bug. Thanks!

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

technically i guess this prepends, not appends, but that's more probably what you want anyway, especially if you're constructing a time-based lookup.

0 Karma

jwestberg
Splunk Employee
Splunk Employee

It will overwrite.

Because of this, the |outputlookup command is well suited to being used in scheduled saved searches, keeping a lookup table up to date with each run.

Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!