How to add a field to an event to verify when it has been output to a csv.

[awedmondson]

Is it possible to add a field to an event to verify when it has been output to a csv? Once I have completed a search, and outputted the results to a CSV file. This is to verify that it has been sent, and not sent again if the search is run again. Thanks

[woodcock · Accepted solution]

If I understand you correctly, you would like to have a single scheduled search that works like this: If there already is a CSV file, then do nothing; otherwise, create the file. If so, try something like this:

| inputcsv YourFileName.csv | stats count AS doesFileExistAlready | addinfo | eval earliestMaybe=if((doesFileExistAlready==0), info_min_time, now()) | map search="search earliest=$earliestMaybe$ latest=$info_max_time$ YOUR SEARCH HERE | outputcsv YourFileName.csv"

If the file already exists (and has at least 1 event/row), the search will generate an error and abort, leaving the file as-is; if it does not exist, it will get created.

[woodcock]

Given this clarification:

The scheduled search runs, and a time stamped output csv file is created (DMY-HMS.csv). Say for example that new events arrive into Splunk , and the search runs again, I only want new events sent to a new csv file, I don't want the events included that were csv'ed earlier. 

Then like this:

Your Search Here NOT [| inputcsv YourFileName.csv] | Your Stuff Here | outputcsv YourFileName.csv

[awedmondson]

Interesting idea, Thanks. Do you know if it is possible to actually add a field to the events called "sent" with a timestamp of when the output csv was created? This would allow to only create the output csv based events that have not been sent already.

[woodcock]

Sure, just add this somewhere:

... | eval sent=now() ...

[awedmondson]

Thank you for your help, much appreciated.

[awedmondson]

Hi, Thank you for your reply. I didn't explain clearly. The scheduled search runs, and a time stamped output csv file is created (DMY-HMS.csv). Say for example that new events arrive into Splunk , and the search runs again, I only want new events sent to a new csv file, I don't want the events included that were csv'ed earlier.