topic Re: How to add a field to an event to verify when it has been output to a csv. in Getting Data In

How to add a field to an event to verify when it has been output to a csv.

awedmondson — Wed, 11 Nov 2015 15:11:31 GMT

Is it possible to add a field to an event to verify when it has been output to a csv? Once I have completed a search, and outputted the results to a CSV file. This is to verify that it has been sent, and not sent again if the search is run again. Thanks

Re: How to add a field to an event to verify when it has been output to a csv.

woodcock — Wed, 11 Nov 2015 16:19:22 GMT

If I understand you correctly, you would like to have a single scheduled search that works like this: If there already is a CSV file, then do nothing; otherwise, create the file. If so, try something like this:

| inputcsv YourFileName.csv | stats count AS doesFileExistAlready | addinfo | eval earliestMaybe=if((doesFileExistAlready==0), info_min_time, now()) | map search="search earliest=$earliestMaybe$ latest=$info_max_time$ YOUR SEARCH HERE | outputcsv YourFileName.csv"

If the file already exists (and has at least 1 event/row), the search will generate an error and abort, leaving the file as-is; if it does not exist, it will get created.

Re: How to add a field to an event to verify when it has been output to a csv.

awedmondson — Sun, 15 Nov 2015 17:05:30 GMT

Hi, Thank you for your reply. I didn't explain clearly. The scheduled search runs, and a time stamped output csv file is created (DMY-HMS.csv). Say for example that new events arrive into Splunk , and the search runs again, I only want new events sent to a new csv file, I don't want the events included that were csv'ed earlier.

Re: How to add a field to an event to verify when it has been output to a csv.

woodcock — Sun, 15 Nov 2015 20:12:49 GMT

Given this clarification:

The scheduled search runs, and a time stamped output csv file is created (DMY-HMS.csv). Say for example that new events arrive into Splunk , and the search runs again, I only want new events sent to a new csv file, I don't want the events included that were csv'ed earlier.

Then like this:

Your Search Here NOT [| inputcsv YourFileName.csv] | Your Stuff Here | outputcsv YourFileName.csv

Re: How to add a field to an event to verify when it has been output to a csv.

awedmondson — Mon, 16 Nov 2015 08:59:04 GMT

Interesting idea, Thanks. Do you know if it is possible to actually add a field to the events called "sent" with a timestamp of when the output csv was created? This would allow to only create the output csv based events that have not been sent already.

Re: How to add a field to an event to verify when it has been output to a csv.

woodcock — Mon, 16 Nov 2015 21:58:44 GMT

Sure, just add this somewhere:

... | eval sent=now() ...

Re: How to add a field to an event to verify when it has been output to a csv.

awedmondson — Tue, 17 Nov 2015 22:29:55 GMT

Thank you for your help, much appreciated.