Getting Data In

How to add a field to an event to verify when it has been output to a csv.

awedmondson
Explorer

Is it possible to add a field to an event to verify when it has been output to a csv? Once I have completed a search, and outputted the results to a CSV file. This is to verify that it has been sent, and not sent again if the search is run again. Thanks

Tags (2)
0 Karma
1 Solution

woodcock
Esteemed Legend

If I understand you correctly, you would like to have a single scheduled search that works like this: If there already is a CSV file, then do nothing; otherwise, create the file. If so, try something like this:

| inputcsv YourFileName.csv | stats count AS doesFileExistAlready | addinfo | eval earliestMaybe=if((doesFileExistAlready==0), info_min_time, now()) | map search="search earliest=$earliestMaybe$ latest=$info_max_time$ YOUR SEARCH HERE | outputcsv YourFileName.csv"

If the file already exists (and has at least 1 event/row), the search will generate an error and abort, leaving the file as-is; if it does not exist, it will get created.

View solution in original post

0 Karma

woodcock
Esteemed Legend

If I understand you correctly, you would like to have a single scheduled search that works like this: If there already is a CSV file, then do nothing; otherwise, create the file. If so, try something like this:

| inputcsv YourFileName.csv | stats count AS doesFileExistAlready | addinfo | eval earliestMaybe=if((doesFileExistAlready==0), info_min_time, now()) | map search="search earliest=$earliestMaybe$ latest=$info_max_time$ YOUR SEARCH HERE | outputcsv YourFileName.csv"

If the file already exists (and has at least 1 event/row), the search will generate an error and abort, leaving the file as-is; if it does not exist, it will get created.

0 Karma

woodcock
Esteemed Legend

Given this clarification:

The scheduled search runs, and a time stamped output csv file is created (DMY-HMS.csv). Say for example that new events arrive into Splunk , and the search runs again, I only want new events sent to a new csv file, I don't want the events included that were csv'ed earlier. 

Then like this:

Your Search Here NOT [| inputcsv YourFileName.csv] | Your Stuff Here | outputcsv YourFileName.csv
0 Karma

awedmondson
Explorer

Interesting idea, Thanks. Do you know if it is possible to actually add a field to the events called "sent" with a timestamp of when the output csv was created? This would allow to only create the output csv based events that have not been sent already.

0 Karma

woodcock
Esteemed Legend

Sure, just add this somewhere:

... | eval sent=now() ...
0 Karma

awedmondson
Explorer

Thank you for your help, much appreciated.

0 Karma

awedmondson
Explorer

Hi, Thank you for your reply. I didn't explain clearly. The scheduled search runs, and a time stamped output csv file is created (DMY-HMS.csv). Say for example that new events arrive into Splunk , and the search runs again, I only want new events sent to a new csv file, I don't want the events included that were csv'ed earlier.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...