topic Re: Importing CSV in Getting Data In

Importing CSV

edgarsilva01 — Fri, 07 Feb 2020 17:30:39 GMT

Hi everyone,

I am ingesting csv files that contain information about views of certain web pages,
These files are updated once a day.

I need that when the file is updated take only the new events.

Example: If the first file ends on day 20, for the second time it is updated I am no longer interested in seeing old events
I just want splunk to take the values from day 21.

Will they have any idea how I can do it?

Thank you

Re: Importing CSV

richgalloway — Fri, 07 Feb 2020 18:16:15 GMT

Is the entire file re-written every day or are the latest results appended to the file?
What are the inputs.conf settings for the file?

Re: Importing CSV

codebuilder — Fri, 07 Feb 2020 18:46:19 GMT

Configure your forwarder to use "batch" instead of "monitor", and set "move_policy =sinkhole".

This will cause the forwarder to delete the file after it's been ingested. Once your process creates a new file, the forwarder will pick it up and you'll get only the new results.

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf#BATCH_.28.22Upload_a_file.22_in_Splunk_Web.29:

Ignore the "Upload a File" portion in the documentation. Not sure why that is there, it's misleading.