topic Re: How to get Splunk to recognise new data added to a CSV File. in Getting Data In

How to get Splunk to recognise new data added to a CSV File.

pjAstroMan — Fri, 30 Apr 2021 17:04:06 GMT

Hi there, I'm experiminting with a single machine/single instance of Splunk Enterprise, using a set of static data in CSV format. I successfully ingested the initial data from the CSV file, however when I add subsequent records to the CSV file, Splunk seems unaware of the new data. How can I set things up so that Splunk will recognise the data, and update the dashboards I have created for monitoring the data dynamically?

Kind Regards

Paul J.

Re: How to get Splunk to recognise new data added to a CSV File.

burwell — Fri, 30 Apr 2021 23:57:17 GMT

Hi. How are you adding more data to the csv?

If you added a file using the lookup files -> new and then you are editing the actual file on disk, that likely won't work. I highly recommend using this app: https://splunkbase.splunk.com/app/1724/

Re: How to get Splunk to recognise new data added to a CSV File.

s2_splunk — Sat, 01 May 2021 01:55:06 GMT

Paul,

how did you ingest the initial file? Did you upload it to your instance via the UI or did you setup a file monitor?

You need to make sure you have an inputs.conf file configured with a

[monitor://....]

stanza that points to your directory/file. If you do that, and that input is enabled, changes to the file should be picked up and indexed.

HTH

Re: How to get Splunk to recognise new data added to a CSV File.

pjAstroMan — Mon, 03 May 2021 09:11:54 GMT

Hi there thank you for your feedback.

The data was loaded via the GUI.

As a newby to Splunk, I am experimenting with its capability to ingest historic data. We are looking to migrate from OSIOsft PI to an alternative data historian and Splunk is one of the products we are looking at. A key requirement for the new historian is the ability to ingest a large amount of historic data (about 10 years worth). To prove this functionality, I have 3 months worth of data across three CSV files and I need to load this into Splunk. Obviously I am not familiar with the config changes that you mention, could you possibly provide a sample entry to the config file.

Kind Regards

Paul.

Re: How to get Splunk to recognise new data added to a CSV File.

pjAstroMan — Mon, 03 May 2021 12:30:29 GMT

Is there no way to create the necessary configuration vua the GUI? I have had a look and tried using the Data Input option, but specified 'Continuously Monitor' instead 'Index Once' as this implied the functionality I was looking for, unfortunately this did not work.

Re: How to get Splunk to recognise new data added to a CSV File.

s2_splunk — Mon, 03 May 2021 15:59:41 GMT

As a user new to Splunk, I suggest you review the video on getting data into Splunk Enterprise for the general approach to monitor a directory using the UI.

There are a myriad of ways to go about getting data into Splunk. For historic data, you would normally not choose a monitor, since you don't expect historic files to be updated. Since the UI imposes a 500MB limit on uploaded files, you can use the CLI and the oneshot or spool commands for a one-time ingest.

If you want to explore configuring file monitoring, please review this part of our documentation, which contains example settings.

One note regarding old data: There are some settings that control how far back of a timestamp Splunk will consider valid. Specifically, MAX_DAYS_AGO is used to discard data that is older than a set amount of time. This defaults to 2000 days, so you may have to adjust this for data older than ~5.5 years.

Re: How to get Splunk to recognise new data added to a CSV File.

pjAstroMan — Mon, 03 May 2021 16:09:54 GMT

Hi there, I have read through the documentation you have suggested. As I have no experience of working under the 'Splunk hood' so to speak I am reluctant to start manually editing files. This strikes me as functionality that should be availabel via a GUI. I guess I am a bit surprised that I would need to resort to manually editing a file in order to get Splunk to recognise new data.

Paul.

Re: How to get Splunk to recognise new data added to a CSV File.

s2_splunk — Mon, 03 May 2021 16:21:05 GMT

You don't need to resort to the command line to do what you want to do; apologies if it sounded like that's a must. Splunk's primary purpose is to ingest data on an ongoing basis, for which the monitor approach is exactly the right thing to do.

Your use case is about loading large amounts of historical data and there are some product features available to support one-time ingest that are not exposed via the UI.

I would suggest you try to configure a monitor input on the directory containing your files. Follow the UI process as shown in the video I linked and instead of selecting a file, select a directory on your Splunk server. Once you have done that, copying/moving files into that directory should cause them to be indexed into Splunk.

If that doesn't work for you, please respond and we can troubleshoot from there.

Re: How to get Splunk to recognise new data added to a CSV File.

pjAstroMan — Mon, 03 May 2021 16:34:08 GMT

Yep, just stumbled across the functionality you suggested, thank you very much for your assistance, much appreciated.

Paul.