Getting Data In

How to update a data input file without uploading the entire file again?

mcohen13
Loves-to-Learn

I have a data input that upload a file on my Splunk server with TSV format and I want to add fields to my index by adding them to the file. How can I do that without Splunk uploading the entire file? I have a limited license of 1GB per day and the file is 5GB.

0 Karma
1 Solution

xpac
SplunkTrust
SplunkTrust

Hey,

you can either upload only the new data, by adding it to a new file, and then upload that, or use the Splunk way - put it in a certain location, and use Splunk file monitors to take care of it. You can find more on this here:
Monitor files and directories

Hope that helps - if it does I'd be happy if you would upvote/accept this answer, so others could profit from it. 🙂

View solution in original post

0 Karma

xpac
SplunkTrust
SplunkTrust

Hey,

you can either upload only the new data, by adding it to a new file, and then upload that, or use the Splunk way - put it in a certain location, and use Splunk file monitors to take care of it. You can find more on this here:
Monitor files and directories

Hope that helps - if it does I'd be happy if you would upvote/accept this answer, so others could profit from it. 🙂

0 Karma

FrankVl
Ultra Champion

Do you want to add those additional fields only to new events that are added in the future, or do you also want to add those additonal fields to already indexed events?

In the first case, you could create a new file, with the desired fields, make necessary changes to Splunk config to handle those new fields if necessary, and then start adding events to that new file, leaving the old file as is.

If you also want to add fields to already indexed data, it will be a bit more difficult to do that without re-indexing the old data with additional fields.

0 Karma

mcohen13
Loves-to-Learn

it's the first case
so i need to:

  1. create new file that will have the old fields + new fields needed - prefer to not separate to two files
  2. add new data input from that file to the index
  3. stop writing to old file - to avoid duplicates

right?

0 Karma

FrankVl
Ultra Champion

Indeed, that should do the trick 🙂

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...