How to update a data input file without uploading the entire file again?

[mcohen13]

I have a data input that upload a file on my Splunk server with TSV format and I want to add fields to my index by adding them to the file. How can I do that without Splunk uploading the entire file? I have a limited license of 1GB per day and the file is 5GB.

[xpac · Accepted solution]

Hey,

you can either upload only the new data, by adding it to a new file, and then upload that, or use the Splunk way - put it in a certain location, and use Splunk file monitors to take care of it. You can find more on this here:
Monitor files and directories

Hope that helps - if it does I'd be happy if you would upvote/accept this answer, so others could profit from it. 🙂

[FrankVl]

Do you want to add those additional fields only to new events that are added in the future, or do you also want to add those additonal fields to already indexed events?

In the first case, you could create a new file, with the desired fields, make necessary changes to Splunk config to handle those new fields if necessary, and then start adding events to that new file, leaving the old file as is.

If you also want to add fields to already indexed data, it will be a bit more difficult to do that without re-indexing the old data with additional fields.

[mcohen13]

it's the first case
so i need to:

1. create new file that will have the old fields + new fields needed - prefer to not separate to two files
2. add new data input from that file to the index
3. stop writing to old file - to avoid duplicates

right?

[FrankVl]

Indeed, that should do the trick 🙂