Getting Data In

How to update a data input file without uploading the entire file again?

mcohen13
Loves-to-Learn

I have a data input that upload a file on my Splunk server with TSV format and I want to add fields to my index by adding them to the file. How can I do that without Splunk uploading the entire file? I have a limited license of 1GB per day and the file is 5GB.

0 Karma
1 Solution

xpac
SplunkTrust
SplunkTrust

Hey,

you can either upload only the new data, by adding it to a new file, and then upload that, or use the Splunk way - put it in a certain location, and use Splunk file monitors to take care of it. You can find more on this here:
Monitor files and directories

Hope that helps - if it does I'd be happy if you would upvote/accept this answer, so others could profit from it. 🙂

View solution in original post

0 Karma

xpac
SplunkTrust
SplunkTrust

Hey,

you can either upload only the new data, by adding it to a new file, and then upload that, or use the Splunk way - put it in a certain location, and use Splunk file monitors to take care of it. You can find more on this here:
Monitor files and directories

Hope that helps - if it does I'd be happy if you would upvote/accept this answer, so others could profit from it. 🙂

0 Karma

FrankVl
Ultra Champion

Do you want to add those additional fields only to new events that are added in the future, or do you also want to add those additonal fields to already indexed events?

In the first case, you could create a new file, with the desired fields, make necessary changes to Splunk config to handle those new fields if necessary, and then start adding events to that new file, leaving the old file as is.

If you also want to add fields to already indexed data, it will be a bit more difficult to do that without re-indexing the old data with additional fields.

0 Karma

mcohen13
Loves-to-Learn

it's the first case
so i need to:

  1. create new file that will have the old fields + new fields needed - prefer to not separate to two files
  2. add new data input from that file to the index
  3. stop writing to old file - to avoid duplicates

right?

0 Karma

FrankVl
Ultra Champion

Indeed, that should do the trick 🙂

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...