Getting Data In

growing file

splunk_gs
Explorer

I have a growing file on a remote location. Can i use the forwarder to monitor this file without having to reupload it everytime it grows? i know initially it will upload the main file but overtime when it grows i just want it to add the changes not the entire file. How do i go about setting this up?

Thanks

Tags (2)
0 Karma
1 Solution

sdaniels
Splunk Employee
Splunk Employee

Yes, you can install a forwarder on that remote server and set up file monitoring. Splunk will handle log rotation without re-indexing data it has already taken in and remember where it was for a given file so that it only takes in the new data. An exception would be archive files like .tar and .gz. Check the docs.

http://docs.splunk.com/Documentation/Splunk/latest/Data/Monitorfilesanddirectories

View solution in original post

0 Karma

sdaniels
Splunk Employee
Splunk Employee

Yes, you can install a forwarder on that remote server and set up file monitoring. Splunk will handle log rotation without re-indexing data it has already taken in and remember where it was for a given file so that it only takes in the new data. An exception would be archive files like .tar and .gz. Check the docs.

http://docs.splunk.com/Documentation/Splunk/latest/Data/Monitorfilesanddirectories

0 Karma

sdaniels
Splunk Employee
Splunk Employee

You'll want to set a sourcetype and index. There are other settings that you can look at with details in the inputs.conf spec. Splunk will remember where it is in a file to continuously monitor it and not duplicate data.

http://docs.splunk.com/Documentation/Splunk/latest/admin/Inputsconf

0 Karma

splunk_gs
Explorer

are there any specific options i would need to select to do this? like tail forwarding?

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Accelerate Splunk App Development with AI

  Learn how to automate the repetitive parts of your build cycle.   Watch On-Demand   Replay the Accelerate ...

Splunk Agent Observability: Measuring AI Value with Splunk Tokenomics on AWS

The AI landscape is moving faster than ever, and our latest 2026 State of Observability report provides the ...

Enterprise Security Watch Party: What’s New in 8.8 - Recap

Splunk Community HQ hosted a Splunk Enterprise Security Watch Party: What’s New In ES 8.8 last Thursday. I had ...