Archive
Highlighted

Splunk 6 Can it index old rotated log files?

Explorer

Hi.

I am testing out splunk and splunk storm for our cluster deployment. In our pilot, I have set up a single host quite well and am receiving data on splunk storm for now.

However, we have old log files that go back almost 4 years. Is it possible to add those files to splunk, by using the same sourcetypes as the ones I have designated for the live data? We have a lot of custom applications logging to /var/log and we have given them all custom types.

Thanks in advance.

0 Karma
Highlighted

Re: Splunk 6 Can it index old rotated log files?

Explorer

To clarify, I can manually upload old files. But can Splunk automatically load old logrotated files?

0 Karma
Highlighted

Re: Splunk 6 Can it index old rotated log files?

SplunkTrust
SplunkTrust

Sure.

Point a Splunk forwarder at the old files like you do with new files, and it'll eat them all up. Two things to keep in mind: Try to do it all in one day, so you only cause one license violation... and, if you have data older than five years, you need to set your sourcetypes to allow such old data, by default Splunk drops data older than 2000 days.

As for Splunk Storm, I'm not quite sure if you need to do additional changes or if there are other restrictions. The above approach is based on Splunk Enterprise.

View solution in original post

Highlighted

Re: Splunk 6 Can it index old rotated log files?

Explorer

License violation???

And yes, Enterprise helps because I will eventually be deploying splunk enterprise at $WORK. Thanks.

0 Karma
Highlighted

Re: Splunk 6 Can it index old rotated log files?

SplunkTrust
SplunkTrust

Yeah, see http://docs.splunk.com/Documentation/Splunk/6.0.1/Admin/HowSplunklicensingworks for reference.

When bulk-backfilling old data it's best to do it all in one go. That's Splunk Enterprise licensing of course, not sure how Splunk Storm works. Isn't that space-based? Maybe also with restrictions on the retention time?

0 Karma
Highlighted

Re: Splunk 6 Can it index old rotated log files?

Explorer

Yeah. Storm is space based. Thanks for the Licensing tip off.

0 Karma