one time upload, correct way help

[lancealotx]

ok, after 3 re-installs due to licensing, etc. I now have a clean install. I have 36G of old logs waiting, but also have the hourly logs going into a folder that splunk is monitoring. It comes to around 350M daily, so I am nice and under the 500 daily, but still would like to get that old stuff in and not run into the licence pool errors, search lock-outs, etc.

The logs are in a separate index, have a custom regex script and I am pleased with what I finally have, but don't want to say, ok take the old and break something. The licence docs aren't helpful, so I am just looking for a simple, this is the best way to get those over. Since I have 100M to spare, I could daily copy over a little a time but that would take forever.

So if I copy that 36G in, will it flag it, then break everything?

Tnx (sorry such a basic question but I need some reports and 3 day's with licence errors, etc is enough)

[NewMilenium]

Just to get it very clearly; does it mean that processing 501M in a day is exactly the same to license violation than processing 50G ?

So, it means that lancealotx in his case, and me in mine, should start processing all the data at midnight of a day and hope it processes it fast enough to eat all the 36G (in his case) in the same day, so that only 1 license violation occurs?

If that's not how it works, please explain, I will then need help on this.

Thanks for any answer!

[fbl_cit]

This is exactly how it works.

Doesn't matter if you put in 501M or 50P if your machine can index it in one day -> One violation.

[jbsplunk · Accepted solution]

I am not sure what you mean by 'flag' it. Splunk is going to eat whatever you tell it to eat, irrespective of the amount of data being indexed. As you noted, it results in license violations and search lock outs. If you're an enterprise customer, support can issue you a license reset. You're allowed 5 violations within a 30 day rolling window, presuming your box is beefy enough, you should be able to get the 36GB of historical data indexed before your sixth violation.

If you absolutely need to ensure you don't hit any violations, the only way I know of to do that would be to copy the data in 100MB at a time.

[jbsplunk]

It depends on how quickly your systems resources can process data. If you do incur more than 3 violations in a 30 day window(since you're using the free product), search is going to be disabled for 30 days. If you can process all of the data more quickly, you'll have less days when you're in violation and it may not be as big of a deal.

[lancealotx]

Tnx, I am a little confused regarding the violation and why I would need a beefy box.

I could take all of May for example which is around 2G. Splunk would eat it,and come back and say Hey, thats over 500M (violation 1). If I did nothing but left it like that, the next day the violation would be gone I would assume (maybe in the permanent until the 30 days is up) but that is just the scenario I was thinking.

The free is only 3 violations not 5, so if it took > 3 day's to process that would explain as well. Can I see start/complete time on indexing as I could get a basline from that.