topic Re: Can't add new data?! in Splunk Search

Can't add new data?!

nickcode — Mon, 06 May 2013 07:40:37 GMT

Hi All! I'm using Enterprise Trial version of Splunk which allows indexing 500MB data a day. I have once specified a directory which contains about totaly 500MB nginx log files for Splunk to index and search. Later I found no more data can be imported any further, that's not strange I thought. So I tried to use "sourcetype="xxx" | delete" command to delete the index for later data import. But it doesn't work, other data still can not be imported or indexed. 😞
Then several days later, I cleaned the index database that stores the 500MB data using CLI "splunk clean ..." command, BUT, BUT, I STILL CAN'T IMPORT MORE DATA...TAT
Any one would be so kindly to help me?

Re: Can't add new data?!

kristian_kolb — Mon, 06 May 2013 08:00:21 GMT

You have probably hit the threshold (500 MB / day) too many times. For the Free version, the limit is set to 3 violations per 30 days (rolling) and for Enterprise it is set to 5 violations per 30 days (rolling). I think Enterprise Trial has the same setting as Enterprise.

So if you index more than 500 MB / day, you'll get a violation, and with enough violations, you'll be locked out (at least for searching 'your' data). Splunk licensing does not care about how much data you have stored, so deleting already indexed data will have no effect.

The only thing that will let you search your data is to;

a) wait until there are less than violations in the last 30 days.
b) request a reset license from support and apply it (doubtful if you'll get that for the free/trial versions)
c) purchase an Enterprise license (presumably of a larger size).

Please read more here;

http://docs.splunk.com/Documentation/Splunk/5.0.2/Admin/HowSplunklicensingworks

/Kristian

Re: Can't add new data?!

nickcode — Mon, 06 May 2013 08:59:37 GMT

Very thankful for your reply! But I haven't found any violation or warning message in 'Manager->Licensing' at all(Maybe my data is just no more than 500 MB). Just now I found after I cleaned up the index db, and added new data to Splunk, Splunk did index it(I found new TXIDX files are created in index db), but no data are showed in search pannel.

Re: Can't add new data?!

kristian_kolb — Mon, 06 May 2013 09:44:42 GMT

Hm, ok.

a) Have you checked that the data is in the index (manager -> indexes). You should see an eventcount and a date per index, which should give you an indication of whether your events have been indexed or not.

b) Have you checked that Splunk can understand your timestamps correctly. If not, a lot of strange things can happen, e.g. events being indexed in the wrong day/month/year etc. Search for 'All Time' to see if you can find them.

c) Do you have access rights to your index? Manager -> Access Rights -> Roles -> . At the bottom of the page you'll see settings for this.

Re: Can't add new data?!

nickcode — Tue, 07 May 2013 09:56:03 GMT

a) I created a new index and added new data to it, the eventcount keeps 0 in manager->indexs page.
b) My data is standard nginx log files, I think it should be ok.
c) I logged in as admin and have the access right.

Re: Can't add new data?!

kristian_kolb — Tue, 07 May 2013 10:41:17 GMT

If you are trying to import the SAME data that you have already indexed once, and now deleted, you might have to clean the fishbucket index as well, as that's where splunk keeps track of the files it has already read.

Some interesting reading;
http://wiki.splunk.com/Community:HowSplunkReadsInputFiles
http://blogs.splunk.com/2008/08/14/what-is-this-fishbucket-thing/
http://splunk-base.splunk.com/answers/54070/btprobe-and-re-indexing-data

Re: Can't add new data?!

nickcode — Wed, 08 May 2013 06:26:33 GMT

OK, thanks for your suggestion! 🙂