topic Temporarily stop indexing in Getting Data In

Temporarily stop indexing

steveirogers — Wed, 05 May 2010 03:58:05 GMT

I would like to temporarily stop Splunk indexing for a couple hours while my QA group runs some volume/performance test on their application. From a practical perspective they are unable to disable logging in the application. I tried mmodifiying the inputs.conf file to "blacklist" those specific log files and that appeared to work. However, when the performance test finished and I removed the "blacklist" SPLUNK automatically went back and indexed all the log data that had not been indexed.
Is there anyway to override this default behavior?

Re: Temporarily stop indexing

the_wolverine — Wed, 05 May 2010 04:30:39 GMT

You could configure Splunk to route the test data to nullQueue. Once you want it to index again. disable routing to nullQueue.

Example:

props.conf:

[source::*]
TRANSFORMS-nullhost = nullhost

transforms.conf:

[nullhost]
REGEX=.
DEST_KEY = queue
queue = nullQueue

If your QA group is running regular tests, you should seriously consider a separate QA instance so that your Production instance is not affected.

Re: Temporarily stop indexing

gkanapathy — Wed, 05 May 2010 05:40:42 GMT

go go wolverine!

Re: Temporarily stop indexing

Lowell — Thu, 06 May 2010 06:18:46 GMT

If your test are on a regularly scheduled interval. You could send your events to the nullQueue using a regular expression that only matches for certain hours, or better yet if there is something in the event's themselves to indicate that it's test data, then you could matching on that.

Re: Temporarily stop indexing

Lowell — Thu, 06 May 2010 06:27:51 GMT

First off. It should be pointed out that splunk resuming indexing where it left of is a highly desirable feature. This is what lets splunk keep going after a splunk restart, for example.

Here is another possible solution. It may not work.

Setup all your inputs with followTail=1
Right before the testing begins. Shutdown splunk. Clear all event data from the fishbucket index.
When the tests are done. Start up splunk again. Now when splunk starts, it should read all of your log files from the end and not from where it left off when it was shutdown.

The biggest problem I see with this approach, is that you can't selectively clear the fishbucket database. So this would take effect for ALL files. If there were instead some way to trick splunk into thinking that it had already read to the end of a file, that would be much more beneficial. Unfortunately, I don't know of a way of doing that. (If you knew the right kind of events to feed into into the fishbucket, this may be possible... but this is way over my head.)

Another option. Forward all your logs via syslog to splunk. Then shutdown splunk during the tests, and start it back up afterwards. Since syslog forwarding will drop any event while splunk is down, this could work pretty easily. Additional, you could just setup a local firewall to b block and unblock the syslog traffic instead of actually bringing splunk up and down; then splunk would still be up for other activities...

This will not work with a splunkforwarder though, because it buffers the log events so you don't loose them. Which again, is almost always what you want splunk to do. Yeah, you're asking for something splunk tries really hard not to do, drop events.

Re: Temporarily stop indexing

steveirogers — Wed, 22 Jun 2011 16:11:52 GMT

Thanks very much. You are absolutely correct in that Splunk is designed to prevent the loss of data. What I did was: (1) disable the index in Splunk (2) after the performance test, I deleted the large log file (3) I re-enabled the index and Splunk started indexing with the current events.