NullQueue and Re-indexing Logs

[asarolkar]

I want to limit the data from one of my universal forwarders from being indexed temporarily (until we get a new Splunk Daily Limit override).

Note that I do not want to block 9997 on the indexer. I do not want to shut down splunk on the forwarder either.

The remedy which involves writing all logs to nullQueues is a potential solution:

http://splunk-base.splunk.com/answers/11617/route-unwanted-logs-to-a-null-queue

My understanding is that when we write logs from a certain sourcetype (or in this case ALL sourcetypes) to a nullQueue, it will mark it for indexing later.

The key here is that I want it to go to a nullQueue NOW, but when then throw-away the nullqueue configuration and re-index everything that was sent to the nullQueue.

Is that how it works ?

Any other suggestions if going about it in this manner is not useful ?

Thanks

[Ayn · Accepted solution]

No, that is not how the nullQueue works.

nullQueue works like /dev/null on *NIX systems - data sent there is gone, never to be seen again. Splunk does not keep track of what's been sent there either. Without shutting down your forwarders or blocking them from accessing the indexer, I don't see an obvious solution to your problem.

[asarolkar]

Forgive me for asking but I am not entirely sure where to set that on the forwarder config.

I am guessing its in one of the conf under /etc/system/local.

[Ayn]

That should work. Just make sure you have proper settings for how events are buffered on the forwarders, so you're not losing events if the buffers fill up.

[asarolkar]

Hi there:

We are assessing the efficacy of a third-way solution.

This involves putting a network firewall between the forwarder and the indexer. Data will keep getting pushed to the indexer on 9997 but will never make it all the way to the indexer.

This way that data is not "lost" and will be re-indexed when this temporary firewall is lifted AND we are not blocking 9997 for all the other forwarders that want to push data.

What say you ?