Getting Data In

NullQueue and Re-indexing Logs

Builder

I want to limit the data from one of my universal forwarders from being indexed temporarily (until we get a new Splunk Daily Limit override).

Note that I do not want to block 9997 on the indexer. I do not want to shut down splunk on the forwarder either.

The remedy which involves writing all logs to nullQueues is a potential solution:

http://splunk-base.splunk.com/answers/11617/route-unwanted-logs-to-a-null-queue

My understanding is that when we write logs from a certain sourcetype (or in this case ALL sourcetypes) to a nullQueue, it will mark it for indexing later.

The key here is that I want it to go to a nullQueue NOW, but when then throw-away the nullqueue configuration and re-index everything that was sent to the nullQueue.

Is that how it works ?

Any other suggestions if going about it in this manner is not useful ?

Thanks

0 Karma
1 Solution

Legend

No, that is not how the nullQueue works.

nullQueue works like /dev/null on *NIX systems - data sent there is gone, never to be seen again. Splunk does not keep track of what's been sent there either. Without shutting down your forwarders or blocking them from accessing the indexer, I don't see an obvious solution to your problem.

View solution in original post

Legend

No, that is not how the nullQueue works.

nullQueue works like /dev/null on *NIX systems - data sent there is gone, never to be seen again. Splunk does not keep track of what's been sent there either. Without shutting down your forwarders or blocking them from accessing the indexer, I don't see an obvious solution to your problem.

View solution in original post

Builder

Forgive me for asking but I am not entirely sure where to set that on the forwarder config.

I am guessing its in one of the conf under /etc/system/local.

0 Karma

Legend

That should work. Just make sure you have proper settings for how events are buffered on the forwarders, so you're not losing events if the buffers fill up.

Builder

Hi there:

We are assessing the efficacy of a third-way solution.

This involves putting a network firewall between the forwarder and the indexer. Data will keep getting pushed to the indexer on 9997 but will never make it all the way to the indexer.

This way that data is not "lost" and will be re-indexed when this temporary firewall is lifted AND we are not blocking 9997 for all the other forwarders that want to push data.

What say you ?

0 Karma