topic Re: How do I pause indexing? in Getting Data In

How do I pause indexing?

talbot7 — Wed, 05 Dec 2012 18:08:04 GMT

I would like to pause indexing when I reach 95% of my license. I have the Nagios check built, I just need the command to pause indexing.

Re: How do I pause indexing?

yannK — Wed, 05 Dec 2012 18:29:49 GMT

Disable the inputs or shutdown splunk.

An alternative is to setup a nullQueue filtering rule and turn it on to trash all your events.

see http://docs.splunk.com/Documentation/Splunk/5.0.1/Deploy/Routeandfilterdatad#Keep_specific_events_and_discard_the_rest

Re: How do I pause indexing?

talbot7 — Wed, 05 Dec 2012 18:44:29 GMT

I have filters in place sending garbage data to nullQueue. Cant shutdown splunk, customers will get upset.

Cant seem to disable the input from the command line:
[root@splk01 bin]# ./splunk disable listen -port 9998
In handler 'cooked': Could not find config id for port 9998

All traffic is sent from forwarders on port 9998 (SSL). Once a connection is established, it moves to some random high level port. If it just stayed put, I would kill it with IPTables.

I know there is a clean way to do this. Splunk does it when it runs low on disk space.

Re: How do I pause indexing?

yannK — Wed, 05 Dec 2012 18:50:27 GMT

To stop indexing, you could change the minFreeSpace in server.conf, But it may also stop searches.
see http://docs.splunk.com/Documentation/Splunk/5.0/Admin/Serverconf
`
[diskUsage]

minFreeSpace =
* Specified in megabytes.
* The default setting is 2000 (approx 2GB)
* Specifies a safe amount of space that must exist for splunkd to continue operating.
* Note that this affects search and indexing
* This is how the searching is affected:
* For search:
* Before attempting to launch a search, splunk will require this
amount of free space on the filesystem where the dispatch
directory is stored, $SPLUNK_HOME/var/run/splunk/dispatch
* Applied similarly to the search quota values in
authorize.conf and limits.conf.
* For indexing:
* Periodically, the indexer will check space on all partitions
that contain splunk indexes as specified by indexes.conf. Indexing
will be paused and a ui banner + splunkd warning posted to indicate
need to clear more disk space.
`

Re: How do I pause indexing?

yannK — Wed, 05 Dec 2012 18:50:57 GMT

Splunk do not throttle the indexing when the license is excessed, it keeps indexing but disable the search.
If you are an enterprise customer, you can go over, and once you have 5 days of warnings ask for a reset key.

To stop a listening port, just edit inputs.conf and disable the input ans restart to reload.

Re: How do I pause indexing?

talbot7 — Wed, 05 Dec 2012 18:54:45 GMT

Tried that:
[root@splk01 bin]# ./splunk set minfreemb 200000
You need to restart the Splunk Server (splunkd) for your changes to take effect.

Restarting the production environment it still not an option 😞

Re: How do I pause indexing?

yannK — Wed, 05 Dec 2012 18:57:52 GMT

so close the input port using the manager !
or block the port using iptables...

Re: How do I pause indexing?

talbot7 — Wed, 05 Dec 2012 19:01:28 GMT

Cant block the port with iptables.

"All traffic is sent from forwarders on port 9998 (SSL). Once a connection is established, it moves to some random high level port. If it just stayed put, I would kill it with IPTables."

I am trying to build the automated approach (Not the manager). But for now, that will have to do.

Re: How do I pause indexing?

kristian_kolb — Wed, 05 Dec 2012 21:37:45 GMT

sorry, are you saying that the forwarder->indexer traffic (i.e. logs) changes ports from what you've defined in inputs.conf (on the indexer) and outputs.conf (on the forwarder)??? I didn't think that was possible.

I still don't.

You should be able to either block it in the local fw on the indexer or edit inputs.conf on the indexer and restart it. (as yannk says)

Also, as for restarting a production environment. You lose 2 minutes of searchability for the restart as opposed to losing 1 day to get a license reset key....

Re: How do I pause indexing?

talbot7 — Wed, 05 Dec 2012 22:47:43 GMT

Having iptables drop anything on 9998 does not effect established sessions.

I wound up editing inputs.conf and restarting the indexer. Only a few customers noticed.

There has got to be a clean way of doing this, with out using iptables or restarting Splunk.

Re: How do I pause indexing?

kristian_kolb — Thu, 06 Dec 2012 11:45:19 GMT

Ok so you have a tricky problem. You have some customers that need access to their data - and they cannot accept downtime (for searches?). But they have no problems with you throwing away their logs once the license is 95% full!?!

Sounds like you need a bigger license. The customers should be happy to pay for it, given the circumstances...or am I missing something.

Re: How do I pause indexing?

talbot7 — Thu, 06 Dec 2012 16:13:10 GMT

Yes, I do understand the hypocritical problems with my customers... 😞

I can get away with an indexer reboot (sometimes). The search heads are different boxes.

Loose data?! HAHAHAHAHAHA. We have forwarders on all systems, and never loose any data. Splunk is real good about catching up the next day when we turn the indexer back on.

Re: How do I pause indexing?

kristian_kolb — Thu, 06 Dec 2012 19:45:59 GMT

So you're just pushing the problem ahead of you until there are days that go below the limit, so you can then catch up on the backlog (no pun intended)

Re: How do I pause indexing?

talbot7 — Thu, 06 Dec 2012 22:01:09 GMT

Our average data is only 50% of our license, so we normally have no problems catching up.

Re: How do I pause indexing?

yannK — Thu, 06 Dec 2012 23:27:33 GMT

What about having each forwarder deployment sending to a different splunktcp port, then you can disable a specific port at a time to block the forwarders.
They will queue, then pause. and restart once the port reopen.

Re: How do I pause indexing?

talbot7 — Fri, 07 Dec 2012 00:02:01 GMT

I was thinking about taking that once step farther and having different indexers for each environment.