Enable/Disable indexing a debug log file on demand?

[mwhitake78]

Is there any easy way to enable/disable indexing of a debug log file so that it can be indexed only when needed? We have some debug log files that are used primarily during rollouts of new features and testing cycles. We would love to have the data in splunk, but most of the time it is not needed.

[gcusello]

Hi @mwhitake78,

two questions:

• would you start/stop indexing of the debug file in manual or automatic way?
• debug logs are on the Splunk server on in another server with Universal Forwarder?

If manually enablement is acceptable for you, you can:

• if local on Splunk Server, manually enable / disable the input using the GUI,
• if remote, you can run a script that enables / disables the input and restarts UF.

If instead you want an automatic intervene is more complicated because you should create an alert that monitors the condition that needs the enablement of the debug log and then:

• if local on Splunk Server, manually enable / disable the input using the GUI,
• if remote, you can associate to the alert an action that runs a script that enables / disables the input and restarts UF.

Ciao.

Giuseppe

[mwhitake78]

The debug logs are on another server with the universal forwarder. What I was hoping for was a way to run a command manually to enable or disable the indexing of these files, but to be able to do so without requiring login to the host the UF is running on. 

All of the config is on the splunk server itself and is sent to the host with the Splunk Deployment Server 

[gcusello · Accepted solution]

Hi @mwhitake78,

In this case, it's easy:

on the Deployment Server, you could manually enable or disable the stanza with the input of these debug logs modifying the parameter 

disable=0/1

than you could push the modified configuration using the command

./splunk reload deploy-server

in this way the updated configuration will be pushed by the Deployment Server to the target server without accessing it.

You could also create a script that automatically makes these jbs and eventually also connect this script to an alert.

Ciao.

Giuseppe

[mwhitake78]

Thank you so much for the quick response. Is there any way to allow an end user to create or run a saved search that would trigger my scripts to enable/disable this indexed file? 

What I would like to do is have a way for our developers to turn on or off indexing for some debug files that are needed at times, but other times would just create a lot of unneeded noise in the indexes. I have been able to create shell scripts to enable and disable the monitoring of the files in question, but have come up blank on a way to allow the user to actually make this change. These users do not have access to the splunk server, just to the web interface.

Any additional help would be very appreciated.

[gcusello]

Hi @mwhitake78,

as @PickleRick said, you have two choices:

• you could manually use the deployment server web interface to deploy an app that enables/disables logs,
• you cud create a shell script to manually launch or to associate to an alert.

Ciao.

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

[PickleRick]

By default - no. Even if you granted your users permissions to edit the deployment server to enable/disable apps (and define the input as a distributable app so it could be enabled/disabled as a whole), you couldn't limit it to just one app.

Also, remember that enabling/disabling input at any given moment doesn't necessarily guarantee that only events from this moment onwards will be ingested.

For example, if you have a directory, let's say, /opt/whatever/logs in which you have normal and debug logs created one file per day, when you enable input monitoring /opt/whatever/logs, it will ingest all files matching the whitelist/blacklist/maxage criteria. So you might end up ingesting quite a big backlog of files that have been created while the input was off.