Splunk Search

Enable/Disable indexing a debug log file on demand?

mwhitake78
Explorer

Is there any easy way to enable/disable indexing of a debug log file so that it can be indexed only when needed? We have some debug log files that are used primarily during rollouts of new features and testing cycles. We would love to have the data in splunk, but most of the time it is not needed.

Labels (1)
0 Karma
1 Solution

gcusello
Esteemed Legend

Hi @mwhitake78,

In this case, it's easy:

on the Deployment Server, you could manually enable or disable the stanza with the input of these debug logs modifying the parameter 

disable=0/1

than you could push the modified configuration using the command

./splunk reload deploy-server

in this way the updated configuration will be pushed by the Deployment Server to the target server without accessing it.

You could also create a script that automatically makes these jbs and eventually also connect this script to an alert.

Ciao.

Giuseppe

View solution in original post

gcusello
Esteemed Legend

Hi @mwhitake78,

two questions:

  • would you start/stop indexing of the debug file in manual or automatic way?
  • debug logs are on the Splunk server on in another server with Universal Forwarder?

If manually enablement is acceptable for you, you can:

  • if local on Splunk Server, manually enable / disable the input using the GUI,
  • if remote, you can run a script that enables / disables the input and restarts UF.

If instead you want an automatic intervene is more complicated because you should create an alert that monitors the condition that needs the enablement of the debug log and then:

  • if local on Splunk Server, manually enable / disable the input using the GUI,
  • if remote, you can associate to the alert an action that runs a script that enables / disables the input and restarts UF.

Ciao.

Giuseppe

mwhitake78
Explorer

The debug logs are on another server with the universal forwarder. What I was hoping for was a way to run a command manually to enable or disable the indexing of these files, but to be able to do so without requiring login to the host the UF is running on. 

All of the config is on the splunk server itself and is sent to the host with the Splunk Deployment Server 

0 Karma

gcusello
Esteemed Legend

Hi @mwhitake78,

In this case, it's easy:

on the Deployment Server, you could manually enable or disable the stanza with the input of these debug logs modifying the parameter 

disable=0/1

than you could push the modified configuration using the command

./splunk reload deploy-server

in this way the updated configuration will be pushed by the Deployment Server to the target server without accessing it.

You could also create a script that automatically makes these jbs and eventually also connect this script to an alert.

Ciao.

Giuseppe

mwhitake78
Explorer

Thank you so much for the quick response. Is there any way to allow an end user to create or run a saved search that would trigger my scripts to enable/disable this indexed file? 

What I would like to do is have a way for our developers to turn on or off indexing for some debug files that are needed at times, but other times would just create a lot of unneeded noise in the indexes. I have been able to create shell scripts to enable and disable the monitoring of the files in question, but have come up blank on a way to allow the user to actually make this change. These users do not have access to the splunk server, just to the web interface.

Any additional help would be very appreciated.

0 Karma

gcusello
Esteemed Legend

Hi @mwhitake78,

as @PickleRick said, you have two choices:

  • you could manually use the deployment server web interface to deploy an app that enables/disables logs,
  • you cud create a shell script to manually launch or to associate to an alert.

Ciao.

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

0 Karma

PickleRick
Ultra Champion

By default - no. Even if you granted your users permissions to edit the deployment server to enable/disable apps (and define the input as a distributable app so it could be enabled/disabled as a whole), you couldn't limit it to just one app.

Also, remember that enabling/disabling input at any given moment doesn't necessarily guarantee that only events from this moment onwards will be ingested.

For example, if you have a directory, let's say, /opt/whatever/logs in which you have normal and debug logs created one file per day, when you enable input monitoring /opt/whatever/logs, it will ingest all files matching the whitelist/blacklist/maxage criteria. So you might end up ingesting quite a big backlog of files that have been created while the input was off.

0 Karma
Get Updates on the Splunk Community!

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...

Security Highlights | January 2023 Newsletter

January 2023 Splunk Security Essentials (SSE) 3.7.0 ReleaseThe free Splunk Security Essentials (SSE) 3.7.0 app ...

Platform Highlights | January 2023 Newsletter

 January 2023Peace on Earth and Peace of Mind With Business ResilienceAll organizations can start the new year ...