Reporting
Highlighted

How can I change the logging level for something I am trying to debug?

Splunk Employee
Splunk Employee

I am trying to figure out why a saved search does not seem to be running on a very busy splunk server, Is it possible to change the logging level only around saved searches to debug so I can get more detailed output?

Highlighted

Re: How can I change the logging level for something I am trying to debug?

Splunk Employee
Splunk Employee

You can do this through splunk manager. From the Manager home page, click on System Setting. From there click on System Logging. There you will find a list of all of the items for which you can configure the logging level, as well as what the current logging level is.

Type saved in the search bar and you will see several items around the saved search manager that you can switch to debug on by clicking the name of the item and from that form changing the setting.

Highlighted

Re: How can I change the logging level for something I am trying to debug?

Splunk Employee
Splunk Employee

Like Ben said you can change these via Manager. These settings will persists until the next reboot or you restore the original log setting. This great when you know the specific area that you are troubleshooting

The majority of Splunk's log settings (including the number of copies to keep and the size at which it rolls) are specified in log.cfg. In 3.x the Splunkweb logs are controlled in $SPLUNK_HOME/etc/SplunkWEB.tac. When you make a modification here you can make the changes to log levels permanent.

You can also start splunk in debug mode (splunk start --debug) which will put all components of the product in debug mode until it is restarted

View solution in original post

Highlighted

Re: How can I change the logging level for something I am trying to debug?

Splunk Employee
Splunk Employee

splunkWeb.tac is not used in 4.x and beyond. The splunkweb appserver debug settings are also in log.cfg now, at the bottom of that file.

See:

define splunk python logging properties

logging classes are defined by a logging declaration at the log of each file.

splunk

splunk.appserver

splunk.search

[python]
splunk = INFO
splunk.appserver = INFO
splunk.appserver.controllers = INFO
splunk.appserver.lib = WARN

(If you want to keep these changes permanently, even after upgrading splunk, copy log.cfg into loc-local.cfg and that file will not be changed on upgrade).

0 Karma
Highlighted

Re: How can I change the logging level for something I am trying to debug?

Splunk Employee
Splunk Employee

You can also set debugging commands through the CLI. I do this for my clustered deployments when I want to debug something on the indexers, and don't have SplunkWeb accessible. You can't view/set Indexer debugs from the SH's GUI, as that just sets debugging on the SH.

So in this example I wanted to check the Splunk2Splunk version of my Windows Universal Forwarder. Those messages are not set at the default INFO level, so I needed to set log-level to DEBUG - so I went to the cluster's indexer and enabled debugging for TcpInputProc.

To see the current debugging settings:

splunk show log-level

or for a specific facility:

splunk show log-level TcpInputProc

to set it:

splunk set log-level TcpInputProc  -level DEBUG

These settings go into effect immediately. Don't forget to set it back to the old level when you are finished!

Highlighted

Re: How can I change the logging level for something I am trying to debug?

New Member

Apologies for jumping on someone elses thread.

Is there a way of doing this on mass?

I've been tasked with changing the logging levels currently set to "WARN" on my splunk to "INFO". Whilst changing them individually I can do, I have 559 components to change. So obviously being able to change them on mass would be preferable.

I've already posed the question of what performance hit my system will take and my factory support are looking into that. I just wanted to be proactive and ready to do it, should I be given the go ahead to do so.

0 Karma
Highlighted

Re: How can I change the logging level for something I am trying to debug?

SplunkTrust
SplunkTrust

Yeah, so this thread has been around for 7.5 years and has an accepted answer. The chances of someone seeing your response is very slim. I suggest you post a new question. Reference this one, if you like.

Having said that, you should be able to use a text editor to modify all instances of "WARN" to "INFO" in the $SPLUNK_HOME/etc/log.cfg file. Make a backup first, of course. Restart Splunk for the changes to take effect.

---
If this reply helps you, an upvote would be appreciated.
0 Karma