Deployment Architecture

How to combine savedsearches.conf from default and local folders to just the local folder?

JDukeSplunk
Builder

I have recently taken over a search head cluster, and am in the process of doing a little cleanup. I have found that /splunk/etc/apps/search/default/savedsearches.conf and /splunk/etc/apps/search/local/savedsearches.conf both contain over 4000 lines of saved searches. It is as if someone copied savedsearches into default not local, but I don't know what I can throw out and what I can keep.
I'm not exactly sure if this is bad or not. Judging that "installed-file-integrity" is complaining about this leads me to lean toward "bad".

The question I have is, can the search app output all of the searches into a new file that I can save as /apps/search/local/savedsearches.conf? Then I can put the default one back where it belongs. I've put them into a compare tool, and it's a mountain of work with tons of potential for human error.

Thanks,
JD

1 Solution

somesoni2
Revered Legend

Try this.

Run the btool command to list content of savesearches.conf on your Splunk search head and then filter only specific to search app. The btool merges the configuration from default and local directory and shows the saved searches configurations that are effective.

./splunk cmd btool savedsearches list --debug | grep "etc/apps/search" | sed -r 's/^(\S+\s+)//'

View solution in original post

alisterwhipp
Path Finder

Came here to find this answer, but you need to beware the mangling of the btool output is wrong from the command listed in the answer;

./splunk cmd btool savedsearches list --debug | grep "etc/apps/search" | sed -r 's/^(\S+\s+)//'

Several chunks of the output have line breaks in the content output, and so grepping for the path gets only the first line of any field. In particular, searches often have line breaks.
You'd be better off just outputting the whole of

./splunk cmd btool savedsearches list --debug

and then manually grabbing just the section you need, then doing a search replace to remove the path.

If you just want to count objects etc, then the original will work fine. But if you need the actual contents, be careful with greps and seds unless you know what you're doing and check the formats of the source.

It also somewhat misinterprets the btool output. If you look at the raw output, it's clear any given saved search is compiled from layers of defaulting, which btool shows, with the path. What worked for me was using fgrep to exclude specific, noisy defaults. In particular;

./splunk cmd btool savedsearches list --debug | fgrep -v "etc/system/default" | fgrep -v display.visualizations.custom >>>>

So, that's basically the raw, minus the defaults across the whole system, minus the paragraphs of settings added by each of the custom viz apps installed that mysteriously get added to every search.
Removing the path at the beginning can be done with a smarter sed than just "something then some spaces, convert to nothing". That will tend to trim the first anyting on the line, which, for saved searches is going to kill a lot of content. something like

cat savedsearchbtool.conf | sed -r 's/\/opt\/splunk\/etc[\/a-z0-9A-Z\-_.]+savedsearches.conf\s+//'

BUT, I'd do the trimming first. Get filtered output with the paths. Manually trim. Then strip the paths.

0 Karma

somesoni2
Revered Legend

Try this.

Run the btool command to list content of savesearches.conf on your Splunk search head and then filter only specific to search app. The btool merges the configuration from default and local directory and shows the saved searches configurations that are effective.

./splunk cmd btool savedsearches list --debug | grep "etc/apps/search" | sed -r 's/^(\S+\s+)//'

verbal_666
Contributor

ATTENTION!!!
With this method you risk losing all the searches that have linefeed...

Ex.

 

[XXX]
...
search = index = _internal\
source = * splunkd\
"ERROR"

 

 

Will be ALL truncated to

 

[XXX]
...
search = index = _internal

 

 

Also the description field has linefeed...

 

description =

 

 

Be warned.

👍👍👍

0 Karma

JDukeSplunk
Builder

Thanks again Somesoni2, As usual your answers are on point.

Question, in the output of the command

 ./splunk cmd btool savedsearches list --debug | grep "etc/apps/search" | sed -r 's/^(\S+\s+)//' >>/tmp/savedsearches.conf 

I can count 260 individual searches, however in the view screen I can only see 122 for the search app. 506 for all apps. What might account for this discrepancy? Might it be Dashboard searches?

0 Karma

somesoni2
Revered Legend

Their may be private searches that show up on the screen (since you're admin). They are stored in location etc/users//search/local/savedsearches.conf. The btool that we checked only checks for location etc/apps/search/ (local or default) /savedsearches.conf. That may be the difference.

0 Karma

JDukeSplunk
Builder

I'm going to have to eyeball the screen/combined savedsearches.conf and consolidate from search-names. It's quite a mess, but the btool command certainly saved a good bit of copy-pasta. I'll mark accepted soonish. I think this solution worked but until the manual eyeball is complete I don't know for sure.

-JD

0 Karma

JDukeSplunk
Builder

So it worked, mostly. Anywhere someone formatted their search with shift-enter's to make clean lines, the command only extracted the first line of the search. So, I had to go back into the old saved search config file. Look for search =, and then eyeball for ones that had multiple lines. I also had to remove the tailing "\" that got inserted.

E.G.
search = index=blah sourcetype=blah\
|eval blah(thing)\
|stats foo bar\

Only had the top line.

Close Enough!

Get Updates on the Splunk Community!

New Splunk Observability innovations: Deeper visibility and smarter alerting to ...

You asked, we delivered. Splunk Observability Cloud has several new innovations giving you deeper visibility ...

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...