How to combine savedsearches.conf from default and local folders to just the local folder?

[JDukeSplunk]

I have recently taken over a search head cluster, and am in the process of doing a little cleanup. I have found that /splunk/etc/apps/search/default/savedsearches.conf and /splunk/etc/apps/search/local/savedsearches.conf both contain over 4000 lines of saved searches. It is as if someone copied savedsearches into default not local, but I don't know what I can throw out and what I can keep.
I'm not exactly sure if this is bad or not. Judging that "installed-file-integrity" is complaining about this leads me to lean toward "bad".

The question I have is, can the search app output all of the searches into a new file that I can save as /apps/search/local/savedsearches.conf? Then I can put the default one back where it belongs. I've put them into a compare tool, and it's a mountain of work with tons of potential for human error.

Thanks,
JD

[alisterwhipp]

Came here to find this answer, but you need to beware the mangling of the btool output is wrong from the command listed in the answer;

./splunk cmd btool savedsearches list --debug | grep "etc/apps/search" | sed -r 's/^(\S+\s+)//'

Several chunks of the output have line breaks in the content output, and so grepping for the path gets only the first line of any field. In particular, searches often have line breaks.
You'd be better off just outputting the whole of

./splunk cmd btool savedsearches list --debug

and then manually grabbing just the section you need, then doing a search replace to remove the path.

If you just want to count objects etc, then the original will work fine. But if you need the actual contents, be careful with greps and seds unless you know what you're doing and check the formats of the source.

It also somewhat misinterprets the btool output. If you look at the raw output, it's clear any given saved search is compiled from layers of defaulting, which btool shows, with the path. What worked for me was using fgrep to exclude specific, noisy defaults. In particular;

./splunk cmd btool savedsearches list --debug | fgrep -v "etc/system/default" | fgrep -v display.visualizations.custom >>>>

So, that's basically the raw, minus the defaults across the whole system, minus the paragraphs of settings added by each of the custom viz apps installed that mysteriously get added to every search.
Removing the path at the beginning can be done with a smarter sed than just "something then some spaces, convert to nothing". That will tend to trim the first anyting on the line, which, for saved searches is going to kill a lot of content. something like

cat savedsearchbtool.conf | sed -r 's/\/opt\/splunk\/etc[\/a-z0-9A-Z\-_.]+savedsearches.conf\s+//'

BUT, I'd do the trimming first. Get filtered output with the paths. Manually trim. Then strip the paths.

[somesoni2 · Accepted solution]

Try this.

Run the btool command to list content of savesearches.conf on your Splunk search head and then filter only specific to search app. The btool merges the configuration from default and local directory and shows the saved searches configurations that are effective.

./splunk cmd btool savedsearches list --debug | grep "etc/apps/search" | sed -r 's/^(\S+\s+)//'

[verbal_666]

ATTENTION!!!
With this method you risk losing all the searches that have linefeed...

Ex.

[XXX]
...
search = index = _internal\
source = * splunkd\
"ERROR"

Will be ALL truncated to

[XXX]
...
search = index = _internal

Also the description field has linefeed...

description =

Be warned.

👍👍👍

[JDukeSplunk]

Thanks again Somesoni2, As usual your answers are on point.

Question, in the output of the command

 ./splunk cmd btool savedsearches list --debug | grep "etc/apps/search" | sed -r 's/^(\S+\s+)//' >>/tmp/savedsearches.conf 

I can count 260 individual searches, however in the view screen I can only see 122 for the search app. 506 for all apps. What might account for this discrepancy? Might it be Dashboard searches?

[somesoni2]

Their may be private searches that show up on the screen (since you're admin). They are stored in location etc/users//search/local/savedsearches.conf. The btool that we checked only checks for location etc/apps/search/ (local or default) /savedsearches.conf. That may be the difference.

[JDukeSplunk]

I'm going to have to eyeball the screen/combined savedsearches.conf and consolidate from search-names. It's quite a mess, but the btool command certainly saved a good bit of copy-pasta. I'll mark accepted soonish. I think this solution worked but until the manual eyeball is complete I don't know for sure.

-JD

[JDukeSplunk]

So it worked, mostly. Anywhere someone formatted their search with shift-enter's to make clean lines, the command only extracted the first line of the search. So, I had to go back into the old saved search config file. Look for search =, and then eyeball for ones that had multiple lines. I also had to remove the tailing "\" that got inserted.

E.G.
search = index=blah sourcetype=blah\
|eval blah(thing)\
|stats foo bar\

Only had the top line.

Close Enough!