You are close. Changes made from the UI will not go into the "default" folder, though; they will go into local. So you can find your changes persisted in
Thank you! Will changes made to files under default be picked up upon restart?
Splunk configuration file precedence is pretty complicated, but this page does a good job of detailing it:
Values in the local folder will take precedence over values in the default folder. Generally speaking, unless you are the developing an app for distribution, you are advised to not make changes to the default folder. Changes you make in the local folder will persist through app updates, but changes you make in the default folder will generally get wiped out if you update an app.
Also, accessible via the GUI under: Settings, User Interface, Navigation menus... (v7.*)