Monitoring Splunk

How do you automate the "Apply Change" to register new servers on the Distributed Management Console?

pdaigle_splunk
Splunk Employee
Splunk Employee

I have a customer who asked:

We need to automatically register new servers on the Distributed Management Console. For this we use the Splunk CLI add search-peer. However, after this, we still need to manually press “Apply Changes” in the UI to complete the setup. Are you familiar with a way to allows us to automate this step?

http://docs.splunk.com/Documentation/Splunk/6.2.8/Admin/ConfiguretheMonitoringConsole


Any help here is greatly appreciated!

Thank you!
PD2

nibinabr
Communicator

I ended up tracking the HTTP requests and automated Apply Changes hitting the following endpoints.

/servicesNS/nobody/splunk_monitoring_console/configs/conf-splunk_monitoring_console_assets/settings -d configuredPeers=<comma separated lists of peers> -d output_mode=json -d disabled=0 -d eai:appName=splunk_monitoring_console -d eai:userName=nobody

/servicesNS/nobody/splunk_monitoring_console/saved/searches/DMC+Asset+-+Build+Full/dispatch -d output_mode=json -d trigger_actions=true -d dispatch.auto_cancel=30 -d dispatch.buckets=300 -d dispatch.enablePreview=true

/servicesNS/nobody/system/apps/local/splunk_monitoring_console -d output_mode=json -d author=Splunk -d check_for_updates=1 -d configured=1 -d description=The+Splunk+Monitoring+Console+application+gives+you+insight+into+your+Splunk+deployment. -d label=Monitoring+Console -d version=7.0.1 -d visible=1

First endpoint will update the splunk_monitoring_console_assets.conf. Second endpoint will run the DMC Asset search which is what I believe is building the assets and the last one to update the app.conf.

UPDATE: There were few more updates that I had to make to get this fully automated. I put this gist together.
https://gist.github.com/nmattam/bcfbc8a4ebd9a520c2ac50ab0137e58f

Aftend1971
Explorer

Do you know what to do for Apply Changes click in Distributed Monitoring Console?
Peers are showing in State: New, instead of State: Configured. And I have no Dasboards on Overview page, unitl I click on Apply Changes

0 Karma

nibinabr
Communicator

I updated my comment above with a gist of everything that I had to do to get this working. Try giving it a shot.

0 Karma

Aftend1971
Explorer

It seems that last step from https://gist.github.com/nmattam/bcfbc8a4ebd9a520c2ac50ab0137e58f is actually /opt/splunk/etc/apps/splunk_monitoring_console/default/data/ui/nav/default.distributed.xml ... so if you rename it and copy it like: splunk cmd /bin/mkdir -p /opt/splunk/etc/apps/splunk_monitoring_console/local/data/ui/nav; /bin/cp /opt/splunk/etc/apps/splunk_monitoring_console/default/data/ui/nav/default.distributed.xml /opt/splunk/etc/apps/splunk_monitoring_console/local/data/ui/nav/default.xml. It should work, but it does not....

0 Karma

nibinabr
Communicator

I was able to get it to work after renaming the xml and then restarting splunk.

0 Karma

altinp
Explorer

To configure or update the DMC in automated fashion, it seems like you have to fully populate all of these files correctly (for Distributed mode):

splunk_monitoring_console/local/app.conf
splunk_monitoring_console/local/assets.csv
splunk_monitoring_console/local/splunk_monitoring_console_assets.conf
splunk_monitoring_console/local/savedsearches.conf
etc/system/local/distsearch.conf

You can copy aside the files after your config management tool (e.g. SaltStack) has generated them and then diff with the version Splunk has modified once you "Apply Changes". Any major changes will tell you what you still need to do 'manually'. Once you have it all, clicking "Apply Changes" (or enabling Distributed mode or 'Enable Monitoring') is no longer needed.

0 Karma

lianwan
Explorer

tried this trick, but does not work. The contents of those files before and after click "Apply Changes" are now identical, except the order in etc/system/local/distsearch.conf may vary. Some how clicking that button results some randomness in order

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Just an idea, but perhaps you could use curl to send the same form the Apply Changes button sends to Splunk.

---
If this reply helps you, Karma would be appreciated.
0 Karma

javiergn
SplunkTrust
SplunkTrust

I don't know the answer for your question but what I've done in the past when I wanted to automate something that wasn't clearly documented was to look at the HTTP requests using a proxy or something similar.
For instance, the Live HTTP Headers add-on for Chrome and Firefox can do that very easily without needing to install or configure anything else.

Once you have the HTTP requests identified you can use the same POST/GET call from CURL or via the SDK and implement this yourself.

Hope that makes sense.

Thanks,
Javier

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...