I have a deployment server from where i have a firewall rule that alows me to reach the 8089 management port of all forwarders. I would like to use the |rest server=xxx command to list porperties of the forwarder.
From command line i can successfully query al rest endpoints of the remote splunk so that part is ok.
When i use the |rest command i get a information "Search filters specified using splunk_server/splunk_server_group do not match any search peer."
I figured i might have to (gasp) configure all remote hosts as distributed search heads? But i encounter an issue doing so:
"Encountered the following error while trying to save: In handler 'distsearch-peer': Error while sending public key to search peer: Connection reset by peer".
What am i fundamentally doing wrong? I don't need to write a custom command to query the splunk endpoints do i? Or how do you guys inspect your forwarders???
I don't think the forwarder license allows it to participate as a distributed search peer. That (could) be the source of your reset error - even though that would be a weird error to get. And even if the forwarder does allow it, I've never heard of anyone trying to do anything like this. "Number of search peers" is a variable in search response time ; adding a bunch of forwarders as search peers could make this really bad. There are tunables that could help here (multi-threaded setup?) but this is probably a losing strategy in the long run.
Most admins I've worked / talked with do not "inspect" forwarders. It's not uncommon to disable the REST port on forwarders entirely, or at least limit it to the loopback interface. Then, if you need it for troubleshooting purposes then you enable it on purpose.
What exactly are you trying to accomplish here?
I made an app to do this: https://splunkbase.splunk.com/app/2775/
I don't think the forwarder license allows it to participate as a distributed search peer. That (could) be the source of your reset error - even though that would be a weird error to get. And even if the forwarder does allow it, I've never heard of anyone trying to do anything like this. "Number of search peers" is a variable in search response time ; adding a bunch of forwarders as search peers could make this really bad. There are tunables that could help here (multi-threaded setup?) but this is probably a losing strategy in the long run.
Most admins I've worked / talked with do not "inspect" forwarders. It's not uncommon to disable the REST port on forwarders entirely, or at least limit it to the loopback interface. Then, if you need it for troubleshooting purposes then you enable it on purpose.
What exactly are you trying to accomplish here?
Um, remote query monitored files and all sorts of settings that a (linux-)Admin might have "tweeked" on the server and thus causing me issues. The rel issue why i started to look into it was weird host names which i found out to be the package name in the zypper.log
Still, it would be nice to be able to use the rest command to query all sorts of properties of a splunk using the standart command, no? The xml from the rest interface is suffiently complex so that i would not want to reimplement it...