SoS : "server to query" pulldown not listing search peers

[MasterOogway]

I have added Splunk on Splunk to my new distributed search environment after having such good luck with the previous installation. I am now running into a bit of a difficulty getting the "server to query" drop-down to fill correctly.
With SoS v2.1 and SideView v1.3.4 installed on my SuSE 11.1 Search head, with (4) Indexers all running Splunk v4.3.2 as part of the distributed search environment, my only option to search SoS is the Search head or "dns". This is odd behavior because my Search head recognizes the (4) distributed Indexers in its list, but for whatever reason SoS only wants to list them in the "Hostname" field directly below the Drop-down menu.

Any ideas as to why this might occur or how to correct SoS to autofill the Drop-down?
Looking closer at the ../sos/lookups/splunk_instances_info.csv file I see three line entries. The first starting with "sos_server"; the second is my Search server hostname; the third starts with "dns".
Each line lists many comma delimited entries, with both lines two and three listing the same values except for the first column where they differ by & DNS.

On the previous installation of SoS where it works great, the ../sos/lookups/splunk_instances_info.csv is actually a "gz" file and the entries within are very simple. Not like the problem installation where there are many, many more column variables to fill in and the file name ends in .csv.

Thoughts? Ideas?

[hexx]

Excellent news! Let me write that as an answer that you can accept 🙂

[hexx · Accepted solution]

I would recommend to upgrade to SoS 2.2 as the logic that populates the "Server to query" pulldown has been rewritten to be faster and configurable (see this Splunk Answer for more details) in that release. That might very well take care of your problem.

[MasterOogway]

This finally corrected the issue of not filling in the server host list found in the SoS App while using SoSv2.1.

[MasterOogway]

Ok....we finally have a winner. Upgrading to S.o.S. v2.2 not only fixed my issue on seeing the Indexers, but also allows me to run the "| btool inputs | search "/var/log/splunk". And reports on all (4) additional Indexers.
Thanks for your patience and persistance, HEXX. A check goes in the WIN column.
I can't vote for my own, so you will have to vote for your answer to my question.

[hexx]

That is very strange, and would somewhat explain why you are not seeing the expected entries in the "Server to query" pulldown. Would you mind trying to upgrade to SoS 2.2 and see if that has any positive effect?

[MasterOogway]

Correct...and as administrator I still get his Permissions error when I run any | btool command. I tried multiple attempts and quoting in case something misfired. Anything else I can try for this troubleshooting exercise?

[hexx]

You need to run that command from the flashtimeline view in the context of the S.o.S app. The "btool" search command is not available to other apps. Please go to http[s]://[search-head hostname]:[splunkweb port]/en-US/app/sos/flashtimeline and run | btool inputs | search "/var/log/splunk" from the search bar there.

[MasterOogway]

Here is the HOST information:

splunk cmd btool inputs list | grep -A20 "///emat/splunk/var/log/splunk"

[monitor:///emat/splunk/var/log/splunk]

_rcvbuf = 1572864

host = c111xbz - or the same as "abc123"

To answer how |btool input failed

At the search line I add this: | btool inputs | search "/var/log/splunk"

and the error that comes back is this:
"Search operation 'btool' is unknown. You might not have permission to run this operation."

I tried adding a single quote around |btool input' -- to no avail.

I even tried to send just |btool input but that failed too.

[hexx]

Could you elaborate on exactly how the search using btool as a search command was unsuccessful? Also, the command line output we need is specifically what you see for the value of the "host" parameter in the file monitoring stanza of inputs.conf that targets $SPLUNK_HOME/var/log/splunk.

In the output you provided, I would be interested in the value of "host =" under the " [monitor:///emat/splunk/var/log/splunk]" stanza.

[MasterOogway]

the "abc123" is only a replacement name for the actual server name that was listed....I simply anonymized the entry substituting the real name for "abc123".

I also tried to run the | btool inputs | search "/var/log/splunk" on the search line but was unsuccessful, so I substituted for what you appeared to be looking for...any entries from the inputs file that had splunk in it.

# splunk cmd btool inputs list | grep splunk

[batch:///emat/splunk/var/spool/splunk]

[batch:///emat/splunk/var/spool/splunk/...stash_new]

[fschange:/emat/splunk/etc]

[monitor:///emat/splunk/etc/splunk.version]

sourcetype = splunk_version

[monitor:///emat/splunk/var/log/splunk]

[script:///emat/splunk/etc/apps/sos/bin/lsof_sos.sh]

[script:///emat/splunk/etc/apps/sos/bin/ps_sos.sh]

[script:///emat/splunk/etc/apps/tagapp/bin/tagapp.pl]

[splunktcp]

or this:

# splunk cmd btool inputs list | grep sos

[script:///emat/splunk/etc/apps/sos/bin/lsof_sos.sh]

index = sos

source = lsof_sos

[script:///emat/splunk/etc/apps/sos/bin/ps_sos.sh]

index = sos

source = ps_sos

If neither of these helped I can run some others.

[hexx]

This is most peculiar. Do you happen to have installed several Splunk instances on the same machine? I am very intrigued by the origin of this "abc123" string. Can we now see the output of the following search?

| btool inputs | search "/var/log/splunk"

I am particularly interested in what you will see for the line containing "host = ".

[MasterOogway]

I only get two columns, one starting with abc123 and the other dns, and nothing else. Should I be seeing a list of the Indexing servers here? Do you need to see a conf file to help explain what might be happening?

sos_server count dc server_label server_role

abc123 2 2 search-head : abc123 search-head

dns 4 2

[hexx]

Oh I see what happened, the formatting ate my back-ticks. The command you should run is:
| `get_splunk_servers`

[MasterOogway]

| get_splunk_servers

is not an acceptable command that I can run at the "search bar" from the ../flashtimeline

I also tried just splunk_servers without luck. I do however get results listed when I run your long command string found within this URL:

http://splunk-base.splunk.com/answers/38452/the-lookup-table-splunk_instances_info-is-invalid-sos?pa...

What else can I attempt to pull the server list?

[hexx]

It seems that your search peers may not be generating the expected results for the getallhosts custom command that allows the app to identify available instances and populate the "Server to query" menu.

What results do you get when searching for | get_splunk_servers from http[s]://[search-head hostname]:[splunkweb port]/en-US/app/sos/flashtimeline?

Do note that the splunk_instances_info.csv lookup table is not involved in the population of the "Server to query" pulldown. It is only used for the "A glimpse of your Splunk instance" in the home view.