All Apps and Add-ons

SoS : "server to query" pulldown not listing search peers

MasterOogway
Communicator

I have added Splunk on Splunk to my new distributed search environment after having such good luck with the previous installation. I am now running into a bit of a difficulty getting the "server to query" drop-down to fill correctly.
With SoS v2.1 and SideView v1.3.4 installed on my SuSE 11.1 Search head, with (4) Indexers all running Splunk v4.3.2 as part of the distributed search environment, my only option to search SoS is the Search head or "dns". This is odd behavior because my Search head recognizes the (4) distributed Indexers in its list, but for whatever reason SoS only wants to list them in the "Hostname" field directly below the Drop-down menu.

Any ideas as to why this might occur or how to correct SoS to autofill the Drop-down?
Looking closer at the ../sos/lookups/splunk_instances_info.csv file I see three line entries. The first starting with "sos_server"; the second is my Search server hostname; the third starts with "dns".
Each line lists many comma delimited entries, with both lines two and three listing the same values except for the first column where they differ by & DNS.

On the previous installation of SoS where it works great, the ../sos/lookups/splunk_instances_info.csv is actually a "gz" file and the entries within are very simple. Not like the problem installation where there are many, many more column variables to fill in and the file name ends in .csv.

Thoughts? Ideas?

1 Solution

hexx
Splunk Employee
Splunk Employee

I would recommend to upgrade to SoS 2.2 as the logic that populates the "Server to query" pulldown has been rewritten to be faster and configurable (see this Splunk Answer for more details) in that release. That might very well take care of your problem.

View solution in original post

hexx
Splunk Employee
Splunk Employee

Excellent news! Let me write that as an answer that you can accept 🙂

0 Karma

hexx
Splunk Employee
Splunk Employee

I would recommend to upgrade to SoS 2.2 as the logic that populates the "Server to query" pulldown has been rewritten to be faster and configurable (see this Splunk Answer for more details) in that release. That might very well take care of your problem.

MasterOogway
Communicator

This finally corrected the issue of not filling in the server host list found in the SoS App while using SoSv2.1.

0 Karma

MasterOogway
Communicator

Ok....we finally have a winner. Upgrading to S.o.S. v2.2 not only fixed my issue on seeing the Indexers, but also allows me to run the "| btool inputs | search "/var/log/splunk". And reports on all (4) additional Indexers.
Thanks for your patience and persistance, HEXX. A check goes in the WIN column.
I can't vote for my own, so you will have to vote for your answer to my question.

0 Karma

hexx
Splunk Employee
Splunk Employee

That is very strange, and would somewhat explain why you are not seeing the expected entries in the "Server to query" pulldown. Would you mind trying to upgrade to SoS 2.2 and see if that has any positive effect?

0 Karma

MasterOogway
Communicator

Correct...and as administrator I still get his Permissions error when I run any | btool command. I tried multiple attempts and quoting in case something misfired. Anything else I can try for this troubleshooting exercise?

0 Karma

hexx
Splunk Employee
Splunk Employee

You need to run that command from the flashtimeline view in the context of the S.o.S app. The "btool" search command is not available to other apps. Please go to http[s]://[search-head hostname]:[splunkweb port]/en-US/app/sos/flashtimeline and run | btool inputs | search "/var/log/splunk" from the search bar there.

0 Karma

MasterOogway
Communicator

Here is the HOST information:

splunk cmd btool inputs list | grep -A20 "///emat/splunk/var/log/splunk"

[monitor:///emat/splunk/var/log/splunk]

_rcvbuf = 1572864

host = c111xbz - or the same as "abc123"

To answer how |btool input failed

At the search line I add this: | btool inputs | search "/var/log/splunk"

and the error that comes back is this:
"Search operation 'btool' is unknown. You might not have permission to run this operation."

I tried adding a single quote around |btool input' -- to no avail.

I even tried to send just |btool input but that failed too.

0 Karma

hexx
Splunk Employee
Splunk Employee

Could you elaborate on exactly how the search using btool as a search command was unsuccessful? Also, the command line output we need is specifically what you see for the value of the "host" parameter in the file monitoring stanza of inputs.conf that targets $SPLUNK_HOME/var/log/splunk.

In the output you provided, I would be interested in the value of "host =" under the " [monitor:///emat/splunk/var/log/splunk]" stanza.

0 Karma

MasterOogway
Communicator

the "abc123" is only a replacement name for the actual server name that was listed....I simply anonymized the entry substituting the real name for "abc123".

I also tried to run the | btool inputs | search "/var/log/splunk" on the search line but was unsuccessful, so I substituted for what you appeared to be looking for...any entries from the inputs file that had splunk in it.

# splunk cmd btool inputs list | grep splunk

[batch:///emat/splunk/var/spool/splunk]

[batch:///emat/splunk/var/spool/splunk/...stash_new]

[fschange:/emat/splunk/etc]

[monitor:///emat/splunk/etc/splunk.version]

sourcetype = splunk_version

[monitor:///emat/splunk/var/log/splunk]

[script:///emat/splunk/etc/apps/sos/bin/lsof_sos.sh]

[script:///emat/splunk/etc/apps/sos/bin/ps_sos.sh]

[script:///emat/splunk/etc/apps/tagapp/bin/tagapp.pl]

[splunktcp]

or this:

# splunk cmd btool inputs list | grep sos

[script:///emat/splunk/etc/apps/sos/bin/lsof_sos.sh]

index = sos

source = lsof_sos

[script:///emat/splunk/etc/apps/sos/bin/ps_sos.sh]

index = sos

source = ps_sos

If neither of these helped I can run some others.

0 Karma

hexx
Splunk Employee
Splunk Employee

This is most peculiar. Do you happen to have installed several Splunk instances on the same machine? I am very intrigued by the origin of this "abc123" string. Can we now see the output of the following search?

| btool inputs | search "/var/log/splunk"

I am particularly interested in what you will see for the line containing "host = ".

0 Karma

MasterOogway
Communicator

I only get two columns, one starting with abc123 and the other dns, and nothing else. Should I be seeing a list of the Indexing servers here? Do you need to see a conf file to help explain what might be happening?

sos_server count dc server_label server_role

abc123 2 2 search-head : abc123 search-head

dns 4 2

0 Karma

hexx
Splunk Employee
Splunk Employee

Oh I see what happened, the formatting ate my back-ticks. The command you should run is:
| `get_splunk_servers`

0 Karma

MasterOogway
Communicator

| get_splunk_servers

is not an acceptable command that I can run at the "search bar" from the ../flashtimeline

I also tried just splunk_servers without luck. I do however get results listed when I run your long command string found within this URL:

http://splunk-base.splunk.com/answers/38452/the-lookup-table-splunk_instances_info-is-invalid-sos?pa...

What else can I attempt to pull the server list?

0 Karma

hexx
Splunk Employee
Splunk Employee

It seems that your search peers may not be generating the expected results for the getallhosts custom command that allows the app to identify available instances and populate the "Server to query" menu.

What results do you get when searching for | get_splunk_servers from http[s]://[search-head hostname]:[splunkweb port]/en-US/app/sos/flashtimeline?

Do note that the splunk_instances_info.csv lookup table is not involved in the population of the "Server to query" pulldown. It is only used for the "A glimpse of your Splunk instance" in the home view.

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...