SoS and clusters

[techn0gichida]

The forwarder points to the peer in the cluster per the instructions. How does the SoS technology add-on point itself to the search head?

[hexx]

Glad to hear that. Feel free to accept my answer, in that case 🙂

[techn0gichida]

hexx, thanks so much. that last suggestion got it working!

[hexx]

Why not enable the input manually in %SPLUNK_HOME%\etc\apps\TA-sos_win\local\inputs.conf then?

[script://.\bin\sospowershell.cmd ps_sos.ps1]
disabled = 0

[techn0gichida]

cannot edit input "./bin/ps_sos.ps1", no input exists with that name
that is the error I get when I use the ps_sos.ps1 with the single quotes removed

[techn0gichida]

still getting the 404 error

[hexx]

Ah! In that case, the scripted input you need to enable is 'ps_sos.ps1', not 'ps_sos.sh'.

As the README file of the S.o.S technology add-on for Windows states:

Enable the scripted inputs that collect information for the SoS Splunk CPU/Memory
Usage and Distributed Searches Memory Usage views:

(...)

b) Run the following from a command or PowerShell prompt:

  %SPLUNK_HOME%\bin\splunk _internal call \
  '/servicesNS/nobody/TA-sos/data/inputs/script/.%252Fbin%252Fps_sos.ps1' \
  -post:disabled 0

[techn0gichida]

This is a windows box. It doesn't have grep

[hexx]

Please show here the output of:

• $SPLUNK_HOME/bin/splunk cmd btool inputs list 'script:' --debug | grep -A7 'ps_sos.sh'
• grep 'ps_sos' $SPLUNK_HOME/var/log/splunk/metrics.log | head -10

[techn0gichida]

It's like the script can't find anything on port 8089

[techn0gichida]

Sorry, the script fails with a 404 error.

[techn0gichida]

I added the input manually and it still isn't showing up. I find the following message in splunkd.log:
splunk-regmon - No enabled entries have been found for regmon or procman in the conf file

[hexx]

Sounds like the ps_sos.sh scripted input was not successfully enabled on the forwarder. I would suggest to use "splunk login" and log in as admin before running that command again.

Alternatively, you can enable that input manually in $SPLUNK_HOME/etc/apps/TA-sos/local/inputs.conf.

[techn0gichida]

This script fails with a 401 error:
$SPLUNK_HOME/bin/splunk _internal call '/servicesNS/nobody/TA-sos/data/inputs/script/.%252Fbin%252Fps_sos.sh' -post:disabled 0

[techn0gichida]

I did now and it still isn't showing any SoS data. Like I said previously, I did the "index=sos sourcetype=ps | stats count by host" test per the installation instructions but it isn't returning the name of the server in the list.

[hexx]

Have you manually added the forwarder to the "splunk_servers_cache.csv" lookup in $SPLUNK_HOME/etc/apps/sos/lookups on the search-head, as recommended?

[techn0gichida]

When I run the test on the search-head it does not return the server name in the list. So the forwarder is not sending any SoS data to the search-head. Although the forwarder is sending splunk data.

[hexx]

I think you're asking "Now that my forwarder is collecting data with the scripted inputs of the S.o.S technology add-on, how do I consult that information in the S.o.S app on the search-head?".

If that is accurate, please consult this Splunk Answer which addresses that scenario.

The short version is: You'll need to manually add your forwarder to the "splunk_servers_cache.csv" lookup.

We have plans to make this an automated step in a future release.