How can I monitor the resource usage of my forwarder using the S.o.S app?

[mctester]

I would like to monitor forwarder indexing performance via S.O.S. The _internal already forwarded to indexer and could be searched as "index=_internal host=fwd" on indexer.
But I could not find it on the dropbox "Server to query" in S.O.S. Overview or any other dashboard.

Do I need to modify anything or any steps I did wrong?

Thanks!

[hexx · Accepted solution]

*/!\ THE FOLLOWING INSTRUCTIONS ARE ONLY VALID FOR SOS 2.2 AND NEWER /!\*

In SoS 2.2, the "Server to query" pulldown present in most views and that lists instances reachable by distributed search will load from a lookup file "splunk_servers_cache.csv", refreshed at 5-minute intervals.

This will considerably speed up the population of this omnipresent pulldown. It also means that you'll be able to scope S.o.S' views to events in the _internal index (or in other indexes, by updating the appropriate macros such as set_internal_index or set_sos_index) accessible by distributed search and belonging to any host, without the need for that host to be a search peer.

In order to do this, you will have to manually append an entry to the "splunk_servers_cache.csv" lookup. The fields required for an entry are:

• sos_server : This should be set to the value of the "host" field with which the events for the particular server are tagged.
• server_role : A description of the role of the server. Typically, this should be "search-head", "search-peer", "standalone indexer" or "forwarder".

Step by step instructions:

1 - Install the SoS technology add-on for Unix and Linux on the forwarder whose resource usage you care to monitor.

2 - Enable the ps_sos.sh scripted input by one of the following methods:

• Edit the `$SPLUNK_HOME/etc/apps/TA-sos/local/inputs.conf with the following content:

[script://./bin/ps_sos.sh]

disabled = 0

or

• Run the following command from a terminal window:

$SPLUNK_HOME/bin/splunk _internal call '/servicesNS/nobody/TA-sos/data/inputs/script/.%252Fbin%252Fps_sos.sh' -post:disabled 0

3 - Verify that the forwarder is now sending events generated by the ps_sos.sh scripted input to the indexer(s) by searching for:

index=sos sourcetype=ps | stats count by host

The hostname of your forwarder should be among the values returned.

4 - Append a new line to the $SPLUNK_HOME/etc/apps/sos/lookups/splunk_servers_cache.csv table composed of two, comma-separated values:

• The value of the "host" field for your forwarder's events in the sos index, as you gathered it in step #3.
• The string "forwarder", which will label that host as such in the "Server to query" pulldown.

Here's an example of the kind of line you would append to splunk_servers_cache.csv:

"forwarder.example.com","forwarder"

5 - The "Server to query" pulldown in the "Splunk CPU/Memory Resource Usage" view should now list your forwarder and enable you to view its system resource usage.

[hexx]

You really only need to populate two fields in that lookup when adding a host manually: sos_server and server_role. The next time the schedule search that maintains this lookup runs, it will derive the other fields from the values of those two. For more details, see the splunk_servers_cache.csv.spec file in the lookups directory of the app.

[yoho]

I'm running 3.X version of SoS. Other entries in splunk_servers_cache.csv already contain 4 columns. Should the forwarder entries only contain 2 columns or 4 columns like the other entries ?

[hexx]

@rroberts: Although this is desirable information, I don't think that the warnings view is the best location for it, as you would have to cycle through all forwarders to check their state. As you mention it, the Deployment Monitor app (which we consider a sister app to S.o.S) is currently delivering this feature. For that reason, we are not looking at adding it to S.o.S at this time.

[rroberts]

Adding to this thread, Can I see "warning" traffic light on Warnings page if a forwarder is missing? Something similar to the deployment app?

[MasterOogway]

Nice feature to add on in order to get more data on the environment. Thanks for the upgrade.