Search query to check 50-100 forwarders phoning home status from thousands of forwarders.

[onkarkore1]

Hi,

We have thousands of forwarders installed. Our requirement is to check status of only 50-100 forwarders from thousands of forwarders on daily basis.

These selective 50-100 forwarders will be new every time.

Now we are trying to write a search query which will check phoning home status for these 50-100 forwarders which will be new on every run.

I will be thankful If any of you can guide or provide search query.

Thanks

[DalJeanis]

mmodestino is right in the long run. However, in the short run, you should go ahead and provide them with their requested sample of statuses.

It's pretty simple. Have them give you the host names each day in a csv file with a fixed name.

Supposing your base search for a single forwarder (bar) was something like this....

index=foo host=bar 
|(your search terms and transforms) 
| table host status

....then your query for all 50-100 forwarders, assuming you got the host names in a csv file in a field called forwardername, would be....

index=foo [| inputcsv todaysforwarders.csv |rename forwardername as host | table host] 
| (your search terms and transforms)
| table host status

The values input from the csv (everything inside the brackets) is transformed by the implicit format command at the end of the brackets to be in the form...

(host=bar OR host=baz OR host=....)

[DalJeanis]

Okay, so consider my "answer" to be a comment on how this section of mmodestino's answer works...

[| inputlookup forwarders.csv | fields hostname] 

[onkarkore1]

Thanks Jeanis for your reply.

[mattymo · Accepted solution]

I recommend you use the Management Console (aka DMC) to report on the status of your forwarders. The Monitoring Console allows you to see the overall status of all your forwarders (active/missing) and can even alert on missing forwarders.

https://docs.splunk.com/Documentation/Splunk/6.5.3/DMC/ForwardersDeployment
https://docs.splunk.com/Documentation/Splunk/6.5.3/DMC/Deploymentsetupsteps

The views in this dashboard make it easy to report on many aspects of your forwarders and will let you know if any go missing.

All you need is to ensure all your hosts are forwarding their internal logs to your indexing tier, then select a host that can act as a search head to host the Monitoring Console. (license master is a common choice).

At the very least you can examine the macros and searches this dash uses in search and use them as you wish!

[onkarkore1]

Hi,

Thanks for the time you put on replying on this issue. We have thousands of forwarders installed. Now our requirement is to check status of only 50-100 forwarders from these thousands of forwarders on daily basis.

These selective 50-100 forwarders will be new on every search query run.

Now I am trying to write a search query which will check phoning home status for these 50-100 forwarders which will be new on every run.

Thanks

[mattymo]

The fact you have thousands of forwarders is all the more reason to be using the Monitoring Console....why check 50-100 when you can report on ALL of your forwarders???

Can you tell us more about the requirement?

Why are you checking 50-100 servers a day? are these random spot checks to make sure the forwarders are still phoning home?? Still connecting to indexers and forwarding data? What will the list you get look like?

There is a big difference between a forwarder phoning home to the deployment server, and a forwarder actually sending you the data you expect.

What problem are you trying to solve here?

Anyways... if you go to settings > searches, reports & alerts you can reuse what the monitoring console already has for searches like this reporting on forwarders

Then go to settings > advanced search > macros to fill in the macros the searches the MC uses

Here is an example search that the Monitoring Console uses to report on forwarders, this would tell you if they are connecting to indexers:

index=_internal sourcetype=splunkd group=tcpin_connections (connectionType=cooked OR connectionType=cookedSSL) fwdType=* guid=* 

This search is the base search MC uses to build it's forwarder table (yes, it already keeps a sort of state table!). If you configure the MC properly all you would need to to is work with it's lookup table:

| inputlookup dmc_forwarder_assets

Then you could just do a subsearch for your list of forwarders against this lookup.

You can also explore using index=_internal source=*splunkd.log deploymentserver and search against the connectionId

You would be looking for messages like this for your forwarders to see them reporting to your deployment server:

04-30-2017 13:05:43.186 +0000 INFO  PubSubSvr - Subscribed: channel=deploymentServer/phoneHome/default/reply/n00b-splkufw-01.n00blab.local/1B30BA3C-495C-4EAC-ADCF-7C12D3423078 connectionId=connection_10.10.31.210_8089_10.10.31.210_n00b-splkufw-01.n00blab.local_<GUID> listener=0x7f75fc5d5600

Anyways, i urge you to use the management console/DMC to set yourself up for success for managing your forwarders, whether it be this ask for checking 50-100 UFs a day, or the next questions that will need to be answered about your UF fleet.

[onkarkore1]

These are the random spot checks to make sure forwarders/hosts are up and they are phoning home.
We will pick random 50-100 forwarders and put these forwarders entries in one document.

Now we would like to design a search query which will read these forwarders entries and check status of each forwarders.

Each time we would like to hit search query which will take input of forwarder instances from document and return status of at least 50 forwarders input listed in document.

We expect output like, forwarder instance name and status (up or down)

[mattymo]

That is completely unnecessary and a will provide a poor level of assurance for your Splunk deployment and is not sustainable long term.

The Monitoring Console will monitor the status of ALL your forwarders (as well as the rest of your deployment) and provides alerts to tell you when forwarders are not reporting as well as many more insights on your UFs. It will be much more accurate/reliable and way less manual than random spot check of a small percent of your forwarder fleet. Best of all it will allow you to use your time for something more valuable!!

I urge you to review the documentation I have provided and rethink your strategy.

"Monitoring Splunk Enterprise". http://docs.splunk.com/Documentation/Splunk/6.5.3/DMC/DMCoverview

I work in multiple environments of 10K forwarders+, and would never support what you are suggesting as acceptable as an assurance strategy, especially knowing Splunk provides you better options out of the box.

[mattymo]

Now that I have done all I can to get you to do the right thing...

If you choose to continue down this path in spite of my begging you to use MC..you may as well do it right...

This is how I would provide this list until I could get the Monitoring Console stood up or convince whoever is asking for this why its a bad idea....Use the Lookup Editor app https://splunkbase.splunk.com/app/1724/ or build a lookup file with these instructions. https://docs.splunk.com/Documentation/Splunk/6.5.3/PivotTutorial/AddlookupfilestoSplunk

I called mine forwarders.csv and it simply had one column called hostname. You will need to ensure the hostnames you use in the csv actually match the hostnames you will see in Splunk....another complication to this approach if you are not careful..you can confirm the hostnames to expect by searching index=_internal sourcetype=splunkd group=tcpin_connections (connectionType=cooked OR connectionType=cookedSSL) fwdType=* guid=*

index=_internal sourcetype=splunkd group=tcpin_connections (connectionType=cooked OR connectionType=cookedSSL) fwdType=* guid=* [| inputlookup forwarders.csv | fields hostname]
| stats values(fwdType) as forwarder_type, latest(version) as version, values(arch) as arch, values(os) as os, max(_time) as last_connected by guid, hostname
| eval status = if(last_connected < (now() - 900), "missing", "active")
| eval last_connected=strftime(last_connected,"%m/%d/%y %H:%M:%S")
| fields guid, hostname, forwarder_type, version, arch, os, status, last_connected

I literally took this search directly from the monitoring console and pared it down to leave only what would be needed to satisfy your ask. Now, this search gives a "status" based on whether the forwarders have connected to indexers in the last 15 mins (900 seconds - change this value in the search if you wish to change the threshold)...not whether they have phoned the deployment server. If that is what you are after....change my search to this

| rest /services/deployment/server/clients 
| fields hostname lastPhoneHomeTime 
| eval status = if(lastPhoneHomeTime < (now() - 900), "missing", "active") 
| search 
    [| inputlookup forwarders.csv 
    | fields hostname]

Hope this helps....did I mention to you that you that you should use the Monitoring Console??? 😉

[onkarkore1]

This query is working as per the expectations thanks. However I have one more concern i.e.
We want to see the status of forwarders as "Not Found", For those forwarders which does not exist in enviornment or For those forwarders which log entries are not present in internal index.
This query is not returning status for those forwarders which does not exist or whose log entries are not available.

[mattymo]

probably warrants a new answers post.

[onkarkore1]

https://answers.splunk.com/answers/526398/search-query-to-check-selective-forwarders-phoning.html?mi...

[onkarkore1]

Here, you can post your answer 🙂