Deployment Architecture

Forwarder Management troubleshooting client errors- Where can I find the client errors?

dspyros
Engager

In forwarder management I get a message stating there are 6 clients with "DEPLOYMENT ERRORS" but cannot find the issue. Searched the _internal index but still do not see what the errors are.

Where can I find the client errors?

Labels (1)

jotne
Builder

Here is a dashboard I have made to find these types of error.

<form version="1.1" theme="dark">
  <label>Deployment status</label>
  <!--
  1.0
  1.1 change name 19.12.2019
  -->
  <search id="base_search">
    <query>
      index=_internal OR index=*_internal
      sourcetype=splunkd
      host="$Host$"
      name="$Server$"
      sc="$Stansa$"
      app="$App$"
      result="$Result$"
      action=Download
      | table _time host name sc app result
    </query>
  </search>
  <fieldset submitButton="false">
    <input type="time">
      <label></label>
      <default>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </default>
    </input>
    <input type="dropdown" token="Host">
      <label>Deployment server</label>
      <search base="base_search">
        <query>
          | eval data=host
          | stats count by data
          | eval info=data." (".count.")"
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>*</default>
    </input>
    <input type="dropdown" token="Server">
      <label>Server</label>
      <search base="base_search">
        <query>
          | rex field=name "bit_(?&lt;server&gt;[^_]+)"
          | eval data=name
          | stats count by data server
          | eval info=server." (".count.")"
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>*</default>
    </input>
    <input type="dropdown" token="Stansa">
      <label>Stansa</label>
      <search base="base_search">
        <query>
          | eval data=sc
          | stats count by data
          | eval info=data." (".count.")"
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>*</default>
    </input>
    <input type="dropdown" token="App">
      <label>Application</label>
      <search base="base_search">
        <query>
          | eval data=app
          | stats count by data
          | eval info=data." (".count.")"
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>*</default>
    </input>
    <input type="dropdown" token="Result">
      <label>Result</label>
      <search base="base_search">
        <query>
          | eval data=result
          | stats count by data
          | eval info=data." (".count.")"
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>Fail</default>
    </input>
  </fieldset>
  <row>
    <panel>
      <chart>
        <search base="base_search">
          <query>
            timechart count by name limit=10
          </query>
        </search>
        <option name="charting.chart">column</option>
        <option name="charting.chart.stackMode">stacked</option>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <table>
        <search base="base_search">
          <query>
            stats count by host name sc app result
            | sort result
            | rename host as "Deplyment server" name as Server sc as Stansa app as Application
          </query>
        </search>
        <option name="count">100</option>
        <format type="color" field="Deplyment server">
          <colorPalette type="sharedList"></colorPalette>
          <scale type="sharedCategory"></scale>
        </format>
        <format type="color" field="Server">
          <colorPalette type="sharedList"></colorPalette>
          <scale type="sharedCategory"></scale>
        </format>
        <format type="color" field="Stansa">
          <colorPalette type="sharedList"></colorPalette>
          <scale type="sharedCategory"></scale>
        </format>
        <format type="color" field="Application">
          <colorPalette type="sharedList"></colorPalette>
          <scale type="sharedCategory"></scale>
        </format>
        <format type="color" field="result">
          <colorPalette type="map">{"Fail":#DC4E41,"Ok":#53A051}</colorPalette>
        </format>
      </table>
    </panel>
  </row>
</form>

 

0 Karma

akocak
Contributor

This is my way of finding out who is that has issue:
1st , search this in deployment server:

index=_internal sourcetype=splunkd record (New OR Updating) result=Fail | head 100

You should be able to see name of the client along with application and server class.
you can get the system name of the server, by Settings > Forwarder Management > Clients Tab, then paste name of the client.

You could continue your troubleshooting from there.

dspencer
Engager

Hello, my deployment server shows 11 errors, however the query doesn't return any results and I have selected all time. Where would I go from here?

0 Karma

aferone
Builder

This works.  Thanks!

0 Karma

splunkreal
Motivator

Thanks!!! Splunk should implement this...

* If this helps, please upvote or accept solution if it solved *
0 Karma

whrg
Motivator

This answer greatly helped, thanks.

0 Karma

jensenh1999
New Member

This is one reason I am starting to NOT like Splunk many unanswered questions. I too am having this problem.

0 Karma

jlongworth
Explorer

run the search
index=_internal sourcetype=splunkd fail

The return will have information to narrow the search for the clients that have problems.

Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In December, the Splunk Threat Research Team had 1 release of new security content via the Enterprise Security ...

Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?

(This is the first of a series of 2 blogs). Splunk Enterprise Security is a fantastic tool that offers robust ...

Index This | What are the 12 Days of Splunk-mas?

December 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...