Forwarder Management troubleshooting client errors- Where can I find the client errors?

[dspyros]

In forwarder management I get a message stating there are 6 clients with "DEPLOYMENT ERRORS" but cannot find the issue. Searched the _internal index but still do not see what the errors are.

Where can I find the client errors?

[jotne]

Here is a dashboard I have made to find these types of error.

<form version="1.1" theme="dark">
  <label>Deployment status</label>
  <!--
  1.0
  1.1 change name 19.12.2019
  -->
  <search id="base_search">
    <query>
      index=_internal OR index=*_internal
      sourcetype=splunkd
      host="$Host$"
      name="$Server$"
      sc="$Stansa$"
      app="$App$"
      result="$Result$"
      action=Download
      | table _time host name sc app result
    </query>
  </search>
  <fieldset submitButton="false">
    <input type="time">
      <label></label>
      <default>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </default>
    </input>
    <input type="dropdown" token="Host">
      <label>Deployment server</label>
      <search base="base_search">
        <query>
          | eval data=host
          | stats count by data
          | eval info=data." (".count.")"
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>*</default>
    </input>
    <input type="dropdown" token="Server">
      <label>Server</label>
      <search base="base_search">
        <query>
          | rex field=name "bit_(?&lt;server&gt;[^_]+)"
          | eval data=name
          | stats count by data server
          | eval info=server." (".count.")"
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>*</default>
    </input>
    <input type="dropdown" token="Stansa">
      <label>Stansa</label>
      <search base="base_search">
        <query>
          | eval data=sc
          | stats count by data
          | eval info=data." (".count.")"
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>*</default>
    </input>
    <input type="dropdown" token="App">
      <label>Application</label>
      <search base="base_search">
        <query>
          | eval data=app
          | stats count by data
          | eval info=data." (".count.")"
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>*</default>
    </input>
    <input type="dropdown" token="Result">
      <label>Result</label>
      <search base="base_search">
        <query>
          | eval data=result
          | stats count by data
          | eval info=data." (".count.")"
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>Fail</default>
    </input>
  </fieldset>
  <row>
    <panel>
      <chart>
        <search base="base_search">
          <query>
            timechart count by name limit=10
          </query>
        </search>
        <option name="charting.chart">column</option>
        <option name="charting.chart.stackMode">stacked</option>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <table>
        <search base="base_search">
          <query>
            stats count by host name sc app result
            | sort result
            | rename host as "Deplyment server" name as Server sc as Stansa app as Application
          </query>
        </search>
        <option name="count">100</option>
        <format type="color" field="Deplyment server">
          <colorPalette type="sharedList"></colorPalette>
          <scale type="sharedCategory"></scale>
        </format>
        <format type="color" field="Server">
          <colorPalette type="sharedList"></colorPalette>
          <scale type="sharedCategory"></scale>
        </format>
        <format type="color" field="Stansa">
          <colorPalette type="sharedList"></colorPalette>
          <scale type="sharedCategory"></scale>
        </format>
        <format type="color" field="Application">
          <colorPalette type="sharedList"></colorPalette>
          <scale type="sharedCategory"></scale>
        </format>
        <format type="color" field="result">
          <colorPalette type="map">{"Fail":#DC4E41,"Ok":#53A051}</colorPalette>
        </format>
      </table>
    </panel>
  </row>
</form>

[akocak]

This is my way of finding out who is that has issue:
1st , search this in deployment server:

index=_internal sourcetype=splunkd record (New OR Updating) result=Fail | head 100

You should be able to see name of the client along with application and server class.
you can get the system name of the server, by Settings > Forwarder Management > Clients Tab, then paste name of the client.

You could continue your troubleshooting from there.

[aferone]

This works.  Thanks!

[splunkreal]

Thanks!!! Splunk should implement this...

[whrg]

This answer greatly helped, thanks.

[jensenh1999]

This is one reason I am starting to NOT like Splunk many unanswered questions. I too am having this problem.

[jlongworth]

run the search
index=_internal sourcetype=splunkd fail

The return will have information to narrow the search for the clients that have problems.