Deployment Architecture

Forwarder Management troubleshooting client errors- Where can I find the client errors?

dspyros
Engager

In forwarder management I get a message stating there are 6 clients with "DEPLOYMENT ERRORS" but cannot find the issue. Searched the _internal index but still do not see what the errors are.

Where can I find the client errors?

Labels (1)

jotne
Communicator

Here is a dashboard I have made to find these types of error.

<form version="1.1" theme="dark">
  <label>Deployment status</label>
  <!--
  1.0
  1.1 change name 19.12.2019
  -->
  <search id="base_search">
    <query>
      index=_internal OR index=*_internal
      sourcetype=splunkd
      host="$Host$"
      name="$Server$"
      sc="$Stansa$"
      app="$App$"
      result="$Result$"
      action=Download
      | table _time host name sc app result
    </query>
  </search>
  <fieldset submitButton="false">
    <input type="time">
      <label></label>
      <default>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </default>
    </input>
    <input type="dropdown" token="Host">
      <label>Deployment server</label>
      <search base="base_search">
        <query>
          | eval data=host
          | stats count by data
          | eval info=data." (".count.")"
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>*</default>
    </input>
    <input type="dropdown" token="Server">
      <label>Server</label>
      <search base="base_search">
        <query>
          | rex field=name "bit_(?&lt;server&gt;[^_]+)"
          | eval data=name
          | stats count by data server
          | eval info=server." (".count.")"
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>*</default>
    </input>
    <input type="dropdown" token="Stansa">
      <label>Stansa</label>
      <search base="base_search">
        <query>
          | eval data=sc
          | stats count by data
          | eval info=data." (".count.")"
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>*</default>
    </input>
    <input type="dropdown" token="App">
      <label>Application</label>
      <search base="base_search">
        <query>
          | eval data=app
          | stats count by data
          | eval info=data." (".count.")"
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>*</default>
    </input>
    <input type="dropdown" token="Result">
      <label>Result</label>
      <search base="base_search">
        <query>
          | eval data=result
          | stats count by data
          | eval info=data." (".count.")"
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>Fail</default>
    </input>
  </fieldset>
  <row>
    <panel>
      <chart>
        <search base="base_search">
          <query>
            timechart count by name limit=10
          </query>
        </search>
        <option name="charting.chart">column</option>
        <option name="charting.chart.stackMode">stacked</option>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <table>
        <search base="base_search">
          <query>
            stats count by host name sc app result
            | sort result
            | rename host as "Deplyment server" name as Server sc as Stansa app as Application
          </query>
        </search>
        <option name="count">100</option>
        <format type="color" field="Deplyment server">
          <colorPalette type="sharedList"></colorPalette>
          <scale type="sharedCategory"></scale>
        </format>
        <format type="color" field="Server">
          <colorPalette type="sharedList"></colorPalette>
          <scale type="sharedCategory"></scale>
        </format>
        <format type="color" field="Stansa">
          <colorPalette type="sharedList"></colorPalette>
          <scale type="sharedCategory"></scale>
        </format>
        <format type="color" field="Application">
          <colorPalette type="sharedList"></colorPalette>
          <scale type="sharedCategory"></scale>
        </format>
        <format type="color" field="result">
          <colorPalette type="map">{"Fail":#DC4E41,"Ok":#53A051}</colorPalette>
        </format>
      </table>
    </panel>
  </row>
</form>

 

0 Karma

akocak
Contributor

This is my way of finding out who is that has issue:
1st , search this in deployment server:

index=_internal sourcetype=splunkd record (New OR Updating) result=Fail | head 100

You should be able to see name of the client along with application and server class.
you can get the system name of the server, by Settings > Forwarder Management > Clients Tab, then paste name of the client.

You could continue your troubleshooting from there.

aferone
Builder

This works.  Thanks!

0 Karma

realsplunk
Motivator

Thanks!!! Splunk should implement this...

0 Karma

whrg
Motivator

This answer greatly helped, thanks.

0 Karma

jensenh1999
New Member

This is one reason I am starting to NOT like Splunk many unanswered questions. I too am having this problem.

0 Karma

jlongworth
Explorer

run the search
index=_internal sourcetype=splunkd fail

The return will have information to narrow the search for the clients that have problems.

Get Updates on the Splunk Community!

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...