Deployment Architecture

Forwarder Management troubleshooting client errors- Where can I find the client errors?

dspyros
Engager

In forwarder management I get a message stating there are 6 clients with "DEPLOYMENT ERRORS" but cannot find the issue. Searched the _internal index but still do not see what the errors are.

Where can I find the client errors?

Labels (1)

jotne
Builder

Here is a dashboard I have made to find these types of error.

<form version="1.1" theme="dark">
  <label>Deployment status</label>
  <!--
  1.0
  1.1 change name 19.12.2019
  -->
  <search id="base_search">
    <query>
      index=_internal OR index=*_internal
      sourcetype=splunkd
      host="$Host$"
      name="$Server$"
      sc="$Stansa$"
      app="$App$"
      result="$Result$"
      action=Download
      | table _time host name sc app result
    </query>
  </search>
  <fieldset submitButton="false">
    <input type="time">
      <label></label>
      <default>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </default>
    </input>
    <input type="dropdown" token="Host">
      <label>Deployment server</label>
      <search base="base_search">
        <query>
          | eval data=host
          | stats count by data
          | eval info=data." (".count.")"
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>*</default>
    </input>
    <input type="dropdown" token="Server">
      <label>Server</label>
      <search base="base_search">
        <query>
          | rex field=name "bit_(?&lt;server&gt;[^_]+)"
          | eval data=name
          | stats count by data server
          | eval info=server." (".count.")"
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>*</default>
    </input>
    <input type="dropdown" token="Stansa">
      <label>Stansa</label>
      <search base="base_search">
        <query>
          | eval data=sc
          | stats count by data
          | eval info=data." (".count.")"
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>*</default>
    </input>
    <input type="dropdown" token="App">
      <label>Application</label>
      <search base="base_search">
        <query>
          | eval data=app
          | stats count by data
          | eval info=data." (".count.")"
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>*</default>
    </input>
    <input type="dropdown" token="Result">
      <label>Result</label>
      <search base="base_search">
        <query>
          | eval data=result
          | stats count by data
          | eval info=data." (".count.")"
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>Fail</default>
    </input>
  </fieldset>
  <row>
    <panel>
      <chart>
        <search base="base_search">
          <query>
            timechart count by name limit=10
          </query>
        </search>
        <option name="charting.chart">column</option>
        <option name="charting.chart.stackMode">stacked</option>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <table>
        <search base="base_search">
          <query>
            stats count by host name sc app result
            | sort result
            | rename host as "Deplyment server" name as Server sc as Stansa app as Application
          </query>
        </search>
        <option name="count">100</option>
        <format type="color" field="Deplyment server">
          <colorPalette type="sharedList"></colorPalette>
          <scale type="sharedCategory"></scale>
        </format>
        <format type="color" field="Server">
          <colorPalette type="sharedList"></colorPalette>
          <scale type="sharedCategory"></scale>
        </format>
        <format type="color" field="Stansa">
          <colorPalette type="sharedList"></colorPalette>
          <scale type="sharedCategory"></scale>
        </format>
        <format type="color" field="Application">
          <colorPalette type="sharedList"></colorPalette>
          <scale type="sharedCategory"></scale>
        </format>
        <format type="color" field="result">
          <colorPalette type="map">{"Fail":#DC4E41,"Ok":#53A051}</colorPalette>
        </format>
      </table>
    </panel>
  </row>
</form>

 

0 Karma

akocak
Contributor

This is my way of finding out who is that has issue:
1st , search this in deployment server:

index=_internal sourcetype=splunkd record (New OR Updating) result=Fail | head 100

You should be able to see name of the client along with application and server class.
you can get the system name of the server, by Settings > Forwarder Management > Clients Tab, then paste name of the client.

You could continue your troubleshooting from there.

aferone
Builder

This works.  Thanks!

0 Karma

splunkreal
Motivator

Thanks!!! Splunk should implement this...

* If this helps, please upvote or accept solution 🙂 *
0 Karma

whrg
Motivator

This answer greatly helped, thanks.

0 Karma

jensenh1999
New Member

This is one reason I am starting to NOT like Splunk many unanswered questions. I too am having this problem.

0 Karma

jlongworth
Explorer

run the search
index=_internal sourcetype=splunkd fail

The return will have information to narrow the search for the clients that have problems.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...