Determine actual forwarder host

[laurie_gellatly]

Hi,
I've taken on an existing Splunk environment that has had some non-standard things happen to it.
In the process of bringing it back to standard I found that some data was appearing in the default index 'main'.
First thing I found was that apps existed on the UF that the Deployment Server didn't know about and they were sending some of this data. I've fixed that but still data is arriving in main.
When I check on any host UF that is appearing in main, there are no enabled inputs.
The only sourcetypes appearing in main are:
ActiveDirectory
Perfmon:Available Memory
Perfmon:CPU Load
Perfmon:Free Disk Space
Perfmon:Network Interface
WinEventLog:Application
WinEventLog:Security
WinEventLog:System
so it looks like windows apps running somewhere, that the DS does not know about, are setting the host field for the incoming data.
How can I find the actual forwarder so I can stop these logs?

Thanks ...Laurie:{)

[JDukeSplunk]

I took over a Splunk environment where many of the windows hosts did not report to the deployer and as such they ended up with whatever app set was copied/cloned to them.

I use the metrics log to get at least a list of hosts that talk to the indexers with this search. Which I think I got from Answers somewhere.

Forwarder list search.

index=_internal source=*metrics.log* group=tcpin_connections | regex hostname!="\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}" | eval sos_server=hostname | stats latest(sourceIp) AS IP latest(arch) AS cpu_arch latest(fwdType) AS forwarder_type latest(os) AS os_name latest(version) AS version  by sos_server

Anyway, I also came up with a dashboard that breaks down sourcetypes by host by index and source. It's not pretty but I've found it useful

<form>
  <label>Splunk Sourcetypes and Hosts</label>
  <description></description>

  <fieldset autoRun="false" submitButton="true">
    <input type="multiselect" token="hostgroup">
      <label>Host</label>
      <choice value="*">All</choice>
      <prefix>(</prefix>
      <suffix>)</suffix>
      <valuePrefix>host=</valuePrefix>
      <delimiter> OR </delimiter>
      <fieldForLabel>host</fieldForLabel>
      <fieldForValue>host</fieldForValue>
      <search>
        <query>|tstats count WHERE index=* by host |table host | eval _sortfield=lower(host) | table host _sortfield | sort _sortfield</query>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </search>
      <default>*</default>
      <initialValue>*</initialValue>
    </input>
    <input type="multiselect" token="idxs" searchWhenChanged="false">
      <label>Indexes</label>
      <search>
        <query>| eventcount summarize=false index=* | dedup index | fields index</query>
        <earliest>-15m</earliest>
        <latest>now</latest>
      </search>
      <fieldForLabel>index</fieldForLabel>
      <fieldForValue>index</fieldForValue>
      <choice value="*">All</choice>
      <prefix>(</prefix>
      <suffix>)</suffix>
      <delimiter> OR </delimiter>
      <valuePrefix>index=</valuePrefix>
      <default>application</default>
      <initialValue>application</initialValue>
    </input>
    <input type="multiselect" token="sourcetypes" searchWhenChanged="false">
      <label>Sourcetypes</label>
      <search>
        <query>|tstats count WHERE index=* by sourcetype |table sourcetype | eval _sortfield=lower(sourcetype) | table sourcetype _sortfield | sort _sortfield</query>
        <earliest>@d</earliest>
        <latest>now</latest>
      </search>
      <fieldForLabel>sourcetype</fieldForLabel>
      <fieldForValue>sourcetype</fieldForValue>
      <choice value="*">All</choice>
      <prefix>(</prefix>
      <suffix>)</suffix>
      <delimiter> OR </delimiter>
      <valuePrefix>sourcetype=</valuePrefix>
      <default>*</default>
    </input>
  </fieldset>
  <row>
    <panel>
      <table>
        <title>Hosts by Sourcetype</title>
        <search>
          <query>| tstats count WHERE $idxs$ by index host sourcetype   |search $sourcetypes$ AND ($hostgroup$) |fields sourcetype,host | stats values(host) AS hosts by sourcetype</query>
          <earliest>@d</earliest>
          <latest>now</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="count">50</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="percentagesRow">false</option>
        <option name="rowNumbers">false</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
      </table>
    </panel>
    <panel>
      <table>
        <title>Sourcetypes by Host</title>
        <search>
          <query>| tstats count WHERE $idxs$ by index host sourcetype |search $sourcetypes$ AND ($hostgroup$)|
 fields sourcetype,host | stats values(sourcetype) AS sourcetype by host</query>
          <earliest>@d</earliest>
          <latest>now</latest>
        </search>
        <option name="count">50</option>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <title>Sourcetype by filename Source and Index</title>
      <table>
        <search>
          <query>| tstats count WHERE $idxs$ by index host sourcetype source  |search $sourcetypes$ AND ($hostgroup$)  | eval host=upper(host) | eval sourcedate=replace(source,"\d\d+","_DATE_") | eval minimal=replace(sourcedate,"ch-(.*)-","ch_USERNAME")|stats values(host) AS hosts values(index) AS index by sourcetype minimal</query>
          <earliest>@d</earliest>
          <latest>now</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="count">100</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="percentagesRow">false</option>
        <option name="rowNumbers">false</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
      </table>
    </panel>
  </row>
</form>

[rjteh_splunk]

If you have access to search, you can potentially find the actual forwarder with this search.

| metasearch index=main sourcetype=<your_sourcetype_here_or_put_*> host=* | stats list(sourcetype) by host

or

index=main NOT (host=sh* OR host=idx*) sourcetype=<your_sourcetype_here_or_put_*> | stats list(sourcetype) by host

[laurie_gellatly]

As I don’t have access to the uf I sent it a run once batch file that does a dir of etc/apps and sends the output to be indexed

[FrankVl]

You should also take a look at the config files in etc/system/local, not just etc/apps. Or even better: have the batch script execute the btool command, to dump all input config on that UF.

[laurie_gellatly]

Turns out that doing the btool with --debug was a good idea. Thanks.

[whrg]

How are you checking for which inputs are enabled? I suggest you use btool for that: "splunk btool inputs list"