Splunk Enterprise

Unable to find the source of Uberagent logs indexing in main index

juhiacc
Explorer

Hi 

We had UberAgent apps installed in Splunk environment and recently we deleted the apps along with the index. We see that due to index deletion , data is getting in main index from very few servers/devices. But not sure where this data is coming from since we have removed the UberAgent apps from everywhere. Any suggestions where should we be looking at to find the source? There are no related HEC tokens OR scripts that is to be found.

Warm Regards !

Labels (1)
Tags (2)
0 Karma

defection-io
Explorer
Hi @juhiacc 

You can do some snooping around in the `_internal` index to see if you can figure out where the data is coming from.

I'm not sure what sourcetype UberAgent uses, but if we assume it's `uberagent`, you can run the following search to get some more info about the origin of the data (just replace `uberagent` with the correct sourcetype):

```
index=_internal sourcetype=splunkd component=Metrics group=per_sourcetype_thruput series="uberagent"
```

In the results that return, you should be able to see all of the hosts that have processed data for this sourcetype. Depending on your environment, you may see multiple hosts in the `host` field, but you should be able to determine which hosts are intermediate steps (like a Heavy Forwarder or Indexer) and which hosts are the original source.
 
From there, you can investigate the hosts `inputs.conf` to see if there are any hints as to where the data is coming from. Sometimes, the `source` field of the data might also indicate where the data is coming from. For example, if the `source` is a file path, it's almost certainly coming from a file monitor input. But it looks like you may have already checked this.
 
There is also a chance that it was data indexed in the past with future timestamps. But since you mentioned that you deleted the index, this is unlikely the case. New data needs to be indexed for it to start appearing in the `main` index now.
 
If none of that helps, let me know and we can try some other things. Good luck!

juhiacc
Explorer

Thanks @defection-io  for responding. The query is returning hosts which are basically our Indexers. We had config files in Indexers taht was removed as part of removing config files from Splunk environment.  Regarding the source column , it is /opt/splunk/var/log/splunk/metrics.log so not of much help. 


0 Karma
Get Updates on the Splunk Community!

A Prelude to .conf25: Your Guide to Splunk University

Heading to Boston this September for .conf25? Get a jumpstart by arriving a few days early for Splunk ...

4 Ways the Splunk Community Helps You Prepare for .conf25

.conf25 is right around the corner, and whether you’re a first-time attendee or a seasoned Splunker, the ...

Enhance Your Splunk App Development: New Tools & Support

UCC FrameworkAdd-on Builder has been around for quite some time. It helps build Splunk apps faster, but it ...