topic Re: Search sourcetypes by forwarders in Getting Data In

Search sourcetypes by forwarders

johnchamp — Mon, 16 Dec 2013 08:13:28 GMT

I need to collect list of sourcetypes for each forwarder using search query. i can get forwarders list from metics.log and sourcetype list from licence_usage log seperately from _internal index.
Is there any way to get all the sourcetypes configured from each forwarder using single search query?

Re: Search sourcetypes by forwarders

lguinn2 — Mon, 16 Dec 2013 12:16:37 GMT

Sort of - if the host name matches the forwarder name:

index="_internal" source="*license_usage.lo*" type!=*Summary 
| stats sum(b) as bytes by h st 
| rename h as host st as sourcetype 
| eval MB=round(bytes/(1024*2014),1) | fields - bytes

Hopefully, there will be some correlation between this and

index="_internal" source="*metrics*" group=tcpin_connections 
| stats sum(kb) as kb by hostname 
| eval MB =round(kb/1024,1) 
| rename hostname as host 
| fields - kb

If you wanted to combine these, you could do something like this

index="_internal" source="*license_usage.lo*" type!=*Summary 
| stats sum(b) as bytes by h st 
| rename h as host st as sourcetype 
| eval MB=round(bytes/(1024*2014),1) | fields - bytes
| join host type=outer [ search index="_internal" source="*metrics*" group=tcpin_connections 
    | stats sum(kb) as kb by hostname 
    | eval totalMBforwarded =round(kb/1024,1) 
    | rename hostname as host 
    | fields status host totalMBforwarded]

Any result that has a "totalMBforwarded" was a match. If the host name is being reset on the forwarder, you could also create a lookup table to define a mapping that could also be used in this search.

Re: Search sourcetypes by forwarders

lguinn2 — Mon, 16 Dec 2013 12:30:38 GMT

BTW - totalMVforwarded may always be larger than the sum of the sourcetypes from a host.

Re: Search sourcetypes by forwarders

johnchamp — Mon, 16 Dec 2013 19:15:50 GMT

Thanks for you help. In my case host names are not matching with forwarders. I need to work on lookup table to do the mapping.

Re: Search sourcetypes by forwarders

somesoni2 — Mon, 16 Dec 2013 20:05:45 GMT

Another option could have been using |metasearch command

|metasearch host=* sourcetype=* | stats count by host, sourcetype

Here host name is the name of forwarder using which data is been indexed into splunk.

Re: Search sourcetypes by forwarders

lguinn2 — Mon, 16 Dec 2013 23:31:10 GMT

That gives a count of the events - and it is very fast - but it still has the problem that it doesn't map back to the actual forwarder name.