topic Re: identifying sourcetypes by index in Getting Data In

identifying sourcetypes by index

cphair — Tue, 14 Jan 2014 19:00:07 GMT

Hello,

I'd like to display all sourcetypes available for each index in my environment. Unfortunately, metadata type=sourcetypes doesn't preserve the index name, and I want to be able to run it on the entire set of indexes on whatever instance the search runs on (i.e. I don't want to hardcode index=a OR index=b, etc, into the search). I tried getting fancy:



| rest /services/data/indexes | rex field=id mode=sed "s/.\/(\w+)$/\1/" | search id!="_" | fields id | map search="|metadata type=sourcetypes index=$id$ | stats list(sourcetype) as sourcetype | eval whereFrom=$id$ | table sourcetype whereFrom"

...but the second $id$ is always null. Can anyone give me a good way to list all indexes and the sourcetypes they contain? As a bonus, if you can explain why my map command doesn't work as expected, I'd appreciate it.

Re: identifying sourcetypes by index

theouhuios — Tue, 14 Jan 2014 19:10:33 GMT

earliest=-5m@m latest=@m index=*|dedup sourcetype|table index sourcetype

I always use this to list all sourcetypes and there respective indexes.

Re: identifying sourcetypes by index

cphair — Tue, 14 Jan 2014 19:11:42 GMT

That only works if all the sourcetypes have appeared in the past five minutes, though.

Re: identifying sourcetypes by index

theouhuios — Tue, 14 Jan 2014 19:14:59 GMT

You can increase the timerange to one hour and try it. It will just take extra time to give you the result. Will depend on your splunk environment on how much time it will take for search to complete

Re: identifying sourcetypes by index

cphair — Tue, 14 Jan 2014 19:15:43 GMT

Understood, but my indexes are huge. I would prefer not to search the data itself if I can avoid it--this is the kind of problem that metadata should solve.

Re: identifying sourcetypes by index

somesoni2 — Tue, 14 Jan 2014 20:18:44 GMT

Try this

|metasearch index=* sourcetype=* | stats count by index, sourcetype | fields - count

Re: identifying sourcetypes by index

cphair — Tue, 14 Jan 2014 20:25:13 GMT

Perfect. Thank you.

Re: identifying sourcetypes by index

koshyk — Wed, 27 May 2015 14:16:44 GMT

you will get all sourcetypes faster if you do
| metadata type=sourcetypes index=*

Re: identifying sourcetypes by index

wrangler2x — Thu, 03 May 2018 23:21:46 GMT

This REST search works great, and it is fast, too. I lists all sourcetypes by index and the associated event count:

|rest /services/data/indexes count=0
| dedup title
| fields title
| map  [|metadata type=sourcetypes index="$title$"
        | eval type="$title$"] maxsearches=1000
| stats values(totalCount) AS EventCount values(sourcetype) AS Sourcetype by type
| rename type as index
| fields index Sourcetype EventCount

Re: identifying sourcetypes by index

sojnv — Sun, 01 Jul 2018 09:16:49 GMT

Thanks alot, This is more accurate and truly fast

Re: identifying sourcetypes by index

MuS — Sun, 01 Jul 2018 20:02:52 GMT

Hi there,

another one would be tstats which is lighting fast, because it does not look at any _raw data :

| tstats count WHERE index=* by sourcetype, index

cheers, MuS

Re: identifying sourcetypes by index

sojnv — Mon, 02 Jul 2018 04:13:17 GMT

Thanks MuS

Re: identifying sourcetypes by index

wrangler2x — Mon, 02 Jul 2018 17:09:42 GMT

I use that as well, but I like the way this looks better:

| tstats values(sourcetype) AS Sourcetype where index=* by index

And yes, as MuS said, this is lightening fast. Run that and then compare against |metasearch index=* sourcetype=* | stats count by index, sourcetype | fields - count which is way slow by comparison.

Re: identifying sourcetypes by index

sojnv — Thu, 05 Jul 2018 03:46:01 GMT

so true, thanks

Re: identifying sourcetypes by index

sdewar83 — Wed, 11 Sep 2019 02:22:31 GMT

do you have to do this search against "All Time" ? i tried running it and got completely different results when searching ALL Time vs 15 minutes.