Splunk Search

how can i list all indexes and sourcetypes?!

r999
Path Finder

i can do

| metadata type=sourcetypes |table sourcetype

but what i would like is the equivalent of:

| metadata type=sourcetypes index=* | table index sourcetype

however this does not work and does not enter data in the index column

How can i achieve this very simple list, preferably without using stats command

moh30ka
New Member

| tstats count WHERE index=* by index sourcetype | stats values(sourcetype) by index,| tstats count WHERE index=* by index sourcetype | stats values(sourcetype) by index

0 Karma

Dallastek
Explorer

When I have tried using the above tstats I don't get all of my indexes/sourcetypes. When I use | eventcount summarize=false index=* index=_* | dedup index | fields index | map maxsearches=100 search=" | metadata type=sourcetypes index=\"$index$\" | eval index=\"$index$\"" | stats values(sourcetype) by index

I get a list of all of them

Micheal_S
Explorer

To expand on this. I had an issue where if I did this empty indexes wouldn't show in my results. I used the following to work around that. 

| eventcount summarize=false index=* 
| search NOT index IN ( <indexes you don't want to include> )
| dedup index 
| fields index
| map maxsearches=100 search="|metadata type=sourcetypes index=\"$index$\" | eval index=\"$index$\""
| fields index sourcetype 
| append [| eventcount summarize=false index=* 
| search NOT index IN (
<indexes you don't want to include>
)
| dedup index 
| fields index] 
| fillnull value="No Known SourceTypes" 
|  stats count as mc values(sourcetype) as sourcetype by index 
| stats count values(mc) as mc by index sourcetype
| eval sourcetype = if(sourcetype=="No Known SourceTypes" AND mc>1, NULL,sourcetype) 
| dedup index sourcetype 
| fields index sourcetype 
0 Karma

esix_splunk
Splunk Employee
Splunk Employee

I think these solutions are overkill, and perhaps less efficient. Let's use tstats and go home early.. (its not the stats command.. 😛 )

| tstats values(sourcetype) where index=* group by index

cmeisch
Explorer

So this is a great search.  But I want to limit the search to a spacific index cluster.  So say I have indexers call na-idx01, na-idx02, na-idx03, ...  This is an idx cluster of a separate business unit that a SH is peering to as well as other BU's cluster.

splunk_server=na* is normally what I would do to only search at that BU.  How can I use the |tstats search but only for that cluster?

0 Karma

marius_gh
Observer

simple and efficient ..it works like a charm. thanks

0 Karma

hardik_d
Engager

with group not work for me:

used this:

| tstats values(sourcetype) where index=* by index

you can also try this one to get indexes based on provide sourcetype in query::

| tstats values(sourcetype) where index=* sourcetype="abc" OR sourcetype="xyz" by index

 

0 Karma

chutuo
Engager

Answer by esix [Splunk] should have been the selected answed and is actually best practice!

0 Karma

sjbriggs
Path Finder

For some reason, I get fewer results with tstats recommendation than I get with the first recommendation. I have one index that has 3 sourcetypes and with tstats, it only shows one of them.

0 Karma

jonuwz
Influencer

Does this involve any setup ? the docs indicate that you need to run tscollect to create the tsidx files that tstats uses. If my answer is out-dated, i'll remove it.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Our eventcount answers still are valid, though tstats can answer the same questions nowadays - no setup needed for indexed fields like sourcetype and index.

furby559
New Member

I downvoted this post because need to run this over all time for this to be accurate and is then significantly slower over larger data sets.

0 Karma

jagadeeshm
Contributor

so what did you end-up doing?

0 Karma

SarahBOA
Path Finder

We used tstats and we only run it on part of the data. We really wanted a list of which hosts send what sourcetype and source to what index. We run it on a small sampling of the data and collect it weekly and add it to our own lookup/csv to keep track.

0 Karma

mrjoshua050
New Member

This was a perfect answer exactly what I needed, and very fast.

0 Karma

tmerry
Explorer

I generally would prefer to use tstats (and am trying to get better with it!), but your string does not return all indexes and sourcetypes active in my environment. When I use this tstats search:

| tstats values(sourcetype) as sourcetype where index=* OR index=_* group by index

I get 19 indexes and 50 sourcetypes.

When i use the accepted answer (eventcount) i get 30 indexes and 295 sourcetypes.

I tried excluding index=_* from both searches and still saw a huge difference in the results. Any thoughts on why there is a discrepancy?

esix_splunk
Splunk Employee
Splunk Employee
index=*

Make sure you use that and not just index=, especially if you have search filters setup so that not all indexes are searched by default.

Regarding excluding index=_*, these are internal indexes for Splunk. Of course if you are skipping these and expecting them to be in the event count, then your numbers will be off.

tmerry
Explorer

Sorry, the asterisks were stripped out of my comment, but they were there when I did my comparison.

| tstats values(sourcetype) as sourcetype where index=* OR index=_* group by index

I added the internal indexes to your proposed tstats search to match the search string in the accepted answer above. If I remove them from both searches, I still see a major discrepancy in results.

0 Karma

jagadeeshm
Contributor

The discrepancy is due to the fact that tstats takes selected time period into consideration. So unless you select ALL TIME, you won't be seeing all indexes and sourcetypes.

martin_mueller
SplunkTrust
SplunkTrust

You can get a list of indexes like this:

 | eventcount summarize=f index=* index=_* | dedup index | fields index

See http://splunk-base.splunk.com/answers/39370/is-it-possibl-to-get-a-list-of-available-indices

Your search doesn't work because metadata does not contain any field "index". It does give you the list of sourcetypes though.

Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!