How to design a data catalogue that captures all source types that are being ingested?

[tirelana]

I'm looking at designing a Splunk data catalogue that captures all source types (and metadata) that are currently being ingested, so that we can quickly see what the current state of the workspace. E.g. a customer who wants access to event X can use the catalogue to check that source type Y exists already. Has anyone done something similar to this or have suggestions? I'm quite new to Splunk but it seemed like it could be a common 'nice to have' for Splunk users. 

Thanks. 

[VatsalJagani · Accepted solution]

Use this search:

| metadata type=sourcetypes index=*

(Timerange picker matters, so run it in the same timerange as what you would consider all the sourcetypes being included. like last 7 days or last 30 days)

| metadata type=sourcetypes index=* | rename totalCount as Count firstTime as "First Event" lastTime as "Last Event" recentTime as "Last Update" | fieldformat Count=tostring(Count, "commas") | fieldformat "First Event"=strftime('First Event', "%c") | fieldformat "Last Event"=strftime('Last Event', "%c") | fieldformat "Last Update"=strftime('Last Update', "%c")

for additional formatting.

[tirelana]

Great thanks. I'll have a read through the docs. 

[bowesmana]

Before you jump in and design one, take a look at TrackMe

https://splunkbase.splunk.com/app/4621/

I have written a data catalog that does similar to what you are looking to do, with the aim of being able to find out about data and ownerships. However, I think that TrackMe, with some additional dashboards that provide a query functionality onto what it captures would be pretty easy to do.

I believe there are other apps out there, but this one is open source and the dev is super helpful.