I'm looking at designing a Splunk data catalogue that captures all source types (and metadata) that are currently being ingested, so that we can quickly see what the current state of the workspace. E.g. a customer who wants access to event X can use the catalogue to check that source type Y exists already. Has anyone done something similar to this or have suggestions? I'm quite new to Splunk but it seemed like it could be a common 'nice to have' for Splunk users.
Thanks.
Use this search:
| metadata type=sourcetypes index=*
(Timerange picker matters, so run it in the same timerange as what you would consider all the sourcetypes being included. like last 7 days or last 30 days)
| metadata type=sourcetypes index=* | rename totalCount as Count firstTime as "First Event" lastTime as "Last Event" recentTime as "Last Update" | fieldformat Count=tostring(Count, "commas") | fieldformat "First Event"=strftime('First Event', "%c") | fieldformat "Last Event"=strftime('Last Event', "%c") | fieldformat "Last Update"=strftime('Last Update', "%c")
for additional formatting.
Use this search:
| metadata type=sourcetypes index=*
(Timerange picker matters, so run it in the same timerange as what you would consider all the sourcetypes being included. like last 7 days or last 30 days)
| metadata type=sourcetypes index=* | rename totalCount as Count firstTime as "First Event" lastTime as "Last Event" recentTime as "Last Update" | fieldformat Count=tostring(Count, "commas") | fieldformat "First Event"=strftime('First Event', "%c") | fieldformat "Last Event"=strftime('Last Event', "%c") | fieldformat "Last Update"=strftime('Last Update', "%c")
for additional formatting.
Great thanks. I'll have a read through the docs.
Before you jump in and design one, take a look at TrackMe
https://splunkbase.splunk.com/app/4621/
I have written a data catalog that does similar to what you are looking to do, with the aim of being able to find out about data and ownerships. However, I think that TrackMe, with some additional dashboards that provide a query functionality onto what it captures would be pretty easy to do.
I believe there are other apps out there, but this one is open source and the dev is super helpful.