Splunk Search

How to design a data catalogue that captures all source types that are being ingested?

tirelana
Engager

I'm looking at designing a Splunk data catalogue that captures all source types (and metadata) that are currently being ingested, so that we can quickly see what the current state of the workspace. E.g. a customer who wants access to event X can use the catalogue to check that source type Y exists already. Has anyone done something similar to this or have suggestions? I'm quite new to Splunk but it seemed like it could be a common 'nice to have' for Splunk users. 

Thanks. 

Labels (1)
0 Karma
1 Solution

VatsalJagani
SplunkTrust
SplunkTrust

Use this search:

| metadata type=sourcetypes index=*

(Timerange picker matters, so run it in the same timerange as what you would consider all the sourcetypes being included. like last 7 days or last 30 days)

 

| metadata type=sourcetypes index=* | rename totalCount as Count firstTime as "First Event" lastTime as "Last Event" recentTime as "Last Update" | fieldformat Count=tostring(Count, "commas") | fieldformat "First Event"=strftime('First Event', "%c") | fieldformat "Last Event"=strftime('Last Event', "%c") | fieldformat "Last Update"=strftime('Last Update', "%c")

for additional formatting.

View solution in original post

VatsalJagani
SplunkTrust
SplunkTrust

Use this search:

| metadata type=sourcetypes index=*

(Timerange picker matters, so run it in the same timerange as what you would consider all the sourcetypes being included. like last 7 days or last 30 days)

 

| metadata type=sourcetypes index=* | rename totalCount as Count firstTime as "First Event" lastTime as "Last Event" recentTime as "Last Update" | fieldformat Count=tostring(Count, "commas") | fieldformat "First Event"=strftime('First Event', "%c") | fieldformat "Last Event"=strftime('Last Event', "%c") | fieldformat "Last Update"=strftime('Last Update', "%c")

for additional formatting.

tirelana
Engager

Great thanks. I'll have a read through the docs. 

0 Karma

bowesmana
SplunkTrust
SplunkTrust

Before you jump in and design one, take a look at TrackMe

https://splunkbase.splunk.com/app/4621/

I have written a data catalog that does similar to what you are looking to do, with the aim of being able to find out about data and ownerships. However, I think that TrackMe, with some additional dashboards that provide a query functionality onto what it captures would be pretty easy to do.

I believe there are other apps out there, but this one is open source and the dev is super helpful. 

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...