Sourcetypes - list of where they're being used?

[sheltomt]

We're migrating from a stand-alone production instance to a clustered environment. As such, we're moving applications over one at a time and testing as we go.

We've come across an app that apparently supports numerous other apps, through field extractions, shared sourcetypes, etc.

We're looking for ways to audit our sourcetypes, and figure out where all they're being used.

Has anyone done this before?

[somesoni2 · Accepted solution]

We do this kind of analysis typically in the case where we're renaming or retiring a sourcetype. We look for saved object where the sourcetype is used, e.g. saved searches (alerts, reports), dashboards, macros, eventtypes etc. Here are the queries that you can use to see if your sourcetype is used in different KOs (knowledge objects) in Splunk.

Query1 (macros)

| rest splunk_server=local /servicesNS/-/-/properties/macros | table id eai:acl.owner | rename eai:acl.owner  as owner | fillnull value="-"  owner | map maxsearches=10000 search="| rest splunk_server=local $id$/definition | eval id=\"$id$\" | eval owner=\"$owner$\"" | where match(value,"YourSourceTypeHere\:") | table id | rex field=id ".+\/(?<search>.+)$" | table search | eval search="search=*\"`".search."`\"*"

Query2 (eventtypes)

| rest /servicesNS/-/-/saved/eventtypes splunk_server=local | search search="*YourSourceTypeHere*"| table title | eval search="search=\"*eventtype*=*".title."*\"" | table search

Query3 (Saved searches)

| rest splunk_server=local /servicesNS/-/-/saved/searches | table title eai:acl.app search eai:acl.owner | rename eai:acl.owner as owner | where match(search,"YourSourceTypeHere") 

Query4 (Dashboards/Forms)

| rest splunk_server=local /servicesNS/-/-/data/ui/views | table title eai:acl.app  eai:data eai:acl.owner| rename eai:data as code eai:acl.owner as owner | where match(code,"YourSourceTypeHere")

Now, there may be people who use the sourcetype in ad-hoc queries (not saved). You can query audit logs to query those. Note that audit logs are limited by retention period on _audit index and may not have all historical data. Also below query gives result only for adhoc searches where sourcetype is referred directly. If sourcetype is used in a macro or eventtype, it won't show here. Adjust the search=... clause accordingly to find those usage.

index=_audit action=search (search="*sourcetype*=*YourSourceTypeHere:*")  user!="splunk-system-user" | timechart span=1d count as "#Searches" dc(user) as "#Users"

[jlarsonq]

For the first query for the Macros I had to add a rex command to get it to work | rest splunk_server=local /servicesNS/-/-/properties/macros | table id eai:acl.owner | rename eai:acl.owner as owner | fillnull value="-" owner | rex field=id mode=sed "s/https:\/\/127\.0\.0\.1:8089//" | map maxsearches=10000 search="| rest splunk_server=local $id$/definition | eval id=\"$id$\" | eval owner=\"$owner$\"" | where match(value,"YourSourceTypeHere*") | table id | rex field=id ".+\/(?<search>.+)$" | table search | eval search="search=*\"".search."\"*"

[sheltomt1]

Thank you! This is the info I was looking for!

[skoelpin]

This is a really good overview. Thanks @somesoni2

[skoelpin]

Check out Settings > Fields > Field Extractions. You can enter your sourcetype in the search and it will bring back all fields to that sourcetype.

You could also run this search..

index=_* sourcetype=splunkd 
| stats count values(user) AS user values(action) AS app by series 
| rename series AS sourcetype