What are some good methods for identifying dependencies between knowledge objects in Splunk?

[mattness]

This issue comes up when you need to delete an obsolete or duplicate tag, event type, transaction, or similar knowledge object. It also turns up in situations where you find you need to change the app context of an object (to change it from the Search app to the *nix app, for example).

If you have a tag that is used in an event type that appears in a saved search that in turn is the basis for a dashboard panel...how do you identify those dependencies and deal with them before deleting the tag?

Solid use case examples would be a great help here, but "best practices" advice would be useful as well.

[jonaclough]

I've written a python script dependency tracker (below) which is really just a fancy search tool. It's designed to run on the Splunk server and run from the directory from where you want to track.

It recurses from the current folder looking for a search term in a .conf file. If it finds it it will return the filename and the KO name in which the search term occurs.

"""
Script which searches from local directory and finds KOs which include a search term
Filters searches files by .conf extension. Starts searching from present working directory
Can be run from any linux host with Splunk config files.

It works by scanning .conf files from pwd. For each file it gets a list of all line numbers with a match. It then
gets a list of all KOs (using pattern match: [some ko]). For each search match it will get the KO match at or above
(i.e. lower than) the search match line number. It will then return a list of files and KOs.

Was designed to be used by python 3.7.1-1

usage:
#ssh to Splunk host
cd $SPLUNK_HOME/etc/apps #apps only
cd $SPLUNK_HOME/etc/ #apps and user config
/path/to/python3.7.x /path/to/dependency_tracker.py --search_term whatever
"""

import argparse
import sys
import os
import glob
import re

def get_parser():
    """
    Build an Argument Parser.

    Returns:
      parser: An argument parser
    """
    parser = argparse.ArgumentParser()
    parser.add_argument("--search_term", help="The name of the KO which is the dependecny", required=True)
    return parser



def get_matching_files(search_term):
    #Use glob to recursively search for .conf files from pwd
    pwd = os.getcwd()
    search_path = '{}/**/*.conf'.format(pwd)
    regex_searchterm = re.compile(search_term)
    regex_ko = re.compile("\[(.*)\]")
    matches = []
    for filepath in glob.iglob(search_path, recursive=True):
        matching_lines = []
        file_kos = []
        for i, line in enumerate(open(filepath)):
            searchterm_matches = re.findall(regex_searchterm, line)
            if searchterm_matches != []:
                matching_lines.append(i)
            ko_matches = re.match(regex_ko, line)
            if ko_matches:
                kos = {"ko_name" : ko_matches.group(1), "ko_line" : i}
                file_kos.append(kos)
        if matching_lines != []:
            matching_kos_in_file = []
            for matching_line in matching_lines:
                for i in range(len(file_kos)):
                    is_match = False
                    if i == len(file_kos)-1:
                        is_match = True
                    else:
                        next_ko_line = file_kos[i+1]["ko_line"]
                        if next_ko_line >= matching_line:
                            is_match = True
                    if is_match:
                        matching_ko = file_kos[i]["ko_name"]
                        if matching_ko not in matching_kos_in_file:
                            matching_kos_in_file.append(matching_ko)
            file_match = {"filepath": filepath, "kos": matching_kos_in_file}
            matches.append(file_match)
    return matches

def main(args=None):
    parser = get_parser()
    args = parser.parse_args()
    matching_files = get_matching_files(search_term=args.search_term)
    print("Searching for {}".format(args.search_term))
    for matching_file in matching_files:
        print("File {} contains following KOs:".format(matching_file["filepath"]))
        for matching_ko in matching_file["kos"]:
            print(matching_ko)

if __name__ == "__main__":
    main()

[sideview]

stupid answer but honestly i usually just grep through $SPLUNK_HOME/etc to find dependencies. to answer little questions 'does anything actually use this savedsearch', or 'is there other view config and savedsearches config here that could benefit from a macro' etc..

But for a while we did have an instance we set up where we would index various splunk conf files and report on them. it would index each stanza as an event and each key as a kv pair etc.
There were definitely some drawbacks but we didnt spent that much time on it (grepping etc is fast)

And we werent monitoring the config live. If we were doing it live we'd have indexed with fschange and probably we would have had to do a little work at the search language level to only report on the most recent versions of everything. It's quite possible we or someone else could put together a little app with some dependency-finder tools like a macrofinder, eventtypefinder, savedsearchfinder, tagfinder etc..

[mattness]

I think a "dependency finder" app or add-on would be great, if you couldn't somehow work the functionality directly into Manager.

[mattness · Accepted solution]

For now it appears there's no easy way to manage this through Splunk Web. Until a better internal device comes along, the best way to identify dependencies with Splunk (if you haven't been keeping track of them with a spreadsheet or similar method--see bfaber's answer) is to search on the knowledge object in question in order to find the knowledge objects that are dependent on it, and then search on those objects to find their dependencies. It could take a bit of detective work, depending on how interconnected your knowledge objects are.

For example, if you have a tag named "roger1" you might search on it and find that it is used in three event types. You would then search on those event types to find the knowledge objects that use them--saved searches, other event types, etc. Along the way you might keep track of the dependent knowledge objects so you can repair the dependent objects if you decide to remove/rename "roger1."

If you want to delete a knowledge object but you're not sure whether you've tracked down all of its downstream dependencies, you could try disabling it first to see what impact that has. If nothing seems to go seriously awry after a day or so, delete it (or ask the object's owner to do so).

For more on the subject of knowledge object management via Splunk Web, see "Curate Splunk knowledge with Manager" in the Knowledge Manager Manual.

[bfaber]

MattNess: My deployment isn't too complex, but what I have been doing is this: I create a dictionary of eventtypes and tags that we have used by application. Before I roll a dashboard into production, I note any dependencies from that dashboard into the dictionary spreadsheet.

This works fine as I am the only developer, but I am not sure it would scale so well.

On my wishlist would be an automated way to manage all of this internally -- but dependencies would have to understand the difference between my prod dashboards and my dev/test ones.

HTH

[mattness]

Seems like as good an approach as any to manage dependencies as they develop at this point. See my answer for the official word on tracking down dependencies if you haven't been keeping track of them already.