Everyone knows Splunk is a powerful platform for running searches and doing data analytics. Your users know this, and therefore love using Splunk for setting up all kind of queries. This is great and creates real value! However, over time this can create performance problems on the platform, especially when “bad” searches are running continuously. This article will give you some specific tips on how to find the worst bad searches in your Splunk environment, and how to fix them.  Key Takeaways  Cost and Resource Benefits: Cleaning bad searches frees resources, which leads to reduced costs, better performance, and more happy users.  Focused Housekeeping: A small portion of searches typically consumes most resources. Fix the worst offenders to quickly improve your Splunk environment.  Optimization Mindset: Inefficient searches can often be improved with better query design and smarter commands, like tstats.  Dependency Awareness: Scheduled searches can power other alerts, reports, and dashboards. Always check for dependencies before making changes.  Housekeeping; boring but necessary  Usually, this is how the story goes: Your Splunk users create all kinds of demanding scheduled searches. This stuff builds up over time and steals valuable resources from your Splunk platform. As a rule of thumb, we say that 20% of the searches use 80% of the resources (the Pareto principle). This means that finding and fixing the “worst” 20% of scheduled searches releases plentiful of resources, you do not need to fix all the searches. Housekeeping might seem boring, but it makes your platform happy, which leads to happy users, which leads to happy Splunking!  What are “bad” searches?  There are a lot of factors that make a search “bad”. In this case we are looking at “bad” regarding performance and are focusing on the following key issues.  Searches that have long (accumulated) run times → Steals CPU  Searches that use a lot of memory → Steals RAM  Searches that run on schedule time peaks → Causes skipped searches  Searches that create huge search artifacts → Causes replication issues  Note that there are a lot more issues one can investigate, but the tips presented in this article should give you a good starting point. Also note that searches can be bad regarding security as well, but that’s a whole other topic.  Watch out for dependencies  Before you start editing and deleting bad searches, be aware of dependencies. Scheduled searches are usually alerts or scheduled reports. Of course, users can be dependent on these reports and alerts, so check with the search owner before making changes! Also, note that other searches and dashboards can be dependent on scheduled searches. Such dependencies commonly are created with commands such as “savedsearch” or “loadjob”. Make sure that you don’t break anything in use! Before you change a search, use REST to look for dependencies for that specific search.  Here is a REST command to find dashboard dependencies:  | rest /servicesNS/-/-/data/ui/views | search eai:data="*<search name>*"  Here is a REST command to find search dependencies:  | rest /servicesNS/-/-/saved/searches | search search="*<search name>*" And with that, without further ado, let's find some bad searches!  Find searches with high CPU usage  Searches with the biggest accumulated run times use the most CPU on your Splunk platform. Accumulated run time is the product of how long a search takes to run and how often it runs.   To find the searches with the highest accumulated CPU usage:  index=_audit action=search search_type=scheduled | stats sum(total_run_time) as run_time by savedsearch_name | sort limit=100 - run_time To improve these slow searches, make sure they are using good search practise. Also, if you need more optimization, you can consider using tstats, TERM, and PREFIX.  Find searches with high memory usage  Certain commands in Splunk are very memory intensive, e.g. sort, join, or transaction.  To find the searches with the highest memory usage:  | rest /services/search/jobs | stats sum(diskUsage) by label eai:acl.app | sort limit=100 - sum(diskUsage) To reduce memory usage, try re-writing the searches to use less memory demanding commands, and fix any unnecessary memory usage (like a sort before a stats).  Find searches with huge search artifacts  Searches with efficient run times can be the cause of huge needless search artifacts. These artifacts lead to more resource usage used for search head cluster replication. Use the same search as above for memory usage, or look under “Jobs” in Splunk web, to find the searches with the biggest artifacts. Note that the REST command doesn't differentiate between memory usage (as in RAM) and disk usage (for storing artifacts).  To find the searches with the biggest search artifacts:  | rest /services/search/jobs | stats sum(diskUsage) by label eai:acl.app | sort limit=100 - sum(diskUsage) To reduce the size of job artifacts, make sure only the fields needed are returned from the search (using table), and for lookup generating searches, add a “stats count” after the outputlookup command (don’t need to store the results as a job artifact in addition to storing it in the lookup).  Find peaks in search run times  Many searches running simultaneously creates resource usage peaks and can lead to skipped searches. Try to distribute searches evenly over time (using cron). Also, don’t run searches on the hour or on the tens, those slots are dedicated to acceleration and other out-of-the-box search activity.  To find the time slots with the biggest number of simultaneous searches:  | rest /servicesNS/-/-/saved/searches earliest_time=-7d | search is_scheduled=1 disabled=0 | eval scheduled_minute=strftime(scheduled_times,"%M") | stats count values(title) by scheduled_minute | sort limit=100 - count Find big, outdated CSV lookups  Users sometimes create huge CSV lookups. Such lookups make the searches using them slow. Also, they lead to more resource usage for search bundle replication.  Linux example command to find the biggest CSV lookups on your environment:  find /opt/splunk/ -n "*.csv" -exec stat -c "%n %.10y %s" {} +  For lookups that are not in use, consider deleting them. If so, remember to disable any lookup generating searches, so that the lookup is simply not re-created. If the lookups are needed, consider making the CSV lookup into a KV store instead.  Final words  These are just a few tips for housekeeping in your Splunk environment. If you haven’t looked for bad searches before in your environment, you are sure to catch some big bad fish! There are of course several apps on Splunkbase addressing this topic, which you can look into, but they often require some setup beyond just installing. If you need to get started and do some quick cleaning, and ad-hoc searching is often best.  About the Author  Martin Hettervik, Senior Consultant and Team Leader at Accelerate Oslo, Splunk MVP   LinkedIn: https://www.linkedin.com/in/martinhettervik/  ... View more

The Foundation of Data: A Practitioner's Guide to Splunk Source Types  Even if you are new to Splunk, you should be somewhat familiar with source types. The idea of source types is one of the first thing you learn about in your Splunk journey. That being said, the concept and its applications might be somewhat hard to fully grasp. With this blog post I’ll try to answer some common questions about source types.  Key Takeaways Technical Core: The concept of source types is to classify logs into defined data structures, and standardize the way these data structures are treated.  Efficiency Engine: They are a key component in parsing, field extractions and CIM-normalization. Stuff that makes your Splunk fast and friendly.  Search Strategy: Source types are default indexed fields that gives an effective way to optimize query performance and allows the use of commands like tstats.  Technical Definition: What is a Source Type Exactly?  Technically a source type is just a default indexed field. Normally the source type field is determined at data ingest, but it can be rewritten later in the pipeline as well. That the field is default means that every log event in Splunk has this field, and that the field is indexed means that the field is stored in the index (as opposed to being extracted at search time). The picture below shows examples of source types, as shown in Splunk web (note the other default indexed fields as well, listed on the left).  Run a search in Splunk to see source types (the actual field name being “sourcetype”). They are pre-selected as an interesting field in the GUI. Conceptual Overview: Classifying Your Log Formats  Conceptually, the source type is supposed to classify the data structure of a group of log events, or in other words, specify the log format. So, for example, you might have a group of sources (e.g. log files) that are sending events to Splunk in JSON format. Then, the common source type for all these events should be JSON. Said in a different way, the source type tells you what the events are. The other default fields “source” and “host” tells you where the events are.  Backend Mechanics: How Splunk Uses Source Types Under the Hood  Rules for parsing and field extractions are defined per source type. This means that you can use source types to decide how different log streams should be treated. Spending some time setting well-defined logical source types gives a better end user experience in Splunk, and better performance utilization for the Splunk platform as a whole.  Configuration Guide: Defining Your Source Types in props.conf  All source types are defined in the configuration file props.conf. The file can be edited directly, or you can add and edit source types through the Splunk web UI. What works best is dependent on your environment and situation. Each source type has its own stanza in props.conf, defining the source type name, parsing rules and field extractions.  Strategy: What Makes a "Well-Defined" Source Type?  A source type that specifies efficient parsing rules and user-friendly field extractions is a good start. If the field extractions are also following the naming standards as defined in the Common Information Model (CIM), used for data normalization and data models, that makes a quite well-defined source type. Make sure to look into the Splunk Great Eight for super-efficient parsing rules. Note that defining source types requires some manual work, but the payoff makes it worth it.  Naming Conventions: Building a Consistent Hierarchy  There is not a single common consensus to how source types should be named, but having a consistent naming convention within your environment is beneficial. Commonly you’ll see source type names using a colon separated hierarchy system, where you “build” the name from “general” to “specific”. See the screenshot above for examples, like "aws:cloudwatchlogs:vpcflow". One benefit from this system, besides being easy to read, is that you can add a wildcard after any colon to search that subset of source types, for example "aws:cloudwatchlogs:*" or "aws:*".  Deployment Logic: When to Split into Different Source Types  When to split a source type? That’s a good question. To add to the JSON example mentioned earlier, you might have two different systems logging JSON events. Even though the log format is the same (JSON), the fields used by the systems might be totally different. This means that rules for field extraction and normalization need to be customised for each system, and thus, each system needs its own source type. You could define the source type names as something like “json:this” and “json:that”, each with its own field extraction rules.  When You Should NOT Split Source Types  If you have different systems, logging essentially the same type of events, but with minor variances, it might not be smart to split all these log streams into different source types. Say that you want to change a field extraction rule for all these log streams, you would have to update all the different source types, one-by-one, every time. In this case a common source type might be a better option. You’ll have to decide what is best case by case.  Search Optimization: How Users Benefit from Source Types  Users can specify a source type field in a query to easily find and filter events. For example, by searching for source type “WinEventLog”, they’ll find events from Windows Event Log. Since the source type field is an indexed field, it gives good search efficiency when specified. Also, note that indexed fields allow for the usage of tstats, which can be used to create super-efficient searches (see example below).  tstats is a highly optimized way to search your data. You can use tstats on indexed fields, but also on other “terms” that exists in your data (see my other blogpost if you are interested). Understanding exactly how this works takes a bit of practice and learning, but combine this knowledge with a set of well-defined source types, and you can create all sorts of Splunk tstats magic.  More source type searching tips  Also, since source type is a special default field, not just any indexed field, you can use the metadata command as well. This command lets you search on bucket metadata for source type information (not even touching the logs on disk). It’s uses are limited, but when it works, it grants huge efficiency benefits. One common use of metadata is to efficiently catch stopped data streams, e.g. see when logs of a certain log format (source type) suddenly stop being ingested to Splunk. This could indicate a failure in the data pipeline somewhere, that needs to be investigated.  The metadata command can be used to find source types that haven’t sent data to Splunk in more than a day. Another special use of the source type field, is to analyse license usage. In Splunk there is a license usage log, which also has some prebuilt views in the Monitoring Console. This log can be split by source type, meaning that you can use this field to identify which “classes” of sources are using the most of your license. The better and more consistent you’ve defined your source types, the better values you can get from this license analysis.  About the Author Martin Hettervik, Senior Consultant and Team Leader at Accelerate Oslo, Splunk MVP  LinkedIn: https://www.linkedin.com/in/martinhettervik/  ... View more

By Martin Hettervik, Senior Consultant and Team Leader at Accelerate at Iver, Splunk MVP The stats command is commonly used and well known. The tstats command is also well known, but for many only associated with searching data models. What if I told you that tstats can be used to gain the performance benefits of searching in tsidx files, without even using accelerated data models? In this post, we’ll explore how to super-optimize your Splunk searches using tstats, TERM, PREFIX, and a deeper understanding of how Splunk handles data under the hood. How do stats work, and why it’s not always optimal The stats command works by retrieving raw data from the indexers (stored in compressed journal files), performing relevant field extractions at search time, discarding the leftover raw data, and formatting a table. The flow is illustrated in the diagram below.   As an example, you could search WinEventLog and create a table of most common computer names. In the picture below, the raw data that you would have so dig through is on the left, while the table you end up with is on the right. As you can see, most of the data in the raw event is not needed. Searching through all that unnecessary text creates longer search times and is not always optimal. What alternatives do we have for optimization? There are some commonly used tricks in Splunk for creating faster searches. Let’s have a quick look at some of them, and pros and cons for each. Accelerated data model Good if your search can utilize data that maps to standard CIM data models Can be rebuilt, if changes in data or fields Complex to create a new data model for a small single use case, e.g. a single dashboard Add continuous resource usage 24/7 Summary index Easy to understand and set up Not flexible for later changes in data or fields Add continuous resource usage 24/7 Accelerated report Easy to set up Not reusable for other searches Add continuous resource usage 24/7   One thing in common with all options above, is that they add continuous resource usage on Splunk. They run regular background jobs to create “summarized” data. This is not a problem if several searches can use the same summarized data, or if the searches are expected to run often, so that it’s worth it performance wise. However, for a single dashboard or report that is only expected to be used occasionally, the cost in resource usage could become higher than the value added by the search optimization. To avoid this continuous cost, can we use tstats directly on the data? Yes. The tstats command only searches in indexed metadata (tsidx files), not raw data. Even so, you don’t necessarily need indexed fields or an accelerated data model to use tstats, it depends on the minor and major breakers in the raw data. Learn how to use TERM and PREFIX together with tstats for amazing results. Let’s look into it! Enter tsidx files and breakers The tsidx files contains indexed fields and segmented keywords (term) from the raw data. The keywords points to the events in the raw data where the fields or keywords are present. This mechanic is used by Splunk under the hood by for search optimization. The diagram below illustrates how the pointers work. What segmented keywords ends up in the tsidx file is dependent on major and minor breakers in the raw data. Major breakers separate keywords. Examples on major breakers are space, tab, brackets and quotation marks. Minor breakers create different segment combinations from keywords. Examples on minor breakers are period, slash, colon and the equal sign. You can look up all the breakers in the Splunk documentation. As an example, se the event below, that shows how a key value pair in the raw data is stored as keywords in the tsidx file (notice the minor breakers). So that means we can use tstats directly raw data? In some cases, yes! Here is where the magic of TERM comes into play. Use TERM to encapsule keywords in tsidx files containing minor breakers. So, instead of writing a normal stats search like this: index=dc_logs source=WinEventLog:Security ComputerName=win-hp-25431 | stats count Try writing a tstats search like this: | tstats count where index=dc_logs source=WinEventLog:Security TERM(ComputerName=win-hp-25431) Note how we do not need to use TERM on the fields index and source, since they are already indexes fields. For the computer name keyword, we need to encapsule the key value pair in TERM because it contains minor breakers. How about if you wanted to create a timechart? The timechart command has the property that it adds empty time slots to the results (useful in visualizations), which is omitted by using simply stats. Can we re-create this effect with tstats? Yes. Instead of writing a normal timechart search like this: index=dc_logs source=WinEventLog:Security ComputerName=win-hp-25431 | timechart span=1d count Try writing a tstats search like this: | tstats count where index=dc_logs source=WinEventLog:Security TERM(ComputerName=win-hp-25431) by _time span=1d | timechart span=1d count Sweet! How about if we want to count by a field? Is this also possible with tstats? Indeed, we can use something called PREFIX. Instead of writing a stats search like this: index=dc_logs source=WinEventLog:Security | stats count by ComputerName Try writing a tstats search like this: | tstats count where index=dc_logs source=WinEventLog:Security by PREFIX(computername=) (Note that PREFIX always uses lowercase in the encapsulation.) Super nice, but when will using tstats not work? This trick by using tstats combined with TERM and PREFIX will not work if there are major breakers in the key value pairs in the raw events. Look at the example below. Note that the logon account is separated between key and value by a space (major breaker). We will therefore not find the key value pair “logon account: pburdytt2i” in the tsidx file. We will however find the username “pburdytt2i”, but cannot know the key context in which the use name exists. Unfortunately, this tstats trick only works if the data is formatted in the “correct” way, but when it does work, the efficiency boost is huge! Final notes Optimizing your Splunk searches with tstats, TERM, and PREFIX can dramatically improve performance and reduce resource usage in Splunk. While not every search can be converted, understanding how Splunk segments and stores data gives you some powerful knowledge to make faster and more optimized queries. If you haven’t done so already, have a look at your dashboards, reports and alerts, and see which searches you can super optimize. Happy splunking! ... View more

These dashboards are part of an app i made, to visualize Nessus security scans i Splunk. The idea is somewhat inspired by the existing Tenable App for Splunk from Tenable, but I wanted to to take the visualizations to the next level, and make the data easier to understand and navigate. The first dashboard is an overview dashboard. The picture below does not show the whole dashboard, but you get the point. It shows data from all vulnerability scans, with color coding differentiating the level of vulnerability severity. It's an easy way of seeing which environments and hosts have the most vulnerabilities, and see which type of vulnerabilities are most widespread. Also note that it shows what period there are scan data from (which might not be the same as the time picker) and how many networks have been scanned (out of the total number of networks). This next picture shows one of many drilldown dashboards in the app. It allows for a more detailed view of vulnerabilities per host, and also the possibility to get more information about a specific host if you click on the top table. This table also uses the same color coding as the overview dashboard. The bottom table links directly to the Tenable website, with more information about the specific vulnerability ID clicked on. All dashboards allows for various types of filtering, or example only show vulnerabilities with a minimum severity, e.g. at least medium. The dashboards are also utilizing the Splunk ES asset list to get more information about the hosts, so that it's possible to sort on vulnerabilities per business group or environment, among other things. Also, there is a lookup of "ignored vulnerabilities", for which the users can add vulnerabilities to ignore them in the dashboards, e.g. by editing it in the Splunk App for Lookup File Editing. Summary of functionality used in the dashboards: Color coding of vulnerability severity Drilldowns to other dashboards with more detailed information Drilldowns to external URLs with information on severity IDs Various filtering options on the dashboards Host enrichment from Splunk ES asset list Dynamic whitelisting of vulnerabilities through lookup file Correlation with other sources to show meta-information about vulnerability scans ... View more