Update So with a high cardinality field where timechart won't work you can use a straight stats then xyseries. index=data | stats count by user, date | xyseries user date count Previous Answer Take a look at timechart. It creates this exact set of data but transposed; the fields are in the columns and the times in the rows. ... View more

Update 2 Ok, I've got everything worked out. So I've found all the environment variables I need to unset so that each version of Python can build using only host packages/libraries and not using things built into Splunk. I've also found that for this specific case (Ubuntu 14), the OpenSSL package in the standard repository is of insufficient version for Python 3.7 (required: 1.0.2, highest available: 1.0.1). So in order to resolve OP's specific issue three steps will be needed: Update PyDen to the latest release (1.2.1) Ensure the following packages are installed on the ubuntu box: build-essential libffi-dev libssl-dev Use Python version 3.6 or lower. Once completed, I've verified I can create new virtual environments and successfully install pandas to them. Let me know if you have any further issues and please accept if this solution worked. Update After some more digging I've discovered two issues at play. The first is what I discussed below is that the cause of failing to create new environments stems from pip being unable to use SSL because the correct OpenSSL library is unavailable on the host. There's a second issue at play when it comes to the libraries as well. When Splunk runs a Python script it creates a set of environment variables that point to some libraries bundled with the core software, one of which is its own OpenSSL. So even if you have the correct packages installed on the host they won't necessarily be used during the make process because of the Splunk environment variables. I'm working on a hotfix to remove those library references so the make process will default to the host shared libraries. Should be forth coming in a day or two. So after digging in to the issue (after getting the OS version from OP) it looks like the issue is most likely one of dependencies for Python compilation. Since PyDen downloads the Python source code and compiles it on the host directly, the host has to have the correct set of packages installed in order for the build to be successful. The behavior described by OP in slack conversation indicates that, while Python is compiling successfully, it isn't bundling pip properly. This is evidenced by the fact that the Python distribution gets created successfully but virtual environments fail to build. CPython compilation is built to be as fault tolerant as possible, so Python can compile successfully with a number of core modules missing. Which of these modules will be unavailable is dependent on the host libraries installed. For Ubuntu, you typically need at least build-essential, libffi-dev, and libssl-dev. But again, this varies by the exact version and I haven't yet found the minimum correct set of packages needed for the version. I'll keep working on finding the minimal set of packages, but in the mean time I'd suggest reviewing the script I use to provision my Vagrant boxes for development here. Try the packages I'm installing with APT and see if you're able to create new versions that can build virtual environments successfully. I use the virtualenv package to create the environments and it's installed through pip, so if you're able to build the environment you should be able to add packages through pip. I'll keep looking for a set of packages I know will work and will update when I find it. ... View more

Based on the scenario you've presented it sounds like you need to "throttle" your alerts until the metric you're measuring returns to normal levels, instead of a time-based throttling that Splunk provides out of the box. The bad news is this is not something you can achieve with Splunk throttling as both variants (once and per result) are governed by time-based approaches. However, you can do this using a lookup to track the metric you're measuring I'll use a trivial alert as an example. Suppose I create an alert based on a simple count going above a set threshold of 300. I'd define a search such as: your search here | stats count And then create an alert trigger like search count > 300 . Then we hit your problem of simple time-based alert throttling. For the more advanced case where we need the result to return to normal before we alert again, we'll need to track the results of each search to see when this happens. First step is to create a lookup, let's call it threshold and tie it to a CSV file with a count field and a row with a 0 to start. Next, we'll need to modify the alert logic of the search to do the following: add the previous count from the lookup set a trigger field with a boolean based on the current count and previous count output the current count to the lookup alter the alert logic to trigger off the trigger field boolean Here's an example: your search here | stats count | appendcols [| inputlookup threshold | rename count as previous_count | table count] | eval trigger=if(count>300 AND previous_count<=300, 1, 0) | fields count trigger | outputlookup threshold Then for the alert logic you'd set a custom alert condition to search trigger=1 . ... View more

The difference is between two versions of how to create a custom command in Splunk. The splunk.Intersplunk module is a relatively simple module baked into the Splunk Python distribution and is the version 1 style of creating custom commands that has been around since about Splunk 3.x. The splunklib package is the Splunk SDK for Python found here: http://dev.splunk.com/python. The SDK provides lots of tools for doing lots of things, including the new version 2 of how to create custom commands. There are lots of advantages to using the version 2 approach but that does come at the cost of bundling the SDK into your app. However, most agree it's worth it and it is considered best practice. There is an excellent .conf talk about the differences between the two here: https://conf.splunk.com/files/2017/slides/extending-spl-with-custom-search-commands-and-the-splunk-sdk-for-python.pdf. ... View more