Another one of the hints you can look for in Splunk's documentation for when something is map-reducible is the phrase "sufficient statistics". Like Martin said, avg() is easily map-reduced by subtotaling count and sum, so a count and a sum are the sufficient statistics for average. The sufficient statistics for a dc() is the set of distinct values. The sufficient statistics for a standard deviation include a sum and a sum of squares. You'll also see similar sufficient statistics talk used in summary indexing (as the problem is very similar), and in the "prestats=true" output of the tstats command. There, you'll see fields beginning with named like prsrvd_* which are preserved sufficient statistics values for use in further aggregation down the pipe... ... View more

Wow, blast from the past from 2014. First of all, it is not possible to tell heavy forwarders NOT to parse events. If you don't want parsing, don't use a heavy - use a universal forwarder instead. Second of all, in 2016 the use of heavy forwarders as a parsing / routing layer is definitely NOT best practice. It can be made to work, but special care must be taken in the design to make sure that adding heavy forwarders does not create a situation where things are worse. The best practice today is to let indexers handle parsing and routing directly, and only resort to using heavies for this role in specific deployment scenarios. ... View more

So the thing you'll get from an ASA in debug mode is per-connection open and close events. These look like this: Dec 11 08:01:31 <IP> %ASA-6-302013: Built outbound TCP connection 447236 for outside:KAV_Update_Server/<port> (KAV_Update_Server/<port>) to dmz:OCSP_Server/<port> (OCSP_Server/<port>) Dec 11 08:01:31 <IP> %ASA-6-302013: Built outbound TCP connection 447236 for outside:KAV_Update_Server/<port> (KAV_Update_Server/<port>) to dmz:OCSP_Server/<port> (OCSP_Server/<port>) Dec 11 08:01:31 <IP> %ASA-6-302014: Teardown TCP connection 447236 for outside:KAV_Update_Server/<port> to dmz:OCSP_Server/<port> duration 0:00:00 bytes 14804 TCP FINs Dec 11 08:01:38 <IP> %ASA-6-302014: Teardown TCP connection 447234 for outside:KAV_Update_Server/<port> to dmz:TSP_Server/<port> duration 0:01:08 bytes 134781 TCP FINs Dec 11 08:01:38 <IP> %ASA-6-302014: Teardown TCP connection 447234 for outside:KAV_Update_Server/<port> to dmz:TSP_Server/<port> duration 0:01:08 bytes 134781 TCP FINs And some others - these are just examples. But, the point being, for forensic purposes these logs are invaluable. But, they do take up a lot of license quota - because of their verbosity and getting two events minimum for every connection through the firewall. The original question asked about the "best" logging level for ASAs, and I took that to mean "the one that provides the most useful forensic details available" ... View more

Given the change of state, I unaccepted my prior answer (that was years old) and marked ogdin's as correct. ... View more

Are you sure this is UDP syslog? It looks more like TCP. I would not be surprised if the HWF's Output Queues are becoming full, blocking, and dropping some data. ... View more

When you hit an assertion or other crash condition, and you are running on something that is not the latest patch level for the release you are on - update first, then seek help from the community and/or support. There have been 8 public releases of Splunk 6.3 over the last year since Splunk 6.3.0 originally dropped.. Currently Splunk 6.3.8 is the latest, and the change logs (http://docs.splunk.com/Documentation/Splunk/6.3.8/ReleaseNotes/6.3.8) show it has dozens of fixes put in cumulatively to solve issues found. The community is glad to help, but make sure you make the most of other people's time they contribute by attempting the easy fixes like upgrading first. ... View more

Note: This is NOT disabling SplunkWeb. A UF does not have SplunkWeb ever at all, period. This is disabling the REST API server. This has side effects on the UF, such that the CLI largely (but not completely) quite working. See Matt's .conf talk: http://conf.splunk.com/files/2016/recordings/universal-forwarder-security-dont-input-more-than-data-into-your-splunk-environment.mp4 http://conf.splunk.com/files/2016/slides/universal-forwarder-security-dont-input-more-than-data-into-your-splunk-environment.pdf ... View more

So to begin with - this is a software development effort - there is some documentation and some sample code available. I would start with http://docs.splunk.com/Documentation/Splunk/6.5.0/Security/ConfigureSplunkToUsePAMOrRADIUSAuthentication and read and understand the sample code. This is a several day effort at best and will require testing and such. ... View more

In a word, yes. Linux on ppc64 / ppc64le are officially supported targets for the Splunk UF. You can find these downloads for Splunk 6.5.0 on the current UF downloads page. You will need to know if your copy of Linux on Power is running in LE mode (little endian) or not, and pick the right binary to install. ... View more

Let's speak a bit more precisely to help explain. Say we have buckets A, B, C, and D and an SF=2 - so we could call the 4 buckets and their redundant copies A1, A2, B1, B2, C1, C2, D1, D2. Within the clustering system, one copy (and only one copy) of each bucket will be flagged as primary. We'll mark that as (P) . So one valid configuration is A1, A2(P), B1, B2(P), C1(P), C2, D1, D2(P). Splunk only ever searches primary buckets. Non-primary copies of buckets are ignored from a search point of view. The reason clustering has the concept of "primary buckets" is to avoid duplicate events during a search. If we use our "average session count" example above - by the time data gets back to the search head, the 'identity' of which bucket(s) on the indexer it came from is lost. And indexers do not have a way to cross-communicate to make sure they don't return duplicate events from different copies of the same bucket. So Splunk avoids the issue by only having one primary copy of each bucket, and only the primary is searched. Historically, Splunk had distributed search before it had clustering. Clustering provides redundancy, distributed search provides map-reduce. There is definitely an effect of "more indexers leads to increased search performance" but it has nothing at all to do with clustering - it is part of the nature of distributed search. The "performance effect" of having more indexers is not a function of the search factor at all - it is a function of the number of indexers and the number of buckets overall. By adding indexers, we cause forwarders to "spray wider" at data ingest time - this causes more buckets overall in the system. More buckets overall in the system still means that each bucket only has a single primary, searchable copy. But, with more buckets in the system overall, we have a situation where the primary copies of the larger number of buckets are spread out over more nodes. This gives each node a smaller amount of data to have to work through from a search perspective and therefore better performance. (The first time I wrote this was better, sorry) ... View more

This is actually a relatively common meme in searches. When you are thinking about state, think lookups. Here is a slight restatement of your problem definition - please let me know if it is accurate. You want to know the earliest value of _time for a given combination of client_ip and token . This will enable you to tell if a particular combination is 'new' relative to 'now'. I know your source data is slightly more complex, but let me approach it as simply as you have a series of events with fields _time, client_ip, and token . (Ignoring your spath and other complexity for the time being). Because Splunk stores time as a time_t , it's an integer and easily comparable. So let's start with: sourcetype=mylog | stats min(_time) as earliest, max(_time) as latest by client_ip, token This is relatively easy, and we can tack on some simple evaluation criteria onto the end to find 'really old' and/or 'really new' combinations, like: <search_above> | eval status = case( now() - earliest < 300 , "really_new", now() - latest > 86400 * 90, "really_old", 1=1, "normal") | search status = "really_new" This works, but requires you to search over a fairly large time range. We can compress that some by caching results in a lookup. There's a 3-step process to setting this up and using it. Step 1 - Generate the lookup the first time. Run this search over a long time period (like all time): sourcetype=mylog | stats min(_time) as earliest, max(_time) as latest by client_ip, token | outputlookup token_cache.csv Step 2 - Set up something to run over a shorter interval to maintain the lookup. Run this like every 5 minutes over the 5 minutes before. sourcetype=mylog | stats min(_time) as earliest, max(_time) as latest by client_ip, token | inputlookup append=true token_cache.csv | stats min(earliest) as earliest, max(latest) as latest by client_ip, token | outputlookup token_cache.csv What we've done here is make changes to our lookup incremental. We run the main data collection search over a smaller time window, but then suck in the cached data and smash it together. Because of how _time is an integer, the primitives of min() and max() make sense for updating the lookup in the most sane way possible. Step 3 - Set up something to use the cached data for alerting purposes. We've made a cached result now it should be easy to use. | inputlookup token_cache | where now() - earliest < 300 Searches 2 and 3 get scheduled to run - search 2 is a background maintainer and search 3 is an alert. Assuming I have properly understood what you were trying to do, this should be much cheaper to use and maintain over time. And as a bonus, you get an indication of when a given IP address stops using a given token too. ... View more

test, I wonder if my other comment got lost. Clearly it did I'll rewrite it. ... View more

For the total picture when it comes to configuring SSL, I would humbly pitch the talk that @starcher and I put together long ago and have given many times over now. The latest version was the April 2016 Splunk Trust virtual.conf -- http://wiki.splunk.com/Virtual_.conf Or you can get last year's .conf recordings -- http://conf.splunk.com/session/2015/recordings/2015-splunk-115.mp4 http://conf.splunk.com/session/2015/conf2015_DWaddle_DefensePointSecurity_deploying_SplunkSSLBestPractices.pdf ... View more

For the purpose of visualizing the problem, think of splunk results and fields in terms of "rows" and "columns". When you do a search, and do something like | stats max(foo) by bar then you get a new "row" for each value of bar, and a "column" for max(foo). If a particular value of bar does not have any related value of foo, then fillnull is perfectly appropriate. But, you can't fillnull what does not exist. Drawing an example, suppose you have a brick-and-mortar store and wanted to compute | stats sum(sales) by day_of_month for December. On December 25th the store was closed. The results of our stats cannot possibly have a row for "25". We cannot fillnull to make a result row that does not exist. Does this make sense? ... View more

EFS uses NFS v4 to mount on Linux hosts. Splunk generally does not recommend the use of NFS for hot buckets. See http://docs.splunk.com/Documentation/Splunk/6.4.3/Installation/Systemrequirements#Considerations_regarding_Network_File_System_.28NFS.29 It might do well for cold buckets. But even then I'm not really sure. From Amazon's FAQs on the product: All file systems deliver a consistent baseline performance of 50 MB/s per TB of storage, all file systems (regardless of size) can burst to 100 MB/s, and file systems larger than 1TB can burst to 100 MB/s per TB of storage. When you have multiple Splunk nodes trying to hit an EFS volume in parallel and you have an aggregate of 100MB/sec/TB across all of your indexers - you would need to do the math to be sure that given the size of your volume and number of indexers that you don't create a bottleneck. ... View more

The requirement is somewhat vague. When you say "Backup copies", is that implying something like (say) LTO tape? Or are you keeping your "backups" on disk? Once you reach LTO-4, LTO is incredibly easy to encrypt using native LTO encryption. If you are leaving everything on disk for your "backups" and performance is not a concern then you might be able to use something like LUKS + dm-crypt to make an encrypted filesystem. Or maybe you run a script that does a 'gpg' encryption of frozen data. You really haven't said a lot about your target environment so any one of these is as valid as any other. One of the important things about planning for encryption of data at rest is identifying what threats you wish to protect that data from. Like in the above examples, in the case of LTO-4 tape the encryption of the tape itself is a powerful way to be sure that if a tape is misplaced it is not trivially read. And an encrypted filesystem with LUKS + dm-crypt is great to protect disk drives attached to servers in the data center from being physically carried offsite and read. But, with the LUKS + dm-crypt approach, the operating system maintains the full ability to read the encrypted storage and decrypt it on behalf of the user. In terms of threat modeling, an OS-level encrypted disk provides almost no protection from a piece of malware that uses the OS features to read the disk on its behalf. You have to know what threats you are trying to protect your data from before you start trying to pick out what encryption systems are going to work for you. ... View more

If you go down the route @jkat54 is talking about with your trusty rusty chrome HTTP request debugger, it looks like it just does a POST of form data directly to the 'app' itself, in my case http://localhost:8000/en-US/manager/launcher/apps/local/search -- which is really not a REST endpoint, but likely a splunkweb controller of some kind. You can't just smash it easily with curl because it doesn't seem to understand HTTP basic auth. I estimate you'd need to hit the auth form and submit to get back a cookie and then hit the form up. ... View more

It helps you plenty. If you are running a Splunk CLI search via the Windows CMD prompt, then you need to understand the quoting rules IT uses. It's the same thing, only different. When you're using a command line shell to run a command - regardless of what the command line shell is - you are subject to how IT interprets and quotes input before it passes it along to the child process it is starting on your behalf. If you are not using CMD, but using PowerShell instead - it may have different quoting rules that you have to account for. A quick google search of "windows command prompt quoting rules" brings these resources: http://ss64.com/nt/syntax-esc.html https://blogs.msdn.microsoft.com/twistylittlepassagesallalike/2011/04/23/everyone-quotes-command-line-arguments-the-wrong-way/ http://stackoverflow.com/questions/7760545/escape-double-quotes-in-parameter http://stackoverflow.com/questions/24173825/what-does-single-quote-do-in-windows-batch-files ... View more

Trying to back-pedal from the example you gave from the CLI, I still think your issue is related to shell quoting. Going back to the updates I gave to your other question, https://answers.splunk.com/answers/449193/how-to-run-rex-commands-from-cli-mode.html#answer-450021 ... using set -x in the shell prompt helps you see better how shell expansion and quoting rules are affecting what is passed on to Splunk. Given what you show above, your most likely to be successful CLI command is probably: splunk search 'index=apps sourcetype=weblogs | rex field=_raw "Source:\s(?<Source>.*)" | search Source | dedup Source | table Source' We wrap the whole SPL command in single quotes ' because that is the environment where the shell does the LEAST amount of interpretation and replacement. From the bash man page: Enclosing characters in single quotes preserves the literal value ofeach character within the quotes. A single quote may not occur between single quotes, even when preceded by a backslash. ... View more

What search exactly are you running successfully in the search bar? Sounds like a quoting issue in the shell. ... View more