This is about the best I could have asked for in this situation. From what I can tell, the new AOB license is almost straightforward enough for a layperson to understand it, and it enables me to do what I think I need to with my derivative works. And all of the 3rd party code is (seemingly) clearly laid out there. I really appreciate the efforts and the clarity brought here. ... View more

The updates to the docs are a really great start. However, I think "more is needed" because I did a quick scan of the minified common.js hairball that AOB ships and I see jQuery and a bunch of other things inside the hairball. Similar to @sideview's concerns I'm not sure that I feel comfortable with Splunk's standard software license terms being attached to something I'm building. That puts me in a potential pickle if I either open-source my app, or try to commercially license it. Granted, I would hope that Splunk would not have an incentive to litigate an ecosystem partner that is releasing apps built on Add-On Builder. However, it would be an expensive thing on my part to deal with. Finally, are there plans to ensure that when AOB-integrated 3rd party code does get updated due to a security issue that there is a public advisory within the Splunk Security Portal so that we know to come repackage our TAs to pick up the updates? Honestly, truly, not trying to kill AOB in a pile of oversight, but I'm deeply concerned about using it for shipping product. ... View more

You really can't. Splunk does not expose how things were matched. Now, for debugging purposes, if you want to be freakishly clever, you can do something like this: (?<common_name>(?<unique1>a+))|(?<common_name>(?<unique2>b+)) Assuming Splunk has the regex library configured to allow for duplicate subpattern names in a single regex (and I assume they do, but don't know this for a fact), then you could extract the field named common_name either as "a+" or "b+" -- but in the "a+" case we would also extract "unique1", and in the "b+" case we would also extract "unique2". This is taking advantage of some oddities of the regex engine. Fun fact: This is also a semi-reasonable approach to replacing some instances of FIELDALIAS and some edge cases for EVAL ... View more

Echoing @rich7177 and @woodcock and adding a little more detail. (A) it's a compressed file. You have to uncompress it to read it. Sometimes less will silently decompress it for you. (B) Even if you do uncompress it, there's going to be binary data in it. Splunk's journal file has your raw events along with binary metadata about them. What you'll see is clumps of plain text event, surrounded by clumps of binary. That is just how it is, because that is how Splunk stores the data. ... View more

So, are the above your whole authentication.conf? There's some things missing. Like I would expect to see: [authentication] authSettings = lassie_np_ldap authType = LDAP Or something similar .. also a [rolemap_lassie_np_ldap] section that specifies the role mappings. As @SloshBurch said prior, "use btool" 🙂 ... View more

Wish I could upvote this three or four times. ... View more

This seems .. unlikely .. I would suspect something unanticipated in your editing of the file. I have edited authentication.conf many times directly via the CLI and had no issues like this, especially after a restart. (It is possible to reload most authentication.props properties w/o a restart via the REST API but that is a little off topic for here) Authentication.conf (like all other conf files) is subject to btool's merging rules across apps. Make certain you are editing the right authentication.conf file(s) to affect your change. I think more information is needed (like a before and after of the file itself) before we can give great guidance on what went wrong. ... View more

Hey Glenn, agreed. If THP is disabled, then defrag on or off is meaningless. Splunk's health check code is misreporting the combination of enabled=never, defrag=always as an issue when it is not, and there is a bug opened to resolve that. Good catch! ... View more

"There's an app for that" https://github.com/georgestarcher/UF-TA-killrest ... View more

I'm going to combine the approaches of @woodcock and of @SloshBurch. [1] Every Splunk server gets the same authentication app (except forwarders because read below) [2] All indexers have the Splunk Web UI disabled, so only search heads, deployment servers, and other infrastructure nodes are log-in-able (except via REST API) [3] UFs have the REST API port disabled entirely - manage them via configuration management and you'll never need to log in. If I'm in a highly secure environment, I might deploy different authorization (not authentication but authorization) for my DS and CM and so forth so that my "most basic generic user role" (might be user, might not be?) has practically no access at all via that node. Sure they can authenticate, but they can't do anything;. ... View more

To add some additional specificity ... iplocation services are provided by a variety of vendors who collect their data in their own unique way. There is no single, universally accurate that "the internet" ties IP addresses to physical locations. Splunk, for their part, use the Maxmind Geolite2 databases. ( https://dev.maxmind.com/geoip/geoip2/geolite2/ ) Geolite2 is great because it is free. Geolite2 is terrible because it has a lower update frequency, and lower accuracy overall. As Matty has mentioned, you can update Splunk's Geolite2 databases relatively easily, or you can accept that they will be updated each time you update Splunk itself. If iplocation data is very important to you, I would suggest subscribing to Maxmind's Geoip2 database feed service. These feeds should be available in a format compatible with Splunk, and will be updated more frequently and more accurate overall. But, it is a separate subscription above and beyond your Splunk purchase. See https://www.maxmind.com/en/geoip2-city ... View more

OH well that changes it a little bit I guess! If the data went through Splunk's Data Integrity features as it was indexed before it wound up in HDFS, then the hashes made when the bucket rolled from hot to warm should still be valid on the copy of the bucket in s3. The question becomes if there is a way to check those. And I'm honestly not sure. ... View more

You really can't use the Splunk Data Integrity feature with Hadoop virtual index data. Splunk did not process the data through its own indexing pipeline, so you can't use the Data Integrity features of the indexing pipeline. You would have to make your own file hashing and signing solution for HDFS and make sure it crosses all of your compliance check boxes. ... View more

To the best of my knowledge, Splunk does not support RELP. This is (as I'm sure you know) an rsyslog specific protocol that, while documented thoroughly, has only a few implementations outside of rsyslog itself. I don't even know how you've gotten network devices to send using RELP, but that might have been easy 😉 ... View more

This seems (to me) to be of dubious actual security benefit. The code base being run by a UF and by an indexer are so substantially similar that this likens to taking a glass window and putting in front of it another glass window in order to keep rocks out. In terms of your hypothetical risk profile of someone using a TCP input on a Splunk to buffer overflow the process (for example) and gain RCE against Splunk, you've not done much to reduce that risk. And, if an attacker can RCE your intermediate forwarder layer, the odds are incredibly good that they can use that same RCE against your indexers, just staged off of your IF layer. But, if this makes sense in your security policy and it works for you, then great. My advice would be to make sure your intermediate layer is "wide enough" to remove any performance concerns. ... View more

Are you doing 'other things' other than monitoring console on your monitoring console host? See http://docs.splunk.com/Documentation/Splunk/6.5.2/DMC/WheretohostDMC. One thing MC does in a distributed environment is set up a number of distributed search groups and use those to restrict where searches get run. This can lead to unintended side effects. One of those 'unintended side effects' controls the searching of data created at the MC node, if the MC node is not listed as an 'indexer'. And, one of the reasons tscollect is deprecated is because it makes tstats files that get stored at the search head. I'd say that by using tscollect at all, you're venturing into a possible source of issues down the road. It seems great, and then kicks you in the face when you least expect it. Datamodels & DM Acceleration are a vastly superior solution to most problems. ... View more

Okay, so let's level-set here. You have IIS logs, arriving into HDFS via some mechanism that you are then searching via Hunk. The IIS events are basic w3c formatted strings inside of a JSON wrapper. Your goal is to extract the "sub-fields" from within the larger field. Let's talk about some things that won't work. First, INDEXED_EXTRACTIONS won't work because Splunk isn't actually indexing your data. Data that is picked up via virtual indexes using Hunk does not do INDEXED_EXTRACTIONS (because it's never actually indexed by Splunk) Second, SOURCETYPE_CLONE won't work either, because same issue. You're not indexing the data. Third, while there is a naming convention of vendor:product:logtype that is common, this is only a naming convention. There is no hierarchical configuration support for this. Fourth, while sourcetype renaming is a useful feature I don't think it actually does what you want to do here. I've got an idea that may work, watch this space. UPDATE! Thanks @dshpritz for solving one final bugaboo for me. Turns out you can do DELIMS extraction inside of existing fields. But, those fields have to have been brought to life using a REPORT or an EXTRACT - JSON and other auto-kv things don't bring the field to life in time. So assuming we make a new sourcetype for this. It doesn't match any of the existing sourcetypes well. [foo] EXTRACT-foo = "Event":"(?<Event>[^"]+)" REPORT-foo = foobarbaz So our new foo sourcetype uses a regex based extraction to get Event from the JSON data, treating it as a quoted string. Then our REPORT uses DELIMS on that new field to extract what needs to be extracted from it. [foobarbaz] SOURCE_KEY = Event DELIMS = " " FIELDS = date,time,s-sitename,s-ip,cs-method,cs-uri-stem,cs-uri-query,s-port,cs-username,c-ip,cs(User-Agent),sc-status,sc-substatus,sc-win32-status Note this is terrible. I'm glad it works, but I'm made sad by its necessity. In an environment where a forwarder is collecting IIS logs, you'd be far better off to let INDEXED_EXTRACTIONS handle the IIS sourcetype natively. But, in your situation where collection is happening elsewhere and everything you have to do is based whollly on search time via Hunk, this may be the best you can get. ... View more

You need to make sure your data has the data for the fields you're trying to use - and that they are being extracted correctly - before you can do analytics operations against them. Echoing the suggestion of @dbcase, for sourcetype=access_combined (the default Apache log format) there is no existing "time spent" or "duration" or "response time" or anything like that. You would need to make sure it gets added. Splunk internally has some log file formats that are substantially similar to access_combined like splunkd_web_access that include more detail in the log events and additional field extractions to get the fields from that data. For example: Here's 1 event: 127.0.0.1 - admin [21/Feb/2017:09:33:37.991 -0600] "GET /services/search/timeparser/tz HTTP/1.0" 200 3390 - - - 1ms And the props.conf for that sourcetype: [splunkd_access] maxDist = 28 MAX_TIMESTAMP_LOOKAHEAD = 128 REPORT-access = access-extractions, extract_spent SHOULD_LINEMERGE = False TIME_PREFIX = \[ And the transforms.conf for extract_spent : [extract_spent] REGEX = \s(?P<spent>\d+(\.\d+)?)ms$ With spent being in the data (1ms) and being properly extracted, it's easy for me now to: index=_internal sourcetype=splunkd_access | stats avg(spent) With Apache logs in general, what I would personally suggest is if you're going to modify the format of access_combined , then add new fields to the END, and do them as key=value . Then you don't have to change anything and Splunk just picks it up. ... View more

One way of doing this is via Sideview Utils and their "multiplexer" module. It is designed to make new dashboard panels based on search results. There may be other ways too, but I know the approach of @sideview works. See http://sideviewapps.com/apps/sideview-utils/ ... View more