Digging a bit further, I have the following config: [sslConfig] enableSplunkdSSL = true sslRootCAPath = /opt/splunk/etc/auth/orion84/ca.pem serverCert = /opt/splunk/etc/auth/orion84/server.pem caTrustStore = splunk,OS caTrustStorePath = /etc/ssl/certs/ca-certificates.crt   Running the verification with that OS truststore works: /opt/splunk/bin/splunk cmd openssl verify -verbose -x509_strict -CAfile /etc/ssl/certs/ca-certificates.crt server.pem server.pem: OK   So I don't get why it wasn't working before I manually added that root certificate to the splunk ca.pem file, as I already configured Splunk to also use the OS truststore, which can be successfully used to validate my server certificate. Does the kvstore process strictly use the Splunk truststore, instead of honouring the caTrustStore setting? ... View more

Had a similar issue today, trying to enable ssl on a fresh Splunk 10.0.0 install. Did everything by the book, but got similar kvstore failures. Running the `splunk cmd openssl verify` command returned a verification error (while running it outside Splunk, the certificate verified just fine). After reviewing the let's encrypt certificate and ca certificate a bit further,  I noticed the ca certificate itself is signed by some root CA ISRG_Root_X1. Digging up that root CA certificate from my OS ssl folders and adding that to ca.pem that Splunk uses for verification solved the error. Bit weird, because I did configure Splunk to use the OS truststore as well, but perhaps I did not configure that perfectly, something to look into. But at least by just manually adding that root certificate to the ca.pem in my Splunk install, it works. ... View more

If I'm not mistaken, the license usage logs are generated by the cluster master, so you could try splitting by host. Otherwise have a look in the raw data for that source if there's any other identifier in there that helps you tell things apart. ... View more

If you're really just looking at getting a list of locations, why not just do something like:   index=app_pcf AND cf_app_name="myApp" AND message_type=OUT AND msg.logger=c.m.c.d.MatchesApiDelegateImpl | spath "msg.logMessage.matched_locations{}.locationId" | search "msg.logMessage.numReturnedMatches">0 | rename "msg.logMessage.matched_locations{}.locationId" to LocationId | stats count by _time,LocationId | table LocationId     ... View more

So effectively you want to group by src_ip ...your existing query... | eval src_ip = if(Direction="Outbound",Internal,External) | stats values(Internal) as Internal values(External) as External count values(Direction) as Direction by src_ip ... View more

Can you clarify a bit what you mean with point a about the non-exact matches? Can you (given this sample data) sketch what the expected output of the query would be? E.g. having Pq3, pq3 and pq_3, do you want the query to just tell you that pq3 exists in the lookup but Pq3 and pq_3 don't? Or should the query try to also match those 'imperfect' cases and if so should the query highlight that it is a non-exact match? What you could do to cover all that is this: First: create a lookup definition "pq" that refers to the pq.csv lookup file and configure that lookup definition to be case sensitive (as it is by default I believe) Then, run this query (not tested, so you may need to debug the SPL a bit in case I made some typo or so): ...your current query... | eval cleanedPQ = replace(lower(PQ),"[^a-z0-9]","") | eval from="query" | append [ | inputlookup pq.csv | eval from="csv" | eval cleanedPQ = PQ] | eventstats dc(from) as exact_match by PQ | eventstats dc(from) as nonexact_match by cleanedPQ | eval match = case(exact_match>1,"exact",non_exact_match>1,"non-exact",true(),"no match") | stats values(from) as from values(match) as match values(ACT) as ACT values(cleanedPQ) as cleanedPQ by PQ ... View more

One important thing to keep in mind is that license usage shows volume as per when it was indexed, which doesn't necessarily have to coincide with when the event took place (which is what you probably are looking at with the len(_raw) queries). So make sure to compare apples vs. apples by restricting that len(_raw) query by _indextime, rather than by _time. Not saying this explains your case, but at least in theory it could explain differences. Also not exactly sure how the difference between rolloversummary and usage data can be explained. ... View more

if you're familiar with Azure Logic Apps, you could consider using those to get events from the Event Hub and push them to a Splunk HEC input. The logic app can then also contain some logic to assign different index/sourcetype etc. to the data (as part of the HEC metadata fields). I just noticed someone also built a Splunk App around that approach: https://splunkbase.splunk.com/app/6223 ... View more

Only looking for devices that show a high volume (in bytes rather than count) of unique DNS requests/responses will indeed probably cause a lot of noise. Take a look at these articles to take a more advanced approach to this: https://lantern.splunk.com/Security/Use_Cases/Threat_Hunting/Monitoring_a_network_for_DNS_exfiltration https://www.splunk.com/en_us/blog/security/hunting-your-dns-dragons.html One of the key tricks discussed there is to also look at the entropy of the domains occuring in the query. Long, high-entropy sub-domain names are a good indicator someone is trying to hide data in DNS messages. ... View more

At that point you could also simply replace that last stats by an eval. (Unless I'm somehow completely missing what you're trying to achieve with that stats)   search "Output MeshRequestId" | stats avg(time) as avgTimeOut | appendcols [search "Input MeshRequestId" | stats avg(time) as avgTimeInt ] | eval TimeInThrottler = avgTimeOut - avgTimeInt     ... View more

If you want to apply the regex to the entire raw event: "originatingAddress": "([^,]+) https://regex101.com/r/b4NMEF/1   If you already have a field originatingAddress and you're applying the regex to that field specifically, you only need the part in between parentheses. Or you use an eval command to split based on , and then take the first entry using mvindex(). ... View more

It was a link to your splunk environment I think. Perhaps @richgalloway was so kind to strip it for you. ... View more

Looks like that has already been fixed (probably a copy paste mistake) 🙂 ... View more

First of all: you may want to remove the URL under the lookup name 🙂 There's multiple ways to do this, but in the end it comes down to doing a | lookup command and then some eval to translate the output of the lookup command to yes and no. One way would be to add a column in your lookup called "HighRiskIP" with value "yes" on each row. Then you can do:   ...your current query... | lookup HighRiskDomain IP as IP_ADDR OUTPUT HighRiskIP | eval HighRiskIP = coalesce(HighRiskIP,"no")   Note: you may need to replace "IP" in the lookup command with whatever the ip address field in your lookup is called. If you don't want to add additional columns to the lookup table, you can also do something like this:   ...your current query... | lookup HighRiskDomain IP as IP_ADDR OUTPUT IP as HighRiskIP | eval HighRiskIP = if(isnull(HighRiskIP),"no","yes")   Note: use above as inspiration, don't just copy paste it and trust my code is bug free, I haven't tested these. ... View more

But that solution means you can't apply sourcetype specific index-time config like timestamping and linebreaking, source/sourcetype overrides etc.. So that's far from ideal and will certainly cause a bunch of headaches with certain types of data? Crafting your syslog config such that it puts data that needs a different sourcetype into separate folders is typically the way to go. That way you can easily write distinct input stanzas for each sourcetype. ... View more

That's how I learned about this Splunk Answers post 🙂 But indeed: for those not on slack yet, check that out as it contains interesting discussion and some more experiences from people who already upgraded their DS. ... View more

Good write-up! One thing to keep in mind for temporary shutdown of DS: not only can you not update config on clients, new clients also can't pull their initial config (for situations where for example new VMs are launched with a UF and only deploymentclient.conf and DS normally then provides all the relevant config upon first phonehome). ... View more

@gordo32  It doesn't really have anything to do with the events. It is all about your query and how Splunk translates that query into the actual search that is executed. Splunk uses all the props and transforms and lookups that are in scope for your query (e.g. based on the sourcetype you're filtering on) and uses those to translate any search criteria in your main query into actual raw search criteria such that it can match those against the indexes/bloom filters that are created on top of the raw data. You can get an idea on how that works by performing a search that includes some criteria that are derived through props/transforms/lookups, then opening the job inspector, scroll down, open the search properties and compare the eventSearch to the normalizedSearch.  ... View more

But that fix actually removes some functionality, where the original "raw value" of the action field is matched against the vendor_action column in the lookup and then a normalized action value (from the action column) is returned, overwriting the action field. That normalization is critical for CIM datamodels and related searches/dashboards to work properly. The real solution is that the TA should extract the raw action value into vendor_action and then use the lookup to translate it to action. Instead of the bizarre mess of evals and lookups with like 4 different action fields that is used currently. ... View more

That isn't index-time, but ok, a REPORT based solution might still be preferred over spath in the query. Except for the fact that json is way to complex to parse with such a simple regex. It doesn't support nesting, it doesn't support arrays: https://regex101.com/r/PJQJp7/1 This would work a bit better already: "([^"]+)":\s*("[^"]+|\[[^]]*\]|\d+) https://regex101.com/r/u8hgTB/1 But that also doesn't extract json the way Splunk's auto extraction would. My solution does take content of nested structures, but without retaining that structure, it just captures all the 'leaves' in the json tree. ... View more