Well, no. At least not entirely. CA certificate affects only a specific connection where it is used. You _can_ (although it's rarely done this way) use several different CA certificates for different purposes (for example, another certificate to present on your inputs and another to use to authenticate on your outputs). But yes, I do agree that it requires careful planning. Luckily, Splunk allows for multiple CA certificates so with a bit of care migration to another CA can be quite "easy" (as if anything around TLS was ever easy). @splunkreal Yes, I mean sslRootCAPath in the part of _adding_ new CA (not replacing the old one because then you immediately stop trusting all old certificates). But if you don't verify the certs at all, why bother in the first place? ... View more

Well, this is not really a Splunk question. It's about knowing your data and knowing what to ingest and it's a question of a properly designed onboarding process. Noone knows your data better than you. For example, some organizations consider full firewall logs important, some don't. Some use them in their detections and/or threat hunting, some don't. So noone can tell you what you should be ingesting. It depends on your use cases, maybe on your compliance needs. It's one of those things I always say you should go through internally and/or with your local friendly Splunk Partner. ... View more

Close but a bit inaccurate. A frozen bucket should not reside on a hot/warm or cold storage. So if it should have been already but it is there something must have gone wrong during freezing. Anyway, a frozen bucket is not searchable so in a typical scenario (no frozen storage configured so buckets are deleted when rolled to frozen), deleting a bucket which should have been frozen but wasn't doesn't hurt you. ... View more

Close but the Splunk message is referring to the "splunk fsck" command, not the general filesystem fsck (which might also be in order but that's another story). ... View more

Not directly as far as I remember. You could try doing some tricks with custom CSS if you're using Classic dashboard. ... View more

Well, as I said - you can have protection against data loss (as everything - it has its limitation, nothing in this world is 100% safe, secure, fool-proof and so on) but there is a possibility of data duplication. As simple as that. And you can't really do anything about it.  ... View more

And how would you like to exploit this vulnerability? What is the attack surface here? ... View more

In your case that probably wasn't that important but I learned about HEC collecting when I needed to quickly (re)insert a  relatively big batch of data with relatively high variance on metadata fields. Specifying them manually with "normal" collect would be a chore and would end up with some ugly map-based SPL monstrosity whereas single collect with hec format took care of it quickly. ... View more

Lookups don't use regexes for matching (unless you're talking about an "external lookup" - there you can implement everything you like). But. If we're talking about csv-based lookups the entries are inspected in the order they're written in the file. So if you set maximum matches to 1 and give a negative match early in the file you can prevent Splunk from matching positively later in the file. It's kinda ACL-like behaviour. But yes, that's ugly. ... View more

Yup, I would have done the same - create a temporary index (possibly on SH), then just search and collect (I'd just use collect with HEC format to replay all data along with metadata) ... View more

I'm not saying it's the only possible reason 😉 ... View more

Those two mechanisms are meant to solve two different problems. And unless the protocol is extremely complicated it is very hard to not have either lost data or duplicates. ... View more

But this returns a sid which has to be queried for completion and you have to fetch results, paginate and so on. You can use the /services/search/v2/jobs/export endpoint to spawn search and stream the results with a "| savedsearch ..." search. But you might want to use the namespace parameter for the call to set proper... well, namespace 🙂 ... View more

Be more specific about what "not reflecting" means. Also check if  <your search> | fields + host shows the field ... View more

There are usually two possible scenarios when data loss occurs in UF communication. 1. The receiver gets the event (or chunk in case of cooked data) but fails to properly process it for some reason whereas the forwarder considers the data already sent and therefore processed. The useAck option is designed to prevent this scenario. It can cause duplicates though. 2. The forwarder fails to send buffered events and then it is shut down or crashes. Upon restart the buffer contents are lost and not retried (the data had already been read from the source file so Splunk resumes reading from after what had been read). Against that you can protect with persistent queues. Persistent queues can incure performance penalty since data is not only held in memory but is written onto disk.   ... View more

I've never set a separate cert just for kvstore but assuming that the config works the same as the general splunkd one, your servercert file should consist of concatenated - subject cert, private key, certification chain.   ... View more

As with any decent TLS implementation - the way to go about migrating from one root CA cert to another is to add the new root CA cert to the trusted certs store, migrating the relevant subject certificates (or just letting them naturally expire and just issue new ones with the new CA) and then optionally remove the old CA. The details will obviously depend on the detailed setup on where (and on which end) you use certs.  The general idea is in the first paragraph. For detailed plan, if you don't feel confident enough to do it on your own, I'd suggest to reach out to your local Splunk partner to thoroughly review your infrastructure and go through the process with you. ... View more

As far as I remember (you might want to double check it though), when kvstore grows too big (as is quite common with bigger ES installations for example), Splunk fails to make point in time backups (due to some timeout?). You might consider enabling kvstore entries expiration if it's a viable option for you to keep kvstore smaller. ... View more

Alternatively you could use Lookup Editor's API. It's not officially documented but there are examples of scripts using it on the web. ... View more

First things first - check status of your input (splunk show inputstatus, check logs). Maybe it's not outputs, maybe you're just not getting inputs (especially since we're talking about Security logs and access to those is sometimes limited). ... View more

You can find bits and pieces about kvstore in Splunk docs. It's "just" a mongodb instance so much of mongodb experience applies here as well. ... View more

rsyslog 8.2112 is over 4 years old. Unless you're just using some basic functionalities, you might want to upgrade to something more recent. ... View more