That is interesting though because I don't think MacOS version would differ here much from the Linux one and I have a running 10.4 instance which launches mongod with cmd parameters (which also seem to be included in your log output) explicitly pointing to PEM keyfile and cert. Also I don't see any keystores within my Splunk installation.  But. I see other threads saying that on Windows mongod relies on the cert being imported to the machine local cert store. Which suggests that on MacOS Splunk might be doing something similar. Also reading further down those threads I'd suspect there is something... not exactly "wrong" as such, but rather unsupported by mongod - either some unexpected characters in certificate attributes or maybe some strange characters in private key password... ... View more

This is not really a Splunk question. It's a question about feasilibity of OS upgrade. Windows in-place upgrade is always tricky and as far as I remember, most windows admins I knew advised fresh installation rather than in-place upgrades. You can try it but you must have a plan B in case it fails. ... View more

PFX? I wasn't aware Splunk components could use PKCS12-formatted crypto material. I was only aware of PEM support. Since you seem to be using self-signed certs, have you tried just recreating the certs using splunk createssl command? https://splunk.my.site.com/customer/s/article/How-to-renew-certificates-in-Splunk ... View more

Yup. This pretty much covers it. Just be aware that if/when the inputs already processed some data which they might be able to re-get or re-read, it might be very difficult to get just those missing events without reingesting a whole lot of other data. Resetting checkpoint will usually do just that - restart from scratch.  ... View more

Adding to @richgalloway 's answer - Splunk works on a per-bucket basis. So the actual retention depends on the age and overall composition of events within each bucket. Splunk rolls whole bucket when the most recent event within the bucket gets older than the pre-set retention time. ... View more

Sorry to say that but I have no idea what LLM you pulled this one from. Splunk has no way of "assigining logs to a specific pipeline". There are no specific settings per pipeline if you have more than one and there is no maxKbps setting at inputs level. There is just one global throughput maxKbps setting set in limits.conf.   ... View more

I'm not sure it works like that. If inputs were tied to the particular queues, scaling processing pipelines would have no effect on single input. And as far as I remember, I had multi-pipeline UFs with lagging inputs and it wouldn't help much. ... View more

I haven't touched it in ages but I was pretty sure the inputs were independent on the processing queues. ... View more

While you can to some extent "stretch" your DS capacity by making the clients phone home less frequently which reduces stress on the DS instance, you still need memory for holding the info about all known DCs so you can't scale it indefinitely. And I wholeheartedly agree that configuring DS with GUI doesn't scale well especially since some settings are unavailable in GUI. ... View more

Yes, currently the DS clustering is the official way but I'm not a big fan, to be honest. Most importantly - it requires shared storage which typically makes it tricky to do over multiple sites. ... View more

Can you please copy-paste your whole event into a preformatted paragraph or code block? It's not obvious what your event looks like. ... View more

Before introduction of DS cluster there used to be advanced DS setups recommended for big installations (and taught on advanced Splunk trainings). But they are not meant for beginner/intermediate admins because they introduce quite a lot of possible issues if not done properly. (and require some different ways of troubleshooting as we can see). I must say I haven't worked with such setups myself. ... View more

The overall idea is sound but you have to remember that OS level name resolver can cache entries. Dealing with it might require different things on different OS-es. ... View more

Assuming the users logged out and back in (just to be sure; I'm not 100% it's required), editing roles should be quite immediate. Question is how how did you edit those permissions? Maybe you did edit the authorize.conf file and did _not_ reload the effective config? ... View more

Generally speaking - no. There is no way to prioritize inputs. And yes, it can have an impact on UFs sometimes. I've had a strange setup with a UF checking huge number of  files from network shares. Every time the UF was restarted it would need about an hour to catch up with the states of all the monitored files. As far as I remember it even lagged ingestion of forwarder's internal events. That was very wrong and luckily has been fixed since. But it shows that you can't prioritize inputs versus each other.   ... View more

Yes, this should work but depending on your data some other solutions might be better in terms of performance. Your solution will always read all events from given time period which might not be a very good thing if you're searching through a big index. In specific cases you could leverage either subsearches or the map command. If your data allows it you could try to use PREFIX() to get the ID field from your event without touching raw data, just using tsidx files. ... View more

If I understand correctly what's going on, you're getting your events as (almost exclusively) metadata - indexed fields. That's not very good on several levels. Firstly, as you've just noticed, you have difficulty searching for your data. Splunk works the way it does for a reason. If you wanted to have structured data only, you might not have bothered with Splunk at all. But there's more to it. When you're getting indexed fields, they are not getting tokenized - they are stored as a key::value token. They are not searchable with typical Splunk matching rules. You can try using TERM() for partial match as @bowesmana showed but this is far from optimal. You could try using Edge Processor to rewrite such events on Splunk's side so that the raw event contains the actual data. ... View more

1. What version do you have and on what platform? (in labels you put Windows but it isn't explicitly stated otherwise). 2. Admin user (and admin role) already has all capabilities (except can_delete) so you didn't need to "give all permissions". 3. What exactly are you trying to add and how? Upload to the web interface? Add input from Settings menu? 4. What is the exact message and at which stage? (screenshot maybe?) ... View more

OK. As usual, there are several angles to this. 1. SVAs and other Splunk architectural documents (including Splunk training materials) contain architectures which are well-known, tested and are supposed to work in most cases (unless you heavily overstress them with a very unusual load or something like that). That's why it may contains safeguards/guidelines like "no more than 50 DCs". They are often created with relatively default settings in mind and are meant as an easy to remember reference point so that you can rely on it in case someone asks you "why should we do it this way". 2. This doesn't mean that other architectures won't work or that you cannot (especially if you tweak some parameters - in case of DS it's the phonehome interval which is most important at the early stage; after a certain size of the environment it might be the memory size) adjust the architecture to your needs. For example, if you have a remote site with just one machine and no physical space to put another box you will do whatever it takes to merge DS and HF functionality on a single box and your point of focus will be making it run as smoothly as it can. 3. It takes experience though to build the intuition what and why would or whould not work efficiently. 4. Additional point is supportability. If you do something unusual and end up with problems (performance problems or functionality problems) Splunk might tell you that it is due to your strange architecture and it is not SVA and thus it will not dig too deeply into it. Could happen. 5. Mixing different roles on a single box raises your risk of something going wrong at one of them affecting other ones (and of making misconfiguration mistakes). So it's not a strict toggle switch - you add 51st deployment client and your DS blows up. But you have to be aware of the potential risks. ... View more

Generally, having a fully 100% proof event delivery is extremelly difficult unless you are ready to have high probability of duplicated events. With syslog reliable delivery generally doesn't exist. With UDP the source doesn't have any way to determine if the event has been received or not. So UDP is not reliable by design. With TCP it _could_ be more reliable (and there are some solutions which do keep track of whether the destination actually received the event at least on the TCP stream level - most notable one being Checkpoint's Log Exporter which can get stuck for quite a long time if there is noone to receive the events) but typically the sources don't care about the destination. There is just a simple send() routine within the source's event handling process and as soon as the event is passed there the main process doesn't care anymore. If the connection to the destination can be established the event will be sent. If it cannot it won't. And that's fine because syslog was always meant to be simple, robust and easy to implement and maintain. So yes, there is no way to make a 100% reliable syslog delivery. (One can argue that there is no way to make 100% reliable delivery because every component which would have to queue events in case of downstream path failure could get overfilled but that's another thing - just physical hardware limitations). ... View more

Adding to @livehybrid 's comment - as a rule of thumb, search time extractions are "better". Unless you have a really border case, it's better to create a search time extraction (and maybe accelerate the search in one of possible ways if you need it) than to create index time extractions. Anyway, my first guess would be that you might by any chance using indexed extractions. Secondly, I can never get my head around when you need the WRITE_META setting and when you don't. But still I'd just go for search time extraction. ... View more

Adding to what has already been said it seems you'd be best off with just streamstats but with a time window. index=your_logs "error" | streamstats time_window=2m count values(_time) as _time | where count>=2 If you have multiple types of error and want to check each of them separately you can add some form of a "by" clause to streamstats like by errorcode  One caveat - for long-lasting errors it will give you several results from each subsequent error event. ... View more

I would suspect that you had ran out of your trial license sufficiently long time ago that you already accumulated license violations from the free license tier. In this case you'd have to indeed wait for a 30 days period without warnings to unlock the search functionality. ... View more

Well... it's a topic which seems easy but gets more and more complicated the further you dig into it. Firstly, there is no single thing called "syslog". Syslog used to be a "standardized" way of logging events in unices but since than grew significantly and can mean anything: - Anything sent to port 514/udp or tcp - Anything sent directly over a network stream without using any fancy protocol overhead (usually just separating items with new line characters) - Anything sent over UDP - one item per datagram - Anything following RFC 3164 (BSD syslog) format or RFC 5424 So right from the start without being much more precise than just saying "syslog" it's difficult to say what we're talking about. What do you mean by "sending TCP data as syslog" vs. "streaming as TCP"? What source solution are we talking about? How are you trying to receive and process the events? ... View more

Well, as I'm reading the specs for those webhooks it seems there are two different mechanisms at play here. One is OAuth which seems a bit silly here since webhooks are supposed to be fast, easy to implement, robust and so on. OAuth seem overly complicated for a simple webhook but let's leave it at this point. OAuth solves the case of authentication - it lets the client say hello properly to the server so that the server can decide whether to authorize the client to use the service (post the webhook) or not. This has nothing to do with non-repudiation. And is indeed more or less the equivalent of the HEC token. The non-repudiation part is indeed performed with event signatures but it's kinda orthogonal thing to Splunk. Splunk is not a general "webhook processing solution". Splunk's HEC is meant to receive events. Just that. I'd say that the way to go would be to put something "in front" of Splunk to capture both the webhook contents as well as the signature and either validate the signature or send both the contents and the signature to Splunk (and have a way to later verify said signature). ... View more