You haven't told us what you want the search to do so I'm only guessing. Probably your hosts log events which have either enforcement_mode=block field or event_type=error field but no single event has both of these fields set. So your "combined" search will not find them because both conditions aren't fulfilled in a single event. That's why you need to correlate multple events by using either transaction or stats (the stats approach is preferred due to transaction command's limitations). ... View more

As both searches invoke the same index, there is probably not much point (unless you have a very very specific use case) to use subsearch here. Just search for index=firewall event_type=error sourcetype=metadata enforcement_mode=block Because that's effectively what your search would do. Having said that - that is probably _not_ what you need. I'd hazard a guess that you're probably looking for something like index=firewall | stats values(event_type) as event_types values(sourcetype) as sourcetypes values(enforcement_mode) as enforcement_modes | where enforcement_mode="block"   ... View more

That's true. Remember that docs pages have feedback form on the bottom. You can use it to... provide feedback. And yes, this feedback (of course if precise and reasonable, not just "I don't like this page" ;-)) is read and the docs pages do get better in time because of that. ... View more

Just to nitpick a little. You can set up a cluster without redundancy. It's not a HA cluster but it has its uses (one advantage of such setup is the ability to rebalance buckets when you add a new peer). But yes, if you set up a cluster with RF>=2, every bucket should have at least one additional copy somewhere in the cluster. ... View more

Disclaimer - I haven't used httpout much so I might be mistaken here. But. httpout is not a HEC output (although it needs an HEC input and valid HEC token; it's complicated). It's s2s protocol embedded in http transport. It is indeed a fairly recent invention mostly aimed at situations like yours - where it's easier (politically, not technically) to allow outgoing http traffic (even if it's only pseudo-http) than some unknown protocol. Having said that I'd expect most of the functionalities normally working with tcpout (like useACK) to work. I'd test it first in the lab before pushing to prod anyway. ... View more

It ain't that bad 🙂 https://www.aplura.com/assets/pdf/hec_pipelines.pdf ... View more

My three cents on general approach to such tasks. Since "last 15 minutes" and "last 10 minutes" can be expressed in terms of 5-minute periods, you can simply either use a timechart with 5-minute bins or bin manually time to 5-minute buckets and do stats over the 5-minute periods. And then - when you have those 5-minute stats - you can aggregate last two or last three stats to get summarized "last 10 minutes" and "last 15 minutes" values. It's often useful to see if the problem containing several "parallel" computations cannot be transformed to a single - maybe a bit more detailed - calculation and some form of aggregation after that. ... View more

Check also the _internal events from component=DatabaseDirectoryManager around that time (not all events have idx= field). There might be different factors at play, like retention period. You could check your buckets with dbinspect and see earliest/latest events in them. Anyway, 10 hot buckets is quite a lot. ... View more

I would be very cautious about such third-party hosted extensions. Even in case of Splunkbase-originating add-ons not written and supported by Splunk I tend to dig into an app and peek around the code before installing (and boy, there are some "interesting" ones; luckily I haven't found anything malicious yet but some badly written python code - why not). And this one is not even hosted on Splunkbase which means it didn't even pass appinspect. ... View more

1. OK. This is _current_ configuration. It would be even better to see the output of splunk btool indexes list mack and | rest /services/data/indexes/mack But the question is what/how did you change. 2. Did you check the reason for bucket rolling? index=_internal component=BucketMover idx=mack   ... View more

HEC on its own doesn't have filtering abilities. You can filter events after receiving them (on any input) using props and transforms but that doesn't change what you're sending over your WAN link. Your question is fairly generic and we don't have a lot of details about your environment so the answer is also only really generic in nature. Anyway, ingesting events to Splunk using Logstash might prove to be complicated unless you properly prepare your data in Logstash to conform to the format normally ingested by standard Splunk methods (otherwise you'd need to put some work to properly extract fields from such logstash-formatted events). But Logstash should give you ability to filter the data before sending it over HTTP to the HEC input. ... View more

The configuration elements work where they are defined (but they may have additional impact on other functionalities due to mutual dependency - for example lowering output bandwidth on forwarder can affect rate of input on some inputs (you can't slow down inputs working in "push" mode - you can just drop events if the the queue is full). So if you were to configure your HEC input to blacklist something, that would be working on the HEC input, not on other components. Having said that - what do you mean by blacklisting on HEC input? I don't recall any setting regarding http input filtering/blacklisting events. The closest thing to any filtering on HEC input would be the list of SANs allowed to connect and that's it. Even if you wanted to filter on the source forwarder, remember that filtering applies only to specific types of inputs - windows eventlog inputs can filter and ingest only some events and file monitor inputs can filter and ingest only certain files (still no event-level filtering). Maybe you could implement some form of filtering on the UF if you enabled additional processing on the UF itself but that's not very well documented (hardly documented at all if I were to be honest) and turning on this option is not recommended. So if you wanted to filter events before sending them downstream, you'd most probably need a HF which would do the parsing locally, fitler some of them out and then send across your WAN link but here we have two issues: 1) While it is called "http output", the forwarder doesn't use "normal" HEC to send events downstream but uses s2s tunelled over http connection. It's a completely different protocol. 2) HF parses data locally and sends the data parsed, not just cooked. That unfortunately means that it sends a whole lot of data more than UF normally sends as it sends data cooked. So "limiting" your bandwidth usage by installing a HF and filtering the data before sending might actually have the opposite effect because even though you might be sending less events (because some have been filtered out) you might actually be sending more data altogether (because you're sending parsed data instead of just cooked stream). Depending on the data you want to ingest, you might consider other options on the source side - if the events come from syslog sources you could set up a syslog receiver filtering data before passing them to Splunk, if you have files, you could preprocess them by external script. And so on. ... View more

"You must use services/collector/raw endpoint of Splunk HEC for data filtering to work." This is not entirely true. In fact it's not true at all 😉 But seriously, while the /event endpoint does skip some parts of the ingestion queue and you can't affect line breaking or timestamp recognition (with exceptions) this way, your normal routing and filtering by means of transforms modifying _TCP_ROUTE or queue works perfectly ok.   ... View more

When you keep hitting the size limit, Splunk will roll the buckets to frozen. That's the point. Some things worth verifying: 1) How did you increase the size limit? Which parameters did you edit and did you restart your splunkd? 2) How do you know buckets are frozen due to size limit? 3) Do you have volume size limits? ... View more

Apart from all that @richgalloway already mentioned, this document shows results of testing on some particular reference hardware. It's by no means a guarantee that an input will work with this performance. Also remember that windows eventlog inputs get the logs by calling the system using winapi whereas file input just reads the file straight from the disk (most probably using memory-mapped files since it's most effective method). And last but definitely not least - as I already pointed out - UF typically doesn't break data into events! ... View more

The whole idea is tricky. If you want to use an embedded report - that's relatively easy and safe - you can embed _scheduled_ report so they are generated using predefined configuration and you can embed the resulting report into your external webpage. With stuff more "dynamic" it's way more tricky because it involves a whole lot of making sure you don't accidentally create a security vulnerability for your environment and don't let the user do much more with your Splunk installation than you initially intended. There is an app https://splunkbase.splunk.com/app/4377 which is supposed to do that or at least help with that but I haven't used it and I don't know if/what limitations it has. Of course another thing would be to use REST api in your app to call Splunk servers and generate proper output which you could then present to your user but due to the security issues I mentioned before, it's something you'd rather do in your backend and only present the user with the results embedded in your web app by said backend than let the browser call the Splunk search-heads directly. ... View more

Apart from all the valid remarks from @gcusello and @richgalloway , think about what you need. If you want to build on the search shown by @gcusello notice that yours groups login attempts by source (which means that it will show multiple attempts to log in using different usernames but from the same IP as one result) whereas the other one groups by username which means that it will aggregate login attempts to the same account launched from different IPs but split different account login attempts from the same IP as separate results. So it's a question of what you are looking for. ... View more

UGH. If you have any say in this - try to force the team responsible for producing these logs to get them in some reasonable format. It's some mix of pseudo-syslog embedded in some pseudo-json, and containing some "kinda delimited key/value pairs". It's not gonna end well. ... View more

We don't know the whole picture so it's a bit difficult to give precise recommendation as to the license issue. There are at least two diferent solutions here - you could just attach your on-prem DS to your LM in Azure (but would need to make sure there is proper connectivity of course) or you could obtain a "zero ingest" license from your local Splunk sales team for the DS alone. And you need to point your deployment clients'... deploymentclient.conf to the new DS. (if your infrastructure was well-designed you probably would just deploy a new version of a proper app from the old DS). ... View more

OK. You're _not_ waiting on the indexing queue so it doesn't seem to be the issue of backpressure from the disk not able to keep up with the rate of incoming events or any other configured outputs. I'd start by checking the OS-level metrics (cpu usage, ram). If _nothing_ else changed "outside" (amount of events, their ingestion rate throughout the day - not only the general summarized license consumption over the whole day, composition of the ingest stream between (split among different sources, sourcetypes and so on)), something must have changed within your infrastructure. There are no miracles 😉 Is this a bare-metal installation or a VM? There could be issues with either oversubscribing resources if that's a VM or even with environment temperature in your DC so your CPUs would get throttled. (yes, I've seen such things). But if the behaviour changed, something must have changed. Question is what. ... View more

Anyway, obsessing about EPS suggests that you might be thinking about replacing some other SIEM/log management solution. Those used to be licensed on a per-EPS basis. With Splunk it doesn't matter. If ingest-based your license allows for indexing specified volume of data _daily_ regardless of whether it's a constant steady data stream or if it's just a few "batches" of high volume peaks of data. ... View more

OK. Have you read anything that has been written in this thread? EPS as such is not a very important concept for Splunk (at least not on the UF level). ... View more

What is your business problem? ... View more