From Splunk's perspective, manipulating peers within the cluster works the same regardless of whether they are on Debian, Centos or gentoo (I wonder if anyone runs Splunk on gentoo ;-)). You should have all nodes on the same distro and release (for example, all nodes on RHEL9) for maintainability purposes but technically - especially during transition period - you can just add Ubuntu peers to AWS Linux-based indexer cluster and they should run just fine. ... View more

You might have completely oposite requirements as well - short retention for performance data, long retention for content for auditing purposes. Depends on your organization. Still, as usual - if your data has different access requirements, different retention parameters or dramatically different volumetric characteristics, split it into separate indexes. Otherwise just leverage search parameters. Differentiate sourcetypes, create eventtypes... be creative. ... View more

Ok. So you're comparing things in different time spans. You should compare your license usage _before_ you had your drop in firewall events (before Jul 14th) with current one. You seem to have had two things happening which both affected your license usage and their combined result is that your license usage raised. But as I see from the graph, the license usage increased roughly by factor of two whereas the number of events per day increased way more 3-4 times. So this might actually mean that the reduction is working properly. As I said - you could disable this pipeline for a day or two and see if the license usage skyrockets again to be 100% sure. ... View more

OK. It's starting to make sense. At least partially. As you can see, while your events indeed got smaller due to the pipeline removing some fields, you're ingesting way more events now. Why is it? It's hard to say. Maybe it's a coincidence and your source simply suddenly started producing more data. Maybe enabling your pipeline triggered some other performance changes so you - for example - stopped losing some events. So generally - volumetrically it makes sense and it adds up - you have smaller events but you have so many of them now that they more than make up for the reduction in size. Therefore your license usage actually grew instead of dropping. But diagnosing why it actually happened could be tricky. You could disable your pipeline temporarily as a test and see whether your event count drops again If it does, that would mean that something in your ingestion path cannot keep up with the raw incoming data stream. ... View more

It's getting more and more interesting 😉 And the count of events? | tstats count where index=* sourcetype=pan:traffic:cloud by _time span=1d   ... View more

Interesting. The event contents indeed seem to have changed although you're getting some embedded json hash of _forwarder_channel_info (we don't see on the screenshot how big it is). You could do some stats how/whether single event size has changed by doing index=<your index> sourcetype=pan:traffic:cloud | timechart avg(eval(len(_raw))) Of course you can adjust span for timechart as needed and add sampling if you have (and you probably do) a lot of data. We're interested about sample stats, not about details for absolutely all events. ... View more

Typically you'd use tcpdump to check whether there are events coming in on your network interface from the source IP to your rsyslog box. ... View more

Well, this is more of a syslog-ng question than a Splunk one so you might get more info about syslog-ng specific tuning and tweaking on some syslog-ng related mailing list/forum. But. I can drop in my three cents (because why not ;-)) 1. I'm not (and I've never been) a big fan of the "separate port for everything" approach. Yes, I know that in the old times when we'd receive syslog directly on Splunk forwarder you'd have a separate port for each product so that you can assign different sourcetype to each port. With syslog-ng or rsyslog you can just use network-level metadata (source IP in this case) to assign target source/index/sourcetype to the event. I find this approach more tidy and the network teams like it because you only ever poke holes in firewalls for the typical 514 port instead of doing a wide range of ports. 2. Having said that - I'm not a big expert on syslog-ng and I'm not sure if you can in a reasonably manageable way attach different (post)processing pipelines to a single "source" depending on the event type (so that - for example - events from Palo Alto are processed differently than events from Fortigate while still being received on the same port). With rsyslog it's trivial, I'm not sure about syslog-ng. 3. There are two schools of thought regarding the "middle" between syslog receiver and Splunk. There is the old-school approach which you have in your config - receive events, write them to files, pick them up with monitor input, send to Splunk. There is also the modern-school approach - use http output/connector/whatever you call it to send it from your syslog receiver directly to HEC input. Both approaches have their pros and cons. The file-based approach requires disk space for the files, requires manual cleaning of the directories but gives you a natural "buffer" if your downstream is not available (your Splunk is down). The HEC output approach is simpler in terms of infrastructure, lets you easily assign additional metadata but is more sensitive to downstream availability (and buffering is more complicated). ... View more

First things first - have you compared the actual raw events before and after you applied this reduction pipeline? ... View more

1. What do you mean by "statistics" here? 2. If you use tstats with datamodel summaries, you don't deal with raw events. ... View more

Yes, but if CM cannot contact LM, it might not be aware of the fact that your license supports clustering. ... View more

Obviously you have some connectivity problems with your LM or simply your LM is down. I haven't had this kind of problems lately but I suppose if you restarted your CM _and_ it wasn't able to contact LM to confirm that your installation is actually licensed for cluster operations it _might_ refuse to work.  ... View more

Even if you had a common field join would probably be a bad idea - there are better solutions. Still, if you have two separate datasets, how would you want to join them? You can't just randomly add fields from another data set? Actually, you can (that's what appendcols is for) but it will not make any sense. So what would be the logic behind those two? Of course, you can simply append those two searches but as @gcusello already pointed out you'll gonna get a lot of empty space in your table since some of your result rows will have one set of fields, some wil have another. ... View more

If you can't find ready-made TA's (which even if exist may be of varying quality), you have several areas to cover. Sometimes there are some choices you have to make on your own, sometimes those are just imposed on you. 1. Ingestion method. Obviously anything that doesn't require writing own code, handling authentication and checkpoints like syslog is preferred over API-based methods provided you can get the same data both ways. 2. Basic parsing - event breaking, timestamp assignment. That should be pretty straightforward. Can get tricky if the same data stream (see p.1; can happen with multiple different "kinds" of data sent over a single syslog stream or written to the same file). 3. Basic field parsing - if you have multiple available formats providing the same set of data you have to decide on the format which will be easy to parse and possibly not overly verbose so you don't waste too much license on non-essential stuff. So you might - for example - prefer json over XML and a well-defined format over CEF (don't touch CEF with a six-foot pole! it's gonna end in tears) 4. CIM-compliance. This is often the most tricky and labor-intensive part. The problem here is that often if you've already dealt with such technology before, you've probably done that for some particular customer (or employer) and are not at liberty to release it freely. Sometimes, some solutions use well-known formats or at least use standard open-source components which produce relatively standard logs. For example, many web-related components produce one of typical access_log formats (like access combined), and many soultions have some standard SMTP daemon embedded in them (often postfix). EDIT: Oh, and make sure _what_ you actually want to onboard. For example - Trellix (formerly FireEye) components have two separate event "streams" one is operational logs which are often overly "chatty" and not used very much in reality, another is security alerts which is probably what you'll wanna go for. ... View more

OK. I've never used dbconnect outputs, so it was a bit confusing for me (typically dbconnect "in my parts" is used either as a source for inputs or to query external DB from within the search). But yes, generally speaking (with some caveats typical for federated search) you can have a local on-prem SH searching against your Cloud stack using federated search and outputting the data into your local dbconnect outputs. Having said that, you could also just spawn a search using API, wait for it to complete and fetch the result (and possibly push to your DB of choice) without the trouble of maintaining local Splunk instance. And not having to deal with all the DBconnect quirks. I'm not saying that it's a _better_ solution. Just showing possible different approach. ... View more

It's not all there is. Remember that per_* metrics only report top X (10 by default) entries for each given statistic. See https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/10.2/configuration-file-reference/10.2.0-configuration-file-reference/limits.conf#metrics-0 ... View more

OK. So with an unsupported driver, that's more or less what you should have done, indeed. I'm not 100% sure about the serviceClass (the example in the docs says "None"). What errors do you get in the webui? And what errors - if any - you get in dbxonnect's logs? ... View more

What have you tried so far? Did you read https://help.splunk.com/en/splunk-cloud-platform/connect-relational-databases/deploy-and-use-splunk-db-connect/4.3/configure-and-manage-splunk-db-connect/create-and-manage-database-connections#ariaid-title5 ? ... View more

Adding to already excelent answers from @bowesmana and @ITWhisperer - depending on what kind of insights you want to provide, you might take a look at splunk premium apps (in this case it probably would be ITSI). While it might not necessarily look like an IT Service, some of your requirements might translate to ITSI KPIs and services. ... View more

1. Certification marked as "legacy" doesn't mean "useless". It just means that its development stopped at some particular version of Splunk software and while it still touches a lot of relevant topics, it might not be using some newly introduced features. 2. Splunk ES certified admin is someone certified in administering Splunk's Enterprise Security. Obviously this certification is as useless for someone wanting to work with another SIEM as another SIEM vendor's certification is useless for Splunk-related jobs. 3. Whether certification is relevant... that hugely depends on the market. Some markets completely (or almost completely) ignore certifications of any kind and rely mostly on experience and demonstrated skills whereas other markets require certifications simply for compliance reasons. So the usual answer applies here - it depends. As an additional opinion (purely my own, mind you) - while one may argue whether you can get Splunk Core knowledge without going through the official certification path (there are some opportunities to do some stuff with core Splunk product on your own, your company can get you a dev/test license and so on), it's not that easy with ES. So proper admin training with ES is simply harder on your own. ... View more

This is a tricky problem if you don't have access to the Splunk itself (if you do - you can create a list of buckets to back up using @richgalloway 's method). As you can see here - https://help.splunk.com/en/data-management/manage-splunk-enterprise-indexers/9.0/indexing-overview/manage-index-storage/how-the-indexer-stores-indexes#bucket-names-0 the bucket directory name contains its ID. You should _not_ have two distinct buckets with the same ID. So that's one thing -  you can "deduplicate" on bucket ID. To be on the safe side you can use ID and start/end times. But it's not all that easy since if you have SF<RF some bucket copies might be searchable, some not. So you would need to check which copy is the searchable one for copying. You could just copy the biggest one but this way you still might be backing up too much because within the bucket directory you might get addiitonal data - datamodel acceleration summaries, saved search acceleration summaries - there can be a lot going on (if you don't have separate volume defined for those). So _I_ would probably go over each bucket ID from each cluster node for a given index and copy over the biggest bucket directory (and then check if there is no "discardable" data contained within). ... View more

OK. That's a more reasonable approach. Unfortunately, it's quite unusual, that's why I tend to expect the "we want everything green" stance. I haven't touched java for ages but from what I remember JMX doesn't use javascript. Fastjson must then be used intenally by the add-on. To what extent and how much it relies on user-supplied content (and if it's at all susceptible to the potential exploitation) - that's beyond my knowledge (and probably most people at this forum). So you should call Splunk support if you have in-depth technical questions about this add-on (it's Splunk-supported so it's perfectly OK to raise tickets about it). As I said - I haven't worked with java for a long time so I don't remember how java linking works but you could try replacing the fastjson jar whch comes with the add-on with a newer version. Still if it worked that would mean that maintaining this walkaround is on you (at least until a next verion with an updated fastjson comes out).   ... View more

OK. Let's start from the top. We have no idea how the data _is supposed to_ get to your Splunk. Judging from the directory layout, you have a forwarder alongside a rsyslog daemon. Rsyslog receives the data from the network and saves events to files in directories. From those directories monitor inputs read events. Am I on the right track? If you have multiple sourcetypes from a single source, you most probably have some configuration (props/transforms) "recasting" sourcetype for a specific type of events. So there are several questions here: 1) Do those "missing" events don't come at all or are they indexed with a "wrong" sourcetype? 2) Do they even reach your rsyslog receiver? 3) Have there been any changes introduced lately in your environment. ... View more

Wait a second. You're mixing several things here. 1. The CVE you provided is about fastjson, it has nothing to do with JMX. 2. Splunk does not ship with java components. As far as I remember It used to have some auxiliary tools (not essential to the general Splunk operations and very rarely used at all) but they were removed in latest releases. So. It's not about Splunk itself. You might be thinking about the add-on for JMX - https://splunkbase.splunk.com/app/2647 which indeed ships with fastjson-1.2.83.jar. You can raise a support ticket asking about whether they are planning on addressing this CVE any time soon (this is a Splunk-supported add-on so you can ask support about it) but the main question here is _how_ and _if_ said vulnerability is applicable to your use case. I'm a bit allergic to the "we just want our vulnerabilty scanner to be all green" approach since it usually involves no thinking about the actual results and no risk assesment. ... View more

This is one of the possible methods - we used a similar approach for some packages in $job-2 - build a custom RPM named <something>-stdconf which would depend on the base package and would perform configuration steps, enable the service, set up users and so on. But I'm not a big fan of this approach - RPMs are not well suited for this and - especially if the base package changes (like Splunk's migration from splunk to splunkforwarder user) you have to handle a lot of transition cases. And it quickly gets messy. ... View more