Yes, but if CM cannot contact LM, it might not be aware of the fact that your license supports clustering. ... View more

Obviously you have some connectivity problems with your LM or simply your LM is down. I haven't had this kind of problems lately but I suppose if you restarted your CM _and_ it wasn't able to contact LM to confirm that your installation is actually licensed for cluster operations it _might_ refuse to work.  ... View more

Even if you had a common field join would probably be a bad idea - there are better solutions. Still, if you have two separate datasets, how would you want to join them? You can't just randomly add fields from another data set? Actually, you can (that's what appendcols is for) but it will not make any sense. So what would be the logic behind those two? Of course, you can simply append those two searches but as @gcusello already pointed out you'll gonna get a lot of empty space in your table since some of your result rows will have one set of fields, some wil have another. ... View more

If you can't find ready-made TA's (which even if exist may be of varying quality), you have several areas to cover. Sometimes there are some choices you have to make on your own, sometimes those are just imposed on you. 1. Ingestion method. Obviously anything that doesn't require writing own code, handling authentication and checkpoints like syslog is preferred over API-based methods provided you can get the same data both ways. 2. Basic parsing - event breaking, timestamp assignment. That should be pretty straightforward. Can get tricky if the same data stream (see p.1; can happen with multiple different "kinds" of data sent over a single syslog stream or written to the same file). 3. Basic field parsing - if you have multiple available formats providing the same set of data you have to decide on the format which will be easy to parse and possibly not overly verbose so you don't waste too much license on non-essential stuff. So you might - for example - prefer json over XML and a well-defined format over CEF (don't touch CEF with a six-foot pole! it's gonna end in tears) 4. CIM-compliance. This is often the most tricky and labor-intensive part. The problem here is that often if you've already dealt with such technology before, you've probably done that for some particular customer (or employer) and are not at liberty to release it freely. Sometimes, some solutions use well-known formats or at least use standard open-source components which produce relatively standard logs. For example, many web-related components produce one of typical access_log formats (like access combined), and many soultions have some standard SMTP daemon embedded in them (often postfix). EDIT: Oh, and make sure _what_ you actually want to onboard. For example - Trellix (formerly FireEye) components have two separate event "streams" one is operational logs which are often overly "chatty" and not used very much in reality, another is security alerts which is probably what you'll wanna go for. ... View more

OK. I've never used dbconnect outputs, so it was a bit confusing for me (typically dbconnect "in my parts" is used either as a source for inputs or to query external DB from within the search). But yes, generally speaking (with some caveats typical for federated search) you can have a local on-prem SH searching against your Cloud stack using federated search and outputting the data into your local dbconnect outputs. Having said that, you could also just spawn a search using API, wait for it to complete and fetch the result (and possibly push to your DB of choice) without the trouble of maintaining local Splunk instance. And not having to deal with all the DBconnect quirks. I'm not saying that it's a _better_ solution. Just showing possible different approach. ... View more

It's not all there is. Remember that per_* metrics only report top X (10 by default) entries for each given statistic. See https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/10.2/configuration-file-reference/10.2.0-configuration-file-reference/limits.conf#metrics-0 ... View more

OK. So with an unsupported driver, that's more or less what you should have done, indeed. I'm not 100% sure about the serviceClass (the example in the docs says "None"). What errors do you get in the webui? And what errors - if any - you get in dbxonnect's logs? ... View more

What have you tried so far? Did you read https://help.splunk.com/en/splunk-cloud-platform/connect-relational-databases/deploy-and-use-splunk-db-connect/4.3/configure-and-manage-splunk-db-connect/create-and-manage-database-connections#ariaid-title5 ? ... View more

Adding to already excelent answers from @bowesmana and @ITWhisperer - depending on what kind of insights you want to provide, you might take a look at splunk premium apps (in this case it probably would be ITSI). While it might not necessarily look like an IT Service, some of your requirements might translate to ITSI KPIs and services. ... View more

1. Certification marked as "legacy" doesn't mean "useless". It just means that its development stopped at some particular version of Splunk software and while it still touches a lot of relevant topics, it might not be using some newly introduced features. 2. Splunk ES certified admin is someone certified in administering Splunk's Enterprise Security. Obviously this certification is as useless for someone wanting to work with another SIEM as another SIEM vendor's certification is useless for Splunk-related jobs. 3. Whether certification is relevant... that hugely depends on the market. Some markets completely (or almost completely) ignore certifications of any kind and rely mostly on experience and demonstrated skills whereas other markets require certifications simply for compliance reasons. So the usual answer applies here - it depends. As an additional opinion (purely my own, mind you) - while one may argue whether you can get Splunk Core knowledge without going through the official certification path (there are some opportunities to do some stuff with core Splunk product on your own, your company can get you a dev/test license and so on), it's not that easy with ES. So proper admin training with ES is simply harder on your own. ... View more

This is a tricky problem if you don't have access to the Splunk itself (if you do - you can create a list of buckets to back up using @richgalloway 's method). As you can see here - https://help.splunk.com/en/data-management/manage-splunk-enterprise-indexers/9.0/indexing-overview/manage-index-storage/how-the-indexer-stores-indexes#bucket-names-0 the bucket directory name contains its ID. You should _not_ have two distinct buckets with the same ID. So that's one thing -  you can "deduplicate" on bucket ID. To be on the safe side you can use ID and start/end times. But it's not all that easy since if you have SF<RF some bucket copies might be searchable, some not. So you would need to check which copy is the searchable one for copying. You could just copy the biggest one but this way you still might be backing up too much because within the bucket directory you might get addiitonal data - datamodel acceleration summaries, saved search acceleration summaries - there can be a lot going on (if you don't have separate volume defined for those). So _I_ would probably go over each bucket ID from each cluster node for a given index and copy over the biggest bucket directory (and then check if there is no "discardable" data contained within). ... View more

OK. That's a more reasonable approach. Unfortunately, it's quite unusual, that's why I tend to expect the "we want everything green" stance. I haven't touched java for ages but from what I remember JMX doesn't use javascript. Fastjson must then be used intenally by the add-on. To what extent and how much it relies on user-supplied content (and if it's at all susceptible to the potential exploitation) - that's beyond my knowledge (and probably most people at this forum). So you should call Splunk support if you have in-depth technical questions about this add-on (it's Splunk-supported so it's perfectly OK to raise tickets about it). As I said - I haven't worked with java for a long time so I don't remember how java linking works but you could try replacing the fastjson jar whch comes with the add-on with a newer version. Still if it worked that would mean that maintaining this walkaround is on you (at least until a next verion with an updated fastjson comes out).   ... View more

OK. Let's start from the top. We have no idea how the data _is supposed to_ get to your Splunk. Judging from the directory layout, you have a forwarder alongside a rsyslog daemon. Rsyslog receives the data from the network and saves events to files in directories. From those directories monitor inputs read events. Am I on the right track? If you have multiple sourcetypes from a single source, you most probably have some configuration (props/transforms) "recasting" sourcetype for a specific type of events. So there are several questions here: 1) Do those "missing" events don't come at all or are they indexed with a "wrong" sourcetype? 2) Do they even reach your rsyslog receiver? 3) Have there been any changes introduced lately in your environment. ... View more

Wait a second. You're mixing several things here. 1. The CVE you provided is about fastjson, it has nothing to do with JMX. 2. Splunk does not ship with java components. As far as I remember It used to have some auxiliary tools (not essential to the general Splunk operations and very rarely used at all) but they were removed in latest releases. So. It's not about Splunk itself. You might be thinking about the add-on for JMX - https://splunkbase.splunk.com/app/2647 which indeed ships with fastjson-1.2.83.jar. You can raise a support ticket asking about whether they are planning on addressing this CVE any time soon (this is a Splunk-supported add-on so you can ask support about it) but the main question here is _how_ and _if_ said vulnerability is applicable to your use case. I'm a bit allergic to the "we just want our vulnerabilty scanner to be all green" approach since it usually involves no thinking about the actual results and no risk assesment. ... View more

This is one of the possible methods - we used a similar approach for some packages in $job-2 - build a custom RPM named <something>-stdconf which would depend on the base package and would perform configuration steps, enable the service, set up users and so on. But I'm not a big fan of this approach - RPMs are not well suited for this and - especially if the base package changes (like Splunk's migration from splunk to splunkforwarder user) you have to handle a lot of transition cases. And it quickly gets messy. ... View more

Ok. Again. Too many words, too little relevant information. Focus on the Splunk side of things firstly. What is your setup? What components are you using? What is their configuration? (and again, you're talking about Deployment Server but then you mention observability and post this in Observability Cloud section of the forum). Also, we have no idea how your app works and what those processing steps mean, how they generate/export events and so on. It's as if I asked you "hey, I got a truck but sometimes I can't get enough speed and have to abort the take off; do you think it can be an issue with tyre pressure, carburetor or the recent clutch overhaul?". ... View more

To be honest, I have no idea what "kickstart file" the docs refer to here. The only kickstart file with RH and RH-based distros I know is the kickstart you can write to automate the OS installation. And generally, at scale, probably the most maintainable solution is to install a clean UF, drop in an app pointing to deployment server (and maybe another one with local Root CA certs if you're using one) and enable boot-start, start service whichever way you feel most comfortable with - shell script, ansible, whatever.   ... View more

And this vulnerability actually affects you how? ... View more

OK. 1. Do not just blindly copy-paste some AI slop. And please use some formating, break the text into paragraphs - generally, make the thing readable. 2. You posted in Observablility Cloud section of the forum but your AI-generated content is pointing to a Splunk Enterprise/Splunk Cloud specific features. So make up your mind - which is it? ... View more

To be fully honest, it was challenging to go through this stream of conciousness. Lack of proper paragraph splitting and almost nonexistent punctuation didn't help. But as I understand, there is an instance of a UF and an instance of a Splunk Enterprise server. There is an output defined, but it returns winsock error 10061 on connection attempts. The error 10061 is "Connection refused" which means that either receiving is not enabled (input is not configured) on the 9997 port on the server, or there is something blocking connectivity on 9997 port. That's it - there's no big mistery to it. ... View more

Depends on where your "inbound" and "outbound" is 🙂 If your DS is "inside" then the connections are "inbound". If you are looking from the UF's point of view, they are outbound. ... View more

Do you have Python for Scientific Computing app installed? litellm is a part of that app. https://splunkbase.splunk.com/app/2882   ... View more

Well, this is how it works. You download the package, you install it and from this moment you have the trial period to evaluate the software. After that period you either use the limited free tier license or - if you need the full functionality - buy the proper commercial one. For some purposes, if you are eligible, you might be able to obtain a special license (Dev, Dev/Test...) but they have their limitations both in terms of allowed use as well as set of features. If you are looking to potentially buy the software in the end but need more time to verify if it fits your business case, you might contact your local Splunk Partner or Splunk sales team directly to obtain an Enterprise Trial license. But this will probably involve a PoC installation, not just "here you go, that's a license for you". But it might depend on your market. ... View more

No. The DS doesn't initiate data transfer over the connection. These are all REST requests over 8089 port. Either the UF reports its "identity" or pulls the needed apps but it is always the DS responding to UF's requests. ... View more

No. You are wrong here. DS never calls the UF. UF phones home with the defined polling frequency. Then if the config bundle that UF has stored locally differs from what the DS reports as the current state, the UF pulls updated apps from the DS. It's always this way - the data flow is bidirectional but the connections are always initiated from the deployment client (usually a UF) to the DS. ... View more