There is no single comprehensive document listing all versions that far back. You have to go to "How to Upgrade Splunk Enterprise" section of the Install Guide (for 10.4 it's https://help.splunk.com/en/splunk-enterprise/get-started/install-and-upgrade/10.4/upgrade-or-migrate-splunk-enterprise/how-to-upgrade-splunk-enterprise) and switch further "back" to earlier versions to build your own upgrade path. It will probably be something like 9.0->9.2->9.4->10.2. But you might need to consult the Compatibility Matrix as well (here is a small problem - I don't think there is a compatibility matrix for unsupported versions available on Splunk website so you have to dig through archive.org) for supportability of premium apps (ES, ITSI...) with specific Splunk versions. And of course remember about verifying which of your apps in general are supported with specific Splunk versions. ... View more

Keep in mind that if you have multiple DS instances you must set [global] crossServerChecksum = true in serverclass.conf on all of them. Otherwise the checksum calculated for an app will be based (also) on modification time which might result in different checksums across nodes (getting you into endless loop of app "updates").  ... View more

Again - it depends 😉 But seriously - if your SIEM works mostly on relatively fresh data you're mostly focused on current data for which constant integrity verification might not be possible or feasible. If you're relying on correlating historical data, performing integrity verification periodically might be useful. OTOH, integrity control is essential or close to essential (local legal requirements might also add another layer to this issue) if you want to be able to use the data as evidence. On yet another hand, remember that depending on the system we're talking about, modifying already ingested data might simply not be easy due to how it's structured. ... View more

It's generally the usual "who watches the watchers" question and an answer in any given scenario depends on threat model, appetite for risk and of course acceptable cost (not limited to direct financial cost). So broadly speaking - sure, additional layer of integrity control provides some technical means which could be used (depending on implementation) for setting up detective controls or corrective ones. I've seen environments where events were pushed into two different systems. I've seen various levels of integrity controls implemented in different systems - they all protect against some scenarios but obviously not against all possible ones. So the general answer to the broad question is - as almost always - it depends. ... View more

This doesn't make much sense. Why not just monitor the file? Noone prohibits you from reading a static file. ... View more

What manipulation are you talking about? At what level? The SIEM's own logs or the logs processed by SIEM? Are you aware how Splunk (and other SIEMs) works and what integrity controls it provides? ... View more

search and where are two different commands and may work differently. That's expected. One obvious difference is that search can be used as an event generating command (the initial search) and can be optimized "into" the initial search if it's relatively early in the search pipeline while where is a streaming command. Another thing is that where can work on any predicate, possibly comparing different values from the same result row, whereas search can only compare to static terms. And finally - I'm not sure if that's sufficiently well documented but as Splunk sometimes does some implicit type casting, in this case the commands differ as to how it's done. In your case where is treated strictly - your condition x_wbrs_score<4 is fulfilled only if there is a field called x_wbrs_score and its numerical value is lower than 4. With search, the search term is treated more loosely - with string values it checks lexicographical order. You can - a bit counterintuitively - get the same result as with search command if you do  | where x_wbrs_score < "4" In this case where will still cast the compared value from a string "4" to a number 4 and compare the field value but will compare lexicographically if your field value is a string. (yes, some implicit type casts are unexpected at times). ... View more

And where does it not work the way you'd expect it to? (I have some lab guide for reference but it's fairly old - 2021, so it might not contain current content). ... View more

It's actually a question about Exchange. Splunk just tries to authenticate to the server. From Splunk's point of view if you want to use OAuth with a specific account used for sending emails, you have to be able to use OAuth on this particular account. Whether on the server's side it means that OAuth must be enabled on whole organization or if it's enough to have it limited to a specific scope (group, account, whatever) is up to the server side. ... View more

If you're really looking for something suitable for this forum, there's no reason why it shouldn't be publicly stated in this thread. If not, please read the community rules. ... View more

That's a very general question and it's deceiptively difficult to get an exhaustive answer, especially since we don't know your data and your search. 1. As @ITWhisperer already pointed out, the string value might have additional whitespaces which are not shown in UI 2. The value might come from a multivalued field. 3. If you are doing "<initial search> | search field=value", the search might get optimized by Splunk to a single "<initial search> field=value" one which in specific cases might introduce some non-obvious effects coming from breaking events into search terms. I'm pretty sure there are more things which can affect the search and matching process. So while we might be able to tell you more if you show us a sample of your data (anonymized if needed) and your search, the answer will most probably be in your job details and search log. ... View more

Contact details are provided on the Splunkbase app page. You can try mailing the author if app's behaviour/configuration is not obvious beyond what is described in the Details and Installation tabs (I admit, the Splunk API key use is not obvious for me either) ... View more

Your SPL is quite straightforward so either: 1. The fields you're trying to use are not extracted and you're trying to use fields which aren't there. Or 2. The fields are ther but aren't numerical. The usual case is when they are multivalued ... View more

I'm not quite sure what you mean that your server has "NAT IP". I suppose you mean that it can be reached by means of connecting to IP A which is by some router NAT-ed to IP B which is your server's address. Generally speaking, in order for the TLS connection to be considered properly validated, the cert presented by the server must: - come from the certification chain originating at CA trusted by the client - the whole chain must not break any constraints and must not contain expired certs (I'm not 100% sure of validity of non-expired certs which had been issued by CAs which already expired) - the subject you're connecting to (either host name or IP, depending on what you are using to initiate the session) must match the data in the certificate presented by the server (either the subject field or one of the Subject Alternative Name (SAN) values) So if you have your server configured at 192.168.101.101 to which your internal DNS points from splunk.local hostname and your certificate is issued for splunk.local hostname only (or for both the hostname and 192.168.101.101 IP although it's a common practice to _not_ issue certs for IPs) it will _not_ be valid for any other name or IP. So if some part of your company would try to connect to your Splunk at 172.16.101.101, the cert will not match properly. If they use a name of splunk.external.local, it won't match either obviously. As far as I remember, while you can bind different certificates to specific inputs (although TLS settings on inputs have their share of problems), I don't recall binding separately (with different certificates) with webui instances. So you must either make sure that all your users connect using the same name(s) which are resolved by DNS to IPs apropriate for the client's environment (which can be tricky) or you must add all "endpoints" (names and IPs) as SANs to your certificate (which is very ugly and possibly leaking information especially if you're mixing private and public names and IPs; whatever private and public means in your case) ... View more

In this case you can just take down servers one by one generally in any order. Just remember to enable maintenance mode on the cluster so it doesn't start rebuilding searchable copies immediately. ... View more

There are some unknowns here. Do you actually have a 1+1 cluster? What is your RF/SF? What is your data flow? I'm assuming you want to have as little downtime as possible, right? Remember that forwarders can and will pause inputs if they cannot send to outputs (if those are the "stoppable" inputs; you can't "hang" syslog packets on a wire). Also you will undoubtedly have downtime during SH upgrade. Generally in a normal scenario you should not be needing to "point" anything anywhere becaue in a well-engineered setup you should be load-balancing outputs over your available indexers so any single-node downtime should not affect ingest flow. Also, be very careful about the in-place windows upgrades. They can go wrong. I would consider doing backup/restore over a clean server. As long as the names and IPs stay the same, the process should be trivial. Especially if you indeed have index storage on a separate storage which can be easily "plugged into" a new/restored instance. ... View more

If you want to anonymize your data, please use another character/string as placeholder. Asterisk has its usual meaning as a wildcard and thus is expected to be taken literally when encountered in such context. That's one. Two - The usual technique of avoiding join is appending two searches and doing stats over the combined result by common field. But. This requires planning since if you simply use the append command you can still hit the limits for subsearches and get incomplete/wrong results. You can use multisearch to fetch data from multiple searches but those searches must be streaming ones (i.e. cannot contain stats command). Three - it still doesn't solve the problem that you're trying to create the sat_host field based on value of a non-existent field. ... View more

OK. There are several things which catch my eye right off the bat. hostname!="*.corp" Two things "wrong" here (I mean, the search will run but it will be terribly inefficient). 1. Inclusion is better than exclusion. Splunk searches events by search terms and only those pre-selected events are verified against field extractions. In other words, if you do hostname=foo  Splunk will find all events where there is a "foo" anywhere and will check which of those has "foo" in place of the hostname field. If you do hostname!=foo Splunk has to check every single event to see if it contains "foo" and if it does, is it in the hostname spot or not. 2. Splunk indexes terms which come from splitting the input data on breakers (spaces, tabs and such). And keeps a lexicon of those terms with entries pointing to the events containing those terms. So if you're looking for "foo" or "foo*" Splunk can find in its lexicon all entries being "foo" or starting with "foo". If you do "*foo", there is no such magic. Splunk has to scan every single event to find this string. Another thing - eventstats is a relatively "heavy" command. I assume that since you're pulling data from some summary, you might have it already relatively well "compacted" but in a general case, be aware that it can be very memory-consuming since it needs to have whole result set to operate on. And probably the main culprit -  join type=inner sat_host [ search index="its-*-*tec-app" sourcetype="*:*:agent:reports" source="D:\\*eData\\*Agent_*.csv" earliest=-1d latest=-1m" | rename "Machine Name" as sat_host | eval sat_host = upper(sat_host) | stats latest(IP) as IP latest(Version) as Version_*, latest("Last Update Received") as last_update_recieved by sat_host] 1. You can't just write "as Version_*". Wasn't it supposed to be "latest(Version_*) as Version_*" (wildcard on right side of AS is allowed only as a match to left-side one). 2. More importantly - join with a raw event search usually ends in tears. Join command is usually best avoided altogether. Sometimes it's ok with a predictably fast search (like a small tstats or inputlookup). It practically never ends well with an index search. Join has a lot of limitations (result count, execution time) so it can get silently finalized and produce wrong/incomplete results without you ever knowing it. 3. You're joining on sat_host field which obviously (if your results before the join really look like what you're showing from the csv dump) does not exist in your data. You're doing  | stats latest(os) as os latest(Last_Status_time) As Last_Status_Time by sat_host Which leaves you with fields: - os - Last_Status_Time - sat_host so when you arrive at | eval sat_host=upper(replace(DeviceName, "\..*$", "")) there is no DeviceName field in your result set. ... View more

This is a "search app" it makes no sense to install it anywhere else. Also, quoting the Splunkbase description: "This add-on is available for Linux (64-bit), Windows (64-bit) and Mac. Make sure you install the appropriate one for your Splunk deployment." Notice the "appropriate for your Splunk deployment", not "for your endpoint". ... View more

There are some unknowns here regarding your detailed use case and data but I see you're already pretty proficient with debugging Splunk's internal workings. 😉 I do not have anything to support my suspicion but I have a feeling that since Splunk has changed over last few big versions around indexed fields autodiscovery I suppose it might be adding that lispy part for indexed field "just in case" and it could be difficult to get rid of.  Anyway, the lispy is sound. Splunk should not just "give up" on scanning the lexicon. What I'd try first (assuming that since your lispy shows just one VALUE) is verifying how many results you'd  get with  index=indexhere sourcetype="json_sourcetype" TERM(VALUE) (Choose a value which is unlikely to appear in other fields). You'll see if Splunk has this term correctly indexed and only has problems with parsing. If your event is big enough, you could be hitting some limits on auto-kv number of fields... ... View more

I don't recall this app having any web-configurable settings in this up. You're supposed to configure your ISE to send the logs to your Splunk via syslog and receive the syslog on your Syslog, possibly using SC4S or another syslog component. There are no modular inputs to configure in this add-on as far as I remember. ... View more