Have fun fiddling with this 🙂 That's how you learn and start "thinking splunk". ... View more

Assuming these are correct (have you checked those?): 1. The upgrades did finish correctly 2. The UF processes are indeed running on the source machines. The things to check are: 1. Do you have any logs from the upgraded machines in the _internal index? 2. Do you have logs about the UFs connecting to your deployment server if you're using one. 3. Do you have any logs about connection attempts from those source hosts in your _internal log? 4. Contents of splunkd.log on source machine(s). As far as I remember I saw some issues when the UF was set to use TLS on output but didn't have any client cert configured (even though client cert authentication wasn't used) - it tried to connect without TLS. Since the input was TLS-enabled, the connection would fail. ... View more

Generally, as @ITWhisperer already pointed out - sudo functionality can be limited to only allow very specific sets of commands (even with predefined set of arguments) but if you're disabling it due to minimizing possible exposure, that's your policy, not ours 😉 Anyway, in order to be able to upgrade a system-wide component (in this case - a Splunk forwarder package), the upgrader must be able to execute commands with elevated privileges. I'm not a very big fan of this upgader anyway, I prefer to manage the UFs (in terms of the software itself of course; for configuration I recommend the Deployment Server) using whatever the customer already uses for managing endpoints. ... View more

Adding a bit of context to @gcusello 's answer - Splunk uses the indexes.conf file for propagating the UI fields. So there is no other way to make sure you "see" those indexes on your HFs than to define them in indexes.conf (which effectively creates an empty instance of those indexes on your HF). If you manage your Cloud programmatically, you just have to extend your playbooks or whatever automation you use to create index both in Cloud and on your HFs (preferably you should use DS to manage your HFs; at least in regard to distributing common configuration items). If you do it by hand... well, you have to create them on-prem as well. ... View more

There are two... relatively independent things here. One is _what_ to look for. And that's unrelated to Splunk itself. You must have relatively good working knowledge of the systems which logs you're gonna be analyzing. Without it, even with the SPL proficiency at the ultimate-guru level you won't find anything because you won't know what you need. Another thing is SPL skills. And here it's getting a bit tricky because it's different when you're working "interactively" with Splunk - like doing threat hunting. You usually don't mind whether your search runs 10s or 30s. If you don't know how to transform the data into the form you want you often can do the same thing in smaller steps, over smaller time ranges, by saving some intermediate results, sometimes doing some parts of your job by hand if you can't directly do stuff in Splunk. Your SPL proficiency is not _that_ critical. Of course you must know basics and the usual "do nots" so that your local Splunk admin won't kill you for searching "index=*" over "All time" and the general "mid-level" skills are good to have. But if you're writing scheduled searches - that's another cup of tea. If you want - for example - your ES detections to be fast and effective so that your environment can run more detections with the same hardware... you need to write as good searches as you can - leverage all the tricks you know. That comes with time. I cannot help you with the "what" part but for the "how" - just do SPL. Install Splunk Free edition at home, get some real data (from your local environment, maybe your home internet router, maybe your laptop...) and try to do some dashboards over this data. Then see if you can make them more efficient. Also, go through old .conf presentations about searching. For me a great source of SPL skills was this forum - I browsed through other people's problems and thought how I would approach them and then confronted my ideas with other people's solutions. ... View more

OK. I didn't have much time to sit on your problem (but I haven't forgotten about it :)) So, rephrasing what I said before - if you don't have anything different than _time to correlate on, you have to assume that your events are sparse enough so you can rely on their sequence. Otherwise, especially if the SQL queries you're trying to track are intertwined, you'll get wrong results. Having this out of the way. After running this | multisearch [ search index=windowsevents EventCode=33205 SourceName=MSSQLSERVER (database_name=prod_prosjekt) (action_id="UP") (session_server_principal_name!=prod_prosjekt AND session_server_principal_name!="sa") ] [ search index=esak-audit ] | sort _time You should get your results in format of <windows event with query1> <before values> <after values> <windows event with query2> <before values> <after values And so on (after values can be listed before the before values; it's important that the windows event with the query itself preceeds those). Now the hard part - extracting the fields we need from the SQL statement. It is an ugly task because it's generally impossible to do  right for a general case with just regexes. You can search for a particular form of SET expression but it might fail if you have some escaped values end so on. So I will have to make some assumptions about your SET syntax - they contain either numbers or strings in single quotes without any commas in them (this is important because I rely on the fact that comma separates set statements. So we have | rex field=message "set\s*(?<set>,?\s*([^=]+\s*=\s*([-\dxA-F.+]+|'[^']*')+)*)" | rex field=set max_match=0 "(^|,\s*)(?<fields>[^=\s]+)\s*=" Now we have the field "fields" having multiple values of names of the fields you wanted to modify. Now the part which is not that hard but a bit tricky. Since we rely on the order of events we have to "copy over" the set of fields in the "fields" field (boy, that's confusing XD). Unfortunately, if we do simple streamstats last(), it will only copy one value, not all of them so we need to combine them | eval fields=mvjoin(fields,",") so we can copy them over to our "audit" events | streamstats last(fields) as fields (if you have the whole SQL statement extracted as well you could do the same with the statement field) and break them up again into a multivalued field | eval fields=split(fields,",") So now we can finally rename our fields as "before" and "after" | foreach * [eval "<<FIELD>>_{operation}"=if(fields="<<FIELD>>",<<FIELD>>,null())] Unfortunately you cannot use the {} notation with mode=multivalue of foreach so you have to iterate over everything and only create before/after fields if the field's name matches our list of fields to list. Now all that's left is to "flatten" those results. It probably could be done differently but I'm relying on the fact that we have 3 events - SQL query, before and after.  | streamstats count as mergecounter | eval mergecounter=floor((mergecounter-1)/3) | stats min(_time) as _time values(*) as * by mergecounter And all that's left is pure cosmetics -  | fields _time statement *_after *_before (assuming you have your statement extracted as a whole; if you don't - just drop this field from the expression). And that's pretty much it. I fancy a guess that it might work without resorting the input stream but I'm not 100% sure in what order multisearch returns the results so it's a safeguard. ... View more

See this article https://help.splunk.com/en/splunk-enterprise/search/search-manual/10.2/manage-jobs/dispatch-directory-and-search-artifacts ... View more

OK. First things first. If you don't have any field which you can use to correlate the cdc logs with the windows log, that's gonna need a bit of guessing and you'll have to rely on the order of what has happened (and it will break if you have things reported out of order or have multiple sql queries running at the same time; but the same thing would happen with your initial map-based approach - you simply have no way of telling which operation did which modification). If you can safely assume that those logged transactions are relatively "sparse", you're fairly safe. Anyway, it now popped into my mind, why not just compare which fields did change? If you have the cdc event which lists _all_ fields before and after the operation, why not just list the fields which changed? (unless you also want to show the case when the SET operation requested an update on a field but it already had the same value). ... View more

OK. What have you tried so far and what are your results/findings? ... View more

OK. And I assume that in this case as a result you'd want to return the time of the operation, the database table name, and (again - in this case), the INNSTBEL value before and after the change. I also see a "client" field in your original attempt but I do not see it in the data. But that's for later. The question is whether there is any field strictly correlating both those logs - not just giving you the rough time approximation but pointing to a specific search.  I see the start_lsn_hex and seqval_hex fields which seem to contain some form of identifiers. Are they unique to an update operation or are they some form of bitmask or similar so they repeat throughout your data? The problem with correlating based solely on time is that you might have two or more concurrent or very closely spaced operations in source 1 which might be difficult to distinguish in source 2 (especially if - for example - source 1 logs something at the beginning of an operation whereas source 2 - at the end or vice versa). ... View more

There are no miracles. If it used to work but suddenly stopped, something must have changed. In case of TLS-secured connectivity it's usually expiration of the certs. Have you checked the usual suspects - trying to connect to the database endpoint from your dbconnect Splunk instance with openssl s_client and checking the certs presented by the server and verifying what certs you have configured on your dbconnect's side if you're using cert-based client authentication (I'm not sure if it can be used with Oracle; just listing the typical issues). ... View more

I believe the add-on should be perfectly fine working with 10.6. Since this is an author-supported add-on, you might call out to them asking them for compatibility info update. ... View more

OK. Let me start from dissecting your search so far and pointing out a few things. index=windowsevents EventCode=33205 SourceName=MSSQLSERVER (database_name=prod_prosjekt) (action_id="UP") (session_server_principal_name!=prod_prosjekt AND session_server_principal_name!="sa") This part is generally OK since you already have several conditions narrowing your search pretty well but remember that exclusion on its own is not a very good way to search. So if you just had the session_server_principal_name on its own, it would be very ineffective. (You can check how many rows are scanned vs. how many results are returned in your job details). | table _time, object_name, action_id, server_principal_name, statement OK. Here's my first objection. The table command in general should not be used early in the search. It moves the processing from the indexers to the search heads. So you no longer benefit from distributed searching. It also gathers all results Splunk has so far so you're not processing data in a streaming fashion. If you want to limit the set of fields processed further (due to memory constraints you might want to remove _raw sometimes for example), use the "fields" command. | eval attrs="extract commaseparated list of the attributes" I suppose this is a placeholder for opration(s) which would result in the "attrs" field being filled with names of several fields extracted from the event, right? | append [ search index=windowsevents EventCode=33205 SourceName=MSSQLSERVER database_name=prod_prosjekt action_id=UP (server_principal_name!=prod_prosjekt AND server_principal_name!="sa") | eval E=_time-3, L=_time+5, tabell="cdc.dbo_"+object_name+"_CT" | map search="search index=esak-audit tablename=$tabell$ earliest=$E$ latest=$L$ | fields _time, tablename, operasjon, client, <here I want to be able use attrs> | fields - _raw | eval result_type='audit_logs' " ] OK. This one has more issues than just map. First and foremost - unless you have a very well-defined case when the appended search is sure to be run quickly and return a small set of result, don't use it. Sometimes append can be optimized out internally to multisearch but it works only for specific searches and you can't count on it always. In this case your appended subsearch does not qualify for changing it to a multisearch. So splunk will try to run the appended subsearch and might hit its limits (60 seconds runtime by default and 10k results returned). Since the appended search contains map, it will be "heavy" - for each row of the "base subsearch" Splunk will try to spawn a whole search - create it, parse, spawn a search job to indexers, fetch results, aggregate them... it's a lot of work. This is definitely _not_ the kind of search you'd want to have as an appended search. Another thing which you overlooked is the scope. You're not only trying to dynamically list fields, you're trying to pass them over _from another search_ - look at it, you're generating the "attrs" field in the "first" search and want to use it in the appended search which - for Splunk - is a completely separate search and Splunk only aggregates results from those searches. Other than that those searches do not share any common "knowledge". This is definitely _not_ how you should approach this. Especially since you're doing two separate "runs" over the same result set (the part  search index=windowsevents EventCode=33205 SourceName=MSSQLSERVER database_name=prod_prosjekt action_id=UP (server_principal_name!=prod_prosjekt AND server_principal_name!="sa") is common to the initial search and the appended search. So the issue here is that you're thinking about the problem as a programming problem - about things to _do_ whereas with Splunk you have to think about data manipulation. Think about it in terms of two data sets and combining them. Please post at short snippet of those events: index=windowsevents EventCode=33205 SourceName=MSSQLSERVER (database_name=prod_prosjekt) (action_id="UP") (session_server_principal_name!=prod_prosjekt AND session_server_principal_name!="sa") and those events: search index=esak-audit Anonymize them of course if they contain sensitive information. It's about the structure and correlation, not about the data as such. ... View more

Ok, for your "disappearing" reply - this forum has an automatic clasifying system which usually works quite well (without it we'd have tons of spam) but once every blue moon can falsely flag something as spam. Do report those. Back to our main issue - this explanation is getting a bit vague without examples 🙂 It's usually best if you post a snippet of your (anonymized if needed) data and expected results. ... View more

You mean you're just testing how the command works as a form of exercise? That's OK. Just remember that in production there can be restrictions on using unsafe commands. ... View more

Again, this is very unlikely that the type of the disk was the root cause. Not entirely impossible but highly unlikely. Especially since cold storage is not heavily used in terms of writes. Maybe you unknowingly touched some other issue. If the storage was too be the culprit here it's way more probable that it would be the hot/warm storage than cold. ... View more

Ingestion latency is hardly ever caused by storage. In very very rare cases it might be affected by search overload and badly tweaked server parameters but it is relatively unlikely. OK, it might be caused by storage if you have severely undersized environment but then you'd see much more bad symptoms (like skipped searches) than just ingestion latency. ... View more

OK. The obligatory warning - map is an unsafe command and is not to be used lightly. In most of use cases map can be substituted for another SPL construct to get the same results. Here be dragons, you've been warned. Now. You can simply use two "variables". | makeresults count=4 | eval a="splunk_server" | eval b="host" | map search="search index=esak-qa sourcetype=eport:access | table _time index $a$ $b$" Anyway, what's the underlying problem? What is it you're trying to do that you're resorting to use map? ... View more

The ingestion latency monitoring mechanics is quite tricky and can (and often does) go sideways. As far as I understand, it's based on a var/spool/splunk/tracker.log file. This file is created by Splunk process (both on a full Splunk Enterprise instance as well as on Universal Forwarder) and is ingested as batch:// input (with a sinkhole policy removing it after read). So with a normally working installation the running process should on the one hand create a timestamped entry in that file, which should get ingested by the input and the file should be removed. Then further down the stream Splunk compares the actual date with the date in this timestamp. There are several things which can happen here. One thing is obviously that the output from the source forwarder might be clogged and there can indeed be significant latency. Another thing is that the source UF is not properly time synced. I also had a case when the input for unknown reasons (I didn't troubleshoot it further at that time since simple restart solved the problem) wouldn't remove the file. In that case the running process would append another timestamp to the end of the file but the input would read the file again and again from the start, with old timestamps.   ... View more

And here's where things get tricky. It's usually not a good idea to use indexed extractions on json data... (but that's a topic for a completely different discussion). If we're talking just parsing and truncating - I believe you should be OK. ... View more

As @kiran_panchavat already mentioned, this is an OS-level "issue". Splunk is just reporting whatever error message it got from the operating system. Since this is Windows I suspect there might be something about "languageness" of a particular release of the OS. I used to have a lot of problems with perfmon counters in one particular version of multilanguage Windows setup. ... View more

You cannot send a custom HTML mail contents with the built-in send email alert action. You might try the sendresults app (but I don't remember if it does that) or the TA-html-email one. Generally speaking however, allowing custom HTML, especially if it would be depending on external content, could prove risky. ... View more

Short answer is no. There is no built-in functionality which does ingestion of a whole file depending on its contents. You could try to tackle this problem by using your own script to move files based on contents into a "spool" directory for ingestion. Or by writing your own scripted input (assuming it's a UF) which would selectively read files. But that would mean handling checkpoints and such; it's not a quick and dirty one-liner. If the contents on the other hand are distinguishable on a per-event basis, you could try filtering them out after initial pickup by the input (if you want to save on bandwidth you could enable additional processing on the UF). ... View more