See this article https://help.splunk.com/en/splunk-enterprise/search/search-manual/10.2/manage-jobs/dispatch-directory-and-search-artifacts ... View more

OK. First things first. If you don't have any field which you can use to correlate the cdc logs with the windows log, that's gonna need a bit of guessing and you'll have to rely on the order of what has happened (and it will break if you have things reported out of order or have multiple sql queries running at the same time; but the same thing would happen with your initial map-based approach - you simply have no way of telling which operation did which modification). If you can safely assume that those logged transactions are relatively "sparse", you're fairly safe. Anyway, it now popped into my mind, why not just compare which fields did change? If you have the cdc event which lists _all_ fields before and after the operation, why not just list the fields which changed? (unless you also want to show the case when the SET operation requested an update on a field but it already had the same value). ... View more

OK. What have you tried so far and what are your results/findings? ... View more

OK. And I assume that in this case as a result you'd want to return the time of the operation, the database table name, and (again - in this case), the INNSTBEL value before and after the change. I also see a "client" field in your original attempt but I do not see it in the data. But that's for later. The question is whether there is any field strictly correlating both those logs - not just giving you the rough time approximation but pointing to a specific search.  I see the start_lsn_hex and seqval_hex fields which seem to contain some form of identifiers. Are they unique to an update operation or are they some form of bitmask or similar so they repeat throughout your data? The problem with correlating based solely on time is that you might have two or more concurrent or very closely spaced operations in source 1 which might be difficult to distinguish in source 2 (especially if - for example - source 1 logs something at the beginning of an operation whereas source 2 - at the end or vice versa). ... View more

There are no miracles. If it used to work but suddenly stopped, something must have changed. In case of TLS-secured connectivity it's usually expiration of the certs. Have you checked the usual suspects - trying to connect to the database endpoint from your dbconnect Splunk instance with openssl s_client and checking the certs presented by the server and verifying what certs you have configured on your dbconnect's side if you're using cert-based client authentication (I'm not sure if it can be used with Oracle; just listing the typical issues). ... View more

I believe the add-on should be perfectly fine working with 10.6. Since this is an author-supported add-on, you might call out to them asking them for compatibility info update. ... View more

OK. Let me start from dissecting your search so far and pointing out a few things. index=windowsevents EventCode=33205 SourceName=MSSQLSERVER (database_name=prod_prosjekt) (action_id="UP") (session_server_principal_name!=prod_prosjekt AND session_server_principal_name!="sa") This part is generally OK since you already have several conditions narrowing your search pretty well but remember that exclusion on its own is not a very good way to search. So if you just had the session_server_principal_name on its own, it would be very ineffective. (You can check how many rows are scanned vs. how many results are returned in your job details). | table _time, object_name, action_id, server_principal_name, statement OK. Here's my first objection. The table command in general should not be used early in the search. It moves the processing from the indexers to the search heads. So you no longer benefit from distributed searching. It also gathers all results Splunk has so far so you're not processing data in a streaming fashion. If you want to limit the set of fields processed further (due to memory constraints you might want to remove _raw sometimes for example), use the "fields" command. | eval attrs="extract commaseparated list of the attributes" I suppose this is a placeholder for opration(s) which would result in the "attrs" field being filled with names of several fields extracted from the event, right? | append [ search index=windowsevents EventCode=33205 SourceName=MSSQLSERVER database_name=prod_prosjekt action_id=UP (server_principal_name!=prod_prosjekt AND server_principal_name!="sa") | eval E=_time-3, L=_time+5, tabell="cdc.dbo_"+object_name+"_CT" | map search="search index=esak-audit tablename=$tabell$ earliest=$E$ latest=$L$ | fields _time, tablename, operasjon, client, <here I want to be able use attrs> | fields - _raw | eval result_type='audit_logs' " ] OK. This one has more issues than just map. First and foremost - unless you have a very well-defined case when the appended search is sure to be run quickly and return a small set of result, don't use it. Sometimes append can be optimized out internally to multisearch but it works only for specific searches and you can't count on it always. In this case your appended subsearch does not qualify for changing it to a multisearch. So splunk will try to run the appended subsearch and might hit its limits (60 seconds runtime by default and 10k results returned). Since the appended search contains map, it will be "heavy" - for each row of the "base subsearch" Splunk will try to spawn a whole search - create it, parse, spawn a search job to indexers, fetch results, aggregate them... it's a lot of work. This is definitely _not_ the kind of search you'd want to have as an appended search. Another thing which you overlooked is the scope. You're not only trying to dynamically list fields, you're trying to pass them over _from another search_ - look at it, you're generating the "attrs" field in the "first" search and want to use it in the appended search which - for Splunk - is a completely separate search and Splunk only aggregates results from those searches. Other than that those searches do not share any common "knowledge". This is definitely _not_ how you should approach this. Especially since you're doing two separate "runs" over the same result set (the part  search index=windowsevents EventCode=33205 SourceName=MSSQLSERVER database_name=prod_prosjekt action_id=UP (server_principal_name!=prod_prosjekt AND server_principal_name!="sa") is common to the initial search and the appended search. So the issue here is that you're thinking about the problem as a programming problem - about things to _do_ whereas with Splunk you have to think about data manipulation. Think about it in terms of two data sets and combining them. Please post at short snippet of those events: index=windowsevents EventCode=33205 SourceName=MSSQLSERVER (database_name=prod_prosjekt) (action_id="UP") (session_server_principal_name!=prod_prosjekt AND session_server_principal_name!="sa") and those events: search index=esak-audit Anonymize them of course if they contain sensitive information. It's about the structure and correlation, not about the data as such. ... View more

Ok, for your "disappearing" reply - this forum has an automatic clasifying system which usually works quite well (without it we'd have tons of spam) but once every blue moon can falsely flag something as spam. Do report those. Back to our main issue - this explanation is getting a bit vague without examples 🙂 It's usually best if you post a snippet of your (anonymized if needed) data and expected results. ... View more

You mean you're just testing how the command works as a form of exercise? That's OK. Just remember that in production there can be restrictions on using unsafe commands. ... View more

Again, this is very unlikely that the type of the disk was the root cause. Not entirely impossible but highly unlikely. Especially since cold storage is not heavily used in terms of writes. Maybe you unknowingly touched some other issue. If the storage was too be the culprit here it's way more probable that it would be the hot/warm storage than cold. ... View more

Ingestion latency is hardly ever caused by storage. In very very rare cases it might be affected by search overload and badly tweaked server parameters but it is relatively unlikely. OK, it might be caused by storage if you have severely undersized environment but then you'd see much more bad symptoms (like skipped searches) than just ingestion latency. ... View more

OK. The obligatory warning - map is an unsafe command and is not to be used lightly. In most of use cases map can be substituted for another SPL construct to get the same results. Here be dragons, you've been warned. Now. You can simply use two "variables". | makeresults count=4 | eval a="splunk_server" | eval b="host" | map search="search index=esak-qa sourcetype=eport:access | table _time index $a$ $b$" Anyway, what's the underlying problem? What is it you're trying to do that you're resorting to use map? ... View more

The ingestion latency monitoring mechanics is quite tricky and can (and often does) go sideways. As far as I understand, it's based on a var/spool/splunk/tracker.log file. This file is created by Splunk process (both on a full Splunk Enterprise instance as well as on Universal Forwarder) and is ingested as batch:// input (with a sinkhole policy removing it after read). So with a normally working installation the running process should on the one hand create a timestamped entry in that file, which should get ingested by the input and the file should be removed. Then further down the stream Splunk compares the actual date with the date in this timestamp. There are several things which can happen here. One thing is obviously that the output from the source forwarder might be clogged and there can indeed be significant latency. Another thing is that the source UF is not properly time synced. I also had a case when the input for unknown reasons (I didn't troubleshoot it further at that time since simple restart solved the problem) wouldn't remove the file. In that case the running process would append another timestamp to the end of the file but the input would read the file again and again from the start, with old timestamps.   ... View more

And here's where things get tricky. It's usually not a good idea to use indexed extractions on json data... (but that's a topic for a completely different discussion). If we're talking just parsing and truncating - I believe you should be OK. ... View more

As @kiran_panchavat already mentioned, this is an OS-level "issue". Splunk is just reporting whatever error message it got from the operating system. Since this is Windows I suspect there might be something about "languageness" of a particular release of the OS. I used to have a lot of problems with perfmon counters in one particular version of multilanguage Windows setup. ... View more

You cannot send a custom HTML mail contents with the built-in send email alert action. You might try the sendresults app (but I don't remember if it does that) or the TA-html-email one. Generally speaking however, allowing custom HTML, especially if it would be depending on external content, could prove risky. ... View more

Short answer is no. There is no built-in functionality which does ingestion of a whole file depending on its contents. You could try to tackle this problem by using your own script to move files based on contents into a "spool" directory for ingestion. Or by writing your own scripted input (assuming it's a UF) which would selectively read files. But that would mean handling checkpoints and such; it's not a quick and dirty one-liner. If the contents on the other hand are distinguishable on a per-event basis, you could try filtering them out after initial pickup by the input (if you want to save on bandwidth you could enable additional processing on the UF). ... View more

Haven't touched 4.21 yet but I would suspect that this functionality has indeed been added in 4.21 but the other part of the docs hasn't been updated yet. But I'd just go and verify it in lab, especially if you want to rely on this in production. ... View more

That's exactly what I was about to point you to 🙂 First thing to do is to check _which host_ the message comes from. With messages about structured parsing I'd be looking into either KV_MODE/auto kv or indexed extractions. Since search-time kv extractions are limited by number of fields rather than event length I'd go for indexed extractions. This seems to match your sourcetype definition. So you need to adjust your parsing/line breaking setting on the UF because with indexed extractions all this happens on the source UF. ... View more

Your answer is very categorical as a general rule, without knowing a specific organization's circumstances and such. Even SVAs and mechanics taught at advanced administration trainings are just guidelines and are described as such. If you have a border case (or one even beyond that border) you adjust your solution to the terms of the scenario. I've seen scenarios where only permitted traffic flow from an environment was outside. And I'm not talking only about connection direction but data flow. What then? Deployment Server? Sorry, can't do that. If you have whole company running Chef and have engineers who can handle it well and you have in-house skills and knowledge you're gonna push for Ansible because "it's the way to go"? I don't think so. So yes, often those things are a quite effective way of managing your Splunk environment but not always. ... View more

You can't do auto-upgrade for software installed from a RPM/deb package. Unless you have an external privileged service which updates whole package (like I believe the current upgrader service does) you could at best overwrite package's own files only but then they wouldn't match the package manager's database so any integrity check will fail. ... View more

It greatly depends on your needs, tools typically used in your organization, your skill set, policies... There is no "objectively better" way of updating the UFs. I haven't personally tried the remote upgrader app but it seems overly complicated and requires a highly privileged service running as root. You can use any of your configuration management tools, you can repack installation packages. For Linux packages you can create local repo and handle it as any other software (but you'd need to adjust installation scripts so that you don't need to manually accept the license after each upgrade). So there are limitless posibilities. ... View more

You can't really use Graph API add-on nowadays. It's deprecated, no longer supported and from my experience it cannot properly connect and authenticate to the service. The recommended (only?) replacement is Event Hub.  ... View more