I've worked with Splunk in financial-services environments for a long time and built observability tooling on top of it, so here's how I'd approach this - Splunk-first, but architected so you never regret it. Get your ingestion architecture right first. The mistake I see most is apps writing straight to HEC with hardcoded endpoints. It works on day one, but you've welded your application code to a specific pipeline. Put an OpenTelemetry Collector in front instead — apps emit structured telemetry, the Collector handles routing, enrichment, and redaction, and HEC becomes one exporter behind it. This is squarely where Splunk itself is heading; Splunk is one of the biggest backers of OTel and it's the recommended path into Splunk Observability Cloud. You get clean ingest into Splunk and the flexibility to add or change destinations later through config, not code. Run the Collector agent + gateway. Lightweight agent near each service for local pickup, a central gateway tier for sampling, PII scrubbing, and routing to your indexers. Drop Kafka in front of the gateway if your volume is spiky. Scrub PII and secrets at the gateway before they hit an index — non-negotiable on a platform handling resumes and payment data. Structure your logs for SPL. Structured JSON, consistent fields, and follow OTel semantic conventions for naming. The thing that pays off most: propagate a trace_id/request_id across every service boundary. Being able to transaction or stats by trace_id across services is the difference between reconstructing a failing request in two minutes versus two hours — and it makes your dashboards and RBA far cleaner. Dashboards and alerting in Splunk. Dashboard Studio for the visuals, and lean into Risk-Based Alerting rather than firing on raw error counts — alert on symptoms (error rate, p95/p99 latency, saturation) so the team doesn't learn to ignore pages. RBA aggregating weaker signals into high-confidence notables is exactly the pattern for this. On the broader question of Splunk vs. open-source — Splunk gives you the strongest ad-hoc search and the lowest operational burden on Cloud, which is usually the right call for a single product where you'd rather not run an Elasticsearch cluster yourself. If ingest cost ever becomes your hard constraint at scale, the OTel layer above means you can route a portion of low-value, high-volume data elsewhere without touching application code. That's the beauty of decoupling early — you keep Splunk for what it's best at and stay flexible on the rest. If you share your rough daily volume, whether you're on K8s or VMs, and whether this is ops-only or also security/compliance, I can get a lot more specific. Happy to share a sample OTel Collector -> HEC config too.   ... View more

Please find details below: Test type: Browser test with synthetic transactions (login → Customer Search → Customer 360). Use Chrome Recorder import to build it fast. Prerequisites: Splunk O11y Synthetics entitlement + role to create tests IP allowlisting in two places: your WAF and Salesforce (Login IP Ranges + Trusted IPs) Test target = your My Domain URL, not login.salesforce.com Service account: Yes, required. Dedicated Salesforce integration user (not a person's login), minimum permissions. Handle MFA via Splunk's built-in TOTP variable — grab the secret from the authenticator QR code, store as a TOTP global variable, reference as {{totp.name}}. Alternative: exempt the user from MFA on the allowlisted IPs (needs security sign-off). Biggest gotcha: Salesforce Lightning has dynamic auto-generated element IDs. Use stable selectors (data-*, ARIA labels, link text), not raw CSS/ID paths, or the test breaks on the next Salesforce release. Add explicit waits for async component loading. Key docs: Auth/TOTP: https://help.splunk.com/en/splunk-observability-cloud/monitor-end-user-experience/synthetic-monitoring/advanced-test-configurations/authentication Browser test: https://docs.splunk.com/observability/synthetics/browser-test/set-up-browser-test.html Transactions: https://docs.splunk.com/observability/en/synthetics/browser-test/set-up-transactional-browser-test.html ... View more

For BOTSv3, don't hand-roll it — install the TAs from the dataset's required add-on list plus Splunk_SA_CIM. Those TAs already carry most of the field aliases. Then per sourcetype, CIM compliance is really four things Field aliases - rename source fields to CIM names (user, src, dest, action, etc.) in props.conf. Tags - the part people miss. Models constrain on tags, not sourcetypes. Add an eventtype + tag (tag=authentication, tag=network, etc.) or the model stays empty. Populate recommended fields, not just required ones. Validate with | datamodel Authentication search | search sourcetype=X. Empty result = almost always a missing tag, not a missing alias. ... View more

Is dashboard studio preferred over simple xml? It depends what you weight. Splunk's strategic direction is Dashboard Studio — that's where new visualization work and feature investment go, and it's what Splunk steers new users toward. Simple XML is still fully supported and not deprecated, so existing dashboards aren't going anywhere. The practical catch for your use case: Simple XML still has a couple of drilldown capabilities Dashboard Studio doesn't replicate cleanly, and conditional drilldown is the big one. More on that below. Capturing the clicked column name Yes, this works in Dashboard Studio. As of 9.4, the table visualization supports predefined drilldown tokens where you choose an event of either name, value, or row.<fieldname>.value — and "name" is the field name of the value/location clicked. So clicking a cell in your ERRORS column gives you name = "ERRORS", and the WARNINGS column gives name = "WARNINGS". That's the column-detection you're after. Splunk You'd configure it under the Interactions section of the visualization's config panel, or directly in JSON:   "eventHandlers": [ { "type": "drilldown.setToken", "options": { "tokens": [ { "token": "clicked_col", "key": "name" }, { "token": "clicked_val", "key": "value" } ] } } ] After a click, $clicked_col$ holds "ERRORS" or "WARNINGS" and $clicked_val$ holds the cell's value. Your detail table's search then consumes them, e.g.:     index=your_index | search severity="$clicked_col$" If the column label doesn't match the raw field value, you map it in SPL with an eval/case rather than needing two tokens. Where you'll feel the difference is your phrasing about setting either `ERRORSERRORS ERRORS` or `WARNINGSWARNINGS WARNINGS` depending on the column. Simple XML does this natively with <condition match="..."> blocks that set one token and unset others. Dashboard Studio's drilldown.setToken has no equivalent conditional matcher - this is a long-standing gap that trips people up. You can't say "if column is ERRORS, set this token; else set that one" inside the event handler. So you have two realistic paths: The Dashboard Studio way — set a single token to the column name (as above) and do all the branching in SPL with eval ... case() or if(). This is clean and is how most people handle it in Studio. One token, logic lives in the search. The Simple XML way — if you genuinely need separate $ERRORS$ and $WARNINGS$ tokens set/unset conditionally (e.g., to show/hide whole panels), Simple XML's <condition match="'click.name2'==\"ERRORS\""> pattern is more direct and less fiddly. My suggestion: build it in Dashboard Studio with the single clicked_col token and branch in SPL. You keep Splunk's preferred platform and avoid the one feature it's missing. Only fall back to Simple XML if you specifically need conditional token unsetting for panel visibility that's awkward to drive from a single token. ... View more

ok, you can also try and increase the size  max_memtable_bytes ... View more

Treat the NULL byte as what it almost certainly is: corruption in the source data. A \x00 in a text CSV serves no purpose, and stripping it lets the file stay on the efficient on-disk indexed path with no memory bloat. Where you strip it depends on how the file is produced: If an external process writes the file daily, add a sanitization pass before it lands in the lookups/ dir: tr -d '\000' < raw.csv > clean.csv If the lookup is generated by a Splunk search piped to outputlookup, clean it in SPL right before writing, so it's all one pipeline:   ... | eval myfield=replace(myfield, "\x00", "") | outputlookup mylookup.csv (use a foreach over fields if you don't know which column carries it). You can check and find which field/source injects the NULL, because the usual culprit is an upstream encoding problem — e.g. UTF‑16 data read as UTF‑8 leaves a NUL between every character, or a binary value leaking into a text column. Fixing that at the source is better than scrubbing downstream forever. If this lookup is large, changes every day, and is read constantly, the more architectural option is to move it to a KV Store collection instead of a CSV ... View more

On the last item, the mc_auto_config section is actually fine as written. The value enabled is correct — it's exactly what the official docs show:     [settings]mc_auto_config = enabled https://help.splunk.com/en/splunk-enterprise/administer/monitor/10.4/configure-the-monitoring-console/configure-the-monitoring-console-in-distributed-mode The message you're hitting — Invalid key in stanza [settings] … mc_auto_config (value: enabled) — is a conf-validation warning, not a failure. That "Invalid key in stanza" wording fires when the key isn't found in the matching .conf.spec file. In several builds mc_auto_config isn't listed in splunk_monitoring_console_assets.conf.spec, so btool/conf validation flags it cosmetically even though splunkd honors it at runtime — same class as the old python.version warning that can be safely ignored. So the config isn't broken. To confirm it's actually applying, don't go by the warning — check whether the MC auto-populates in distributed mode (Settings > General Setup, asset table). Per the docs, with mc_auto_config = enabled it takes effect on restart, or automatically every hour with no restart required. ... View more

The issue might be - TLS/cipher mismatch between UF 10.0.5 and DS 10.2.0. Different minor versions can ship different sslVersions / cipherSuite / ecdhCurves defaults, and a FIPS-on-one-side-only setup fails identically. also check if UF tries to connect to DS ove unecrypted HTTP.   ... View more

The Deployment Server does NOT handle data flow. It only manages configurations for your forwarders.  So your forwarders send data directly to Splunk Cloud, but receive their configuration from the Deployment Server. What the Deployment Server manages inputs.conf (what data to collect) outputs.conf (where to send data) props.conf, transforms.conf (parsing rules) Any apps/add-ons you want deployed to forwarders This is incredibly useful when you have many forwarders, since you can push configuration changes to all of them centrally instead of logging into each one. ... View more

Rename/remove Splunk's OpenSSL libraries cd /opt/splunkforwarder/lib sudo mv libcrypto.so.3 libcrypto.so.3.bak sudo mv libssl.so.3 libssl.so.3.bak   Then try starting the forwarder again: sudo systemctl start SplunkForwarder ... View more

I do not think you can do this using lookup but you can achieve  this like  | eval email="[email protected]" | inputlookup user_emails | where like(all_emails, "%" . email . "%") | table all_emails user_address ... View more

@maheshnc  Yes that is possible , you can configure the syslog device to send data to HF and then forward it to the indexers. ... View more

@richgalloway  I am looking for sampling like 10:1 or 100:1 ratio in events to filter out events , not looking for simple filter that we can do in heavy forwarder ... View more

Thanks @richgalloway : can we do the sampling in forwarder in splunk versions 7.2  or old version ... View more

I am looking for data returned to the SH and data returned to the user,I want to know that how much data is queried(not total but unique) vs how much data is not used or not queried by any user or scheduled search. ... View more

move your current search head(etc/apps) to new search head , and change current (search head + indexer) as indexer ... View more

you can extract fields by using props.conf and transforms.conf or extract from web-gui, you can also give different names like param1, param2 etc ... View more

there are multiple ways to know this search directly in indexer in command line : http://docs.splunk.com/Documentation/SplunkCloud/latest/SearchReference/CLIsearchsyntax ... View more

Use dynamic dropdown, take help from: https://answers.splunk.com/answers/148500/how-to-set-up-three-dynamic-dropdowns-on-dashboard-in-simple-xml-on-splunk-6-0-1.html ... View more