Checkout the btool. It will show you where setting is coming from and which .conf is winning.  https://help.splunk.com/en/splunk-enterprise/administer/troubleshoot/9.0/first-steps/use-btool-to-troubleshoot-configurations     ... View more

Hey Telvin, This is a pretty common headache when you're changing multiple things at once new Splunk version, new OS, and potentially updated DoD cert chains. The good news is that "Unauthorized" XML error is actually a useful clue. It tells us Splunk is receiving the CAC cert from the browser but failing to validate it against your trusted chain. So you're close. I'm leaning towards caCertPath file as being the issue. This is the one that trips people up the most. The caCertPath needs to contain every CA certificate required to validate the client (CAC) certificates, not your server certificate chain. Your server cert chain and the user's PIV auth cert chain might share the same root CA, but they typically diverge at the intermediate level. If you only included the intermediates from your server cert, that's why it's failing. What you want to do is export a known-good user's PIV Authentication certificate from their CAC, open it up, and trace the full chain. Every CA in that chain (except the end-entity cert itself) needs to be in your caCertPath file. You're looking at things like DOD ID CA-59, DOD ID CA-62, the DOD Root CAs, etc. Easiest approach: grab the full DoD CA bundle from DISA's PKI-PKE page (public.cyber.mil/pki-pke), pull the P7B out, and convert it: openssl pkcs7 -in certificates.p7b -print_certs -out dod_ca_bundle.pem Use that as your caCertPath or make sure all the relevant CAs from it are in your chain file. The .key vs .pem thing for privKeyPath, don't worry about it. That's not a breaking change. Splunk has always accepted PEM-formatted private keys regardless of the file extension. Your existing .key file will work fine as long as it's PEM-encoded and matches your server cert. The documentation just shifted to referencing .pem as a naming convention. Not a functional difference. That said, if you want to be safe, you can convert to PKCS#8 format: openssl pkcs8 -topk8 -nocrypt -in server.key -out server_pkcs8.pem   A few Splunk 10.x things to double-check in your web.conf: Make sure requireClientCert = true is explicitly set. Without it, Splunk won't even ask the browser for the CAC cert. Also check whether sslCommonNameToCheck is set. For CAC auth you typically want to leave that unset or it can cause validation failures. And you might need allowSslRenegotiation = true since Windows Server 2022 handles TLS renegotiation a bit differently than 2016 did. Windows Server 2022 gotchas: 2022 ships with TLS 1.3 on by default and has stricter Schannel behavior. A couple things to look at: make sure the Splunk service account actually has read permissions on the cert and key files (Windows ACLs are tighter by default in 2022), and check Event Viewer for any Schannel errors during your login attempts. Group policies enforcing specific TLS settings can also conflict with Splunk's internal OpenSSL handling. For troubleshooting: You can test your chain independently with: openssl verify -CAfile dod_ca_bundle.pem user_piv_auth_cert.pem If that comes back OK, your chain file is good. If it fails, you're missing CAs. Also check the Splunk logs: index=_internal sourcetype=splunkd_ui_access "Unauthorized" index=_internal sourcetype=splunkd log_level=ERROR ssl OR certificate OR CAC The splunkd_ui_access log should tell you whether the client cert DN was even received. If it's blank, Splunk never got the cert from the browser, which is a different problem. My best guess on root cause: Since your HTTPS/TLS is working fine for the web portal itself, the server cert setup is good. The failure is happening when Splunk tries to validate the client certificate from the CAC. Build out a comprehensive caCertPath from the full DoD CA bundle, restart Splunk, and I'd bet that gets you there. Let us know how it goes, happy to help dig deeper if it's still not cooperating after that. ... View more

Here's a couple of projects from Splunk's github page: https://github.com/splunk/docker-splunk https://github.com/splunk/splunk-operator I'm working through this as well, so I don't have a review or either yet ... View more

This is a really solid set of requirements, and honestly, you’re asking the right questions. You’re well past “does the product have feature X” and deep into “does this actually work at scale, under real constraints, without breaking policy or budgets.” At a high level, yes, the kind of UXM you’re describing is achievable. But a lot of what you’re asking about lives in the gray space between product capability, architecture, deployment patterns, and licensing. Things like true “at the glass” experience, meaningful change correlation across domains, privacy-safe data collection, and enterprise-wide data sharing aren’t things the community can answer with a simple yes or no. That’s also where you’re probably starting to hit the ceiling of community help. These are the kinds of questions that really need to be walked through with your Splunk account team and solutions architects who can talk specifics around supported designs, scale limits, and how this plays out in real environments. If you want to go deeper or compare notes (or flat out running into a wall with who to contact), feel free to message me directly. I’m working with a other similar groups in the tackling very similar UXM problems, and I’m always happy to nerd out on architectures, tradeoffs, and what actually works in practice. ... View more

Thanks for the screenshot. That helps a lot. I'm thinking that the metric is ingesting fine and SignalFx is creating time series, but they’re likely keyed on high-cardinality Kubernetes dimensions, such as pods or pod UID,  instead of a stable Kafka identity. That’s why Usage Analytics shows activity but charts come back empty. How about try adding a stable Kafka dimension for example: kafka.cluster, broker.id, or broker.name. That way you're not relying on k8s.pod.uid as the primary identity. Then, in the chart, explicitly group by the Kafka dimension and use mean or max. Definitely locked into finding out if this helps or not. ... View more

The line application = vault is the issue. It's not supported as a stanza in the TA. The link below has the supported values.  Configure the Splunk Add-on for Google Workspace - Splunk Add-on for Google Workspace That being said, what you could do is write a short script that uses the Google API to pull the Vault audit event into a custom input Vault Audit Activity Events  |  Admin console  |  Google for Developers ... View more

@173022  edit: I just realized we're in the AppD section, so this probably wont work as it's the Splunk Enterprise answer Are you talking about the servers that Splunk is running on, or your enterprise servers? Identity servers, files servers, email servers, etc?   If it's your Splunk servers, you can use the metrics. If it's the others, you first have to get that data into Splunk. The add-ons for windows and linux with grab these metrics, but you may have to enable it in each stanza.  Once you know the data is going in, it's a simple search that you turn into a report and post it to your dashboard  ... View more

As pointed out, you should contact your Splunk account team. While the Splunk ML toolkit is a free add-on, ITSI is a paid app. In theory, yes, you could just use the ML toolkit and build out what you want. However, the ITSI group has done the heavy lift, and they provide the support and maintenance of the app.   ... View more

This is a fun one, and your logs actually show the collector is doing its job. Since the metrics show up in Usage Analytics, SignalFx is receiving them. When charts show “0 time series,” it’s usually a dimensions problem, not an ingestion problem. Kafka JMX metrics often come in with attributes like type=produce or type=fetchconsumer, but they’re missing stable identity fields like host.name, service.name, or a cluster identifier. SignalFx needs a consistent set of dimensions to form a time series. If those change between scrapes, the metric exists but won’t chart. A second gotcha is that many of these metrics are reporting 0 or constant values. If you’re using rate or delta functions, or the wrong rollup, the chart can look empty. Make sure you’re treating them as gauges and using something simple like “Last value” or “Mean.” One more thing to note: the Kafka JMX receiver you’re using is still alpha. It’s known to emit inconsistent metadata for some MBeans, which makes this behavior more likely. That’s why queueSize works. It’s a simple internal metric with clean, stable dimensions. The Kafka and JVM JMX metrics need a bit of normalization first. Try this: -Add explicit service.name, host.name, and a Kafka cluster tag in the OTel Collector -Normalize dimensions so every datapoint looks the same -Chart with gauge semantics, not rates ... View more

What Splunk products are you using? Just Enterprise? ... View more

Splunk doesn’t behave like a traditional data lake, so it helps to frame the answers in Splunk terms. Splunk is fundamentally schema-on-read. Data lands as raw or semi-structured events, and structure is applied at search time using field extractions, knowledge objects, data models, and CIM. If you need enforcement, that happens upstream or during ingest through disciplined sourcetypes, DSP pipelines, validation, and standardized TAs. Splunk itself does not enforce schemas at rest. From a schema perspective, native Splunk indexes can store essentially any event format. Structure is implicit and defined by extractions, not storage. When using Federated Search for Amazon S3, schemas are defined externally in AWS Glue Data Catalog tables (for formats like JSON, CSV, or Parquet). In that case, the schema lives in Glue, not in Splunk. Splunk does not automatically normalize time-series data across sources. Normalization is achieved by mapping fields to CIM (or OCSF when working with Security Lake–aligned data), either at ingest or at search time. Data models, macros, and calculated fields are what provide a consistent analytical view across different producers. For ingest, you use the standard Splunk mechanisms: forwarders, syslog, HEC, DSP, and app-based collectors. Queries are executed with SPL via the Search REST API or SDKs. These APIs are designed for analytics and automation, not bulk lake-style exports. Practical limits are driven by search concurrency, quotas, job lifetimes, and result pagination rather than a single hard rate limit. External tools do not directly query Splunk-managed storage. They query Splunk through its APIs. If data physically resides in S3, external analytics tools query S3 directly using Glue schemas, independent of Splunk. Splunk acts as an analytics and correlation layer, not a general-purpose lake endpoint. Splunk can federate to external object storage. Federated Search for Amazon S3 allows SPL queries over data stored in S3 without ingesting it into Splunk indexes. This is commonly used for long-term retention, compliance data, or very large volumes where ingesting everything isn’t cost-effective. At large scales (for example, 10 TB/day), Splunk requires deliberate architecture and licensing. The typical pattern is to ingest a curated, high-value subset for fast detection and correlation, while leaving the bulk of the data in S3 and using federated search for longer lookbacks. Retention is managed through index policies, and cost optimization is achieved through federation and archiving rather than treating Splunk as the primary data lake. I can assure you though, once built out right, Splunk has no issues soaring about 10TB per day. Integration with external data lakes and analytics platforms usually follows two models: federate when you need access without ingest, and curate-and-ingest when you need speed, correlation, and operational context. Most mature environments use both. Access control uses standard Splunk RBAC. Roles are explicitly granted access to native indexes and federated indexes. The same permission model applies across on-platform and federated data. From a compliance standpoint, Splunk Cloud Platform is FedRAMP High authorized. DoD and federal customers should validate the exact authorization boundary and region, and remember that external S3 buckets fall under their own authorization and controls. Auditability is handled through Splunk’s built-in _audit index and the Audit Trail app. This covers user activity, searches, and configuration and knowledge-object changes. In Splunk terms, “schema changes” are changes to extractions, data models, or configurations, all of which are auditable. ... View more

This is super easy to do, but you'll need to have your apps setup right on the deployment server. There You would create an app that has the index location, and the then the right inputs.conf containing the stanzas to pull the windows info and then what index it should go to.  How you structure the apps is up to you. The important piece is getting then setup on the deployment server. The when you install the UF on the windows hosts, use SCCM and only give the msiexec installer command  the address of the deployment server. Don't put in an indexer here.  When the agent loads, the hosts will check into the deployment server, pull the right apps, provided you've got it set right, and all your data will got where you want it out of the gate and not hit main. ... View more

Hey Looking at this and have a few questions that hopefully should help 1. I see you have ESXi sending syslog to SC4S TCP and RFC6587.  Is it going to a dedicated VMware port or a shared one? 2. For the misclassified events, what does the program field show (for example sshd)? 3. Can you paste one raw auth.log or shell.log event as it hits Splunk? 4. Did you restart the SC4S container after adding it? ... View more

FortiGate isn’t handled the same way as iDRAC in SC4S. SC4S doesn’t have a Fortinet specific TLS listener, so swapping TCP to TLS in the env vars won’t actually do anything. Essentially, there is no SC4S_LISTEN_FORTINET_TLS_PORT, which is why your logs stopped showing up. For Fortinet, you should enable TLS globally and the logs should be sent to the default TLS syslog listener, not a Fortinet-named port. The correct setup would be to enable TLS and configure a default TLS port, for example 6514 or whatever custom port you want to use. FortiGate then sends syslog over TLS to that port. Because you’re no longer using a Fortinet-specific listener, SC4S may not automatically tag the events as Fortinet based on port. The right way to handle that is to use source-based metadata overrides (by FortiGate IP or hostname) to force the correct sourcetype and index. That’s a pretty common when multiple vendors share the same TLS listener. Also make sure the FortiGate side matches what SC4S is listening for. Fortinet TCP syslog often uses RFC6587 framing, and switching to TLS can change behavior depending on the options you pick. If the framing or mode doesn’t match, SC4S will accept the connection but not parse the data correctly. ... View more