Thanks for the clearer shot. I had one of the numbers wrong: the HTTP call is 232 ms, not 4 ms. That's worth correcting because it actually splits the answer into two different reasons. Reading the flow map properly: the HTTP call from ServiceA to ServiceB is 232 ms (async), the ADO.NET/SQLite call to test.db is 4 ms (async), and ServiceA's own work is about 5 ms. So you've got one exit well above the call graph threshold and one well below it, and they're missing from the Exit Calls/Threads column for different reasons. The SQLite call (4 ms) is the simple one. It's below the call graph's minimum capture threshold (min-duration-for-jdbc-call-in-ms, 10 ms by default for SQL/ADO.NET), so it never gets drawn in that column even though the agent recorded it. Drop that threshold toward 1 ms on the test node and it'll start showing. The HTTP call (232 ms) is the interesting one, and the threshold doesn't explain it, because 232 ms is far over any limit. This is the async continuation behavior. When you await the HttpClient call, the method yields and the real 232 ms wait happens on a detached continuation, not inline on the synchronous frames the sampler is showing you (the AsyncTaskMethodBuilder and TaskAwaiter rows). For async exits the agent doesn't nest the call under your method, it surfaces it as an "await@TierName" link in the Exit Calls/Threads column, and it lists every exit in the snapshot's DB & Remote Service Calls tab regardless. So the reliable place to confirm the 232 ms call is that DB & Remote Service Calls tab and the flow map. One thing to check first: make sure the snapshot you're drilling into is actually a full /call instance that contains the HTTP hop (something around the 252 ms total), not a short instance that never made the call. If the Exit Calls/Threads column is still empty on a snapshot that clearly includes the 232 ms call, that points at how your agent renders async rather than at a missing exit. So to your four questions, now with the right numbers: Expected for async? Yes. A fast exit (your DB call) is hidden by the threshold, and a slow async exit (your HTTP call) is handled as an async continuation rather than nested inline in the call graph. Neither means anything is broken. Correlated differently from threads? Yes, and the 232 ms call is the clean example. If it were synchronous it would block the calling thread and get annotated inline against your method. Because it's awaited, it detaches to the continuation and shows as the await link instead. Async backend tracking and thread correlation are separate mechanisms in the agent. Config to populate the column? For the SQLite call, lower min-duration-for-jdbc-call-in-ms. For the HTTP call there's nothing to lower, it's already above threshold, so it's about opening the right snapshot, not a setting. Also confirm you're on a full call graph rather than a partial one, since partials can omit exit calls. The thread-correlation node property only matters if you spawn real threads. Different for synchronous? Yes, meaningfully. Run that 232 ms HTTP call synchronously and it would sit on the stack and show inline in the call graph against your method, with the exit in that column. Async moves it to the continuation. The threshold still applies either way, so the 4 ms DB call would stay hidden even synchronously. One caveat worth knowing: the await link in the Exit Calls/Threads column is documented for the .NET Framework agent. The .NET Core agent can represent async exits differently, so if you never find an await link, treat the DB & Remote Service Calls tab and the flow map as the source of truth for that 232 ms call. Worth a read: Splunk AppDynamics: Call Graphs (the minimum capture threshold) AppDynamics: Asynchronous Exit Points for .NET (the await@tier link) On the drill-down, if you open a snapshot that actually shows the full /call (around the 252 ms total, not the 8 ms or 23 ms instances), do you see the 232 ms HTTP call or an await@ServiceB link in the DB & Remote Service Calls tab or the Exit Calls/Threads column? That'll tell us whether your agent renders async exits with the await-link style at all, or only lists them in the exit-call tab. ... View more

Nice writeup, and good call flagging that the symptoms send you the wrong way. That "certificate is not yet valid" line is the tell every time, because a genuinely bad cert throws a different error (bad signature, unknown CA, expired). "Not yet valid" almost always means time, not TLS. One thing worth adding for the next person who lands here: you actually had two different clock checks in play, enforced by different layers. The cert error you saw is pure X.509, where notBefore is a hard line with zero tolerance, so the instant the validating node's clock sits earlier than the peer cert's start time it fails. That also gives you a direction, since "not yet valid" means the SH was reading time behind the cert, so the drift isn't symmetric and it points you at the box that jumped rather than making you check them all blind. Separately, Splunk has its own skew check when peers connect, and that one is tunable: addpeer_skew_limit defaults to 600 seconds, and there's [search] max_tolerable_skew in limits.conf on the SH. Same root cause, but the cert layer bites first and harder, which is why yours surfaced as a cert error instead of the friendlier "skew_seconds=..." message. On the chrony side, since the trigger was chronyd coming back without re-syncing, the durable fix beyond verifying tracking is making sure it self-corrects on boot. RHEL ships chrony.conf with makestep 1.0 3 and iburst, but if your internal-NTP setup replaced the stock pool lines it's easy to lose those, and without makestep a large boot-time offset just slews slowly forever instead of stepping. So confirm your source lines still carry iburst and that makestep is present. The other common cause of "came back unsynced" is the source not being reachable at boot, a blocked UDP 123 or chronyd starting before the network is up, so that's worth ruling out as well. And since this is a multi-site cluster, the same skew reaches further than distributed search. The cluster manager heartbeats and bucket replication between peers depend on synced clocks too, so it's worth confirming the CM and every peer, not just the SH to peer hops. A skew large enough to fail cert validation on the search head was almost certainly stressing replication at the same time. Worth a read: Splunk: distributed search troubleshooting (clock sync requirement) Out of curiosity, did you pin down whether it was the SH or the peers that drifted at boot? The cert direction points at the peers jumping ahead, but I've seen the search head be the odd one out too. ... View more

I have also tagged the developer (@efheem) on this thread to raise visibility and get his input. ... View more

Hey 0xAli, I haven't run this add-on myself, so rather than guess I pulled the package off Splunkbase and read through the collector code (bin/input_module_guardium_online_report.py). That answers both your questions and turned up a couple of things you'll want to know before you lean on it. 1. Token refresh: no, it won't. The only auth setting is the global auth_token, a bearer token you paste in. There's no client_id, no client_secret, and no Guardium GUI login anywhere in the config or the code. The collector just stamps Authorization: Bearer with that stored token on every call. Since the password grant gives you no refresh_token (check your output, there's only an access_token) and the add-on has no credentials to re-authenticate with, the token expires at the ~3 hour mark (expires_in 10799 is Guardium's default) and the calls start failing. And here's the part that matters more than the expiry itself. When a run fails, the code still advances its checkpoint to the current time and saves it. It does that on the unreachable path, the JSON-parse path, and the error-response path. So once your token expires, every 60 second run fails, logs a warning, and then moves the checkpoint forward to now anyway. By the time you notice and paste in a fresh token, everything between expiry and the fix is gone, with no backfill. So the token issue isn't just "ingestion stops", it's "ingestion stops and silently drops the gap". Worth knowing before you put this in front of anything like compliance reporting. Until the developer adds credential-based auth, the realistic options are to script the token regeneration and write the fresh token back into auth_token before it expires, or to stretch the lifetime with setOAuthTokenExpirationTime. That last one is a global appliance setting with the obvious security tradeoff, and the token still expires eventually, so treat it as a stopgap. Either way, watch the logs for "Guardium API unreachable after retries". That line is how an expired token actually shows up, because the 401 gets caught and retried rather than reported as an auth failure. 2. The online_report call. Two things, and the second is why your command behaves the way it does. The tiny one: it starts with "url" instead of "curl", a paste slip. The real one: your body has no reportParameter block, and online_report returns ErrorCode 1001 without it. The reason the add-on works but your hand-run doesn't is that the collector builds that block for you. Straight from the code, it sends QUERY_FROM_DATE and QUERY_TO_DATE plus the report's runtime params: ServerIP, DBUserName, OSUser, DBName, DBUser, SHOW_ALIASES, REMOTE_SOURCE, SourceApp, FullSQL. So for a manual test, add the same block (corrected call below). reportName also has to match an existing report exactly and the token's GUI user needs access to it, and fetchSize tops out at 30000. A few more things I noticed in the code while I was in there: It treats Guardium error responses as empty. If the report name is wrong or a parameter is off, Guardium replies with a JSON object, and the code logs that as "No data returned" at INFO and stops. So if you ever see "No data returned", don't assume the window was empty, it can be a hidden error, and per the checkpoint behavior above it still advances past that window. Dedup is a one second watermark, not real dedup. It tracks the highest Timestamp at second resolution and keeps only rows greater than it. In a busy SQL feed, events that share that boundary second across two runs can get dropped. Times are local. The query window end uses the forwarder's local clock, and event time is built from the row timestamp as local time, so if the forwarder and Guardium aren't on the same timezone, the window is offset and your event times are skewed. Small one: paging starts indexFrom at 1 instead of 0, so the first row of a result set gets skipped. None of that means it can't work. For a single timezone, lower volume setup with a long lived token it's fine. The checkpoint-on-failure behavior is the one I'd be careful about, especially paired with the 3 hour token. Corrected manual call below, fill in your real date range: curl -k -X POST https://GUARDIUM_HOST:8443/restAPI/online_report \ -H "Authorization: Bearer TOKEN" \ -H "Content-Type: application/json" \ -d '{ "reportName": "SQL Activity", "indexFrom": "1", "fetchSize": 30000, "reportParameter": { "QUERY_FROM_DATE": "2026-05-01 00:00:00", "QUERY_TO_DATE": "2026-05-07 23:59:59", "ServerIP": "%", "DBUserName": "%", "OSUser": "%", "DBName": "%", "DBUser": "%", "SHOW_ALIASES": "true", "REMOTE_SOURCE": "%", "SourceApp": "%", "FullSQL": "%" } }' Docs: IBM: Using Guardium REST APIs (auth flow and token lifetime) IBM: Create online report REST API reference For a token-only input like this, the durable fix is a small scheduled job that re-mints the token and writes it into auth_token before the old one expires, which also sidesteps the checkpoint gap. If you want to stay on this add-on, that's the piece I'd add first. ... View more

Hey licadiw273, fun project. Your instinct is right: OS-level metrics aren't good enough here, and you've already put your finger on why. Once a disk sits behind the PERC, the OS only sees the virtual disk. It has no idea about controller cache pressure, BBU charge and learn cycles, patrol reads, or per-physical-disk predictive and media errors. High iowait with healthy-looking OS counters is the classic "the controller is doing something you can't see" signature, so yes, getting controller-level data is worth it. The catch that trips people up: the H710P is a PERC Series 7 controller (an LSI 9265-8i under the hood). Dell's perccli/storcli only replaced MegaCLI starting at Series 8 (H730P and up), and storcli/perccli usually return "no controller found" on the H710P. For your card the tool is the old MegaCLI (MegaCli64). It feels ugly and unsupported, but it's the one that actually talks to that controller, and it's a single binary, nothing like the full OMSA stack. MegaCLI is where all the granular stuff lives: BBU health and learn cycles: MegaCli64 -AdpBbuCmd -GetBbuStatus -aALL. Watch this one closely. When the battery runs a relearn it silently flips the cache to write-through, write latency jumps, and that is a very common cause of the "intermittent latency I can't pin down" you're describing. Per-drive predictive and media errors: MegaCli64 -PDList -aALL (look at Media Error Count, Other Error Count, and Predictive Failure Count per drive). This is where a flaky SATA disk shows up long before the OS notices. Rebuild and patrol-read progress: MegaCli64 -PDRbld -ShowProg and MegaCli64 -AdpPR -Info -aALL. Virtual disk cache policy and overall controller state: MegaCli64 -LDInfo -Lall -aALL and MegaCli64 -AdpAllInfo -aALL. For the light footprint you want, drop just MegaCli64 on each host and run those as Splunk scripted inputs on an interval. The one bit of work is that MegaCLI output is plain text, not key-value, so you wrap it in a small script that emits field=value (or grab one of the community MegaCli/storcli parsers floating around) so Splunk gets clean fields instead of a wall of text. If you genuinely want agentless, go through the iDRAC instead of the host. Remote racadm (racadm -r IDRAC_IP -u user -p pass storage get controllers, then pdisks, then vdisks) pulls controller, drive, and BBU state straight off the iDRAC with no agent on the OS at all, and you can run it from one collector box against every iDRAC. Redfish is the other agentless route, but iDRAC7's Redfish is old and partial, so confirm the storage endpoints are actually populated on your firmware (2.65.65.65 is the last iDRAC7 build) before betting on it. On the SNMP mess: don't lean on raw traps for steady-state data, they're noisy and unordered by design. Poll the specific storage OIDs on a schedule for the metrics, and keep traps only for the asynchronous events (BBU failure, drive fault, rebuild started or finished). If you haven't seen it, Splunk Connect for SNMP does the MIB normalization that makes raw iDRAC SNMP such a headache, and it's a much cleaner front door than dumping traps straight onto the indexer. So the honest take for a home lab: skip full OMSA. MegaCli64 scripted inputs give you the deepest data per host for almost no footprint, remote racadm if you'd rather stay fully out-of-band, and SNMP through Splunk Connect for SNMP for the event stream. Once the fields are clean, a dashboard for rebuild percentage, BBU state, and per-drive error counts is easy to build. What OS are the R720s running, and are you leaning host-side (MegaCLI) or fully out-of-band (iDRAC)? That decides which of these I'd stand up first. ... View more

Hey shocko — your Procmon read is spot on: this is the metadata/permission merge getting hammered by on-access scanning. The asymmetry is the giveaway. Searches hit big sequential index I/O and feel fine, but opening any config dialog makes splunkd walk the whole knowledge-object tree across system, app and user contexts — hundreds of thousands of tiny .meta/.conf opens plus a Get ACL on each. That small-file-plus-ACL workload is exactly what Windows on-access AV punishes hardest, and your trace shows it: the .meta reads and the Get ACL storm across etc\users, etc\apps and etc\slave-apps. Two things to go after, in order: 1. Get Splunk out of Defender's on-access path. This is Splunk's own recommendation for Windows and almost certainly your biggest win. Exclude the Splunk processes (splunkd.exe and the splunk-*.exe helpers) and the whole install dir ($SPLUNK_HOME, plus $SPLUNK_DB if it's separate) from Defender real-time/on-access scanning — and make sure your EDR layer honors the same exclusions, since EDR often keeps scanning even when the plain AV exclusion is set. Quick proof: in a maintenance window, temporarily turn off real-time protection, open a macro dialog, and see if it snaps open. If it does, you've found it. 2. Shrink what splunkd has to scan every time. Exclusions cut the per-file scan cost; you can also cut the number of files: etc\users is usually the culprit. Every user directory carries its own local.meta and any private knowledge objects, and the permission merge walks all of them on each config load. Count them (dir etc\users) — if you've got hundreds of stale/orphaned user dirs from people who've left or from SSO churn, that's a lot of dead weight. Archive the orphaned ones (back up first), and reassign or clean up private KOs that don't need to be per-user. Trim unused apps. More apps in etc\apps means more .conf/.meta to merge on every load — remove anything you're not actually using. slave-apps means this box is also an indexer cluster peer (or has leftover peer apps from one). Those .meta are in the scan too, so it's worth being clear on the box's full role — a combined search head plus peer is carrying both footprints. One more to rule out: make sure $SPLUNK_HOME\etc is on fast local disk, not a SAN or network volume — the many-tiny-files merge is brutal over any added latency. Worth a read: Splunk Enterprise and anti-virus products (9.4) — the exact processes and directories to exclude Optimize Splunk Enterprise for peak performance — AV, fast local disk, and related guidance Antivirus causing performance issues in Splunk (Splunk support article) How many user directories are under etc\users, and roughly how many apps in etc\apps? That'll tell us whether the AV exclusion alone does it, or whether you've also got an etc\users cleanup on your hands. ... View more

Hey — first off, your diagnosis is dead on, so trust it. "missing 1 required positional argument: 'token'" is a Python TypeError thrown right at the call site in flow_runs_collector.py: the code calls get_flows_by_environment() without the token it requires, and Python bails before any API request even goes out. That's a code defect in the collector, not anything you misconfigured. It also explains why environment discovery and manual flow listing work fine — those are separate code paths that pass the token correctly. The flow-runs path is the one with the broken call. To your three questions: Known issue / is it you? It's a bug in the add-on, full stop — you can't configure past a missing positional argument, and the traceback is definitive. Patch / newer version? As far as I can see on Splunkbase, 1.1.3 (released 2025-05-25) is the current version, so there isn't a newer build to jump to right now. Worth double-checking the app page in case something newer dropped, but don't expect a fix to already be sitting there. Where the fix comes from. This is a third-party add-on (published by Spex), so Splunk Support won't patch it — the fix has to come from the vendor. Report it to Spex Support (https://www.spex.lu) and paste the exact traceback you posted here; you've basically written them a perfect bug report. On their end the fix is just passing the token into that call. One config thing to verify in the meantime, because it'll bite you even after the code is fixed: Flows, Flow Runs and Actions need more than the app-registration credentials. Per the add-on's own notes, those inputs require both the App Registration creds and a Service Account configured with the password OAuth2 flow — environments only need the app-registration token, which is why that part works. Make sure a service account is set on the flow-runs input and that it's excluded from MFA/Conditional Access so the password flow can complete. Without that service-account token, flow runs won't ingest regardless of the bug. If you're comfortable and need it working before Spex ships a build — you're on-prem on 10.4.0, so this is possible (it wouldn't be on Cloud) — you can patch the Python locally as a stopgap. In flow_runs_collector.py around line 31, the get_flows_by_environment() call needs the token passed the same way the working flow-listing path does: grab the service-account token and hand it into the call. Caveats: back up the file first, it's unsupported, and it'll be overwritten on the next upgrade, so treat it as a bridge until the vendor fix lands. I'd only bother if the service account above is already configured, otherwise you'll just trade this error for an auth failure. If you paste the flow_runs_collector.py snippet around that line, I can point at exactly what to change — but the clean path is the Spex bug report plus confirming the service account is in place. ... View more

Hey kedar — your read is correct, and the screenshots pin it down. The error is building the request as "0/events/query", which means the analytics input is taking your Analytics URL value (saved as "0") and tacking "/events/query" onto it — so it never picks up your On-Prem URL. That's why the scheme is missing, and why SaaS works but on-prem doesn't. What's happening under the hood: the analytics search input doesn't talk to the controller — it talks to the AppDynamics Events Service, which is a separate endpoint (default port 9080, https). That's exactly why your other inputs (high level status, license usage, snapshots) are fine — those hit the controller, and your controller connection is good. Only analytics goes to the Events Service, and that's where the URL handling breaks. The "Analytics URL" column is a pick-list of the SaaS analytics domains. On-prem has no matching entry, so it ends up stored as "0", and the input uses that instead of falling back to your "Optional On-Prem URL". That lines up with what you're seeing — it looks like a genuine bug in the add-on's on-prem analytics path, not something you misconfigured. Two things worth trying: Confirm the endpoint you should be pointing at. On-prem analytics is the Events Service, not the controller — typically https://your-events-service-host:9080 (use the load-balancer VIP if the Events Service is clustered), and make sure the Splunk host can actually reach it on 9080/https. A common trip-up is putting the controller URL in the on-prem field; that won't work even once the "0" is sorted. Set the URL directly in the conf as a workaround. These UCC-based connections are stored in the add-on's local conf (look under $SPLUNK_HOME/etc/apps/Splunk_TA_AppDynamics/local/ — folder name may differ — and grep the conf files for the analytics account stanza; you'll find the field sitting at "0"). Back it up, set that field to your full https://your-events-service-host:9080, and restart. If the input reads that field, it sidesteps the broken pick-list. Given it reproduces across multiple versions and only on-prem, I'd also open a Splunk Support case (or report it on the add-on's Splunkbase page) with the error plus the conf showing the analytics URL stored as 0 — that's a clean repro for them. To aim the workaround right: which add-on version are you on, and what did you put in "Optional On-Prem URL" — the controller URL, or the Events Service URL on 9080? ... View more

Hey John — your instinct is right: that parameter is the standard MXBean Map-to-TabularData mapping. A Map on an MXBean (here a String-to-String map) comes across JMX as a TabularData whose rows are CompositeData with two items literally named "key" and "value", and the table is indexed by "key". For setRecordingSettings the map is JFR setting-name to value, e.g. "jdk.ExecutionSample#enabled" maps to "true". The catch — and the real answer to your question — is that a generic MBean console like the one in your screenshot (same with jconsole) can't build a TabularData from that text box. Those fields only take simple scalars; there's no string form the console will parse into a CompositeData/TabularData. So for this operation, through that UI specifically, you basically can't hand it the parameter. It's not a syntax you're missing — the console just doesn't construct open types. What actually works, depending on what you've got: A few lines of JMX client code is the reliable route. Connect with a JMXConnector, build a TabularDataSupport from the operation's own TabularType (rows are CompositeDataSupport with "key"/"value"), then call invoke(). jshell or a quick Groovy script is enough — no full project needed. Jolokia/HawtIO, if that console happens to be Jolokia-based: its exec endpoint takes JSON and converts it into the open types, so a key/value map goes in as a JSON object instead of being hand-built. For JFR specifically, sidestep TabularData entirely. FlightRecorder has String-based operations the console can drive directly: setPredefinedConfiguration(id, "profile") (or "default") applies a built-in profile, and setConfiguration(id, ...) takes a .jfc profile as a plain XML string. If your real goal is "configure this recording," those are far easier than reconstructing the settings map. And outside JMX, jcmd with the JFR.* subcommands, the jfr tool, or -XX:StartFlightRecording do the same with no MBean wrangling. So — what are you actually trying to do: configure a JFR recording (then I'd reach for setPredefinedConfiguration/setConfiguration or jcmd), or do you specifically need to drive arbitrary TabularData operations through a console? And is that console plain JMX-over-HTTP or Jolokia/HawtIO? That decides which path fits. ... View more

Hey javier — your breakdown is solid, and you've landed on the right primary path. Option 1 (the C++ wrapper) is the one that actually gets you COBOL-level visibility in APM, because it's the only approach that creates spans from inside the ACUCOBOL runtime. Option 2 is real but complementary — the Java agent only sees the JVM boundary, so it captures a Java tier and its DB calls, not what's happening inside your COBOL programs. I'd build Option 1 and treat Option 2 as a bonus if you already have a Java front end. A few things I'd lock in before writing much code: Export through a local Collector, not straight to Splunk. Point your C++ wrapper's OTLP exporter at a local Splunk OTel Collector (OTLP gRPC 4317 / HTTP 4318) and let the Collector hold the access token, batch, add resource attributes, and forward to APM. That's exactly your step 4, and it keeps secrets and retry logic off your COBOL hosts. You can bypass the collector and hit the ingest endpoint directly if you must, but I wouldn't for a first build. service.name is everything in APM. Whatever you set as the service.name resource attribute becomes the box on the APM service map, so set it deliberately (one service per COBOL app is a sane start), and add deployment.environment so you can filter prod vs test. Model spans to your code. Program = span, paragraph/section = child span or span event, transaction/batch id as attributes. Set span status to error on abends so failures surface. If this COBOL is called by or calls other instrumented services, propagate W3C traceparent so the spans stitch into the same distributed trace instead of standing alone. One correction on the interop: CALL "C$SYSTEM" runs an OS command — it's not the hook for invoking your wrapper. What you want is ACUCOBOL-GT's C subroutine interface: expose your functions with a C ABI (extern "C" so the names aren't C++-mangled), make them callable from the runtime, then CALL "start_span" etc. from COBOL. The "Interoperating with C" section of the AcuCOBOL-GT (OpenText/Micro Focus) docs covers the linkage and parameter passing. The gotcha that bites people: the batch span processor buffers, so on a short-lived batch run you'll lose spans unless you explicitly flush/shut down the tracer provider at program exit. Init the provider once per runtime process and shut it down cleanly on the way out. And don't span every paragraph if volume is high — instrument at meaningful boundaries (program entry/exit, transactions, DB calls) or you'll drown in overhead and cardinality. How I'd actually get started, smallest provable steps first: Stand up the Splunk OTel Collector locally with OTLP receivers and confirm it forwards to your APM (send a throwaway test span with otel-cli first). Build a minimal extern "C" wrapper over the C++ SDK: init plus one start_span/end_span, service.name set, exporter pointed at the local collector. Prove a single hand-made span lands in APM. Wire the CALL interop from a tiny COBOL test program, one span per run. Confirm the service shows on the map. Expand: attributes, child spans, error status, then traceparent propagation if you're multi-service. Add metrics last. Traces are the APM payoff; the C++ metrics SDK works, but get tracing solid first, and you can always push metrics over OTLP to the same collector. On Option 2: if any COBOL is already fronted by Java (web services, Acu4GL through a JVM), attaching the Splunk Java agent there is basically free APM for that tier and its JDBC calls. Just note APM only sees the DB dependency if the driver runs inside that instrumented JVM — if Acu4GL talks to the database natively from the COBOL runtime, the Java agent won't see it, so that visibility still has to come from your Option 1 spans. Happy to keep going on the thread — better to keep it public so it helps the next person who tries this. For real-time back-and-forth there's the Splunk community #observability Slack, and for a production COBOL rollout it's worth looping in Splunk PS or a partner SE. Docs: Send traces to Splunk APM Compatible span formats / OTLP ingest Get started with the Splunk OTel Collector OpenTelemetry C++ SDK What does the COBOL runtime look like — standalone batch, or fronted by a service tier? That changes whether you need traceparent propagation from day one. ... View more

Hey dspencer — short version: team-based queues are a newer feature (they landed in the 8.4 line), so jumping 8.2 to 8.5 means you crossed the version where this whole thing was introduced. A blank queue right after an upgrade, in a cluster, almost always comes down to one of three things rather than the queue feature itself: the load balancer, KV store state, or knowledge-object replication across the search head cluster. The tell that "new queues create fine but don't populate" says your write path (KV store) is healthy but the read/populate path isn't returning anything — so that's what I'd chase. I'd work through these in order: Bypass the load balancer. Hit one SHC member directly (SSH tunnel to :8000 if you have to) and open Incident Review. If the queues populate there but not through the LB, it's a sticky-session/LB config problem, not ES. Check KV store on every SHC member. Queues live in KV store. Run "splunk show kvstore-status" on each member and check mongod.log for errors — after a big upgrade in a cluster it's common for one member to be stale or for collections not to have migrated cleanly. Confirm the feeding knowledge objects replicated. A classic post-upgrade-in-a-cluster gotcha is the macros/lookups/eventtypes behind Incident Review not making it to every search head; people have fixed blank IR by getting those into the replication bundle. btool the relevant macros on each member and confirm they match. Clear it out. Browser cache, then hit /debug/refresh (or _bump) and reload, to rule out stale UI assets from the upgrade. Check team membership/scoping. Team queues are scoped to teams and roles — if the role-to-team mapping didn't carry across the upgrade, the queue can render empty for your user even though it exists. I checked the 8.5.x known issues and there's nothing listed for this specifically (the only queue-adjacent note is detection versioning being temporarily off in 8.5.0/8.5.1). So if none of the above surfaces it, this is worth a Support case — grab "splunk show kvstore-status" from every SHC member plus the browser network calls behind the empty queue (the API response that feeds it) so they have something concrete to work from. Couple things that'd narrow it fast: is this a search head cluster, and does the queue populate if you hit a single member directly (bypassing the LB)? And is all of Incident Review empty, or just the team queues? ... View more

Hey shenglc — there are two things tangled together here: one is expected behavior, the other looks like a bug. The error first. In ES 8.x a detection can only be mapped to one automation rule at a time — On or Off doesn't matter. So "Failed to create automation rule mapping as detection is already mapped to an automation rule" means that detection is already attached to another rule (or a stale mapping for it exists). To move it, remove it from the rule that currently owns it, save, then add it to the new one. Quick way to find where it lives: open the detection itself (Edit event-based / finding-based detection) — the Details section names the automation rule it's tied to. A second thing that can make detections "vanish": every detection in a rule has to be status = On. If detection versioning is on and a save spins up a new version that isn't enabled (new versions don't auto-turn-on), or a detection otherwise flips off, it drops out of the rule's valid set. Worth confirming all your mapped detections are still On and you're not looking at a superseded, disabled version. Now the part that isn't expected: previously-saved detections disappearing when you add a new one. The 8.5 UI is supposed to gray out detections already used elsewhere so you can't even pick them — so hitting the "already mapped" error and losing other detections on save looks like the mapping save is partially failing and rewriting the rule with a subset instead of cleanly rejecting just the conflicting detection. That's not something you can config around. Practically: Edit one detection at a time, save, then re-open the rule to confirm the list actually persisted before adding the next. Avoid two tabs or two admins editing at once. Before adding any detection, check its detail page to confirm it isn't already mapped elsewhere, and remove it there first if it is. If a detection shows as "already mapped" but you can't find which rule owns it, that's an orphaned mapping — Support needs to clean that one up. Check the 8.5.x release notes and known issues, and since this reproduces, open a Splunk Support case with exact repro steps. There have already been automation-rule regressions reported across the 8.x upgrades, so it's worth putting in front of them directly. Doc: Configure automation rules (ES 8.5) Couple things to narrow it down: are the detections that disappear ones you'd also added (or tried to add) to a different rule, and do you have detection versioning turned on? ... View more

Hey ankit13 — the reason your custom parser "loads but defaults back" is that it's hooked into the wrong stage. An [sc4s-syslog] app that just re-calls app-syslog-sophos_firewall_xg() is competing with the built-in vendor parser at the content-classification step, and the vendor parser's own selection logic still wins. That's working as designed — you can't out-filter it from there. For your exact situation (you know the source IPs, you can't change the payload or hostnames), the supported fix is vendor_product_by_source (VPS). It runs before content classification and pins a source to a vendor/product by host or CIDR, so the Sophos firewall log path gets chosen by IP regardless of the device="SFW" check. It's the same mechanism Sophos's own Web Appliance source relies on. Drop this in the parsers directory — and note your path is likely the first problem: it should be the plural app-parsers (some docs show app_parsers), not app_parser: # /opt/sc4s/local/config/app-parsers/app-vps-sophos_firewall.conf # filename just needs to be globally unique application app-vps-sophos_firewall[sc4s-vps] { filter { netmask(10.0.103.20/32) or netmask(10.10.11.129/32) or netmask(10.10.11.65/32) or netmask(10.10.10.73/32) or netmask(10.10.10.78/32) or netmask(10.10.10.82/32) or netmask(10.10.11.97/32) }; parser { p_set_netsource_fields( vendor('sophos') product('firewall') ); }; }; Two things to confirm: The exact product() key. Web Appliance uses product('webappliance'); the XG firewall is almost certainly vendor('sophos') product('firewall'), but pull the exact vendor/product key from the Sophos firewall source page so it matches the built-in log path's metadata key. Restart SC4S, then check events land as sophos:xg:firewall / :web_filter / etc. If after VPS the events route to Sophos but still all land as sophos:xg:idp (selected but not segregated by log_type), then the problem is inside the parser version you're running, not the selection — which fits the device_name="SFW" behavior you found. Check your SC4S version, update if you're behind, and if it's genuinely the parser keying on the wrong field, raise it on the SC4S GitHub with a raw sample. And before any of that: confirm your real logs actually carry device="SFW" — that's the constant product tag in every XG event, while device_name is the hostname. Set SC4S_SOURCE_STORE_RAWMSG=yes and look at RAWMSG. If a relay in front of SC4S is altering the message, that's a different root cause and no parser tweak will fix it. On Option 2 (props/transforms on the indexers/HFs): it works as a last resort, but you'd be reimplementing what SC4S + the Sophos TA already do, and forcing the sourcetype at index time won't restore the field extractions the correct sourcetype gives you. Keep it in your back pocket and get VPS working first. Docs: VPS pattern (Sophos Web Appliance source) Configure metadata / by-source context Raw samples + troubleshooting What SC4S version are you on, and does a RAWMSG sample show device="SFW"? ... View more

Hey Konrad,  I am so glad this helped you figure out the issue. Please consider adding a Karma point, so it is easier for others to see if they have a similar issue.   ... View more

This is pretty cool. I will have to try it out in my lab. Good work! ... View more

Hey 0xAli — quick version: ES is what drives your sizing, not the 130 GB/day. Correlation searches and accelerated data models hit CPU and disk IOPS far harder than plain indexing, so size for the search load. Rough starting point: Indexers — plan ~80–100 GB/day each with ES (vs. ~300 for plain Splunk), so 130 is ~2 indexers of load. With RF2 and wanting to survive a node loss, make 3 your floor. 16 cores / 32 GB each on SSD/NVMe. On ES, IOPS bites before CPU does. ES search head — give it its own dedicated box, nothing else co-located. 16 cores / 32–64 GB, scaled up as correlation/acceleration load grows. Ad-hoc SH — separate from ES, sized to your concurrent-user count (roughly a core per active search). HFs + CM / LM / DS — easy at this volume. The single-instance reference spec is fine; just scale the DS by how many forwarders phone home. One design note: pointing network gear straight at an HF over UDP syslog drops events under load. Put SC4S (or syslog-ng/rsyslog writing to disk with a UF reading it) in front instead. For real procurement numbers, don't eyeball it — run your inputs through a sizing calculator, match the topology against the Validated Architectures, and for an ES build it's worth a quick capacity-planning session with Splunk PS or a partner before you buy. Docs worth referencing: Reference hardware (core/RAM baselines) Summary of performance recommendations Estimate your storage requirements ES performance reference Splunk Validated Architectures Splunk Lantern – platform capacity considerations Storage sizing calculator Splunk Connect for Syslog (SC4S) ... View more