Hi @sakib , Short answer: yes, this is normal, and no, that row is not one host. 1. Why a squashed row can be 100+ GB Squashing isn't a rounding artifact — it's an aggregation. The license manager tracks usage per (host, source, sourcetype, index) tuple. When the number of distinct tuples in a reporting interval exceeds squash_threshold (default 2000 in server.conf [license]), Splunk drops the h (host) and s (source) values and keeps only st (sourcetype) and idx (index). So that blank-host row is the sum of every host that got squashed for that sourcetype/index. If you have a few thousand forwarders, or syslog data where each device IP becomes its own host, 100+ GB/day collapsing into one squashed row is entirely expected. The volume is real; the attribution is what's lost. Two consequences worth knowing: Your total license usage is still accurate. Squashing only affects breakdown, never the billed total. Your query hides useful detail. You're grouping by h, s, st but not idx. Add the index — it survives squashing and is your best lead: index=_internal source=*license_usage.log* type=Usage | stats sum(b) as bytes by idx, st, h | eval gb=round(bytes/1024/1024/1024,2) | sort - gb To confirm squashing is actually happening and see how much is affected: index=_internal source=*license_usage.log* type=Usage | eval squashed=if(h="","squashed","attributed") | stats sum(b) as bytes by squashed | eval gb=round(bytes/1024/1024/1024,2) Also check for the warning on the license manager: index=_internal source=*splunkd.log* squash 2. Finding the real hosts Since idx and st survive, use them to narrow, then attribute from the raw data instead of the license log. Option A — metrics.log (fast, approximate) index=_internal source=*metrics.log* group=per_host_thruput | stats sum(kb) as kb by series | eval gb=round(kb/1024/1024,2) | sort - gb Caveat: metrics.log reports only the top 10 series per interval and lumps the rest into other. Great for spotting a dominant talker, unreliable for a long tail. Run it per indexer (by host, series) since each indexer reports its own. Option B — measure raw bytes directly (accurate, expensive) Take the idx and st from the squashed row and run over a short window: index=<the_index> sourcetype=<the_sourcetype> earliest=-1h latest=now | eval raw_len=len(_raw) | stats sum(raw_len) as bytes, count as events by host | eval gb=round(bytes/1024/1024/1024,3), avg_size=round(bytes/events) | sort - gb One hour, then multiply by 24. Run it off-peak — it scans raw data. The avg_size column often reveals the culprit immediately (a misconfigured input producing huge events, or a debug-level source). Option C — tstats for a fast ranking (event counts, not bytes) | tstats count where index=<the_index> by host, sourcetype | sort - count Doesn't give bytes, but it's near-instant and usually points straight at the offender. Combine with Option B on just the top few hosts. Option D — stop the squashing On the license manager, server.conf: [license] squash_threshold = 5000 Restart required. This costs memory on the license manager (it holds all those tuples), so raise it deliberately rather than to an arbitrary large number. Splunk's own guidance is to keep it as low as you can live with. Useful as a temporary measure while you investigate, then put it back. Practical tip Squashing that appears suddenly, alongside a usage spike, is itself a clue: it usually means a new source started creating many distinct hosts — syslog with per-device hosts, a Kubernetes/container input where each pod becomes a host, or host_segment/host_regex misconfigured so every file path yields a new host value. Comparing distinct host counts before and after the spike often finds it faster than chasing bytes: index=_internal source=*license_usage.log* type=Usage earliest=-14d | timechart span=1d dc(h) as distinct_hosts, sum(b) as bytes If distinct_hosts jumped on the same day as the volume, that's your answer. Hit Karma, if you found the your answer. ... View more

The key question is how you want them combined, since there's no relationship — but either way you don't need join. Two clean patterns: Option 1 — Stacked in one table (simplest, most correct) If there's genuinely no relationship, just pull both sourcetypes in one base search and table all fields. Each row fills only the columns it has: (sourcetype=sourcetypeA OR sourcetype=sourcetypeB) | table user_id user_role device_id device_type Rows from A populate user_id/user_role; rows from B populate device_id/device_type; the rest are blank. One search, one panel, no join, no subsearch limits. (append works too, but a single OR search is more efficient — append runs a capped subsearch.) Option 2 — Side-by-side, aligned by position If you actually want the Nth row of A next to the Nth row of B (a positional merge — arbitrary pairing, only meaningful if order matters), give each source its own row counter and merge on it — single search, no join, no append: (sourcetype=sourcetypeA OR sourcetype=sourcetypeB) | eval src=if(sourcetype="sourcetypeA","A","B") | streamstats count AS row BY src | stats first(user_id) AS user_id first(user_role) AS user_role first(device_id) AS device_id first(device_type) AS device_type BY row | sort row | fields - row streamstats count by src numbers A's rows 1,2,3… and B's rows 1,2,3… independently; then stats … by row collapses each pair (one A event + one B event) into a single row with all four fields. Which to use No correlation at all → Option 1 (stacked). It's the honest representation. Want them visually paired column-blocks → Option 2, but remember the pairing is positional/arbitrary, not a real relationship. ... View more

Hi @Andre_ , What you're hitting is the difference between numeric (dynamic) coloring and categorical coloring in Dashboard Studio. The color block you're using (dataColorConfig with from/to) is the rangeValue encoding — that's for a numeric gradient, and the "enable dynamic colouring → data field" option in the UI only gives you that path. Since it expects a numeric measure, it ends up competing with your duration measure — which is exactly why selecting category_integer gets applied to duration. To color by category, you want the matchValue (categorical) encoding instead, and you'll need to set it in the Source (JSON) editor — the timeline's visual editor doesn't expose color-by-category. Point dataColors at your category series (not duration) and map each value to a fixed color: "context": { "categoryColorConfig": [ { "match": "ok", "value": "#00FF00" }, { "match": "warn", "value": "#FFFF00" }, { "match": "crit", "value": "#FF0000" } ] }, "options": { "category": "> primary | seriesByName('category_name')", "duration": "> primary | seriesByName('duration')", "dataColors": "> primary | seriesByName('category_name') | matchValue(categoryColorConfig)", "additionalTooltipFields": ["category_integer", "eventcount"], "legendDisplay": "bottom" } A few notes: Remove the old dataColorConfig (from/to) block — that's the numeric one causing the clash. dataColors now references the same field you're displaying as the category, so color is driven by category and is fully independent of duration. matchValue works with strings or integers, so if you'd rather key off category_integer, use seriesByName('category_integer') with { "match": 1, "value": "#00FF00" }, etc. Values not listed fall back to the default palette, and the legend picks up your assigned colors. In short: rangeValue = numeric gradient (the dynamic-colouring UI); matchValue = one fixed color per category (what you want), set via the JSON source. Hope that helps! ... View more

Hi @0xAli, The UF always initiates the connection — it phones home to the DS on TCP/8089 asking for its apps. The DS never reaches out to the UF; it just responds. So it's outbound/unidirectional from the UF's side, and you only need to allow UF → DS on 8089. One small note: the connection is initiated one-way, but once it's open, data flows both ways over that same session (that's how the DS sends the app bundle back). A stateful firewall handles the return traffic automatically, so you don't need a separate inbound rule to the UF. ... View more

This isn't your config — it's a default the UF ships with. The internal-log monitor stanzas in $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/default/inputs.conf come with _TCP_ROUTING = *, which means "send to all tcpout groups." So the moment you added group2, _internal started cloning to both. Your override was close but missed two things: [splunkd] is a sourcetype, not an input stanza — _TCP_ROUTING only works on the actual [monitor://...] stanza, so that line does nothing. _internal is fed by multiple monitors (splunkd.log, metrics.log, etc.), each with its own _TCP_ROUTING = *. You only overrode splunkd.log; the rest keep cloning to both groups. (Side note: props/transforms routing won't work on a UF — it doesn't parse. _TCP_ROUTING in inputs.conf is the only lever. And internal indexes require an explicit _TCP_ROUTING, so override it to group2 — don't delete it.) Fix. Enumerate what your version actually ships:   $SPLUNK_HOME/bin/splunk btool inputs list --debug | grep -i "TCP_ROUTING" Then in $SPLUNK_HOME/etc/system/local/inputs.conf, override each internal stanza (header + routing line is enough; index/sourcetype still layer in from default):   [monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log] _TCP_ROUTING = group2 [monitor://$SPLUNK_HOME/var/log/splunk/metrics.log] _TCP_ROUTING = group2 Add a block for every _internal stanza btool listed — any you skip keeps going to group1. Same applies to _audit (audit.log) and _introspection if you want those off group1 too. Restart, then re-run the btool grep to confirm they all resolve to group2. If you meant something other than a reply to post, let me know. ... View more

Hi @dairontorresz  You can do this entirely over REST on port 8089. 1. Check the object's ACL first: curl -k -u admin:PASSWORD -X GET \ "https://SEARCHHEAD:8089/servicesNS/-/SplunkEnterpriseSecuritySuite/saved/searches" \ --get -d search="My Custom Alert" -d output_mode=json Look at removable and can_write in the acl block. If removable is false, no API call will delete it — the stanza is in a default/ directory and can only be disabled. 2. Grant write if can_write is false (owner and sharing are required on every ACL post): curl -k -u admin:PASSWORD -X POST \ "https://SEARCHHEAD:8089/servicesNS/nobody/SplunkEnterpriseSecuritySuite/saved/searches/My%20Custom%20Alert/acl" \ -d owner=admin -d sharing=app -d perms.write=admin 3. Delete: curl -k -u admin:PASSWORD -X DELETE \ "https://SEARCHHEAD:8089/servicesNS/nobody/SplunkEnterpriseSecuritySuite/saved/searches/My%20Custom%20Alert" \ -w "\nHTTP %{http_code}\n" Two gotchas: URL-encode the search name (spaces to %20) — that's the usual cause of a 404 — and use nobody in the namespace for app-shared objects, or the owner's username if it's still private. A 403 means you need admin_all_objects or write on the app itself. if it's work give the kerma ... View more