Hi Splunkers, today I have a curious fact I want to share with you, related on Splunk UF on PrxMox virtualization. Setup  Proxmox VE 9.1.6 (QEMU/KVM), single host. On this exact host I already run a full Splunk Enterprise cluster (SH, CM, 3x IDX, version 10.2.2) plus other tool, with zero issues. It is a 128 GB RAM system with AMD CPU 8 core-16Threads, so I have no resource problems. The env works fine: Splunk ingest data from different sources and trigger UC and relad actions. SO, the general env is totally healthy. The problem When I install a Universal Forwarder on a fresh VM on the same host, splunkd starts, the service/process is reported as running (systemd active (running) on Linux, Windows SCM state: running on Windows), but port 8089 never actually enters a listening state. No error anywhere — splunkd.log shows HTTPDispatch completing SSL/certificate setup normally and then simply goes quiet, no bind failure, no exception, nothing. What I've tested (all reproduce the identical symptom): UF versions: 10.2.2, 10.4.2, 9.4.2, 9.3.2 Linux: Ubuntu 24.04, Debian 13, Oracle Linux 9 (kernel range 5.14 → 6.8+) Windows Server 2022 (Server Core), installed via MSI, but same result: service "running", nothing listening on 8089 (confirmed via netstat -ano) Ruled out: AppArmor, SELinux (enforcing and permissive), firewall (none present, confirmed via a control python -m http.server 8089 binding fine on the same VM), ulimits/ThP tuning, IPv4/IPv6 mgmtHostPort ambiguity, corrupted download (checksum + manifest validation OK). I tryed both automated installazion, like Ansible Playbook, and  zero-config/manual install with no custom conf, same result every time. What stands out (untile now) The fact that it reproduces identically on Windows, which has completely different network stack/service model from Linux, points away from anything Linux-kernel-specific and toward either something in cross-platform Splunk code governing when the REST/mgmt bind happens, or something specific to this Proxmox/QEMU host that both guest OS families are hitting the same way. Has anyone seen this specifically with UF on Proxmox/KVM? Any known gotcha with HTTPDispatch/mgmt server bring-up I might be missing?  Thanks in advance! ... View more

Good morning, Splunkers, I have a question about RBAC segmentation in Splunk. Specifically, my question is: can role-based segmentation be performed within the same index? Let me try to clarify my doubts with an example. I know for sure that I can segment different indexes. For example, if I have the following Splunk groups: Security ITAdmin And the following indexes: alerts performance I could create a segmentation like this: Security -> only accesses the "alerts" index and sees ALL its data. ITAdmin -> only accesses the "performance" index and sees ALL its data. So far, no doubt, I've done it countless times. If, however, I had a scenario where within the same index, for example "alerts," I had two different types of data such as: cybersecurity_alerts it_hardware_alerts distinct in some way (tag?), I could implement the following scenario: Security -> accesses the "alerts" index -> sees ONLY the data tagged "cybersecurity_alerts" ITAdmin -> accesses the "alerts" index -> sees ONLY the data tagged "it_hardware_alerts" ? ... View more

Hi Splunkers, I have some doubt about SSL use for S2S communication. First, let us remark what is sure, with no doubts: 1. SSL provide a compression ratio better than default one: 1:8 vs 1:2. 2. SSL Compression does NOT affect license. In general, ALL Compression on Splunk does not affect license. This means that: if before compression data has dimension X + Y and, after it, X, consumed license will be X + Y, not X. 3. From a security perspective, if I have multiple Splunk components, the best way to configure flows should be encrypt all of them. For example if I have UF -> HF -> IDX, for security purpose the best is to encrypt both UF -> HF flow and HF -> IDX one. Now, for a customer we have the following data flow: Log sources -> Intermediate Forwarder -> Heavy forwarders -> Indexers I know that when possible we should avoid HF and IF but, for different reason, we need them on this particular environment. Here, 2 doubt rise: Suppose we apply SSL only between IF and HF. 1. Data arrive compressed on HF. When they leaves it and goes to IDXs, they are still compressed? So, for example suppose we have original data with a total dimension of 800 MB: Between IF and HF exist SSL, so in HF there is a tcp-ssl input on port 9997 SSL compression is applied: now data have 100 MB dimension When they arrive to HF, they have 100 MB dimension When they leave the HF to go on IDXs, they still have 100 MB dimension? Suppose now we apply SSL on entire S2S data flow: between IF and HF and between HF and IDXs. In addition to a better security posture, which other advantage we should achieve going in this direction?     ... View more

Hi Splunkers, I have a very simple question. When I configure a Splunk indexes.conf, I know that one parameter I can configure is repFactor. In a scenario where SmartStore is used, we know that repFactor must be set equals to "auto", for each configured index. Here the question is this: following Splunk official documentation, repFactor is put under "Per index Options". Does it means that I cannot put it under [default] stanza? Because if, for SmartStore requirements, I need to configure it equals to auto for EVERY index, it could be fast and smart put it as a global setting. Luca   ... View more

Hi Splunkers, I have a doubt about a specific Splunk Alert triggered actions: the log event one. From doc I can see, on the end: "You must also define the destination index on both the search head and the indexers. " Does it means that, even if I am in a distributed environments, I must created index used to save alerts on both Indexers and search heads? ... View more

Hi Guys, we have a doubt reagarding the user that execute Splunk on a linux environment. Until now, we have always avoided use of root user; instead, we have always  installed and configured Splunk on Linux in the following way: Create a dedicated user, eg. "splunkuser" Change ownership of splunk installation folder to that user Configure splunk for boot autorun with dedicated user What is not clear for us, and we didn't found on doc, is: suppose this user belongs to sudoers group. Here 2 question rise: It can be removed from sudoers, or some Splunk related process require it belong to that group? If it cannot be totally removed from sudoers, which process requires it maintain such kind of privileges? ... View more

Hi Splunkers, today I have a strange situation that require a well done data sharing by my side, so please forgive me if I'm goint to be long. We are managing a Splunk Enterprise Infrastructure previously managed by another company. We are in charge of AS IS management and, at the same time, perform the migration to new environment.  The Splunk new env setup is done, so now we need to migrate data flow. Following Splunk best pratice, we need to temporarily perform a double data flow: Data must still go from log sources to old env Data must also flow from log sources to new env. We already faced, on another customer, a double data flow, managed using Route and filter data doc and support here on community. So the point is not: we don't know how it works. The issue is: something is not going as expected. So, how the current env is configured? Below, key elements: A set of HFs deployed on customer data center. A cloud HF in charge of collecting data from above HFs and other data input, like network ones. 2 different indexers: they are not on cluster, they are separated and isolated indexer. The first one collect a subset of data forwarded by cloud HF, the second one the remaining one. So, how cloud HF is configured for tcp data routing? In $SPLUNK_HOME$/etc/system/local/inputs.conf, two stanza are configured to receive data on ports 9997 and 9998; configuration is more or less:   [<log sent on HF port 9997>] _TCP_ROUTING = Indexer1_group [<log sent on HF port 9998>] _TCP_ROUTING = Indexer2_group   Then, in $SPLUNK_HOME$/etc/system/local/outputs.conf we have:   [tcpout] defaultGroup=Indexer1_group [tcpout:Indexer1_group] disabled=false server=Indexer1:9997 [tcpout:Indexer2_group] disabled=false server=Indexer2:9997   So, the current behavior is: Log collected on port 9997 of Cloud HF are sent to Indexer1 Log collected on port 9998 of Cloud HF are sent to Indexer2 Everything else, like Network Input data, is sent  to Indexer1, thanks to default group settings. At this point, we need to insert new environment hosts; in particular, we need to link a new HFs set. At this phase, as already shared, we need to send data to old env and to new one. We can discuss about avoid to insert another HFs set, but there are some reason about using it and the architecture has been approved by Splunk itself. So, what we have to perform now is: All data are still sent to old Indexer1 and Indexer2. All data must be sent also to new HF set. So, how we tried to perform this? Below there is our changed configuration. inputs.conf:   [<log sent on HF port 9997>] _TCP_ROUTING = Indexer1_group, newHFs_group [<log sent on HF port 9998>] _TCP_ROUTING = Indexer2_group, newHFs_group      outputs.conf:   [tcpout] defaultGroup=Indexer1_group, newHFs_group [tcpout:Indexer1_group] disabled=false server=Indexer1:9997 [tcpout:Indexer2_group] disabled=false server=Indexer2:9997 [tcpout:newHFs_group] disabled=false server=HF1:9997, HF2:9997, HF3:9997     In a nutshell, we tried to achieve: Log collected on port 9997 of Cloud HF are sent to Indexer1 and new HFs Log collected on port 9998 of Cloud HF are sent to Indexer2 and new HFs Everything else is sent, thanks to default group settings, to Indexer1 and new HFs So, what went wrong? Log collected on port 9997 of Cloud HF are sent correctly to both Indexer1 and new HFs Log collected on port 9998 of Cloud HF are sent correctly to both Indexer2 and new HFs Remaining log are not correctly sent to both Indexer 1 and new HFs. In particular, we should see the following behavior: All logs not collected on port 9997 and 9998, like network data input, are equally sent to Indexer1 and new HFs: a copy to Indexer1 and a copy to new HFs. So, if we have outputs of N logs, we must have 2N logs sent: N to Indexer1 and N to new HFs What we are seeing is: All logs not collected on port 9997 and 9998, like network data input, are auto load balanced and splitted between Indexer1 and new HFs. So, if we have outputs of N logs, we see N sent: we have more or less 80% sent to Indexer1 and remaining 20% to new HFs. I underlined many times that some kind of logs not collected on port 9997 and 9998 are the Network ones, because we are seeing that auto lb and log splitting is happening most of all with them. ... View more

Hi Splunkers, currently we are managing an Enterprise Splunk environment previously managed by another company. As sadly often occurs, no documentation has been released and so we had to discover almost information about architecture by ourselves. We successfully managed many tasks related to this big problem, but few ones remain; in particular, the one for what I open this discussion. The point is this: almost total ingested data are collected flowing to a couple of HF. This means that data flow is, typically: Log sources -> On prem HF -> Cloud HF (on IaaS VM) -> Cloud Indexer (on IaaS VM). With a search discovered here on community, based on internal logs, I found how to understand what Splunk component send data to another Splunk one. I mean: suppose I have HF on prem 1 -> Hf on cloud 2 I know how to discover this analyzing the internal logs. But what about if I want to discover which HF on prem collect data sent to a specific index? Let me do an example. Suppose I have this host set: Log sources (with NO UF installed on them) Log source 1 Log source 2 Log source 3 On prem HF HF on prem 1 HF on prem 2 HF on prem 3 On cloud HF (IaaS VM) HF on Cloud 1 On cloud indexer Indexer on cloud 1 (IaaS VM) Indexes index1 index2 index3 At starting point, I know only that all 3 On prem HF collect data and send them to HF on Cloud: then, data are sent to the Indexer. I don’t know which On prem HF collect data from which Log source, and in which index data are collected once they arrive on indexer; for sure, I could ask to system owner what configuration has been performed on log sources, but the idea is to discover this with a Splunk Search. Is this possible? The idea is to have a search where I can specify the exact flow. For example, suppose that 1 of the above flow is: Log source 1 -> On Prem HF 2 -> On Cloud HF -> On Cloud Indexer -> index3 I must be able to discover it.         ... View more

Hi Splunkers, I have a doubt about users that run scheduled searches. Until now, I now very well that, if a user own a knowledge object like a correlation searches, when it is deleted/disabled, we can encounter some problems, like the Orphaned object one. So the best pratice is to create a service user and assign it to KO. Fine. My wondering is: suppose we have many scheduled correlation searches, for example more than 100 and 200. Assign all those searches to one single service user is fine, or is better to create multiple one, so to avoid some performance experience? The question is made based on a case some colleagues shared with me once: due there were some problems with search lag/skipped searches, in addiction to fix searches scheduler, involved people splitted their ownership to multiple users. Is that useful or not? ... View more

Hi Splunkers, I'm deploying a new Splunk Enterprise environment; inside it, I have (for now) 2 HF and a DS. I'm trying to set an outputs.conf file on both HF via DS; clients perform a correct phoning to DS, but then apps are not downloaded. I checked the internal logs and I got no error related to app. I followed doc and course material used during Architect course for references. Below, configuration I made on DS. App name:      /opt/splunk/etc/deployment-apps/hf_seu_outputs/       App file     /opt/splunk/etc/deployment-apps/hf_seu_outputs/default/app.conf [ui] is_visible = 0 [package] id = hf_outputs check_for_updates = 0       /opt/splunk/etc/deployment-apps/hf_seu_outputs/local/outputs.conf [indexAndForward] index=false [tcpout] defaultGroup = default-autolb-group forwardedindex.filter.disable = true indexAndForward = false [tcpout:default-autolb-group] server=<idx1_ip_address>:9997, <idx2_ip_address>:9997, <idx3_ip_address>:9997     serverclass.conf:   [serverClass:spoke_hf:app:hf_seu_outputs] restartSplunkWeb = 0 restartSplunkd = 1 stateOnClient = enabled [serverClass:spoke_hf] whitelist.0 = <HF1_ip_address>, <HF1_ip_address>   File and folder permission are right, owner is the user used to execute Splunk (in a nutshell, the owner of /opt/spluk). I suppose it is a very stupid issue, but I'm not able to figured it out. ... View more