Hi Splunkers, I have some doubt about SSL use for S2S communication. First, let us remark what is sure, with no doubts: 1. SSL provide a compression ratio better than default one: 1:8 vs 1:2. 2. SSL Compression does NOT affect license. In general, ALL Compression on Splunk does not affect license. This means that: if before compression data has dimension X + Y and, after it, X, consumed license will be X + Y, not X. 3. From a security perspective, if I have multiple Splunk components, the best way to configure flows should be encrypt all of them. For example if I have UF -> HF -> IDX, for security purpose the best is to encrypt both UF -> HF flow and HF -> IDX one. Now, for a customer we have the following data flow: Log sources -> Intermediate Forwarder -> Heavy forwarders -> Indexers I know that when possible we should avoid HF and IF but, for different reason, we need them on this particular environment. Here, 2 doubt rise: Suppose we apply SSL only between IF and HF. 1. Data arrive compressed on HF. When they leaves it and goes to IDXs, they are still compressed? So, for example suppose we have original data with a total dimension of 800 MB: Between IF and HF exist SSL, so in HF there is a tcp-ssl input on port 9997 SSL compression is applied: now data have 100 MB dimension When they arrive to HF, they have 100 MB dimension When they leave the HF to go on IDXs, they still have 100 MB dimension? Suppose now we apply SSL on entire S2S data flow: between IF and HF and between HF and IDXs. In addition to a better security posture, which other advantage we should achieve going in this direction?     ... View more

Hi Splunkers, I have a very simple question. When I configure a Splunk indexes.conf, I know that one parameter I can configure is repFactor. In a scenario where SmartStore is used, we know that repFactor must be set equals to "auto", for each configured index. Here the question is this: following Splunk official documentation, repFactor is put under "Per index Options". Does it means that I cannot put it under [default] stanza? Because if, for SmartStore requirements, I need to configure it equals to auto for EVERY index, it could be fast and smart put it as a global setting. Luca   ... View more

Hi Splunkers, I have a doubt about a specific Splunk Alert triggered actions: the log event one. From doc I can see, on the end: "You must also define the destination index on both the search head and the indexers. " Does it means that, even if I am in a distributed environments, I must created index used to save alerts on both Indexers and search heads? ... View more

Hi Guys, we have a doubt reagarding the user that execute Splunk on a linux environment. Until now, we have always avoided use of root user; instead, we have always  installed and configured Splunk on Linux in the following way: Create a dedicated user, eg. "splunkuser" Change ownership of splunk installation folder to that user Configure splunk for boot autorun with dedicated user What is not clear for us, and we didn't found on doc, is: suppose this user belongs to sudoers group. Here 2 question rise: It can be removed from sudoers, or some Splunk related process require it belong to that group? If it cannot be totally removed from sudoers, which process requires it maintain such kind of privileges? ... View more

Hi Splunkers, today I have a strange situation that require a well done data sharing by my side, so please forgive me if I'm goint to be long. We are managing a Splunk Enterprise Infrastructure previously managed by another company. We are in charge of AS IS management and, at the same time, perform the migration to new environment.  The Splunk new env setup is done, so now we need to migrate data flow. Following Splunk best pratice, we need to temporarily perform a double data flow: Data must still go from log sources to old env Data must also flow from log sources to new env. We already faced, on another customer, a double data flow, managed using Route and filter data doc and support here on community. So the point is not: we don't know how it works. The issue is: something is not going as expected. So, how the current env is configured? Below, key elements: A set of HFs deployed on customer data center. A cloud HF in charge of collecting data from above HFs and other data input, like network ones. 2 different indexers: they are not on cluster, they are separated and isolated indexer. The first one collect a subset of data forwarded by cloud HF, the second one the remaining one. So, how cloud HF is configured for tcp data routing? In $SPLUNK_HOME$/etc/system/local/inputs.conf, two stanza are configured to receive data on ports 9997 and 9998; configuration is more or less:   [<log sent on HF port 9997>] _TCP_ROUTING = Indexer1_group [<log sent on HF port 9998>] _TCP_ROUTING = Indexer2_group   Then, in $SPLUNK_HOME$/etc/system/local/outputs.conf we have:   [tcpout] defaultGroup=Indexer1_group [tcpout:Indexer1_group] disabled=false server=Indexer1:9997 [tcpout:Indexer2_group] disabled=false server=Indexer2:9997   So, the current behavior is: Log collected on port 9997 of Cloud HF are sent to Indexer1 Log collected on port 9998 of Cloud HF are sent to Indexer2 Everything else, like Network Input data, is sent  to Indexer1, thanks to default group settings. At this point, we need to insert new environment hosts; in particular, we need to link a new HFs set. At this phase, as already shared, we need to send data to old env and to new one. We can discuss about avoid to insert another HFs set, but there are some reason about using it and the architecture has been approved by Splunk itself. So, what we have to perform now is: All data are still sent to old Indexer1 and Indexer2. All data must be sent also to new HF set. So, how we tried to perform this? Below there is our changed configuration. inputs.conf:   [<log sent on HF port 9997>] _TCP_ROUTING = Indexer1_group, newHFs_group [<log sent on HF port 9998>] _TCP_ROUTING = Indexer2_group, newHFs_group      outputs.conf:   [tcpout] defaultGroup=Indexer1_group, newHFs_group [tcpout:Indexer1_group] disabled=false server=Indexer1:9997 [tcpout:Indexer2_group] disabled=false server=Indexer2:9997 [tcpout:newHFs_group] disabled=false server=HF1:9997, HF2:9997, HF3:9997     In a nutshell, we tried to achieve: Log collected on port 9997 of Cloud HF are sent to Indexer1 and new HFs Log collected on port 9998 of Cloud HF are sent to Indexer2 and new HFs Everything else is sent, thanks to default group settings, to Indexer1 and new HFs So, what went wrong? Log collected on port 9997 of Cloud HF are sent correctly to both Indexer1 and new HFs Log collected on port 9998 of Cloud HF are sent correctly to both Indexer2 and new HFs Remaining log are not correctly sent to both Indexer 1 and new HFs. In particular, we should see the following behavior: All logs not collected on port 9997 and 9998, like network data input, are equally sent to Indexer1 and new HFs: a copy to Indexer1 and a copy to new HFs. So, if we have outputs of N logs, we must have 2N logs sent: N to Indexer1 and N to new HFs What we are seeing is: All logs not collected on port 9997 and 9998, like network data input, are auto load balanced and splitted between Indexer1 and new HFs. So, if we have outputs of N logs, we see N sent: we have more or less 80% sent to Indexer1 and remaining 20% to new HFs. I underlined many times that some kind of logs not collected on port 9997 and 9998 are the Network ones, because we are seeing that auto lb and log splitting is happening most of all with them. ... View more

Hi Splunkers, currently we are managing an Enterprise Splunk environment previously managed by another company. As sadly often occurs, no documentation has been released and so we had to discover almost information about architecture by ourselves. We successfully managed many tasks related to this big problem, but few ones remain; in particular, the one for what I open this discussion. The point is this: almost total ingested data are collected flowing to a couple of HF. This means that data flow is, typically: Log sources -> On prem HF -> Cloud HF (on IaaS VM) -> Cloud Indexer (on IaaS VM). With a search discovered here on community, based on internal logs, I found how to understand what Splunk component send data to another Splunk one. I mean: suppose I have HF on prem 1 -> Hf on cloud 2 I know how to discover this analyzing the internal logs. But what about if I want to discover which HF on prem collect data sent to a specific index? Let me do an example. Suppose I have this host set: Log sources (with NO UF installed on them) Log source 1 Log source 2 Log source 3 On prem HF HF on prem 1 HF on prem 2 HF on prem 3 On cloud HF (IaaS VM) HF on Cloud 1 On cloud indexer Indexer on cloud 1 (IaaS VM) Indexes index1 index2 index3 At starting point, I know only that all 3 On prem HF collect data and send them to HF on Cloud: then, data are sent to the Indexer. I don’t know which On prem HF collect data from which Log source, and in which index data are collected once they arrive on indexer; for sure, I could ask to system owner what configuration has been performed on log sources, but the idea is to discover this with a Splunk Search. Is this possible? The idea is to have a search where I can specify the exact flow. For example, suppose that 1 of the above flow is: Log source 1 -> On Prem HF 2 -> On Cloud HF -> On Cloud Indexer -> index3 I must be able to discover it.         ... View more

Hi Splunkers, I have a doubt about users that run scheduled searches. Until now, I now very well that, if a user own a knowledge object like a correlation searches, when it is deleted/disabled, we can encounter some problems, like the Orphaned object one. So the best pratice is to create a service user and assign it to KO. Fine. My wondering is: suppose we have many scheduled correlation searches, for example more than 100 and 200. Assign all those searches to one single service user is fine, or is better to create multiple one, so to avoid some performance experience? The question is made based on a case some colleagues shared with me once: due there were some problems with search lag/skipped searches, in addiction to fix searches scheduler, involved people splitted their ownership to multiple users. Is that useful or not? ... View more

Hi Splunkers, I'm deploying a new Splunk Enterprise environment; inside it, I have (for now) 2 HF and a DS. I'm trying to set an outputs.conf file on both HF via DS; clients perform a correct phoning to DS, but then apps are not downloaded. I checked the internal logs and I got no error related to app. I followed doc and course material used during Architect course for references. Below, configuration I made on DS. App name:      /opt/splunk/etc/deployment-apps/hf_seu_outputs/       App file     /opt/splunk/etc/deployment-apps/hf_seu_outputs/default/app.conf [ui] is_visible = 0 [package] id = hf_outputs check_for_updates = 0       /opt/splunk/etc/deployment-apps/hf_seu_outputs/local/outputs.conf [indexAndForward] index=false [tcpout] defaultGroup = default-autolb-group forwardedindex.filter.disable = true indexAndForward = false [tcpout:default-autolb-group] server=<idx1_ip_address>:9997, <idx2_ip_address>:9997, <idx3_ip_address>:9997     serverclass.conf:   [serverClass:spoke_hf:app:hf_seu_outputs] restartSplunkWeb = 0 restartSplunkd = 1 stateOnClient = enabled [serverClass:spoke_hf] whitelist.0 = <HF1_ip_address>, <HF1_ip_address>   File and folder permission are right, owner is the user used to execute Splunk (in a nutshell, the owner of /opt/spluk). I suppose it is a very stupid issue, but I'm not able to figured it out. ... View more

Hi Splunkers, I have an inssue with a line breaking use case. I know it is very simple to fix, but I still have the problem, so there is something I'm not getting in the right way.  First, a little bit of info about env. Log source: custom application Input type: File monitor Input File monitoring: via UF, so a deployed app has been deployed with a DS Final flow: Log Source with UF -> HF -> Splunk Cloud Data are ingested? Yes. Issue: once log are collected, we got a unique big log. So, we need to separate logs in different events. So I thought: Ok fine, I did a lot of custom addon, I know how do do it. By the way, I did not performed initial configuration about UF, so I check related deployed app and logs . That's the summary: Single event ends with "platform":"ArcodaSAT"} UF deployed app is very simple: it has an app.conf, an inputs,.conf and a props.conf. inputs.conf file works fine due logs are ingested from the right source Below, settings in I found in props.conf:             [<sourcetype_name>]             CHARSET=AUTO             LINE_BREAKER = (\"platform\"\:\"ArcodaSAT\"\})             SHOULD_LINEMERGE = true Observation: Regex is fine; I tested it on regex101 with a log sample and it catch fine. I tried, in the LINE_BREAKER, both using round brackets - cause documentation say that parameter use the capture group to check where new log starts - and without. Same result. SHOULD_LINEMERGE has be set both as true and false: same result Let me say again: I know this is some nonsense I'm missing, but I can't find it. ... View more

Hi Splunkers,  we have a Windows log source with a UF installed on it. We have no access to this log source: we only know that we collect Windows logs via UF and it works properly. Collected logs are the usual one: Security, Applications, and so on. Starting from today, we need to add a monitor input: some files are stored in a folder and we need to collect them. So, on our DS, we created another app, inside deployment-app folder, with a proper inputs.conf and props.conf and then we deployed it. Why we created another app and does not simply added a monitor stanza in inputs.conf for Windows addon? Simply because Windows addon is deployed on many host; on the other side, we need to monitor the path only on 1 specific host, so we preferred to deploy another dedicated app, with its server class and so on. DS give no error; app is shown as deployed with no issues. At the same time, we got no error looking on splunkd.log and/or _internal index. By the way, logs are not collected. For sure, we are going to reach Host owner and perform basic checks, like: Is provided path the right one? User in charge of execute UF has read permission on that folder? In UF app folder, is the one deployed by us viewable?  But before this, there is a doubt I have: above point 2, in case of permission denied, I should see in _internal logs some error message, right? Because currently I don't see any error message related to this issue. The behavior is like the inputs.conf we set in deployment app is totally ignored: searching on _internl and/or splunkd.log, I cannot see anything related to path we have to monitor. ... View more