Hi Splunkers, today I have a problem about understanding how and where Log Sources sends logs to Splunk. In this particular Splunk On Prem environments, no documentation has been done, except the HLD. So, we have to understand, for each log source, what Splunk component it reaches and how. For example, if I have a Domain Controller, we must establish: Where it sends logs? Directly to Indexers? To a HF? A UF is installed on it? If not, how it send logs? WMI? WEF? Other And so on. Now, List of servers sending logs to Heavy forwarder is a community discussion where I started from @scelikok suggested search, changed it in:     index=_internal component=TcpOutputProc | stats count values(host) as host by idx | fields - count     and it helped me a lot: I'm able, for each Splunk Component of env (IDS, HF and so on) to understand what Log sources send them data. So, what's the problem? The above search return data forwarded by another Splunk component. I mean, in the output, field idx has always format ip/hostname:9997, so it means that data are coming from a server with UF or from another Splunk host (we have some intermediate forwarder, so sometimes I can see data ingested by an HF coming from another HF). What about data sent not with a Splunk agent/host? For example, suppose I have this flow: Log source with Syslog -> Splunk HF receive on port 514 With above search, I cannot see those sources (and I know for sure they exist on our env). How can I recover it? The syslog is only an example, the key point here is: I must complete my search with all log sources that do not use UF and/or any other Splunk element, but other forwarding tool/protocol (syslog, API, WEF, and so on). ... View more

Hi Splunkers, I have to calculate daily ingested volume in a Splunk Enteprise environment. Here on community I found a lot of post, and related answer, to a similar question: daily license consumption, but I don't know if it is what I need. I mean: we know that, once data are ingested by Splunk, compression factor is applied and, in a non clustered environment, it is more or less 50%. So, for example, if I have 100 GB data ingested by day, final size on disk will be 50 GB . Well, I have to calculate total GB BEFORE compression is applied. So, in my above example, search/method I need should NOT return 50 GB as final result, but 100 GB. Moreover, in my current env, I have an Indexers cluster.  So, what is not clear is: daily consumed License, is what I need? I mean: when I see daily consumed license by my environment, GB returned are the ingested one BEFORE compression, or the Compressed one?   ... View more

Hi Splunkers, today I have a very strange case to manage. I'm going to try right now to be more clear possible. The scenario is a full on prem Splunk Enterprise environment, with many components. For this customer, we are not the starting provider; another company was on charge before us and developed a full custom app. About this application: No doc has been shared by previous provider It states now some error messages that are not completely clear. So, in a nutshell, we have to try to understand why we got those errors and try to fix them. Now of course I'm not here to ask you "Ehy magic guys, give me the magic solution!"; the purpose of this topic is ask your help to understand data we have (we have only a GUI little dashboard with a short app description and how it works) and try to understand how we can fix those errors. The app analyze Indexers and their indexes. Its purpose is to understand if indexes are retaining the correct amount of historical data; do achieve this, it investigate the index retention status. So, how this investigation is done? The app analyze the currentTimePeriodDay value against the frozenTimePeriodDay. To state if an error is found, the app consider 2 possible cases: currentTimePeriodDay > frozenTimePeriodDay + 45: this case is considered unhealthy because indexes are retaining more historical data than expected currentTimePeriodDay < frozenTimePeriodDay:  this case is considered unhealthy because indexes are retaining insufficient historical data. For both cases, the suggested workaround is a generic retention and disk space settings tuning. Of course there are more specific error message for each index on every Indexers (we have a menu to select specific Indexers) but this, by my point of view, is a further analysis step; what is not clear, for my team and me, is the foundation logic of app. I mean: how comparison between currentTimePeriodDay and frozenTimePeriodDay should help us to check a good index retention? How are they related? Why if one of them is greater than the other one, this could be an unhealthy symptom?  ... View more

Hi Splunkers, I have a problem with timestamp on our platform. Here some assumption and acquired knowledge. Knowledge _time =  is the event time (the time which is present in the event. In other words: the time when the event was generated. _indextime = is the index time or, if you prefer, the time when the events have been indexed. Issue with timezone shown can be related to user settings, that can be changed under username -> Preferences -> Timezone. Environment: a Splunk Cloud SaaS platform with logs ingested in different ways: Forwarder (both UF and HF) API Syslog File monitoring Issue: If I expand the event and I examinate the _time field:  Why, in my case, time event and time shown are different? Important additional Info Our user settings timezone are set on GMT+1 (due we are in Italy) for all users. You see a Windows events as sample, but the problem is present on all logs: it doesn't matter what log source I consider and how it is sending events to Splunk. Every log show time difference. The difference between _time and time shown is always on 1 hour, for every events on every log sources. I searched here on community and I found other topics about this issue, some of them has been very useful to gain a basic knowledge like Difference Between Event Time and _time  but, due we are on cloud (with limited chance to set some file and parameter that are involved) and the issue is for all events, I'm still locked on this problem.    ... View more

Hi Splunkers, I have a doubt about setting for Splunk Enterprise Security. As usual when I put a question here, let me share a minimal of context and assumption. Environment: A completely on prem Splunk Enterprise (no Slunk Cloud SaaS). Currently, only one SH Clustered indexers Task:  Install and configure a SH with Splunk Enterprise Security. Assumption: I know the full installation procedure (doc + Splunk Enterprise Admin course) I know how to manage a cluster environment (doc + Architect course). For example, I know that if I have to set a Splunk instance as SH I can use, from CLI: > splunk edit cluster-config -mode searchhead -manager_uri https://<manager node address> -secret <cluster secret>   Questions: This syntax is still valid to add a SH with ES installed on it? The doubt is if the ES presence should lead me to use a different approach to tell "Hey, SH wth ES: indexers to query are those". SH with ES component should be  add as single SH (so, decoupled from already existing SH) or should I create a SH Cluster with normal SH + ES ES? ... View more

Hi Splunkers, I must recover Splunk version for all component in a particular environment. I have not access to all GUI and/or .conf files on all machine, so the idea is to try to recover those info with a Splunk search. Here: How-to-identify-a-list-of-forwarders-sending-data I got a very useful search that I used and return me a lot of info about Forwarders, all ones: UF, HF and so on. Due I'm not on a cloud env but an on prem one, I have also to recover Splunk version used on Indexers and Search Heads. So, my question is: how should I change search got on above link to gain version from IDXs and SHs? ... View more

Hi Splunkers, today I have a question related not on a "technical how": my doubt is related to a "best practice". Environment: a Splunk Cloud combo instance (Core + Enterprise Security) with some Heavy Forwarders. Task: perform some field extractions Details: addon for parsing are already installed and configured, so we have not to create new ones, we should simply enrich/expand existing ones. Those addons are installed on both cloud components and HFs. The point is this: due we already have some addon for parsing, we could simply edit their props.conf and transforms.conf files; of course, due we have addon installed on both cloud components and HFs, we have to perform changes on all of them.  For example, performing addon editing only on cloud components with GUI Field Extraction imply that new fields will be parsed at index time on them, because they will be not pre parsed by HFs. Plus, we know that we should create a copy of those file on local folder, to avoid editing the default one, etcetera, etcetera, etcetera.  But, at the same time, for our SOC we created a custom app used as container to store all customizations performed by/for them, following one of Splunk best practice. We store there reports, alerts, and so on: with "we store there" I mean that, when we create something and choose an app context, we set our custom SOC one. With this choice, we could simply perform a field extraction with GUI and assign as app context our custom one; of course, with this technique, custom regex are saved only on cloud components and not on the HFs. So, my wondering is: when we speak about field extraction, if we consider that pre parsing performed by HF is desired but NOT mandatory, what is the best choice? Maintain all field extractions on addon or split between OOT one and custom one, using our custom SOC app? ... View more

Hi Splunkers, I'm performing some test on my test environment and I'm curious about observed behavior. I want to add some network inputs, so tcp and udp ones, to my env. I found easily on doc how to achieve this: Monitornetworkports and it works fine, with no issues. Inputs are correctly added to my Splunk. I can confirm this with no problem on both web GUI and from CLI using btool. My wonder is: if I use the command in the above link, inputs are added on inputs.conf located in SPLUNK_HOME\etc\apps\search\local. For example, if I use: splunk add tcp 3514 -index network -soucetype checkpoint   And then, I digit  splunk btool inputs list --debug | findstr 3514   The output is: C:\Program Files\Splunk\etc\apps\search\local\inputs.conf [tcp://3514]   And, checking manually the file, confs related to my add command are exactly on it. So, I assume that search is the default app if no additional parameter are provided. Now, I know well that if I want edit another inputs.conf file, I can simply manually edit it. But what about if I want edit another inputs.conf from CLI? In other words: I want to know if I can use the splunk add command and specify which inputs.conf file modify. Is it possible?  ... View more

Hi Splunkers, I have a strange situation about a some universal forwarders. On some Windows host, a colleague has installed the UF using the graphical wizards. Those forwarders must be managed with a Deployment server. He has NOT used the "customize" options; so, he has not set which logs must be sent to HF (Application, Security and so on) and a destination HF/Indexers. He has only inserted: Admin username and password Deployment server IP address and port As wrote above, he didn't inserted HF and/or Indexers; the idea is that once the UF has spoken with the Deployment server, 2 apps that contains inputs.conf and outputs.conf are downloaded and, after that, logs are sent. On Deployment server (we checked), the apps that should to be downloaded form UF have been created and contains the above 2 files. So, why I wrote "the apps that should be downloaded?" Well, due logs are not collected and sent to HF, we performed some troubleshoot and we found that apps has not been downloaded.  I mean: on host where UF is installed, if we go on $SplunkUFHOME$\etc\apps, the 2 apps are not present. So, that means that no custom inputs.conf and outputs.conf are present on UF. Only the default provided with installation are present. First thing we thought: ok, we have network issues. But it seems not: we are perfectly able, from host with UF, to ping and telnet deployment server on its port. At same time, we can access firewall that manage this traffic and we don't see, on firewall logs, any evidence of blocked/truncated connections. UF can reach DS and vice versa without issues. We tried so to manually copy folders with apps inside UF (I know, very bad things, don't blame me please...) but the situation is always the same. So, the question is: if no network issues are present, what can be the root cause about no downloaded apps?   ... View more

Hi Splunkers, I have a doubt about a custom app customization. For a customer, we created with Splunk Addon Builder a simple app to use as "container": every customization we perform, such as Correlation rules, reports and so on, is assigned to this app. So, in its first release, the app has no particular panel, features and so on; let's say that just "exist". To be clearer: if I login and open the app, what I see is  this: and that's totally fine, due we did not perform any kind of customizations. So now, the question is: if I want to include the search function inside this app, how I can achieve this? I mean, we want avoid, when when we need to perform a search, to go on Search and Reporting app; we would be able to perform searches inside our app. For now, we don't need panel with specific charts, based on particular query: we want simple to be able to use (if it is possible of course) the Search and Reporting app/its functionality inside our app. ... View more

Hi Splunkers, I have a request by my customer. We have, like in many prod environments, Windows logs. We know that we can see events on Splunk Console, with Splunk Add-on for Microsoft Windows , in 2 way: Legacy format (like  the original ones on AD) or XML. Is it possible to see them on JSON format? If yes, we can achieve this directly with above addon or we need other tools? ... View more