Hello community I am looking at TA/apps and trying to figure out what to use, where to use it and how to use it optimally. Potentially there are some best practices for this? Just to set the stage, let’s take as an example. Wanting to collect only some logs from hosts using a universal forwarder, using the “Splunk_TA_nix” and setting up a “/local/inputs.conf”, cherry-picking a few sources/folders from default/inputs.conf seems reasonable. Though pushing the “entire” app to a UF seems a bit “overkill”. There should be lots of things not being used in a setup like this. Not all scripts will be needed etc. Then on the search heads, field extraction should be performed for these sources. I assume you need a/the app installed on the these as well for search time extraction. However, things like “/bin”, inputs.conf, outputs.conf, etc. seem unnecessary. Generally it seems like keeping excerpts from props.conf and transforms.conf could suffice? I should formulate a question which can be answered. Say I have a few logs/sources being indexed and searchable. On the search heads, assuming I am only interested in field extraction for the affected source types, no dashboards, reports, log collection on SH etc. Should I just install the entire application/TA regardless of how much or little of it will actually be used? Or, can I remove things which fills no purpose for function outlined? If so, what is the most efficient way to sort out which things to keep and what to discard (like will it always be enough to keep props.conf/transforms.conf)? Looking forward to your feedback regarding this, all the best ... View more

Hello community In our distributed environemnt we have a few Hello community In our distributed environment we have a few heavy forwarders set up to deal with zone boundaries and whatnot. Silly enough of me, I assumed these would all be configured and humming along, though it turned out that not a single one of them where actually being used. I have looked through the manual, as well as the forum here, though I am still somewhat confused regarding the setup and configuration needed. So I’ll take this step-by-step. We have a universal forwarder set up in a Linux machine set to collect some sys/os logs and a filewatch for application log. Now, the UF connects to the deployment server and fetches the configuration, so far so good. Though nothing shows up on indexers and/or search heads. First of all, I noticed that “Receive data” on the HF was empty, I assume there should be a port listed here so I added the standard port. After this, the server could “curl” connect to the HF, so this seemed like a fantastic start. However, still no log. The local splunkd log in the UF shows: 07-08-2022 13:36:07.718 +0200 ERROR TcpOutputFd [4105329 TcpOutEloop] - Connection to host=<ip>:<port> failed 07-08-2022 13:36:07.719 +0200 ERROR TcpOutputFd [4105329 TcpOutEloop] - Connection to host=<ip>:<port> failed So traffic is allowed though still the UF cannot connect to HF. From what I can tell from other threads, I also need to have the same apps as deployed on the UF installed on the HF? Or am I misinterpreting this? Could this explain the failed connections? I have the inputs correct on the UF, I have the outputs.conf pointing at the HF. The HF sends _internal to indexers so that seems ok. It is just not accepting connections from the UF. What exactly do I need to have on the HF so that log can be “redirected” from UF to IX? ... View more

Hello community After a small "snafu" with new dashboards and version number, I noticed that after the rollout in our distributed environment there was, what seemed like, a local backup present:     /opt/splunk/etc/apps/<appname>/default.old.20220705-235555/ The date lines up with the rollout of dashboards receiving a "This dashboard view is deprecated and will be removed in future versions of Splunk software" error.  Hence, I suspect these are connected in some way. So the dashboards were "repaired" by just dropping the version number by "1", though the "backup files" are still there. The only difference I notice are the install_source_checksum and the changes made to dashboards. So, is it OK to just delete this "backup" folder? If so, is there a preferred way to do so or just remove it? ... View more

Today I noticed that one of the heavy forwarders in our distributed environment was not calling back to the deployment server, fetching config. Checking the logs on the HF I noticed: DC:DeploymentClient [3909 MainThread] - target-broker clause is missing. DC:DeploymentClient [3909 MainThread] - DeploymentClient explicitly disabled through config. DS_DC_Common [3909 MainThread] - Deployment Client not initialized. DS_DC_Common [3909 MainThread] - Loading and initializing Deployment Server... DeploymentServer [3909 MainThread] - Attempting to reload entire DS; reason='init' DSManager [3909 MainThread] - No serverclasses configured. DSManager [3909 MainThread] - Loaded count=0 configured SCs I tried the  "splunk display deploy-client" telling me that the "Deployment Client is disabled." I am pretty sure this is why the HF is not phoneing home or fetching new config, though I cannot figure out why? The "deploymentclient.conf" file is identicall for all our HFs, stored in /etc/apps/xxx/default/deploymentclient.conf A grep-search for "target-broker" revealed no duplicate/hidden/conflicting files locally generated. Traffic is allowed as I am able to telnet to DS:8089. I have tried restarting splunk on the HF with no success, same "DC:DeploymentClient" problems. Why is this only affecting the one HF and not the others? How can I resolve this issue? Best regards // G ... View more

Hello community Trying to figure out what is blocking/affecting UF on Windows Agent was installed using CLI msiexec.exe /i splunkforwarder-<version>-x64-release.msi DEPLOYMENT_SERVER="<ip>:8089" SPLUNKPASSWORD=<password> AGREETOLICENSE=yes /quiet Agent is installed, connects to deployment server and fetches apps/configuration. Looking at the log, it seems that the configuration is read properly as I see configuration in there, blacklist/whitelist and other things. Setup is UF -> HF -> IX, IX cannot be reached directly. Everything looks good but here’s where the issues start. Trying to execute Splunk commands including runtime does not work wile the service/agent is running. I get a blinking prompt and nothing happens. Shutting down the service/agent I can run commands, though then runtime commands do not work, and I can’t diagnose. This is one thing which seems off. Then, the system can reach both DS and HF, traffic is allowed. However, watching the traffic from/to the system I see regular traffic with the DS though nothing against the HF. Not even attempts to establish connections. This does not seem reasonable either, I would expect at least failed connections. We suspect that this is caused by managed configuration of the computer itself in some manner. However, any suggestions regarding possible ways to try to diagnose/solve this would be much appreciated. ... View more