Why is splunk not forwarding on windows?

[fatsug]

Hello community

Trying to figure out what is blocking/affecting UF on Windows

Agent was installed using CLI

msiexec.exe /i splunkforwarder-<version>-x64-release.msi DEPLOYMENT_SERVER="<ip>:8089" SPLUNKPASSWORD=<password> AGREETOLICENSE=yes /quiet

Agent is installed, connects to deployment server and fetches apps/configuration. Looking at the log, it seems that the configuration is read properly as I see configuration in there, blacklist/whitelist and other things.

Setup is UF -> HF -> IX, IX cannot be reached directly.

Everything looks good but here’s where the issues start. Trying to execute Splunk commands including runtime does not work wile the service/agent is running. I get a blinking prompt and nothing happens. Shutting down the service/agent I can run commands, though then runtime commands do not work, and I can’t diagnose. This is one thing which seems off.

Then, the system can reach both DS and HF, traffic is allowed. However, watching the traffic from/to the system I see regular traffic with the DS though nothing against the HF. Not even attempts to establish connections. This does not seem reasonable either, I would expect at least failed connections.

We suspect that this is caused by managed configuration of the computer itself in some manner. However, any suggestions regarding possible ways to try to diagnose/solve this would be much appreciated.

[fatsug]

Looking for documentation/information regarding admon I found a page stating the following:

"If your data is flowing from Universal Forwarder to a Heavy Forwarder then you have to install the Splunk Add-on for Windows on your Heavy Forwarder."

ABC's of Splunk Part Ten: Reduction of Attack Surface AreaWindows and Microsoft Active Directory - C...

Is this correct? It also claims that add-on for windows needs to be installed on indexers when log shipping from UF to IX. To the best of my knowledge this is not the case in our environment though we are still indexing logg correctly.

Even though there obviously is a local issue with execution, I wanted to eliminate this as an option.

[gcusello · Accepted solution]

Hi @fatsug,

the phases before indexing (merge, parse) are done on Indexers or (when present) on Heavy Forwarders, because the pass through a full Splunk instance (as HF) coockes the data that cannot be modified on Indexers.

So, you have to install the TA-Windows on Indexers for the data that directly arrive fron UFs to IDXs and on HFs for the other data.

Ciao.

Giuseppe

[fatsug]

@gcusello I see

Even if this will not solve the issue where the lokal UF cannot run/execute as intended, I also need to ensure that the Windows app is present on the bridging HF.

Just as an update, there seems to be a permission issue with the UF. Providing not only the DS IP during install but also the HF as IX, then at least the agent log indicates a failure to connect and watching egress traffic, there is at least one call to the HF.

Supplying only DS during install and fetching config with HF connect information there are no calls made out whatsoever.

It seems that none of the configuration which is supplied from the DS is allowed to execute somehow, though there are still no ERRORs logged locally by the agent.

In any case, this has not really solved my issue "Why is splunk not forwarding on windows?" though at least now I can likely confirm this is a client issue and not a Splunk configuration issue, and the app should be installed on all "receiving" instances (HF and/or IX) in any case.

I'll mark this as solution as I do not expect this forum to solve local managed client configuration issues.

[gcusello]

Hi @fatsug,

if you have one or more (always better to have at least two!) intemediate HF, you have to configure your UFs to send their logs to the HFs, so you have to deploy a different outputs.conf than the one on the HFs.

My hint is to create two custome TAs, called e.g. TA_Forwarders and TA_Forwarders_HF, containing only two files:

• deploymentclient.conf,
• outputs.conf.

the first is the same for both the TAs and it's used to address the Deployment Server,

intead the second contains the addressing of the destination Splunks addresses:

• the HFs for the UFs,
• the IDXs for the HFs.

I hint to use this method so you can more easily manage all your Forwarders (UFs or HFs).

Ciao.

Giuseppe

[gcusello]

Hi @fatsug,

at first check if you're receiving internal logs from that UF, you can do it running on Splunk Enterprise a simplesearch:

index=_internal host=your_host

if you have results, the problem in in inputs management, if you don't have results, you have to che ck the connection.

I suppose that you already configured forwarding (outputs.conf) on UF and receiving on Splunk Enterprise [Settings -- Forwarding and Receiving -- Receiving].

Ciao.

Giuseppe

[fatsug]

@gcusello 

Hello and thank you

While I can see the UF connecting to the deployment server. Configuration has whitelisted _internal though  there is nothing being forwarded so there is nothing on the recieving end.

outputs.conf is present on the UF though I see no traffic to the defined host at all. One thought was that there was an issue with the "intermediate HF", though I'd expect the UF to still try to connect/send data even if there was a problem on the recieving end? Or am I missing something?

[gcusello]

Hi @fatsug,

I suppose that you configured your Intermediate HF to receive logs from UFs and forward them to IDXs,

Check if you have in Splunk Enterprise internal logs from HF and UF.

Then check if there are other UFs that are sending logs to iDX passing through the HF, if yes, use the same outputs.conf.

Ciao.

Giuseppe

[fatsug]

@gcusello I'll start there, thank you.

Any experience/thoughts regarding the failure to run splunk commands? This, more or less, has to be an local OS configuration issue rather than a splunk problem, right?

Running "splunk list monitor" should produce output. I strongly suspect something is blocking someting, though I cannot find any obvious errors indicating a problem either on the system or the agent log.

[gcusello]

Hi @fatsug,

a very stupid question:

do you runned CLI commands on a cmd windows using "Run as administrator" or not?

Ciao.

Giuseppe

[fatsug]

Not stuped at all. And yes, running as administrator.

While the background service is running the exectution of commands from cmd just "stalls" and does not execute. Shutting down the service, I can run commands though then I cannot get runtime information as the service has been shut down

[gcusello]

Hi @fatsug,

are you meaning that running a command as 

splunk status

in the $SPLUNK_HOME\bin folder you don't have any message?

What are the commands you're trying to execute?

Ciao.

Giuseppe

[fatsug]

Exactly, tried several different:

splunk list forward-server

splunk list monitor

splunk stop

 Now, running just "splunk" does output the help message. Everything other than that gives a new line with the blinking prompt and nothing happens/executes.

If I shut down the service (using the service manager as the CLI does not execute) I can run 'list forward-server' and 'start' successfully. Though 'list monitor' does not work of course, as the service is now not running.

I am assuming this has to be a windows/os issue and not expected behaviour of the UF

[gcusello]

Hi @fatsug,

have you already seen the following posts? maybe helps:

https://community.splunk.com/t5/Getting-Data-In/splunk-list-forward-server/m-p/46038

I have the same behavious on my machine: using some commands Splunk asks for the authentication but after few seconds

I suppose that if you run "splunk version" or "splunk status" it responds immediately.

Ciao.

Giuseppe

[fatsug]

A few seconds would have been OK, this is what I am seeing when the service is turned off.

I have let the prompt sit for several minutes without getting any response at all. I would expect the "authentication" prompt though I get nothing at all.

This is why I suspect something is blocking the exectution as the commands which are working, like "list forward-server", does produce output when the service is not running.

[gcusello]

Hi @fatsug,

I don't know but, for my knowledge, if you're running the cmd window as "Administrator" (using the "Run as Administrator" feature) you shouldn't have any problem.

Ciao.

Giuseppe

[fatsug]

Thank you @gcusello, then I likely have a bigger problem after all

[gcusello]

Hi @fatsug,

tell me if I can help you more, otherwise, please, accept my answer for the other people of Community.

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated 😉