What are the pain points with deploying your Splunk architecture on Windows OS?

[woodcock]

I am growing very tired of being asked to justify my "undocumented" and "bigoted" best-practice of NEVER deploying splunk infrastructure (Search Heads, Indexers, License Manager, Cluster Master, Deployer, Deployment Server, Monitoring Console, etc.) on any Windows OS. I am sure many of you have faced the same challenge. I have created this question so that we can create a canonical list from which we can all share the same URL where the best and brightest of us can share our past pain with the kind intention of helping others avoid the windows path of perfectly-avoidable regret. If you think that you will use this Q&A as a reference point, then please do me-too the question. If you have just cause to avoid Windows then P*L*E*A*S*E post your answer. Remember, friends don't let friends deploy on Windows: let's give them the facts that they need to successfully push back. Please include links to documented disasters when possible. Keep in mind that I probably will never accept any answer to this question (to encourage others to participate in perpetuity). Let's do one objection per answer and vote on the best objections so that the most-important ones will filter to the top.
ATTENTION!!! ATTENTION!!!!
THERE ARE NOW MORE ANSWERS THAN FIT ON A SINGLE PAGE (NOTE PAGINATION CONTROLS AT THE BOTTOM)!

[Richfez]

I'm not disagreeing this was a problem ages ago nor am I in any way suggesting running Splunk on Windows, but I think this is a problem long past now.

I can say without any reservation that you can get years of up-time on Server 2008 and newer easily. Though obviously if you patch - which applies to both Linux and Windows - you'll be rebooting them at least occasionally.

[woodcock]

To be fair, though, THP was a huge disaster that was (is) a big black-eye for Splunk on *NIX.

[woodcock]

True, but in most cases, *NIX can be patched/upgraded without a reboot.

[woodcock]

Like it or not, whether Splunk (the company) will admit it or not, Windows is the "afterthought" option. It gets considered last, tested least, deployed rarely, more crash/down-time, and exploited frequently. From a user-base-size perspective, it is very hard to get help anywhere if you are using Windows just because so few people do; I am not normally a "jump on the bandwagon" kind of person, but in this case it very much makes sense.

[woodcock]

Even app developers may not take time to test on Windows. You can easily find apps like URL Receiver:
https://splunkbase.splunk.com/app/1863/#/details
That say things like this:

Dependencies
* Linux, MacOSX or Windows (Windows is untested but should work)

Just the other day I downloaded OpenDNS Detective and found that it was incompatible with Windows:
https://splunkbase.splunk.com/app/3393/#/details
I worked with the developer to fix it.

[Richfez]

I distinctly recall one of the sessions at .conf in 2015 that involved Windows Events. The speaker, a Splunk employee, made a point of regaling us with how much time and effort they had to expend in just finding a windows box to test some things one. No wonder Windows takes a back seat.

[niketn]

@woodcock, I would concur to this point because during alpha pre-release testing I had to wait for a week to get windows installer since the package for testing was initially only compatible with *NIX or MAC.

I also faced similar issue with Splunk's Custom Visualization API while working on a Customer requirement for a new visualization and the the API failed to build as per documented step on Windows machine. It was confirmed to be an issue on Windows system. However, fix from Splunk did not come out. I ended up finding a fix myself.

https://answers.splunk.com/answers/509420/why-does-splunk-custom-visualization-api-result-in.html

[woodcock]

Windows is architecturally incapable of dealing with *NIX file permissions as documented here (among MANY other places):

https://answers.splunk.com/answers/514357/changing-an-existing-splunk-forwarder-into-a-deplo.html

This means that if you ever try to use a Windows-hosted Deployment Server for *NIX-hosted Deployment Clients (Forwarders), you will have trouble that you would not otherwise have if you were deploying from a *NIX-hosted Deployment Server.

Permissions are retained from *Nix DS to *Nix clients, and Windows DS to Windows clients, and a *Nix DS to Windows clients.

However a Windows DS deploying to Linux will not retain permissions file system permissions.

This typically is only a problem in the *Nix world, where you're running your DS as user splunk (or any non-root id), and you actually want it to be root or a different uid on the client.

In the case where you are deploying to different OS's, it's best to setup a script to chown the $SPLUNK_HOME/etc/apps and run that frequently as you need to.

Windows works a bit differently, as you can setup the filesystem to inherit from the parent folder and this typically works as designed.

[Splunker]

In the lab i used what i had handy, and had a Windows DS push the Splunk Nix Add-on to an OSX machine.

After the app deployed, and wondering why i wasn't getting data from inputs that were turned on, i found the scripted-inputs weren't set to +x and couldn't be executed.

A chmod +x *.sh in the app's bin directory got things going again, but that wasn't nice..

I suspect it's because Windows doesn't grok UNIX permissions and just deployed them as regular files.

[woodcock]

Yes, that is exactly what I mean here, and that is the problem that it causes.

[adonio]

One issue comes to mind regarding Splunk Deployment Server on Windows attempt to deploy to *nix clients.
Here you will find an answer about that challenge: https://answers.splunk.com/answers/70039/windows-deployment-server-to-nix-deployment-client-permissi...

[woodcock]

I moved this to to a comment under the other answer because both answers were describing the same thing. Also because we now have so many answers that if there is just one more, we will begin to paginate (merging this brought it back to a single page). No disprespect intended to you, @adonio! We just said the same thing at the same time!

[jkat54]

When I ran splunk in a windows environment I used winrm, powershell, and a file share as my deployment server. Worked so good I did a speech on it at .conf 2013 which never made it to print on the website... but i swear it happened.

[woodcock]

A while back I noticed the "endpoint" setting in the "serverclass.conf.spec" which *probably* could be used to create a functional Windows-based DS by directing the clients to retrieve the files from a non-Windows repository such that file modes are preserved:
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Serverclassconf