What are the pain points with deploying your Splunk architecture on Windows OS?

[woodcock]

I am growing very tired of being asked to justify my "undocumented" and "bigoted" best-practice of NEVER deploying splunk infrastructure (Search Heads, Indexers, License Manager, Cluster Master, Deployer, Deployment Server, Monitoring Console, etc.) on any Windows OS. I am sure many of you have faced the same challenge. I have created this question so that we can create a canonical list from which we can all share the same URL where the best and brightest of us can share our past pain with the kind intention of helping others avoid the windows path of perfectly-avoidable regret. If you think that you will use this Q&A as a reference point, then please do me-too the question. If you have just cause to avoid Windows then P*L*E*A*S*E post your answer. Remember, friends don't let friends deploy on Windows: let's give them the facts that they need to successfully push back. Please include links to documented disasters when possible. Keep in mind that I probably will never accept any answer to this question (to encourage others to participate in perpetuity). Let's do one objection per answer and vote on the best objections so that the most-important ones will filter to the top.
ATTENTION!!! ATTENTION!!!!
THERE ARE NOW MORE ANSWERS THAN FIT ON A SINGLE PAGE (NOTE PAGINATION CONTROLS AT THE BOTTOM)!

[woodcock]

Hunk, Splunk's integration for Hadoop is not supported for windows; the very first line on this page is Note: Hunk is not supported on Windows.:
http://docs.splunk.com/Documentation/Hunk/6.4.8/Hunk/Install

[Richfez]

If you would like to use the new-for-Splunk-7 metrics facility to read/store metrics from Windows systems, too bad. There are some rather convoluted mechanisms by which you can do this, but they are fairly obscure, not particularly high performance and not even mentioned anywhere let alone documented.

As of now (three weeks after official release of 7) there is still no activity on this front for Windows, not even a timeline of when Windows may become a full fledged participant of metrics.

[woodcock]

This goes along with my "afterthought" answer.

[nplamondon]

Several SSL directives are now deprecated... but not under Windows. This is not reflected in the server.conf.spec.

[woodcock]

Correct; this means that your must either: Deploy only to one OS or the other, deploy an unsupported configuration to *NIX (so that you can deploy a single app/file) or manage/deploy different configurations based on OS, which, thankfully, is supported by DS but not Deployer/CM.

[maraman_splunk]

This has positively changed with version 7.2.0 look at sslRootCAPathHonoredOnWindows in server.conf
short story, il will use the new parameters in both linux and windows with 7.2+ (that is by default)
It has been backported in minor versions of 7.0 and 7.1 but in that case you have to opt in to use them with this setting.

[woodcock]

When configuring some credentials in some applications, there are sometimes problems saving them. This is rarely, if ever, a problem on *nix systems (HFs, SHs). You will get an error like this when you click save:

Encountered the following error while trying to update: In handler 'localapps':
Cannot update application info: /nobody/SomeAppHere/app/install/is_configured = 1:
Could not find writer for: /nobody/SomeAppHere/app/install/is_configured [0] [D:\Splunk\etc]

Witness these sad tales of woe (all of which you will note remain without accepted answers):
https://answers.splunk.com/answers/127087/error-while-trying-to-update-in-handler-localapps-pci.html
https://answers.splunk.com/answers/441961/why-am-i-getting-error-cannot-find-item-for-post-a-1.html
https://answers.splunk.com/answers/372914/why-am-i-getting-error-could-not-find-writer-for-n.html
I just had this problem myself here is what I had to do to get it to work.
Run a search like index=internal sourcetype=splunkd YourAppNameHere ERROR. In my case, it clearly showed file write failures (permissions problems). It also showed the encrypted password which was unable to be saved. So I simply created the passwords.conf file in the local directory and saved the password there. Then I put this into local/app.conf:

[install]
is_configured = 1

Then I rebooted splunk and it worked.
I was also able to get it to work through the GUI by completely deleting the app, then using the Install app from file to put it on disk that way, instead of manually or via the Deployment Server, both of which did not work.

[mattymo]

Hadoop data roll (the best native archiving solution IMO) is not supported on Windows.

[woodcock]

In the docs, it says in a highlighted warning Hadoop Data Roll is not supported for Windows here:
http://docs.splunk.com/Documentation/Splunk/7.0.0/Indexer/ArchivingindexestoHadoop

[woodcock]

Now this one is pretty silly and not really Microsoft's fault at all, but it is a valid risk and it has happened to us TWICE now. If your Splunk infrastructure is running windows and you are shutting down your laptop after a very long day or two of long hours, you just might forget that you are still in your RDP session window and power off the Splunk server instead of your laptop. This is more serious than it might appear because in some cases, you are not working with anybody who knows whom to contact to go into the server room to push that button! Don't laugh: this is a tragedy, not a comedy! Has this ever happened to any of you?

[jonny_lyse]

I downvoted this post because this is not an argument for or against windows/linux

[woodcock]

I prefaced my answer with exactly the same disclaimer.

[woodcock]

When you are editing a file on a Windows Deployment Server, Windows creates some kind of lock that makes the file unreadable by the packager inside the Deployment Server. What this means is that if you forget to exit the editor (even if you have saved your changes) and you do a splunk reload deploy-server, the file that you are editing will be quietly dropped from the deployed app package completely but the app will still be deployed! This, of course, results in total failure of that file's function in the app, with no errors of any kind logged anywhere.

[woodcock]

The Windows Splunk MSI installer (there is no "install as an archive of files" option for Windows) does 2 terribly obnoxious things:

1: If it has problems installing, it goes all the way to the end, barfs, destroys most of it's logs' vomit, leaving no trace of the problem, but not all of it's registry vomit, before rolling back telling you basically "Error when installing". This is always a permissions problem, usually a problem with write access to disk, but how would anybody know? There is a way to force it to leave enhanced error/debug logs behind but it will take you a while to find it (somebody please document it here, maybe @martin_mueller) but that is only if you can figure out where it gets logged. When there is this kind of a problem (or any other) with the *NIX install, it is much more obvious and you can easily capture all the install/launch output.

2: It automatically starts splunk post install. If you need to do something like use a common splunk.secret file, you have to know to use the LAUNCHSPLUNK=0 CLI argument or you'll have to reinstall.

http://docs.splunk.com/Documentation/Splunk/latest/Installation/InstallonWindowsviathecommandline#Su...

[martin_mueller]

There is a default log file in AppData/Local/Temp/splunk.log, and you can force more logging with $ msiexec /I <splunk-MSI> /l*v <log-file>

[Richfez]

As a follow on to this (might even be its own answer but I think it'll do as a comment here for now)...

On a few versions of the UF [1], certain very common options to pass to the installer when push- or silent-installing it would cause this exact problem. Unfortunately, one vomit trail it would leave behind was semi-orphaned registry entries that would cause any future attempts to fail.

What's worse was that it took several MORE versions before the "new UF you were installing" could successfully "ignore" or clean up the old munged registry entries, and indeed I'm not positive it does consistently. [2]

I ended up having to create a script that psexec'd the deletions from the registry and the rest of the cleanup across my environment . [3]

[1] The SOURCE UF versions that would cause the problem: 6.2.x, 6.3.x, it was fixed in (IIRC) 6.4.0 or 6.4.1, but was broken again on or around 6.4.3. Fixed after that.

[2] The versions of the UF that is unable to be installed/upgraded to include practically all of them in the 6.2, 6.3, 6.4 and 6.5 series. 6.6.x finally fixed this, at least mostly (might've been 6.6.2), though I just had an upgrade fail for this reason from an older 6.4 to a 6.6.3 a few weeks ago so it's not consistent.

[3] If anyone needs a better outline of this problem and the fixes/scripts I used (they're all pretty simple), ask and I can send them to you.

[woodcock]

This is still happening in v7*.

[woodcock]

Thank you @martin_mueller. That's the ticket. My emotional scars create mental blox.

[woodcock]

Look what just showed up today!
https://answers.splunk.com/answers/567088/splunk-enterprise-setup-wizard-ended-prematurely-a.html?ch...

[woodcock]

This is the first question that I have ever noticed that has so many answers that they don't fit on a single page so that pagination is necessary! Keep up the good work!

[woodcock]

Wow, 3 pages and growing!