topic Re: What are the pain points with deploying your Splunk architecture on Windows OS? in Getting Data In

What are the pain points with deploying your Splunk architecture on Windows OS?

woodcock — Tue, 28 Mar 2017 23:09:52 GMT

I am growing very tired of being asked to justify my "undocumented" and "bigoted" best-practice of NEVER deploying splunk infrastructure (Search Heads, Indexers, License Manager, Cluster Master, Deployer, Deployment Server, Monitoring Console, etc.) on any Windows OS. I am sure many of you have faced the same challenge. I have created this question so that we can create a canonical list from which we can all share the same URL where the best and brightest of us can share our past pain with the kind intention of helping others avoid the windows path of perfectly-avoidable regret. If you think that you will use this Q&A as a reference point, then please do me-too the question. If you have just cause to avoid Windows then P*L*E*A*S*E post your answer. Remember, friends don't let friends deploy on Windows: let's give them the facts that they need to successfully push back. Please include links to documented disasters when possible. Keep in mind that I probably will never accept any answer to this question (to encourage others to participate in perpetuity). Let's do one objection per answer and vote on the best objections so that the most-important ones will filter to the top.
ATTENTION!!! ATTENTION!!!!
THERE ARE NOW MORE ANSWERS THAN FIT ON A SINGLE PAGE (NOTE PAGINATION CONTROLS AT THE BOTTOM)!

Re: What are the pain points with deploying your Splunk architecture on Windows OS?

woodcock — Tue, 28 Mar 2017 23:20:20 GMT

Windows is architecturally incapable of dealing with *NIX file permissions as documented here (among MANY other places):

https://answers.splunk.com/answers/514357/changing-an-existing-splunk-forwarder-into-a-deplo.html

This means that if you ever try to use a Windows-hosted Deployment Server for *NIX-hosted Deployment Clients (Forwarders), you will have trouble that you would not otherwise have if you were deploying from a *NIX-hosted Deployment Server.

Permissions are retained from *Nix DS to *Nix clients, and Windows DS to Windows clients, and a *Nix DS to Windows clients.

However a Windows DS deploying to Linux will not retain permissions file system permissions.

This typically is only a problem in the *Nix world, where you're running your DS as user splunk (or any non-root id), and you actually want it to be root or a different uid on the client.

In the case where you are deploying to different OS's, it's best to setup a script to chown the $SPLUNK_HOME/etc/apps and run that frequently as you need to.

Windows works a bit differently, as you can setup the filesystem to inherit from the parent folder and this typically works as designed.

Re: What are the pain points with deploying your Splunk architecture on Windows OS?

adonio — Tue, 28 Mar 2017 23:20:21 GMT

One issue comes to mind regarding Splunk Deployment Server on Windows attempt to deploy to *nix clients.
Here you will find an answer about that challenge: https://answers.splunk.com/answers/70039/windows-deployment-server-to-nix-deployment-client-permission-issues.html

Re: What are the pain points with deploying your Splunk architecture on Windows OS?

woodcock — Tue, 28 Mar 2017 23:31:26 GMT

Like it or not, whether Splunk (the company) will admit it or not, Windows is the "afterthought" option. It gets considered last, tested least, deployed rarely, more crash/down-time, and exploited frequently. From a user-base-size perspective, it is very hard to get help anywhere if you are using Windows just because so few people do; I am not normally a "jump on the bandwagon" kind of person, but in this case it very much makes sense.

Re: What are the pain points with deploying your Splunk architecture on Windows OS?

woodcock — Tue, 28 Mar 2017 23:37:25 GMT

It is still a Windows "best practice" to have a monthly reboot (if not more frequently). I have seen Linux indexers that have an uptime of YEARS. Who can afford monthly Indexer downtime just so that the host OS doesn't crash?

Re: What are the pain points with deploying your Splunk architecture on Windows OS?

woodcock — Tue, 28 Mar 2017 23:37:34 GMT

Most Splunk admins that I know have had many cases where Splunk Indexers and Search Heads have crashed due to memory leaks in the OS. I have NEVER seen this happen in *NIX (although I am sure that on rare occasion it has). Many *.0 releases of Splunk on Windows have contained a memory leak that made it through testing, but not *NIX.

Re: What are the pain points with deploying your Splunk architecture on Windows OS?

Richfez — Wed, 29 Mar 2017 00:55:59 GMT

I'm not disagreeing this was a problem ages ago nor am I in any way suggesting running Splunk on Windows, but I think this is a problem long past now.

I can say without any reservation that you can get years of up-time on Server 2008 and newer easily. Though obviously if you patch - which applies to both Linux and Windows - you'll be rebooting them at least occasionally.

Re: What are the pain points with deploying your Splunk architecture on Windows OS?

Richfez — Wed, 29 Mar 2017 01:00:24 GMT

All but one - I ran a single SH/Indexer box on Windows for years, from version 4.3 to - Oh, I may have even skipped 5.x entirely! - 6.0. I had no significant problems (possibly really none at all - I can't remember in that much detail!, but certainly nothing serious).

Re: What are the pain points with deploying your Splunk architecture on Windows OS?

Richfez — Wed, 29 Mar 2017 01:01:21 GMT

The python interpreter in Windows is far slower than that in the Linuxes. I believe it mainly affects Enterprise Security - but nothing is broken just a lot of things take longer to run.

Re: What are the pain points with deploying your Splunk architecture on Windows OS?

woodcock — Wed, 29 Mar 2017 01:01:30 GMT

True, I do know you, @rich7177!

Re: What are the pain points with deploying your Splunk architecture on Windows OS?

woodcock — Wed, 29 Mar 2017 01:03:58 GMT

True, but in most cases, *NIX can be patched/upgraded without a reboot.

Re: What are the pain points with deploying your Splunk architecture on Windows OS?

woodcock — Wed, 29 Mar 2017 01:05:33 GMT

To be fair, though, THP was a huge disaster that was (is) a big black-eye for Splunk on *NIX.

Re: What are the pain points with deploying your Splunk architecture on Windows OS?

Richfez — Wed, 29 Mar 2017 01:20:26 GMT

Yes. 🙂

Note I'm STILL not recommending Splunk on Windows* , just saying I had no problems in several years of running Splunk on Windows.

When might I suggest windows?

Small shop running Splunk Free with no use for more than 10 or 20 GB/day of license because they simply don't have that much stuff going on, then a Windows all-in-one box would probably be fine.

Small shops that have no Linux experience. Again, with a maybe up to 50 GB/day limit and no replication requirements.

Places with no real IT people, just a guy in the place that can take care of the few day to day things....

Wait, I see the common thread - Very small places (data-wise) with little to no Linux experience.

Re: What are the pain points with deploying your Splunk architecture on Windows OS?

woodcock — Wed, 29 Mar 2017 01:25:10 GMT

Most of the splunk documentation (and especially the training documentation) is *NIX-focused. Things are much better now but even so, in most classes that I attended (even last year) there was somebody on Windows whose cut-and-paste would not work because it was wrong. This is obvious because the instructors chat heads-up warnings about these problem to everyone.

Re: What are the pain points with deploying your Splunk architecture on Windows OS?

lycollicott — Wed, 29 Mar 2017 14:40:51 GMT

Windows permissions and file ownership, particularly on on indexers. I have had too many Bucketmover inflight errors, because either LocalSystem or an MSA could not create, delete or rename folders. There are workarounds and you can routinely icacl.exe it, but who has ever had to cron chmod or chown commands on their *NIX indexers? No one.

Re: What are the pain points with deploying your Splunk architecture on Windows OS?

woodcock — Wed, 29 Mar 2017 18:03:42 GMT

There is some kind of intractable race-condition between the Windows Splunk service and many logging services such that a standard installation of Splunk can come up in such a state that events cannot be forwarded without experiencing corruption. The work-around is to delay the start of the Splunk service but even this does not always prevent the problem (although it usually does). Keep in mind that you need to monitor the OS on your Splunk infrastructure, too, so problems forwarding in security logs there are big problems. See here:

https://answers.splunk.com/answers/200924/formatmessage-error-appears-in-indexed-message-for.html

This makes Windows a risky option for Heavy Forwarder or Syslog+UF.

Re: What are the pain points with deploying your Splunk architecture on Windows OS?

woodcock — Thu, 25 May 2017 20:54:02 GMT

If you ever have blocked queues, you may find that your Indexers suddenly refuse to receive data from forwarders requiring the whole Indexer tier to be rebooted (does not happen on Linux Indexers):

https://docs.splunk.com/Documentation/Splunk/6.6.0/Forwarding/Receiverconnection#Receiver_does_not_accept_new_connections_on_its_receiving_port

Re: What are the pain points with deploying your Splunk architecture on Windows OS?

niketn — Sat, 02 Sep 2017 05:19:31 GMT

@woodcock, I would concur to this point because during alpha pre-release testing I had to wait for a week to get windows installer since the package for testing was initially only compatible with *NIX or MAC.

I also faced similar issue with Splunk's Custom Visualization API while working on a Customer requirement for a new visualization and the the API failed to build as per documented step on Windows machine. It was confirmed to be an issue on Windows system. However, fix from Splunk did not come out. I ended up finding a fix myself.

https://answers.splunk.com/answers/509420/why-does-splunk-custom-visualization-api-result-in.html

Re: What are the pain points with deploying your Splunk architecture on Windows OS?

niketn — Sat, 02 Sep 2017 05:20:35 GMT

Sometimes the problem isn't a bug, rather things are unexpectedly "just different" in Windows. The following is a good Splunk documentation link to get to know differences between Unix and Windows operations:

http://docs.splunk.com/Documentation/Splunk/latest/Admin/DifferencesbetweenunixandwindowsinSplunkoperations

Re: What are the pain points with deploying your Splunk architecture on Windows OS?

martin_mueller — Sat, 02 Sep 2017 14:47:48 GMT

A rolling reboot in a cluster shouldn't pose a big issue.