Can I put splunk installation on NAS?

[melonman]

Hi,

I was wondering if I can put splunk installation (SPLUNK_HOME) on NAS?
Is it supported?

Thank you,

[MasterOogway]

Taking a view from the other side, I would say yes. You absolutely can install Splunk on NAS and have your Indexers Search heads reside there. The only caution I have is to make sure your storage team has a very "tuned" environment and your servers mounting NFS with the correct options.

Case in point. I have just completed an upgrade of (4) separate Splunk environments running SuSE LINUX and all the storage requirements were met by using NetApps Filers, to a consolidated shared environment for 1) Platform; and a second Shared environment for 2) Applications. Again, all based on NetApp 6210 Filers.

I confirmed any concern of not handling throughput to the filer by running Bonnie++ testing against the NFS mounts. My results came back like this:

Sequential Output: 86.6MB/sec

Sequential Input (random): 104.2MB/sec

When compared to the measured indexing rate I see on a single Indexing server I believe the NAS NFS can stand on its own. This server by itself sees in excess of 15-20GB/day of data with the following rates:

Events/Sec: 1350

Indexing rate(KBps): 370

The server in question is setup with the following:

(1) 10GB interface; 48GB Memory; (24) 1600MHz CPU's

NFS mount options: nfs rw,bg,hard,intr,rsize=32768,wsize=32768,timeo=60,nfsvers=3 0 0

With this knowledge and the understanding of our future growth rates (which are large), I won't hesitate to continue using Tier I & Tier II NAS as the backend for both hotdb/warmdb (Tier I NAS), and colddb (Tier II NAS). But let me reiterate, don't walk into this blind. Make sure your NAS can handle the load Splunk will put on it when indexing larger amounts of data. Do some extensive testing with Bonnie++ and compare it against what you can get against local or SAN disk.

Good luck.

[melonman]

Thanks, this is also great information!

[tfussell]

Thanks MasterOogway for the detailed explanation. I was hoping you could share even more details on the NetApp configurations. for example (although maybe I'm missing something else)
-What are the disk/SSDs that you are using (any flexcache or flashpool)?
-With 2 x 10GB network cards using LACP?
-Just a 2-node cDOT cluster or more (4, 6, ...)?

Do you have an statistics on the IOPs that the Netapps are providing?

Any info that you can provide on the configs would be helpful. thanks in advance!

[Damien_Dallimor · Accepted solution]

You can safely use NAS storage ie:NFS, for Search Head Pooling and Mounted Bundles, no problem.

But you would not want to run your Splunk Indexers on NAS storage because the IOPS latency would be detrimental to performance.

Considerations regarding NFS

Alternatively to local disk , I've used SAN block based storage over a fibre channel and achieved acceptable IOPS on my Splunk Indexers.

[Damien_Dallimor]

In a distributed architecture it is fine to use an NFS volume as your shared storage for Search Head Pooling and Search Peer Bundle Mounting (etc/users , etc/apps , etc/system directorys )

I would not however install your Splunk application instance(either Standalone Splunk, Search Head, Search Peer etc..) on an NFS volume for the performance reasons alluded to in my first reply.

[melonman]

I can see the part of config files and indexes can be in NAS or NFS, but splunk installation itself is OK to put on NAS/NFS and poot from mounted NAS/NFS?
e.g. /mount/nfs1/splunk/bin/splunk start