how to make a temporary backup/restore of Splunk on Windows server ?

[ruiaires]

We've been having severe Splunk performance issues on the following system:

• Windows 2008 R2 Enterprise 64 with a 2 CPU Xeon E5405 2,0Ghz with 12GB RAM
• The relevant index has only 1Gb of size and about 17 million events.

Searches on everything (including _internal and other small indexes) are very slow... Machine resources (CPU, RAM, Disk) are all OK.

Performance was never great when Splunk was installed and started indexing data... but it got really bad as more data was being indexed (it's been just a few months)

As a method of eliminating the current Splunk configuration and indexes from the problem I'm thinking of a fresh new install of Splunk to see how it handles.

There is no problem with downtime or loosing data (it's monitoring log files from a network share) so, I thought about

1. making a full backup of /etc and /var (or even everything)
2. uninstall splunk
3. install clean version
4. make tests
5. remove and re-install
6. copy over the restore

My questions are with: - is there any "windows dependency" with registry or other problems I can expect ? - should I roll the hot buckets as indicated in other answers here ? - will all the configuration be kept (users, etc) ?

[ruiaires · Accepted solution]

After a full re-install, the performance was back at the expected level. We kept a full backup of the Slunk installation but I hope never to need "post-mortem" analysis on that.. However, in the end, we never found out what the problem was in the first place 😞

[gkanapathy]

There are no outside Windows dependencies other than the Windows Services (which are removed by the uninstall). There is no need to roll hot buckets if you copy them when Splunk is not running.

I don't really know why you'd have such performance problems, and I kind of doubt that this uninstall/reinstall will help. What will help a lot is if you identify for us what the disk you have Splunk on is. In particular, for both the %SPLUNK_HOME%, %SPLUNK_HOME%\var, and the %SPLUNK_DB% file locations (by default these will be all in the same place), please let us know the type of filesystem, the disk size, the disk speed, disk type, RAID configuration, whether it is local or remote, and whether there might be other applications using it.

[ruiaires]

It's a local 130Gb SCSI disk with NTFS. I don't now the disk speed. We have the following "big" indexes:
1. _internal, 800Mb, >153Million events 2. production, 1Gb, >18Million events (this is the main data)

searches are slow across Splunk, even on _internal, even for small timeframes... data is very simple log files that, when added in laptop installations yield thousands of results in a few seconds... we think it's somehow "server/os" related and splunk is not getting enough resources to "run properly"... any ideias ? – ruiaires 0 secs ago