We use virtualization *a lot* in our infrastructure. The advantages are numerous but for sure it saves costs, makes management easier and improves security by isolating services on their own virtual machines (and you can make a very specific configuration for the service).
Putting that aside we were moderately happy about Xen. We have been using Xen since 3.0 and lets say the battle for the optimal configuration was not an easy one. Xen has regularly given us bad magic in the past. In the end it the magic mostly disappeared but it still does have a nasty side (i hate the xen network scripts, even though we disable them nowadays) and many bugs. The latest version 3.2 is definitely the best till now and Xen is stable nowadays, at least in our “optimal configuration”.
The most unpractical things with Xen after you get rid of the bad magic effect are the limited available choice of kernels (a pain with new hardware support and security). We sticked to paravirtualization till now because of its performance advantages. Which brings another pain with Xen: upgrading to new versions or migrating guests between Xen servers running different version (you can copy new kernel modules in the guest everytime).
A curse here and there is definitely on its place.
Then meet KVM. If your used to Xen (open source version) then it seems like virtualization like god intented. We tested KVM for a while now and everything is just like you expect and no nasty scripts everywhere (at least I didn’t see them yet).
Virsch is quite handy too (although you also seem to able to use it with Xen, we never did). Although I must curse shortly about virt-install for its bugs and ubuntu-vm-builder for the lack of support for installing debian (which are both not part of kvm, but can be used to “easy install” guests).
In any case we are giving kvm a proper testing now (for at least a month) and it looks something that might replace Xen entirely in our infrastructure on the long term. It is just sweet.
One reason to blog about this is the following:
- What are you experiences with running kvm in production?
I am also very interested in experiences with running a windows *server* guest with continuous reasonable load. Notes about any quirks or limitations encountered by experience are very welcome as well.
No related posts.
I installed 2 hardy 8.04.1 servers today, and then installed KVM 72 (there is a ppa out there) and VDE. So far so good, works well, I am going to need to write some scripts to start / stop vms etc, but it seems simple enough.
Take a look at kvmctl:
http://kvm.qumranet.com/kvmwiki/HowToConfigScript
Could you switch your captcha to reCaptcha? Not only is it more secure (2 captcha words instead of 1), it helps manually ocr old books and puts them into the public domain.
Take a look at the wordpress plugin for it here:
http://recaptcha.net/plugins/wordpress/
There is also a white theme for reCaptcha that would match your blog better. Take a look at the example on my blog:
http://digitalprognosis.com/blog/2008/02/14/troubleshooting-running-systems-with-lsof/
The reason I/we prefer to use kvm in production that, at least in paravirtualization mode, that nearly all our testing results showed its faster under load – which seems quite sensible given its paravirtualization nature.
In our area (database, webservers, proxies) the difference was quite big in some areas – up to 100% more throughput.
Our windows instances run under kvm though – there was some problem with full virtualization of windows instances under xen. Unfortunately I dont recall what kind of problem it was. All our production windows machines are win32 2003 running relatively smooth – they dont suffer really high load though – we noticed somewhat reduced io performance.
Given that the machines hosting windows are different from the linux ones the xen/kvm mix isnt much of a Problem…
+speling
I switched our Xen server at work to KVM last week. The stock Xen kernel (still at 2.6.18) just doesn’t support the RAID controller, and none of the other kernels for Ubuntu were stable. With KVM, the box is rock-stable now, and some users report that it is even a little bit faster.
Well that’s nice… We’re looking at XEN instead of KVM right now because of the should-be-better support and stability. Martin’s reply gets me wondering if XEN is such a good choice when running Ubuntu. Stability is a number one prioritiy for the to-be-virtualized server, so I’m wondering if I should skip XEN and switch to KVM now, or stay with XEN and switch to CentOS (or Debian?), like my colleague would like me to do.
Is KVM’s performance still okay when the hardware doesn’t have VT? The machine that’s going to be virtualized is a dual Xeon 3GHz with 5GB RAM. The CPU’s are probably the last non-VT enabled Xeons produced by Intel (which I only found out last week), but I’m hoping they’re still powerfull enough to be hosting at least 2 virtual machines (one linux, one Win 2003). Would this be a realistic setup for XEN or KVM?
Mattijs: Our best results were with Debian Etch with Xen 3.2 from Xensource (and not from the repositories).
Maybe Xen got better in the ubuntu/debian repositories but we always had some extra funky stuff.
We also ran Xen on CentOS for a while (i think we tried as from source and from repo) and i remember we sticked with 3.2 from Xensource there as well. But CentOS just drives you nuts after a while.
About KVM, its only VT right? Else your just running qemu, no?
In your case Xen still looks the best option without VT but remember that the open source Xen version doesn’t have PV drivers for Windows (at least it did not 1,5 year ago), if so CPU will be fine but network and disk i/o will be shabby.
Herman: I just did a quick google on ‘xen pv drivers windows’ and this brought up the following link:
http://wiki.xensource.com/xenwiki/XenWindowsGplPv
I guess that means that disk and network I/O shouldn’t be much of a problem anymore?
Then again, looking at the known issues: “0.9.8 appears to cause problems with BSOD’s and filesystem corruption under some circumstances”
Perhaps we should just start using Lotus Domino for Linux instead of the Windows version. Problem is, I don’t have any Domino experience, and my colleagues who do, don’t have that much experience with Linux. So we figured virtualisation could bring us the best of both worlds, saving us a lot of time.
But all in all I guess we’ll switch the dom0 from Ubuntu to Debian. This will have the extra benefit of being able to follow this (IMHO very promising) project:
http://hannibal.solstice.nl
Maybe in time, when all our hardware supports VT, we can make the switch to Ubuntu with KVM, but right now it’s just a bit too early.
Nice to see that things progress at least. If available I suggest running Xen 3.2 instead of 3.1, we didn’t have any stability problems with 3.2 yet.
Hannibal looks nice, from what I read quickly i like their approach. Although I didn’t take a look at the details yet.
Well… turns out we can’t use XEN after all. To use Windows you definitely need VT. So we’re now ‘stuck’ with VMware. I tried ESXi and to be honest – it works like a charm. Where I spend days trying to get XEN working (probably my bad), ESXi was set up in one evening, including a fully functional Win2003 vm. Just too bad VMware doesn’t have an Infrastructure Client for Linux, so I’m stuck with using RDP to connect to a Windows host and using the Infrastructure Client from there to connect to ESXi. So much for open source
Just buy a new server. Better then standarize on something you actually don’t want. It will haunt you later.
I just ordered a second server for KVM. So we can start migrating from Xen.