Thicket data repository for the EEG
anil.recoil.org
1{
2 "id": "https://anil.recoil.org/notes/shedding-some-light-on-xenapp-on-xenserver-performance-tuning",
3 "title": "Shedding light on XenApp on XenServer performance tuning",
4 "link": "https://anil.recoil.org/notes/shedding-some-light-on-xenapp-on-xenserver-performance-tuning",
5 "updated": "2008-08-04T00:00:00",
6 "published": "2008-08-04T00:00:00",
7 "summary": "<p>You won\u2019t be surprised to hear that we spend a lot of time improving\n<a href=\"http://www.citrix.com/XenApp\">XenApp</a> performance when running on\n<a href=\"http://www.citrix.com/XenServer\">XenServer</a>. Although there are some\ngood benchmark comparisons available (such as the <a href=\"http://community.citrix.com/x/_4ENAg\">Tolly\nGroup</a> report), I still get a lot\nof customers asking about what the \u201csecret sauce\u201d is. I sat down with\nGeorge Dunlap, the lead XenServer performance engineer to chat about the\nvery first optimisation we did back in XenServer 4.0 last year.</p>\n<p>Before we dive in, we first need to explain how a normal operating\nsystem handles memory. George explains:</p>\n<blockquote>\n<p>Modern desktop and server processors don\u2019t access memory directly\nusing its physical address. They use \u2018<a href=\"http://en.wikipedia.org/Virtual_Memory\">virtual\nmemory</a>\u2019 to separate the\naddresses that processes use to read and write memory from the actual\nmemory itself. This allows operating systems to hide from processes\nall the dirty details of how much memory there is, where in physical\nmemory the process needs to write to, and so on.</p>\n<p>However, the actual processor still needs to translate from a\n<a href=\"http://en.wikipedia.org/wiki/Virtual_address\">virtual address</a> to the\nphysical memory address in order to actually read and write any\nmemory. This translation is done with something called <a href=\"http://en.wikipedia.org/wiki/Page_tables\">page\ntables</a>.</p>\n<p>Page tables are used to implement virtual memory by mapping virtual\naddresses to physical addresses. The operating system constructs page\ntables using physical memory addresses, and then puts the physical\naddress of the \u201ctop-level\u201d page table into a hardware register called\nthe \u2018base pointer\u2019. Then the processor will read these page tables to\ntranslate virtual addresses to physical addresses as needed, before\nreading and writing to physical memory.</p>\n</blockquote>\n<p>Most modern processor types have some sort of paging mechanism, although\nXenServer is specifically tuned for\n<a href=\"http://en.wikipedia.org/wiki/X86-64\">x86-64</a> CPUs. An excellent book on\nthe general topic is <a href=\"http://en.wikipedia.org/wiki/Special:BookSources/0130313580\">Modern Operating\nSystems</a> by\n<a href=\"http://www.cs.vu.nl/~ast/\">Andrew Tanenbaum</a>. When XenServer creates\nWindows VMs, it takes advantage of the <a href=\"http://en.wikipedia.org/wiki/X86_virtualization\">virtualization\nextensions</a> in modern\nCPUs, which requires special memory handling in Xen. George explains\nthis further:</p>\n<blockquote>\n<p>When we create a virtual machine, we virtualize the memory as well;\nthat means that the guest operating system\u2019s idea of physical memory\ndoes not match up to real physical memory on the host. Traditionally,\nwhat the guest thinks of as physical memory is called \u201cphysical\nmemory\u201d, and what the hypervisor thinks of as physical memory is\ncalled \u201cmachine memory\u201d. Since this terminology is a bit confusing,\nXen tends to call what the guest thinks of as physical memory as\n\u201cguest physical\u201d memory, just to help make things more clear.</p>\n</blockquote>\n<blockquote>\n<p>This means that any fully-virtualized operating system, like Windows,\nwill create page tables using guest physical memory, and will point\nthe base pointer at the guest physical address of the top-level page\ntable. Unfortunately, the hardware still needs to map from virtual\nmemory address to machine addresses, not guest physical addresses.</p>\n</blockquote>\n<blockquote>\n<p>In order to allow this to happen, the hypervisor sets up <strong>shadow\npage tables</strong>. These page tables are generated by the hypervisor are\ncopies of the guest page tables, but with the guest physical addresses\nconverted into machine physical addresses. The guest cannot access\nthem directly, and they don\u2019t reside in the guest\u2019s physical memory;\nthey\u2019re generated out of a pool of memory that the hypervisor\nallocates when a VM is created, called shadow page table memory.</p>\n</blockquote>\n<blockquote>\n<p>What this means is that whenever the guest operating system wants to\nmap some new memory, after it writes the data into the page table but\nbefore it can actually use it, the hypervisor needs to translate the\nchange to the guest page table into changes to the shadow page table.\nSo any workload that involves a lot of this will necessarily involve\nthe hypervisor a lot, which causes overhead.</p>\n</blockquote>\n<p>So shadow page tables are our mechanism of giving a guest an interface\nwhich is identical to real hardware (so it doesn\u2019t need to be modified),\nbut still intercepting changes before they reach the real hardware. You\ncan find more details from the <a href=\"http://www.xensource.com/files/summit_3/XenSummit_Shadow2.pdf\">XenSummit 2006\ntalk</a> or\nfrom the 2005 <a href=\"http://www.cl.cam.ac.uk/research/srg/netos/papers/2005-migration-nsdi-pre.pdf\">NSDI\npaper</a>.\nSo how is this all relevant to XenApp performance? Back to George\u2026</p>\n<blockquote>\n<p>The hypervisor allocates a certain amount of memory for each VM to\nuse for shadow page tables; this is called <strong>shadow page table\nmemory</strong>. As new page tables are created and old ones aren\u2019t used\nanymore, the hypervisor cycles through this shadow page table memory.\nWhen it needs a new page and there isn\u2019t enough, it will \u2018unshadow\u2019\nthe guest page tables that haven\u2019t been used for the longest time to\nreclaim shadow memory, so that it can use more.</p>\n</blockquote>\n<blockquote>\n<p>We don\u2019t know ahead of time how much shadow memory a given workload\nwill use, but we can estimate based on the amount of memory that the\nVM has. We allocate enough shadow memory for each page to be mapped\nonce, more or less, then add an extra 50% to have some slack. For all\nthe workloads we\u2019ve tested, that\u2019s been enough \u2013 except XenApp.</p>\n</blockquote>\n<blockquote>\n<p>XenApp is the one workload we\u2019ve found that requires more shadow page\ntable memory than our standard default. Because XenApp generally\nstarts hundreds of copies of the same process, the same memory ends up\nmapped in hundreds of different processes. What happens when all of\nthose processes are active is that XenServer is continually\nunshadowing one process\u2019 page tables in order to shadow another\nprocess\u2019 pagetables; only to have to re-shadow the original ones a\nsecond or two later when it runs again! This is called\n<a href=\"http://en.wikipedia.org/wiki/Thrash_(computer_science)\">thrashing</a>,\nwhen there\u2019s not enough of a limited resource.</p>\n</blockquote>\n<p>Once the bottleneck was discovered, the solution was simple. In\nXenServer 4.1, we created a special XenServer application template\ncalled <em>\u201cCitrix XenApp\u201d</em>, which has an increased shadow multiplier that\nreserves more shadow memory for the guest when it starts. This is also a\ngood example of how templates hide the complexities of performance\ntuning from the user, but still permitting custom modifications if they\nare required. For example, on your XenServer host with a VM called\n\u201cXenApp\u201d, you could view the shadow multiplier by using the CLI:</p>\n<pre><code># xe vm-list name-label=XenApp params=HVM-shadow-multiplier\n HVM-shadow-multiplier ( RW) : 4.000\n</code></pre>\n<p>The same value is also available from XenCenter in the Optimization\npane, but of course do remember that the default value was chosen\nthrough extensive testing and doesn\u2019t need to be changed. Most of the\nother templates in XenServer also have carefully tuned settings (e.g.\nthe hardware platform flags) to ensure smooth running, or in the case of\nLinux templates, to support <a href=\"http://docs.xensource.com/XenServer/4.1.0/1.0/en_gb/sdk.html#id2553443\">para-virtual\ninstallation</a>.\nThis is why it\u2019s so important that you not use the <em>\u201cOther Install\nMedia\u201d</em> template in preference of a more specialised one!</p>\n<p>I mentioned at the beginning of this post that this was the first of\nmany XenApp optimisations. We\u2019ve just released the <a href=\"https://www.citrix.com/English/ss/downloads/details.asp?downloadId=1679827&productId=683148\">public\nbeta</a>\nof the latest XenServer (\u201cOrlando\u201d) which is even faster. The story of\nwhat those improvements are, and the tools which George and his team\nuses to analyze the inner workings of Xen, are a topic for a future\npost. For now, get downloading XenServer and start virtualizing your\nXenApp installations! Or if you\u2019re feeling inspired, go over to\n<a href=\"http://xen.org/\">xen.org</a>, check out the source, and get coding\u2026</p>",
8 "content": "<p>You won\u2019t be surprised to hear that we spend a lot of time improving\n<a href=\"http://www.citrix.com/XenApp\">XenApp</a> performance when running on\n<a href=\"http://www.citrix.com/XenServer\">XenServer</a>. Although there are some\ngood benchmark comparisons available (such as the <a href=\"http://community.citrix.com/x/_4ENAg\">Tolly\nGroup</a> report), I still get a lot\nof customers asking about what the \u201csecret sauce\u201d is. I sat down with\nGeorge Dunlap, the lead XenServer performance engineer to chat about the\nvery first optimisation we did back in XenServer 4.0 last year.</p>\n<p>Before we dive in, we first need to explain how a normal operating\nsystem handles memory. George explains:</p>\n<blockquote>\n<p>Modern desktop and server processors don\u2019t access memory directly\nusing its physical address. They use \u2018<a href=\"http://en.wikipedia.org/Virtual_Memory\">virtual\nmemory</a>\u2019 to separate the\naddresses that processes use to read and write memory from the actual\nmemory itself. This allows operating systems to hide from processes\nall the dirty details of how much memory there is, where in physical\nmemory the process needs to write to, and so on.</p>\n<p>However, the actual processor still needs to translate from a\n<a href=\"http://en.wikipedia.org/wiki/Virtual_address\">virtual address</a> to the\nphysical memory address in order to actually read and write any\nmemory. This translation is done with something called <a href=\"http://en.wikipedia.org/wiki/Page_tables\">page\ntables</a>.</p>\n<p>Page tables are used to implement virtual memory by mapping virtual\naddresses to physical addresses. The operating system constructs page\ntables using physical memory addresses, and then puts the physical\naddress of the \u201ctop-level\u201d page table into a hardware register called\nthe \u2018base pointer\u2019. Then the processor will read these page tables to\ntranslate virtual addresses to physical addresses as needed, before\nreading and writing to physical memory.</p>\n</blockquote>\n<p>Most modern processor types have some sort of paging mechanism, although\nXenServer is specifically tuned for\n<a href=\"http://en.wikipedia.org/wiki/X86-64\">x86-64</a> CPUs. An excellent book on\nthe general topic is <a href=\"http://en.wikipedia.org/wiki/Special:BookSources/0130313580\">Modern Operating\nSystems</a> by\n<a href=\"http://www.cs.vu.nl/~ast/\">Andrew Tanenbaum</a>. When XenServer creates\nWindows VMs, it takes advantage of the <a href=\"http://en.wikipedia.org/wiki/X86_virtualization\">virtualization\nextensions</a> in modern\nCPUs, which requires special memory handling in Xen. George explains\nthis further:</p>\n<blockquote>\n<p>When we create a virtual machine, we virtualize the memory as well;\nthat means that the guest operating system\u2019s idea of physical memory\ndoes not match up to real physical memory on the host. Traditionally,\nwhat the guest thinks of as physical memory is called \u201cphysical\nmemory\u201d, and what the hypervisor thinks of as physical memory is\ncalled \u201cmachine memory\u201d. Since this terminology is a bit confusing,\nXen tends to call what the guest thinks of as physical memory as\n\u201cguest physical\u201d memory, just to help make things more clear.</p>\n</blockquote>\n<blockquote>\n<p>This means that any fully-virtualized operating system, like Windows,\nwill create page tables using guest physical memory, and will point\nthe base pointer at the guest physical address of the top-level page\ntable. Unfortunately, the hardware still needs to map from virtual\nmemory address to machine addresses, not guest physical addresses.</p>\n</blockquote>\n<blockquote>\n<p>In order to allow this to happen, the hypervisor sets up <strong>shadow\npage tables</strong>. These page tables are generated by the hypervisor are\ncopies of the guest page tables, but with the guest physical addresses\nconverted into machine physical addresses. The guest cannot access\nthem directly, and they don\u2019t reside in the guest\u2019s physical memory;\nthey\u2019re generated out of a pool of memory that the hypervisor\nallocates when a VM is created, called shadow page table memory.</p>\n</blockquote>\n<blockquote>\n<p>What this means is that whenever the guest operating system wants to\nmap some new memory, after it writes the data into the page table but\nbefore it can actually use it, the hypervisor needs to translate the\nchange to the guest page table into changes to the shadow page table.\nSo any workload that involves a lot of this will necessarily involve\nthe hypervisor a lot, which causes overhead.</p>\n</blockquote>\n<p>So shadow page tables are our mechanism of giving a guest an interface\nwhich is identical to real hardware (so it doesn\u2019t need to be modified),\nbut still intercepting changes before they reach the real hardware. You\ncan find more details from the <a href=\"http://www.xensource.com/files/summit_3/XenSummit_Shadow2.pdf\">XenSummit 2006\ntalk</a> or\nfrom the 2005 <a href=\"http://www.cl.cam.ac.uk/research/srg/netos/papers/2005-migration-nsdi-pre.pdf\">NSDI\npaper</a>.\nSo how is this all relevant to XenApp performance? Back to George\u2026</p>\n<blockquote>\n<p>The hypervisor allocates a certain amount of memory for each VM to\nuse for shadow page tables; this is called <strong>shadow page table\nmemory</strong>. As new page tables are created and old ones aren\u2019t used\nanymore, the hypervisor cycles through this shadow page table memory.\nWhen it needs a new page and there isn\u2019t enough, it will \u2018unshadow\u2019\nthe guest page tables that haven\u2019t been used for the longest time to\nreclaim shadow memory, so that it can use more.</p>\n</blockquote>\n<blockquote>\n<p>We don\u2019t know ahead of time how much shadow memory a given workload\nwill use, but we can estimate based on the amount of memory that the\nVM has. We allocate enough shadow memory for each page to be mapped\nonce, more or less, then add an extra 50% to have some slack. For all\nthe workloads we\u2019ve tested, that\u2019s been enough \u2013 except XenApp.</p>\n</blockquote>\n<blockquote>\n<p>XenApp is the one workload we\u2019ve found that requires more shadow page\ntable memory than our standard default. Because XenApp generally\nstarts hundreds of copies of the same process, the same memory ends up\nmapped in hundreds of different processes. What happens when all of\nthose processes are active is that XenServer is continually\nunshadowing one process\u2019 page tables in order to shadow another\nprocess\u2019 pagetables; only to have to re-shadow the original ones a\nsecond or two later when it runs again! This is called\n<a href=\"http://en.wikipedia.org/wiki/Thrash_(computer_science)\">thrashing</a>,\nwhen there\u2019s not enough of a limited resource.</p>\n</blockquote>\n<p>Once the bottleneck was discovered, the solution was simple. In\nXenServer 4.1, we created a special XenServer application template\ncalled <em>\u201cCitrix XenApp\u201d</em>, which has an increased shadow multiplier that\nreserves more shadow memory for the guest when it starts. This is also a\ngood example of how templates hide the complexities of performance\ntuning from the user, but still permitting custom modifications if they\nare required. For example, on your XenServer host with a VM called\n\u201cXenApp\u201d, you could view the shadow multiplier by using the CLI:</p>\n<pre><code># xe vm-list name-label=XenApp params=HVM-shadow-multiplier\n HVM-shadow-multiplier ( RW) : 4.000\n</code></pre>\n<p>The same value is also available from XenCenter in the Optimization\npane, but of course do remember that the default value was chosen\nthrough extensive testing and doesn\u2019t need to be changed. Most of the\nother templates in XenServer also have carefully tuned settings (e.g.\nthe hardware platform flags) to ensure smooth running, or in the case of\nLinux templates, to support <a href=\"http://docs.xensource.com/XenServer/4.1.0/1.0/en_gb/sdk.html#id2553443\">para-virtual\ninstallation</a>.\nThis is why it\u2019s so important that you not use the <em>\u201cOther Install\nMedia\u201d</em> template in preference of a more specialised one!</p>\n<p>I mentioned at the beginning of this post that this was the first of\nmany XenApp optimisations. We\u2019ve just released the <a href=\"https://www.citrix.com/English/ss/downloads/details.asp?downloadId=1679827&productId=683148\">public\nbeta</a>\nof the latest XenServer (\u201cOrlando\u201d) which is even faster. The story of\nwhat those improvements are, and the tools which George and his team\nuses to analyze the inner workings of Xen, are a topic for a future\npost. For now, get downloading XenServer and start virtualizing your\nXenApp installations! Or if you\u2019re feeling inspired, go over to\n<a href=\"http://xen.org/\">xen.org</a>, check out the source, and get coding\u2026</p>",
9 "content_type": "html",
10 "author": {
11 "name": "Anil Madhavapeddy",
12 "email": "anil@recoil.org",
13 "uri": "https://anil.recoil.org"
14 },
15 "categories": [],
16 "rights": "(c) 1998-2025 Anil Madhavapeddy, all rights reserved",
17 "source": "https://anil.recoil.org/news.xml"
18}