VMware memory management: Memory Tax

I'm looking into the specifics of how the ESX hypervisor handles memory, and how resource allocation is performed.

One of the things that has kept me busy was the so-called 'memory tax'. This concept is explained in esx3_memory.pdf but was not clear to me when reading this document. This is a quote from the document:

If a virtual machine is not actively using its currently allocated memory, ESX Server charges a memory tax — more for idle memory than for memory that is in use. That is, the idle memory counts more towards the share allocation than memory in use. The default tax rate is 75 percent, that is, an idle page of memory costs as much as four active pages. This rate can be changed by modifying a parameter setting.

Looking further on the web, I found that in 2002 at, a conference, Carl A. Waldspurger was awarded the best paper award for the following: Memory Resources Management in VMware ESX Server. The slides of the presentation can be found here. Maybe it is because I'm used to reading papers (or at least I used to), but I found the explanation much more clear in this document.

I encourage everyone interested in memory management to read the paper.

Capacity Monitoring for Desktops?

With the rise of VDI (Virtual Desktop Infrastructure, see here for a comprehensive overview) and the momentum it has, one starts to ask similar questions as with conventional server consolidation: what type of virtualization platform is required, how many users/desktops can I host, will there still be room for scaling, etc.

The way to approach this in server virtualization projects is by means of capacity monitoring and planning using virtualization scenarios. We refer to earlier posts for more information about this topic.

The question we are asking: can we use the same concepts and ideas for desktop virtualization? My answer is 'NO', because:

1. A user acts completely different than a process or service: less predictable, depending on our mood, depending on the time of the day, etc.
2. Some end-user applications ask 1OO% of the CPU even while they are not doing anything. The classic example used to be the game 'Pinball'. Even a screensaver can take a large amount of CPU power.
3. In a VDI context, this becomes even more important. For instance: why would you scale your virtual desktop CPU and memory to include desktop search if really nothing personal can be found on the hosted desktop?
4. When starting and running 10 applications at the same time, we will probably only use 5 of them later, but still... they require CPU power (and memory) while seeming idle.
5. If a user has meetings half the time, does that mean his session is closed? Does it still require processing power? How can this be analyzed?
6. Etc.

In other words, a desktop environment is inherently different from a server environment. This is why in my opinion it is harder to maintain a Citrix farm than a VMware farm: applications and users tend to be less predictable and 'stable' than servers.

Does that mean that we can not do any sizing or planning in a VDI context? On the contrary, we should only keep in mind that applying exactly the same reasoning as with server consolidation planning is not a good idea.

A few more tips:

• Make sure you have an overview of running processes and their CPU/memory utilization. This helps in deciding what is really important.
• Be careful with averages and peaks: a PC that is running all the time will have a low average, but may be used heavily during the day and a PC may be 100% used during the day because of some applications.
• Take into account inactive time during the day.
• Do not try to analyze the 'average' user, instead create classes (say 5 or so) that each have typical characteristics. Use these classes to create the virtualization scenarios.

Chargeback

On the VMworld website, you can find a featured presentation concerning chargeback. The presentation is by the CEO of VKernel, a company that I have mentioned on this blog before.

Chargeback, as pointed out in the presentation, is about 'measuring' and accounting for two types of costs: Consumables and non-consumables. The latter has to do with floor space, licenses, support and administration, etc. Consumables refers to utilization of CPU, memory, power, etc. Obviously, performance monitoring is critical in this aspect. Check out the presentation (registration required)!

Performance Monitoring: Correlations

Although I have been arguing that performance monitoring and capacity planning require a decent server montoring environment, it also requires more. This extra part comes from the fact that often services depend on each other. A web service connects to a database (hosted on a different server) and fetches data from the file server (local, or SAN/NAS). Often, one part in the chain is a bottleneck for the whole of the process. This is a shame and can be avoided by careful analysis of the correlations between performance data.

Again, this is an argument in favor of what I called a 'load profile' earlier. By modeling a server by means of a load profile, we get a representation of that server in terms of measurable quantities. Statistics and mathematics in general can then help us analyze the correlations between those load profiles.

Performance Monitoring: More about Peaks

In a previous post, we talked about averages (types of averages) and peaks and how peaks can tell you something about the spreading (variance, standard deviation) of the data.

Information about peaks is required (especially in capacity planning situations) to understand the sizing of the platform you're running on. On the other hand, having a peak utilization of (say) 80% and an average of 20% still does not tell you that much: how long was the system running at high CPU levels? Maybe only for 10 seconds during the day (a scheduled database operation, backup procedures, etc.)? Is it crucial for our service that this high level of CPU can be guaranteed at that moment, or is it affordable to let the application/server wait a little longer for CPU requests? Think of a mail server, for instance, where it wouldn't be a big deal if the server would forward your mail a few milliseconds later or earlier (would it?).

Basically, what we need is a load profile for a server. A load profile contains information like:

• Load during hour, day, week, month (or any other relevant period for this server)
• Expected response times instead of observed response times (basically, a cutoff on the resources)
• Current hardware inventory
• Current 'scaled' hardware inventory (20% CPU usage is different for a quad core than a single core, a scaled inventory takes that into account and enables easy comparison of systems)
• Etc.

More about load profiles later...

Performance Monitoring: Averages and Peaks

Now that we are into the topic of performance (of capacity) monitoring and planning, let us continue with something that has kept me busy the last couple of days: averages of performance data (and other statistical information) versus peaks.

This goes back to a classic textbook example in statistics, where the mean value of a series of data points is completely irrelevant as a representation of the data points itself. Let us consider the following example. Given a series of data like in the table below:

X Y
A 1
B 2
C 4
D 7
E 100
F 4
G 9
H 7
I 3
J 5

These may, for instance, represent scores (0-100) given to students (by a very strange teacher). In a graph, this is presented below:



The red line represents the average value. It is clear that everyone (except the teacher's little friend with 100 points) is below the average. As a consequence, the average is not a good representation of the data as a whole. Some say it is too much influenced by the extreme values. In fact, this average (sum of the data values divided by the total amount of data points) is called the arithmetic mean. There is another notion of 'an average' which is called 'geometric mean'. In the example above, the value of it would be 5.4 which is much more relevant. The median would even better define the data set, but that would lead us too far.

Basically, the fact that the arithmetic mean does not give a good indication of the data set is caused by the large spread of the data. In statistics, there is another indicator for this: the variance, or standard deviation. It is a measure for how close or far apart the values are. In the example above, the standard deviation. In our example above, it would read 30.2. Suppose the value of E would be 10 instead of 100, the standard deviation becomes 3. In others words, the lower is the standard deviation, the more the data is 'close'.

The above brings me back to performance monitoring. Somehow, I want to summarize the performance data of a server by a small set of indicators (averages of time etc.) that give a reasonable picture of the actual performance or in other words: that are representable for the system's actual performance. Typically, when looking at the percentage a CPU is used over time, we see fluctuations that are similar to the figure above (don't believe me, below is an actual example of a system with some high peaks and further sitting idle for most of the time - this server has 4 cores by the way). We conclude that, therefore, it does not make much sense to look at simple averages of the performance counters in order to get an idea of the behavior of the system..



Coming back to VMware Capacity Planner: the tool keeps track of the average value (yes, the simple average that does not say much) but also internally uses the geometric mean but not the variance (as far as I can tell). From this perspective, the whole performance gathering using this tool would be worthless. Luckily, it also keeps track of peak values (and calculates averages over these peaks but that is another story). Comparing the peak values with the average tells us a lot about the spread of the data points. The system behind the graph above, has an average CPU value of less than 20% while the peak CPU utilization is higher than 90%! This tells us that the variance/spreading in data points is large.

In a virtualization/consolidation assessment, these cases have to be taken into account, as we do not want our systems to become unresponsive because they have their peaks at the same time. More about this and other topics later...

Capacity Planning: What to monitor and how to interpret

Capacity planning starts with capacity (of performance) monitoring.

Everybody who is involved in the monitoring of systems will acknowledge that the most difficult aspects in monitoring a server (or set of servers) are:

1. Finding the proper indicators for the performance of the system (CPU usage, CPU cycles, memory usage, paging, etc.)


2. Making sure they are queried regularly, but not too much in order to avoid impacting the performance of the system by monitoring it.


3. Storing the resulting data


4. Summarize, create views, average, etc. (this also depends on what you want to know about the system)


5. Analyze, interpret, etc.

Did I say the most difficult aspects? Are there any other aspects? Well, not really... capacity monitoring (and planning as a further step) is not an easy task:

• Are you aware of the utilization of your systems? Even of your workstations?


• Would you have any idea how many of your servers could be placed on a virtualization platform with a specific set of hardware characteristics?


• Would you know when your mail server had the hardest time managing mail boxes the last couple of weeks?

Probably the answer is 'no'. Maybe the answer is 'I don't care'?

Most companies do care, because of several reasons: cost, manageability, flexibility, scalability, environment, space, etc.

There are already some players on the market: VMware Capacity Planner (see earlier posts), PlateSpin PowerRecon, Veeam Monitor, etc. I'm mostly used to VMware Capacity Planner (VMCP) but recently, I have also evaluated both PowerRecon and Veeam Monitor. More about this later.

VMware Capacity Planner: taking the data offline (Part 3: Getting CSV data)

This is part 3 of a series concerning VMware Capacity Planner.

First a general remark about the update to version 2.6 of the tool: it was not always pleasant to use the web interface the last week because of downtime (scheduled maintenance), errors in the interface, latency, etc. It all feels normal again, and some of the new features where definitely worth it.

Back to business: What interests me most when it comes to downloading data from the website is the pure performance data. VMware averages this information over a week, so data can be downloaded on a weekly basis.

I'm not going into the full details of everything that can be tweaked, configured or queried but most of it will be clear when looking at the actual command-line to get a CSV file. Most important is understanding that some of the choices you make are based on the session that is open on te server, and are not selected using POST or GET. Some of these options can be configured on the html view, but not the export view (although the syntax is available) so we have to apply a trick: first get the html view for the type of data we want and then export the data.

Below, you find a typical command-line to download the core VMware statistics for week 45 of a company with ID 1234:

wget --keep-session-cookies --load-cookies cookies.txt
--save-cookies cookies.txt https://optimize.vmware.com/inv_report.cfm?page=ViewStatsMain2.cfm&action=export§ion=View&sort=ObjectName&sortDir=ASC&YearWeek=200745&Mon=2&ISO=2&opt=P&PerfGroupID=1&grpt=S&HostType=0&menutab=Performance --post-data CID=1234&YearWeek=200745 --no-check-certificate -O Stats.csv

Note that quite some things have to be specified: week, company, type of average, type of metrics, system group, etc.

Drop me a line or leave a comment in case you want to know more about the precise parameters that are important or in case you're interested in my little script that does the magic for me.

This concludes the practical part of this series. I plan to write more about capacity planning and capacity monitoring in general later...

VMware Capacity Planner: taking the data offline (Part 2: Cygwin & wget)

I started thinking about avoiding the manual exports from the VMCP website and remembered I used WGET in the past for this kind of stuff. The difference with before was that this time, I needed to login to the website before being able to get data from it.

Login in to the VMCP website is done using a POST method and luckily wget supports that. The next thing is to understand what the post data needs to be. This can be fetched from the source of the logon page:


The relevant POST data is derived to be

fuseaction=Security
Page=users_login.cfm
username=???
password=???


Since I run Windows XP on my laptop, I use Cygwin to run wget. Cookies are used to store a session ID for you logon session, so wget has to be told to store those cookies in a file. This is the resulting command line:


wget --keep-session-cookies --save-cookies cookies.txt 'https://optimize.vmware.com/index.cfm' --post-data 'fuseaction=Security&Page=users_login.cfm&username=???&password=???' --no-check-certificate -O output.html


Parsing the output.html file, you should be able to see whether logon was succesful. The success depends on many factors (local proxy, network settings, username/password, etc.). You can get more info by adding suitable options to wget.

This concludes part 2 of this series of articles. In this part, we used wget to logon to the VMware capacity planner website.

VMware Capacity Planner: taking the data offline (Part 1: Introduction)

Lately, I have been involved in some VMware Capacity Planner (VMCP) tracks. VMCP deals with monitoring a set of (physical) servers in order to assess possible virtualization scenarios.

VMCP works in the following way: a monitoring server is set-up with a tool that does (but is not limited to) regular performance measurements. This tools sends the data to a VMware database through the web (over a secure channel). Via a web interface, one can then query the information, get reports, view trends, show graphs, configure and create consolidation scenarios, etc. Usually, we let the system run for around 4 weeks to get a realistic idea of the performance characteristics.

What I like about VMCP is that the data is not located at the customer site, and available at all times (once it has been uploaded). This gives me the opportunity to regularly check on the status of the performance measurements.

The biggest disadvantage of VMCP is that the web interface is not the most flexible and fast interface around. Some things I would like to do are not available (lack of flexibility) but could easily be done in, e.g., Excel and at times when everyone in the world is awake it takes ages to refresh a page and get the result of a query. Moreover, it is not easy to get good-looking information to paste in a document.

When it comes to writing a report, the customer is obviously not only interested in a statement like: you will need 5 ESX server of type X to cope with the load. Therefore, I like to add tables with the most useful metrics (CPU, network, Disk I/O, ...) for later reference. I add this information as an appendix.

This is where I would spend at least half a day exporting CSV files from the VMCP website, loading them in Excel, laying it out as a nice table and paste it in the document. I started thinking about automating some of the steps required, and I covered the most time-consuming already: exporting the information from the website as a CSV file.

In the following part, I'll explain how I started this little adventure...