I mentioned earlier that one of the presentations at VMworld Europe 2008 was about measuring at the level of the virtual SCSI adapter of a VM. A wealth of information is available when looking at this kind of information.
A tool is available on ESX 3.5 that creates histograms by default (and complete traces if wanted) is VscsiStats. As an option, one provides the vSCSI handle ID and the VM World ID. In order to get some statistics at all, one first needs to start the monitoring:
./vscsiStats -s
After some time, the relevant statistics can be fetched by issuing a command like:
./vscsiStats -i 8260 -w 1438 -p ioLength
This, for instance, yields an histogram of the size of IO pacakges sent to the virtual SCSI adapter (and thus to the storage array). To finalize the monitoring, it has to be stopped as well:
./vscsiStats -x
The result of the command above, in my test, was a graph like the one below:
This is a VM running Windows 2003. Remember this is a histogram, we put the measures points in 'buckets' according to their size and plot their relative frequency.
Remark that there is some IO with sizes 4095 bytes 8191 bytes. This is a sign that the file systems are not aligned properly. The fact that 4K sizes are the majority is nice, because VMFS is optimized for IO of 4KB.