Blog Posts

Last weekend, I presented a paper on storage workload characterization and consolidation at VPACT '09. In the paper, we did extensive characterization of some of the well known enterprise workloads such as MS exchange, OLTP, DVDStore, Decision support, in terms of various parameters such as seek distances, IO sizes, read/write ratios, outstanding IOs etc.

One of the main conclusion was that: Most workloads exhibit random access pattern. Furthermore, sequential cases such as decision support, backup or virus scanner, seem to be mainly the alternative workloads running on the same data-sets which are commonly accessed in random manner by actual applications. Detailed tracing also revealed that most of them were quite bursty in terms of arrivals.

Based on the two main observations, we tested the effect of consolidation on two different workloads, where they are first run on separate raid-groups and then on  a single raid-group consisting of the physical disk from both of them. The results from consolidation were quite encouraging as the workloads saw a big reduction in 90th percentile latency values. This is mainly because higher degree of burstiness leads to larger gains from statistical multiplexing and combining the underlying physical disks helps a great deal in absorbing the peaks during bursty intervals.

This suggests that administrators should re-think their rule of thumb of placing workloads on different spindles for better performance isolation. This does lead to better predictability in most cases but potentially causes severe over-provisioning as well. Isolation based on software mechanisms both at OS and array level would be much more desirable and cost-effective.

There is also an issue of semantic information gap, where array vendors may not get workload specific information in the request stream that they see. Hence the overall solution may involve cooperation between both end-points of the SAN or Ethernet and some sort of interface between OS and array vendors for stronger isolation.

Here is something to chew: What sort of support or interface should array vendors expose to support various performance isolation requirements for applications in terms of throughput, latency or perhaps burstiness? Are weights (or shares) good enough or we need other parameters such as latency specifications?

This paper was presented by Alexandros from NetApp. The main idea is to adaptively schedule asynchronous requests in order to favor synchronous ones to improve application's performance. For example, when the server is congested, the clients would delay asynchronous writes, read-ahead and send only the reads, metadata requests to the server. This obviously requires the client to consume more memory to keep those dirty pages for longer interval. To handle the pressure on other resources, authors propose a mechanism that can look at utilization of all resources, assign a cost to them and compare those to make such decisions.

The main issue is deciding the cost of an operation based on multiple resources.
They use a previously known result which says that increasing the price exponentially with increase in utilization of a resource leads to a competitive ratio of (log k), where k is the ratio of benefit obtained by and offline vs an online algorithm. They use the pricing function that takes a number K_i per resource and utilization u_i to compute the price. The price is computed for each resource based on its utilization and the highest value is used to depict the current bottleneck. The price calculated by server is sent to clients as part of FSSTAT call. The prices are computed every second and FSSTAT is send along with every tenth read/write request. This is a non-blocking call and doesn't impact clients' performance.
Finally they showed using both micro-benchmarks and real apps that this can lead to about 20% better performance.

Here is something to chew : Can we extend this mechanism to provide prioritization among different clients, in terms of proportionate shares or weights?  

http://www.usenix.org/events/fast09/tech/full_papers/batsakis/batsakis.pdf

The approaches that DynamoRIO and Pin take to

instrumentation are fundamentally different.  Pin's

interface is not so different from tools that modify the

original code, such as ATOM, Dyninst, Vulcan, or Detours.

The only instrumentation option in Pin is to insert a

callout or trampoline to instrumentation code at a

particular point in the application's code stream.  The

application code stream itself cannot be modified directly.

Pin does use its code cache to improve over non-code-cache

tools that modify original application code in two respects:

code discovery and transparency.  The code cache design

allows for incremental dynamic discovery of code, making it

simple to examine all code that is executed, while

non-code-cache tools must statically (or at load time)

determine which code to instrument based on the application

binary and library files.  The code cache also eliminates

safety issues of modifying variable-length IA-32/AMD64 code:

direct modification by inserting a 5-byte jump instruction

can overwrite an entry point midway through those 5 bytes,

as well as suffer from races with multiple threads and

complications if those 5 bytes are later examined or

modified by the application (essentially, the displaced 5

bytes are a "miniature code cache" that incur all the same

complications as a regular code cache).

DynamoRIO leverages its code cache to provide a much broader

interface.  DynamoRIO allows modification of the runtime

code stream of the application: modifying, inserting, or

removing individual instructions.  This is in stark contrast

to Pin's observe-and-callout model.  DynamoRIO's interface

is only possible with a code cache, and it takes full

advantage of the cache's power.  Not only does this general

interface let DynamoRIO support non-observational-only uses,

such as optimization or translation, but it also gives full

control over instrumentation to the tool writer.  While Pin

does automatically inline and optimize simple callouts, the

client has little control or guarantee over the final

performance, and a minor change in the instrumentation

routine can have an order of magnitude impact in performance

if it prevents inlining.

For a real-world example, consider the PiPA memory profiler

and cache simulator:

Implemented as a DynamoRIO tool, PiPA improved the

performance of the Pin dcache tool by a factor of 3.27x.

Implemented in Pin the speedup was only 2.6x, due to the

inability to fully optimize the instrumentation.

As requested I have gathered updated numbers for the bbcount

comparison using the new Pin release 22117, which has

improved inlining support.  Several benchmarks such as

perlbmk, crafty, and eon improve noticeably.  The harmonic

means are now 226% (a 5% improvement from 231%), 233%, and

185%, respectively.

I've been using VMware in the typical classroom setting now for two years. I first starting using it to overcome the issue of how to teach a Linux class without having access to a server. Becuase of an HP grant that I was apart of, our college was awarded 22 TabletPC's. The Tablet's were first used in a Digital Electronics class, Telecommunications Class and a Calculus 1 class. What a great opportunity to now use the Tablet's with VMware.

At first the class was a bit hesitant, but when I told them they could take the Tablet's home for the entire semester, they said.. sign me up.

We installed VMwareserver along with a condensed virtual appliance of Fedora. What a great learning experience. A few minor issues arose, installing IIS.. but basically it went smoothly.

Now I have taken the procedure to my on-line class. So far it is working. Along with using Linux Labsim, I am having the students from around the country install VMware, the virtual appliance of Fedora, and a 4 disk version of Red Hat.

So far, about eight students have tried the task. I must say, there are more issues when you can't see the other person's computer. But so far I have been able to guide them through the process.

To continue the comparison between DynamoRIO and Pin, below I show the memory usage of the two on the same benchmark suite, with no tool:

The average additional working set (resident memory) beyond the native working set of the application is 14.6MB for Pin and 2.6MB for DynamoRIO.

Both Pin and DynamoRIO in these experiments have unlimited code caches, so that is not a factor.  A significant portion of Pin's usage probably comes from its per-indirect-branch hashtables and function cloning.

DynamoRIO by default reserves 128MB of address space up front.  This is to ensure it does not run out of memory when targeting applications like MS SqlServer or Exchange that like to gobble up all available address space.  However, on Linux, DynamoRIO's reserved address space also commits swap space (one area where Linux's features are not as rich as Windows, even with MAP_NORESERVE).  For measuring virtual memory usage, I disabled this (which future versions of DynamoRIO may do by default, at least for 32-bit where reachability is not a concern) with the runtime option "-no_vm_reserve".  (Note that this undocumented option should not be used with the 9601 build on multithreaded Linux applications.)

It does not look like Pin is also reserving up front, given that it uses little memory on tiny test programs.

DynamoRIO's primary data structures have been compressed to save as much space as possible without sacrificing performance.  DynamoRIO's -opt_memory runtime option further reduces data structure usage significantly by eliminating all per-basic-block data structures.  However, we have not yet integrated -opt_memory with a performance optimization that also happens to save memory.  That means that in this 9601 release, on applications with little code (which includes most of SPECCPU2000), -opt_memory's savings are minimal since they disable that performance optimization.

I just got back from the second annual VMAP workshop hosted at VMware's annual user conference, VMworld in Las Vegas. The user conference boasted over 14,000 attendees - which included our members, student posters and university representatives using virtualization to support complex and often diverse campus infrastructures. I think the VMAP workshop attendees would agree that the conference was "eye opening" to say the least! The countless number of vendors developing software solutions to leverage virtualization was staggering. Be on the lookout for VMAP member blogs talking about their experiences from the conference. We'll also send a link out to the poster abstracts and presentation materials in the next couple of weeks.

Meanwhile, a few thoughts of mine about this year's event...

The student posters got first-hand experience explaining their scientific approach to complex problems to industry participants at VMworld's opening reception on Monday night. I enjoyed listening to our rising stars in academia trying to distill complicated problems into a language that the non-scientific, yet technical, observers can understand. Several industry attendees at the conference told me that they were pleased to see students able to present their ideas in a way that allowed them to appreciate how hard some of these problems are to solve.

At this year's full-day workshop, we had faculty from Georgia Tech, Columbia University, Northeastern, PolyTechnic/NYU, Cambridge University, George Mason, and University of Toronto. Carnegie Mellon, MIT and Olin College of Engineering were also represented by our student posters. Industry researchers from VMware and AMD were also present.

After lunch, which included a closer look at the student posters, we had three terrific speakers. We invited each speaker based on their varied use of virtualization at their respective schools. Ada Gavrilovska of Georgia Institute of Technology discussed her work on managed virtualized platforms. This work began using Xen and is now using VMware's ESX source code as a proof of concept to validate performance statistics of power management techniques. Discussion after her talk highlighted the question of cloud vs. grid and how large clusters should be leveraged for research and computation.

Jeffrey Dwoskin of Princeton University, talked with the group about his use of virtualization to test his security architecture research. As Jeff explained, virtualization software resolves the need to reproduce hardware environments to test the architecture he is developing.

Jason Nieh of Columbia University came from a completely different perspective - using and teaching virtualization in computer science. Jason painted an interesting picture of how Columbia has increased enrollment in the operating systems course over the past nine years and as a result has produced several interesting research projects/papers from students much earlier than typical in their academic careers. He attributes virtualization as a means to get more hands on experience and appreciation for systems. Students are not bogged down with crashing machines when doing kernel hacking and they don't have to worry about complexities of different operating systems. Jason also talked about the success of Columbia's first virtualization course offered this past spring. A complete set of the materials from this course can be found in govirtual.org under courseware.

Overall, I'd have to say that I am quite pleased with how our second annual event turned out and I'll share some additional thoughts on discussions from the workshop in my next blog. Also, I want to extend a special thanks to Melissa Wood, VMAP Program Manager extraordinaire, for her tireless work pulling together the poster session and workshop. For those who didn't know, she also helped me coordinate the attendance and schedules of over 200 engineers from VMware for the VMworld conference. Hats off to you Mel!

Security and virtualization often do not come hand-in-hand; merely running a virtualized environment does not automatically provide a guarantee of increased security over dedicated hardware.

As we have mentioned before, even the basic isolation properties of a VM framework are questionable. Relying on a piece of software to enforce isolation on the x86 platform is risky; it is entirely unclear whether a VMM is going to get that sort of job (a job that OS designers have been grappling with for years) right, especially in the absence of hardware primitives that would make things a lot easier (this argument is the essence of our recent VMSec paper).

This KernelTrap thread captures a vigorous discussion of this exact point.

From a certain point of view, the amalgam of virtualization technology and information security techniques represents a rather strange blend. What does emulation or multiplexing of physical devices have to do with the enforcement of a wide variety of security properties?

The easiest answer (and the most traditional one) is that virtualization provides an effective means of isolating execution environments; virtualization seems like a natural way to provide isolation between execution containers. As we've seen previously, however, customizing the communication between such containers --- in many situations, they do need to communicate: complete isolation is the exception rather than the rule --- presents a challenge. Thus, even the "obvious" security application of virtualization is fraught with difficulty.

As organizations increase their adoption of virtualization environments, and with the current industry focus on information security, it is natural to wonder just how a virtualization framework might pull double duty by improving a security posture as well as easing management burden and infrastructure costs.

Besides isolation, virtualization frameworks seem to provide a natural place to implement a reference monitor: a formal, well-defined security construct. A reference monitor provides a low-complexity, trusted (and trustworthy) environment from which to observe the execution of another system and measure a certain set of security-related properties.

Unfortunately, it appears that little thought has been given to what the best way is to combine the twin roles of resource provider and reference monitor within a single virtualization framework. As a result, virtualization environments can find themselves attempting to measure security-relevant properties of a system in
ways that are both creative and convoluted. In essence, the set of events that are interesting from a security viewpoint (and this depends on what type of "security" you're interested in measuring...from integrity of control flow or data items to information flow to authorization and access control) are not necessarily the set of events that the virtualization framework was built to intercept and observe with a minimal performance impact.

Karger and Safford's article in the upcoming issue of IEEE S&P magazine details the I/O complexities of most of the popular approaches to providing virtualization. I and my colleagues Bratus, Ramaswamy, and Smith have a paper at the upcoming VMSec workshop (held with ACM CCS 2008) identifying the problem of designing an efficient event trapping system of use for both security policy enforcement and virtualization.

While the suggestion that the design of current virtualization solutions is actually a hindrance to providing security solutions may not sit well with folks interested in touting a particular virtualization solution's security capabilities, I would argue that we have a unique opportunity to make sure that VM platforms are designed to do the things we're asking them to do. Now is also a good time to note that the stunning complexity of VMM I/O subsystems, the performance hacks therein, and the backdoor management interface all suggest that even the basic isolation story rests on somewhat shaky ground.

We find ourselves at a unique point in time: we can try to identify the right design for doing these two disparate tasks at once, or we can muddle through by abusing a framework meant for resource multiplexing rather than program supervision. In either case, we still must balance the tradeoff between the virtualization framework's I/O architecture and subsystems and the trustworthiness of the reference monitor. Ironically, as we depend on VM frameworks to implement more security functionality, these systems become less trustworthy even as they become more trusted.