CSC 469F | Assignment 0 | Fall 2006

Extended to: 5:00 p.m., October 10, 2006.

1. Introduction
2. Effect of Hardware
3. Effect of OS Structure
4. Submission Instructions
5. References

GOAL: To analyze the impact of architectural changes and operating system structure on performance.

In 1989, John Ousterhout, a professor at UC Berkeley, published a technical report titled "Why Aren't Operating Systems Getting Faster as Fast as Hardware?"[1]. The abstract of this paper reads:

This note evaluates several hardware platforms and operating systems using a set of benchmarks that test memory bandwidth and various operating system features such as kernel entry/exit and file systems. The overall conclusion is that operating system performance does not seem to be improving at the same rate as the base speed of the underlying hardware.

In this assignment, you will explore similar questions using the lmbench microbenchmark

Before beginning the assignment, make sure you have read the Ousterhout paper and the lmbench[2] paper.

The source distribution for lmbench is available on CDF at /u/csc469h/fall/pub/lmbench-3.0-a7.tgz. Unpack and build it in your own directory (you will need about 5MB of space).

The man pages for lmbench can be found online at http://lmbench.sourceforge.net/man/index.html. Section 3 describes the internals of lmbench, while Section 8 describes the individual microbenchmarks.

When you configure a benchmark run, you will be asked for the amount of memory to use. Larger amounts take more time, but should be more accurate. Experiment and try to find the smallest amount that gives you stable results. As a starting point, 20 MB should be more than enough to blow out any cache, yet should still be available on even the weakest machine in the test set. You will also be asked if you want to mail in your results. Say no -- you'll be doing a lot of tests and there is no need to mail them all in. At the end of the assignment, we can mail in a single set of results, if there is interest.

The CDF computing system provides an interesting benchmarking environment -- an identical software setup can be found installed on several generations of Intel x86 processors. This will allow us to observe how changing hardware affects OS performance. The following table gives some rough information about the machines available for benchmarking:

Choose at least 4 of the available architectures (feel free to include results for more, if you like). Make sure you include at least one of the < 1GHz Pentium 3 systems, and the Intel Celeron system. The more variety you have, the more interesting your results are likely to be. Tread lightly on the main compute servers - you will not get good results if they are heavily-loaded, but you should be able to find times (like early morning) when they are not busy. Also be aware that the OS is not identical to the workstations -- the servers are running the SMP (symmetric multiprocessor) version of the same kernel.

When you configure a benchmark run, select option 1 (Allow scheduler to place jobs) for the "Job placement selection" configuration (job placement gives control over how processes are scheduled to CPUs on multiprocessors). There are 2 reasons for this. First, letting the scheduler choose shows what would happen to real, ordinary processes for a given OS. Second, the version of Linux installed on CDF doesn't support the scheduler control system calls anyway, and you'll just get a lot of error messages in your output.

1. Changes in hardware

Run the lmbench "HARDWARE" tests on each target system. Plot the changes in CPU clock speed, and memory and disk bandwidth and latency as a function of time. For the time scale, use a rough estimate of when each of the CPUs was first introduced to the market (the Intel website and other internet resources should be able to help). Discuss the performance trends that you see.

2. Predicting performance

Run the lmbench "OS" tests on your slowest/oldest target system. Using the data for these OS tests, and the hardware performance data you collected, predict results for each of the other (newer, faster) target systems. Explain your methodology. What piece of the underlying hardware is most likely to affect the performance of each of the OS tests?

3. Validating predictions

Run the lmbench "OS" tests on your remaining target systems. Plot the results, together with your predicted performance. Is the Linux operating system getting faster as fast as hardware? Speculate on causes for the results that you see.

In this part of the assignment, you will explore whether different operating systems executing on identical hardware show noticeable performance differences at the microbenchmark level.

We are setting up two comparison systems on the Systems research network. You will be given instructions on how to access them in class on Tuesday.

In the first comparison, Mac OS X will face off against Yellowdog Linux, both running on an Apple XServe. Collect lmbench performance results for both systems as in the first part, and discuss your findings.

UPDATE: As discussed in class, you do not need to compare OS X (aka Darwin) against Yellowdog Linux due to problems creating your user accounts. You should still include the Yellowdog Linux results and discuss whether there seem to be any differences due to the PowerPC hardware vs. Intel hardware (beyond the clock speed differences). If you are interested, I have generated results for the OS X system which you can use, but it is not required. The first three runs (.0, .1, and .2) use a 128 MB range of memory, the second three runs (.3, .4, and .5) use a 32 MB range.

In the second comparison, Linux and BSD variants will face off on 64-bit dual-processor Xeon systems. You will also have the opportunity to compare performance on native vs. virtualized hardware. Once again, you should collect data using lmbench, and plot and discuss the results.

UPDATE: You do not need to repeat all the hardware tests on each of the virtualized platforms. The underlying hardware on bench-intel-1 to bench-intel-4 is the same as bench-intel-0. Comparing the hardware performance on bench-intel-0 (Linux 2.6 native) and bench-intel-1 (Linux 2.6 virtual) is sufficient to discuss apparent hardware performance on a virtualized platform.

What

Write a technical report containing the results of your experiments and your discussions. Your report should be well-organized and easy to read -- you should have an introduction, a description of what you are trying to do and why (preferably other than "for the marks!"), a description of your experimental methodology, your results, and a discussion. Include tables describing your hardware platforms (include all relevant details - CPU family and speed, cache size, memory size, disk parameters, etc.). Present your benchmark results in both tabular and graphical form (for those who want to see the detailed results, and those who want to spot trends in the data). Include as an Appendix your lmbench configuration files used for each experiment.

Create a compressed tar file (tar -czf a0.tgz ) of the benchmark output (the raw output files in the lmbench "results" directories).

How

Submit hardcopy of your report using the assignment drop box in BA 2220 (The box is labelled CSC469, and is located on the lower right side of the array of boxes).

Submit your data using 'submit -c csc469h -a A0 a0.tgz' on CDF.

1. Why aren't operating systems getting faster as fast as hardware?
   J. Ousterhout, WRL Technical Note TN-11, digital Western Research Laboratory, October 1989, 10 pages. (also published in the 1990 Usenix Summer Technical Conference)
2. lmbench: an extensible micro-benchmark suite
   C. Staelin, Software--Practice and Experience, vol. 35, no. 11, May 2005, pp. 1079-1105.