Talk:Hypervisor

From Wikipedia, the free encyclopedia

You have new messages (last change).

Contents

[edit] Relationship with microkernels

This article and the microkernel article ought to say more about each other's subjects. IBM's VM and Xen are really microkernels. (Not sure about Parallels Workstation). Also, what Xen calls "paravirtualization", Parallels Workstation calls a "hypercall", and IBM VM calls a "DIAGNOSE code" are all really the same thing, a system call to the operating system below. Nagle (talk • contribs)

[edit] History paragraph 2: IBM System/360

The S/360 was introduced in 1964, not "the late 1960s." It had "supervisor" and "problem" states, storage-key protection and interrupt handling, but not memory management (i.e. dynamic address translation, or DAT). Multitasking was possible from the beginning with S/360, but not full virtualization.

The S/360 Model 67, introduced in 1966, included DAT for the first time, making VM possible.

The 2nd paragraph should be specific about the 360/67, or go for a broader stroke and just talk about the S/370 architecture (intro'd in 1970), which had all of the necessary features for virtualization from the get-go.

[edit] Open-source ???

I don't think VM/CMS was ever open-source, as stated in the article. Whoever wrote this, care to elaborate? —The preceding unsigned comment was added by JFG (talk • contribs) 09:36, 27 May 2006.

The comment was added by Linas at 15:10 on 29 July 2005, and Linas knows whereof he speaks. The early releases of VM/CMS, known as VM/370 at the time, were in the public domain as per the famous IBM consent decree, which held sway until the Regan-era Dept. of Justice. How much more "open souce" can you get than public domain? On top of that, if you dig into the VM history (see especially Melinda Varian's excellant "VM and the Community" paper), you'll find that many of the important facilities in CP and CMS were designed, built, and shared around in the VM user community. That was true even long after the VM source code was resticted to licensees only. RossPatterson 15:39, 27 May 2006 (UTC)
Linas obviously knows much – as do you, Ross – but I think these details aren't quite correct. Let me have a go at summing up what I believe to be the case.
The first releases of VM/370 were not public domain, but were an IBM-proprietary reimplementation of CP/CMS. CP/CMS was distributed as IBM Type-III software, which was not public domain but was distributed to IBM customers at no cost and with no support. (This was before the famous IBM software unbundling event of 1969.) For all practical purposes it was in the public domain. I have been unable to find legal details on the dividing line.
I don't think IBM's 1956 Consent Decree had anything to do with placing software in the public domain. It did create restrictions on IBM's ability to sell services, and segrated IBM service organizations. The Justice Department continued with antitrust investigations until 1982 (some details are at www.hagley.lib.de.us).
I expect that what both you and Linas are referring to is the peculiar status enjoyed by CP/CMS in its early days as Type-III software, after its submittal by MIT's Lincoln Laboratory to the library – in what Dave Tuttle characterizes as clever "collusion" to work around the IBM politics that had dismissed time-sharing in preference to batch processing as the way of the future. As Type-III software, it could be used freely by any IBM customer, and quickly spread a time-sharing culture through commercial and academic sites, and spawned at least two notable time-sharing vendors (National CSS and IDC) using the OS as a basis for reselling computer time. (NCSS transformed the CP/CMS base so much that it became a new proprietary OS, VP/CSS, which continued to be developed independently of the VM world until the mid 80s. It outperformed contemporary VM systems by a factor of perhaps two to one.)
It has been asserted (and seems likely) that the growth of the time-sharing industry and the VM user community was a major factor in IBM's 1972 addition of virtual memory to the S/370 product line – something that had originally and contentiously been omitted. With the announcement of virtual storage, IBM also announced VM/370 as a product, which Dave Tuttle describes as involving a complete reimplementation of CP-67. I have been unable to determine whether there was a period when VM/370 was Type-III software, but I don't think so; the history of IBM product announcements makes it look like a normal product.
As I believe you know, I am working on a revision of these various articles (VM/CMS, S/360-67, etc.), to try to get the historical timeline and details correct, and especially to get them cited back to good references, since there is a lot of folklore about the events. You are correct in calling attention to Varian's excellent SHARE paper. I have been speaking with her about trying to get it placed on WikiSource or some other permanent home, instead of leaving it on her Princeton personal webpage. (She retired last year.) Please do pass along any good historical resources that you run across for me to include. I'm especially interested in "reputable" citations, i.e. books and academic journals. Unfortunately most of the material of interest is from just before the onset of on-line documents, making it hard to track down article text from, say, the early 70s. Fortunately IBM posts its research journal text back to the 50s. Trevor Hanson 18:57, 12 October 2006 (UTC)

[edit] Dave Tuttle speaks

Trevor, I happened upon this just today. I do have some knowledge of the subject, but little experience with Wikipedia. -- Dave Tuttle, http://home.comcast.net/~Viewpt 71.248.178.228 22:29, 12 October 2006 (UTC)

Terminology first: The original S/360-era virtual machine system was CP-40, a research project developed at the IBM Cambridge Scientific Center (CSC), based on an experimental, modified S/360 Model 40. Next came CP-67/CMS, an adaptation of CP-40 for the S/360 Model 67, developed at CSC with active collaboration from MIT Lincoln Labs and one or two other outside parties. CP-67/CMS version 3 was the base release contributed to the Type III Library in June, 1969. After the June 23, 1969 unbundling, the CP-67/CMS Development Group split off from the CSC. VM/370 (Virtual Machine Facility/370) was developed by that group, as an official IBM System Control Program (SCP) release that was announced on August 2, 1972 along with all of the System/370 "VS" (Virtual Storage) hardware and software.

Note that before the June 23, 1969 "unbundling", all IBM software was freely available to IBM customers - distributed and supported primarily in binary form. All of the major elements of CP-67/CMS and the first 'N' releases of VM/370 were distributed in full source form, along with the tools to (re)build the kernels and many of the command-line programs. Language processors and some other user-mode programs were distributed in binary form, since they were "borrowed" from the OS/360 software. Updates were provided as source patches, to be applied or not at the cutomer's choice. Problem reports (APAR's - Authorized Program Analysis Report) were often received with the fix provided, in source form; one variety of APAR was an Enhancement Request. A large number of fixes and extensions were originated "in the field" and in other parts of IBM, then verified and redistributed on a subsequent PLC (Program Level Change) tape. If that does not qualify as "Open Source", I don't know what would. It predated the GPL by several decades.

According to the terms of the 1969 unbundling, System Control Program (SCP) software was available without additional charge to any IBM customer. System Product Program (SPP) software was extra-cost, ordered separately, and supported for a fee. The source restrictions and charging for operating system software came about much later.

BTW, the term "hypervisor" was most likely coined by me in late 1971, during CP development for VM/370. We had to pick new source filenames and external symbol names for all of the CP functions, to conform to IBM SCP nomenclature. One of the modules I wrote was the handler for the Diagnose instruction, and I called it DMKHVC, for Hypervisor Call. Dave Tuttle 00:56, 13 October 2006 (UTC)

Hokey smokes, yes you do indeed have "some knowledge" of the subject! Melinda Varian said I should look to you for input; but I hadn't expected you to pop up here, like a genie, in answer to my wish. What a nice surprise. You have provided much good detail here, for which I am grateful. And of course your past input elsewhere, to people like Varian and Tom Van Vleck, has been invaluable.
As I said above, I'm trying to pull together a coherent series of articles about CP-40, CP-67, CSC, Lincoln Labs, etc. that put this important period in the correct light, and hopefully assemble correct facts backed up by good citations. A couple of Wikians have jumped in as well, so I have high hopes for achieving good coverage within a couple of months. The text is currently in private 'sandboxes' while I am pulling things together, but I hope to start pushing material out shortly. (I would also like to lift some of your comments verbatim, and plunk them in.) Much source material is not easily available on-line, and some first- and second-person memoirs have some factual details wrong, so this has been an interesting project.
I would greatly appreciate the chance to pick your brain as questions develop. Perhaps email would be more convenient? Anyway, perhaps I can begin with four questions:
  1. One comment you were quoted as having made (on Tom Van Vleck's The IBM 360/67 and CP/CMS page) was that "VM/370-CP was a complete redesign and new implementation." I was surprised to read that, because I had thought a good deal of CP-67 code remained in VM/370. When you say "new implementation" can you characterize the amount of actual new source code? 100%? 75%? I realize that your "carried over significant portions of CMS" but I want to be clear on the CP situation.
  2. An old colleague of mine, who was at MIT in 'the day,' maintains: a) that CP/CMS development inadvertently received some MIT or DoD or public funding; b) that this clouded the ownership of the source code; and c) that this was one reason CSC had Lincoln Lab submit the system for Type-III listing (in addition to what you describe as friendly 'collusion' to evade IBM politics over time-sharing). I'm sure my friend tells the folklore correctly, though that doesn't make it true. I can't find any other mention of this issue, but it is plausible. Perhaps MIT contracts with CSC were used to pay some IBM headcount? Or perhaps there were tangible project contributions made by non-IBM personnel? Anyway, does the topic of 'CP/CMS as public domain' ring any bells?
  3. I find two conflicting descriptions of Type-III software: "customer-submitted" and "IBM employee submitted." My recollection is fuzzy; the key practical issue was of course that it was unsupported, "as-is" software, but I seem to recall seeing contributions from the Labs and from IBM SEs, as well as from customers. I can't seem to find any definitive IBM documents about the Type-III library, nor any decent list of major Type-III software through the years (I remember HASP, CP/CMS, the JOVIAL compiler, and I think a version of SCRIPT, but I know there were lots of others). Any thoughts about where to turn for more about the history of the Type-III library?
  4. I know that the unbundling event was pivotal in all of this, but unfortunately I have forgotten most of the details I used to know about it. (It was just a little before my time, though I certainly remember it and its consequences.) Do you know of any good source material that could clarify what it meant in terms of the CP/CMS-to-VM/370 timeline? In fact, source material on all these topics would be a boon. I am especially fascinated by CP-40, which must have been the most amazing project, really breaking new ground.
I am very grateful that you have weighed in here. This will make my job easier. By the way, let me encourage you to submit any substantive papers or memoirs to WikiSource so that they'll have a permanent home. If you do, kindly let me know so I can cite them in the appropriate places. I have asked Melinda to consider doing this with her excellent SHARE paper.
BTW I was at National CSS in the 70s, where of course I knew Mike Field, Bob Jesurum, Dick Bayles, and other familiar CSC/MIT names.

Thanks again - Trevor Hanson 20:47, 13 October 2006 (UTC)

(I've been fighting with the formatting codes; still a newbie vis-a-vis Wiki markup... Dave Tuttle)

  1. CP-67 vs VM-CP - If you really want a percentage, it's most likely in the range of 98-99% "new" code. The basic functions were largely dictated by the S/360 and S/370 hardware architecture, but conforming to the SDD standards for SCP development meant that we had to revisit and update every line of code. Also remember that CP-40 was a research project for a unique piece of hardware; the S/360-67 had a different DAT architecture, and the S/370-AF machines were even more different. The VM/370-CP was designed to be a production operating system with a full collection of error handling, exception recovery, logging, and serviceability features that were never present in the earlier software. CP-40 supported one machine, period, with a fixed maximum number of virtual machines. CP-67 (as released) supported one model with known characteristics. VM/370 was required to support a wide range of machine speeds, configurations, and model-dependent characterisitics. The decision to start over with a new CP structure was both deliberate and well-advised.
  2. CP-67 Code Status - Mostly nobody cared about "ownership" of software in today's (litigious, often silly) terms. Software patents did not exist and Intellectual Property lawsuits were not part of the software landscape. All that aside, it was before the days of standardized data exchange formats and common interfaces. If you owned an IBM computer, you could get software for it. If you didn't own one, the software for it was of limited use. If you were a member of a computer-related organization (AFIPS, SHARE, GUIDE, ACM, etc.), you had access to a large collection of software -- in machine-specific formats. There may have been a standard copyright notice in the CP-67 code, but I doubt it. Software source was either handled as Trade Secret (undisclosed) or handled as any other academic / professional publication. Citations and attribution were expected, but there were no legal barriers to use, reuse, or derivative works. Significant portions of CP-67 and CMS, over the years, were contributed by other groups, both within IBM and outside.
  3. The Type III Library - When I started corresponding with Melinda back in the 1980's, I didn't have the Internet as an easy reference and source of reminders. Most likely the distinction between Type I, Type II, and Type III programs was in the level of warranty and support provided by IBM. Type I was fully warranteed and supported (e.g. OS/360); Type II was an intermediate level, perhaps for packages such as ASP (Attached Support Processor); Type III was clearly caveat emptor material, to be used on your own recognizance. In the same timeframe, both SHARE and GUIDE maintained large libraries of software that was freely available to members, supported (or not) by the authors or by the SHARE/GUIDE project. I do know for a fact that the CP-67 code submitted to the Type III library in June of 1969 was not submitted directly from the CP-67/CMS support group at CSC; there may have been an internal cutoff date (for "employee" contributions) that was earlier than the external cutoff (for "customer" contributions). I was only 20 at the time and had been working at CSC for only 9 months, so I was not personally involved in the details -- but Dick Meyer and Love Seawright certainly were involved; I heard the news from them.
  4. Events of June 1969 - Melinda Varian's paper has some information about it, I am sure; the earlier versions (1989 and 1991) are longer than some of the later ones. From the inside, the unbundling event was initially viewed as the begining of a death sentence for the Scientific Centers and the open collaboration with universities which had been the modus operandi for the preceding years. I know that it stifled wider release of some of my own work, and that led fairly directly to my decision to 'jump ship' from the CSC to the CP-67 group. The key element was a recategorization of all IBM software into either SCP (System Control Programming: essential to the normal operation of the hardware and, thus, still available at no additional charge) or PP (Program Product: non-essential, business-related, for a fee, subject to competition from other suppliers, etc.) The great Microsoft Web-browser debate was a replay of the same generic issue. The difficulty was the lack of a third category; internally, IBM did not have a mechanism for determining what should or could be a Program Product, nor did it have an established set of lighter-weight procedures for smaller scale projects. Many useful things, such as Script/370 Version 2, ended up as 'IBM Internal Use Only' because there was no mechanism to make them generally available.

Dave Tuttle 21:04, 15 October 2006 (UTC)

[edit] References on Early Hypervisor Work

The 1989 and 1991 versions (which I have in hardcopy and PDF, respectively) of Melinda Varian's paper, VM and the VM Community: Past, Present, and Future, contain a wealth of references to original material, but finding extant online sources is more difficult. The IBM Systems Journal is a valuable resource, but all of the earlier articles are scanned images in PDF form. It is time-consuming and somewhat error-prone to transcribe the citations. These few are selected items from an article published in 1972, one of the best bibliographies:

R. P. Parmelee, T. I. Peterson, C. C. Tillman, and D. J. Hatfield, "Virtual storage and virtual machine concepts," IBM Systems Journal, vol. 11, no. 2 (June 1972)

47. R. J. Adair, R. U. Bayles, L. W. Comeau, and R. J. Creasy, A Virtual Machine System for the 360/40, IBM Corporation, Cambridge Scientific Center, Report No. 320-2007 (May 1966).

One of the first papers describing the implementation of the virtual machine concept. An IBM System/360 Model 40 was modified with an associative memory to provide dynamic translation of addresses, and a control program was developed to allocate resources of the host machine to the virtual machines. Multiprogramming and multiprocessing tasks could then be studied in terms of machine utilization.

53. CP-67/CMS, Program 360D-05.2.005, International Business Machines Corporation, Program Information Department, Hawthorne, New York (June 1969).
68. R. A. Meyer and L. H. Seawright, "A virtual machine time-sharing system," IBM Systems Journal, vol. 9, no. 3, pp199-218 (Septmber 1970).

Describes the CP-67/CMS system, outlining features and applications.

70. D. Sayre, On Virtual Systems, IBM Thomas J. Watson Research Center, Yorktown Heights, New York (April 15, 1966).

An early paper emphasizing the multiprogramming aspects of virtual machines. Results are shown for multiprogramming with the IBM M44/44X system.

83. R. W. O'Neill, "Experience using a time-shared multiprogramming system with dynamic address relocation hardware," AFIPS Conference Proceedings, Spring Joint Computer Conference 30, pp611-621 (1967).

Describes the experimental IBM M44/44X system, reporting performance measurements taken to establish dependence of core requirements on page size and running time on core size (parachor curve). Also discussed are effects of multiprogramming and time-sharing on performance.

Dave Tuttle 17:25, 15 October 2006 (UTC)

Thanks for more useful input. I have been mining the old scanned IBM journals, which I discovered just a couple of weeks ago. There are many good sources there – scanned images are far better than nothing. (I wish I still had all the old AFIPS Proceedings that used to sit on my shelf; their space was reclaimed many years ago, along with old IBM manuals. I still have a TSS manual though.) Any other input you care to provide will be gratefully received, particularly about the transition from CP/CMS to VM/370, and the folklore about CP's public funding/public domain status. Trevor Hanson 18:10, 15 October 2006 (UTC)
As time permits, I will try to fill in some of the blanks. I happen to have had personal experience with the Yorktown M44/44X system as a terminal user, during a 1966 summer job (Research Department, Tunnel and Bridges Division, Port of New York Authority). My unofficial and (later) sanctioned work at the MIT Computation Center, 1966 to 1968, gave me hands-on experience with CTSS, IBSYS, the SHARE library, OS/360-PCP, and ASP v1. Much more recently (1999) I started a job at NorthStar Internetworking and discovered VMware v1.0 running on my Linux desktop, but I never paid much attention to the internals of it (old hat to me, though they had to solve some problems with the Intel-based PC architecture). From 2000 to 2003 I had the opportunity to do a lot of work with ExOS and XOK, from the MIT ExoKernel project. I also spent most of a year at Katana Technology (now Virtual Iron Software) working on a different flavor of virtualization. Dave Tuttle 22:07, 15 October 2006 (UTC)

[edit] Mainframe origins

The following statement really wasn't right, so I have replaced it:

Hypervisors were originally developed in the early 1970s, when cost reduction was forcing multiple scattered departmental computers to be consolidated into a single, larger computer — the mainframe — that would serve multiple departments.

Scattered departmental computers as suggested by this statement really only emerged in the 80s. The mainframes of the late 60s and early 70s were a logical progression from the large machines of the previous generation, e.g. IBM 7xx/7xxx. Computer expenditures during this period were not subject to cost reductions, but cost expansions – as new functions from throughout enterprises were being automated. If you disagree, please cite examples. Trevor Hanson 20:46, 26 October 2006 (UTC)

Great job of editing! Thanks. As discussed in the early IBM Research and IBM CSC reports, development of the virtual machine approach was very much a part of the broader time-sharing trend. The underlying S/360 architecture was primarily designed for User / Supervisor protection for multiprogramming and multitasking; the fact that it was completely virtualizable was a natural but perhaps unintended consequence. Remember also that the IBM System/360 family of processors were among the first microcode-controlled computers. The concept of an invalid instruction did not exist in the IBM 7xx/7xxx series; instruction decode was hard-wired, leading to 7 or 8 editions of the IBM 70x0 Principles of Operation manual as new, useful instructions were discovered and characterized. Dave Tuttle 20:41, 28 October 2006 (UTC)

I wasn't happy with the characterization of the -67 as having been built for virtualization; that was obviously a fortuitous outcome, due to a number of very smart, thoughtful people who were in the right place at the right time. (It's amazing how many times I read conflated descriptions of the -67, CP/CMS, and VM, with no reference to such important topics as TSS, CP-40, or MIT.) However, I wasn't trying to rewrite the section – just to address some fact-related issues. It's easy to start to fix something that seems minor, but to wind up opening a can of worms. (This happened with me on the S/370 article recently.) Eventually, we should be able to get all these related articles in synch, and accurate. (It is exciting to think that this same process is going on in ten thousand separate WikiPedia threads – in twenty years, if flame wars and text vandalism don't kill such efforts, we could really have an Encylopedia Galactica of high accuracy. Trevor Hanson 09:04, 30 October 2006 (UTC)
By the way, if you are inclined, you should leap in and correct the text. The WikiPedia advice to editors is: Be Bold. Some of the comments you made above would be good additions to the article itself, rather than just here in the discussion. Trevor Hanson 09:12, 30 October 2006 (UTC)
Retrieved from "http://en.wikipedia.org../../../h/y/p/Talk%7EHypervisor_12ae.html"