TeX

TeX

Developer(s)	Donald Knuth
Initial release	1978 (1978)
Stable release	3.1415926 / March 2008 (2008-03)
Development status	Maintained
Written in	Originally WEB
Operating system	Cross-platform
Type	Typesetting
License	Permissive
Website	www.tug.org

TeX (pron.: / ˈ t ɛ x / as in Greek, but often pronounced / ˈ t ɛ k / in English) is a typesetting system designed and mostly written by Donald Knuth and released in 1978. Within the typesetting system, its name is formatted as TeX.

Together with the Metafont language for font description and the Computer Modern family of typefaces, TeX was designed with two main goals in mind: to allow anybody to produce high-quality books using a reasonably minimal amount of effort, and to provide a system that would give exactly the same results on all computers, now and in the future.

TeX is a popular means by which to typeset complex mathematical formulae; it has been noted as one of the most sophisticated digital typographical systems in the world. TeX is popular in academia, especially in mathematics, computer science, economics, engineering, physics, statistics, and quantitative psychology. It has largely displaced Unix troff, the other favored formatter, in many Unix installations, which use both for different purposes. It is now also being used for many other typesetting tasks, especially in the form of LaTeX and other template packages.

The widely used MIME type for TeX is application/x-tex. TeX is free software.

History

When the first volume of Donald Knuth's The Art of Computer Programming was published in 1969, it was typeset using hot metal type set by a Monotype Corporation typecaster with a hot metal typesetting machine from the 19th century which produced a "good classic style" appreciated by Knuth. When the second edition of the second volume was published, in 1976, the whole book had to be typeset again because the Monotype technology had been largely replaced by photographic techniques, and the original fonts were no longer available. However, when Knuth received the galley proofs of the new book on 30 March 1977, he found them awful. Around that time, Knuth saw for the first time the output of a high-quality digital typesetting system, and became interested in digital typography. The disappointing galley proofs gave him the final motivation to solve the problem at hand once and for all by designing his own typesetting system. On 13 May 1977, he wrote a memo to himself describing the basic features of TeX.

He planned to finish it on his sabbatical in 1978, but as it happened the language was not frozen until 1989, more than ten years later. Guy Steele happened to be at Stanford during the summer of 1978, when Knuth was developing his first version of TeX. When Steele returned to MIT that autumn, he rewrote TeX's I/O to run under the ITS operating system. The first version of TeX was written in the SAIL programming language to run on a PDP-10 under Stanford's WAITS operating system. For later versions of TeX, Knuth invented the concept of literate programming, a way of producing compilable source code and cross-linked documentation typeset in TeX from the same original file. The language used is called WEB and produces programs in DEC PDP-10 Pascal.

A new version of TeX, rewritten from scratch and called TeX82, was published in 1982. Among other changes, the original hyphenation algorithm was replaced by a new algorithm written by Frank Liang. TeX82 also uses fixed-point arithmetic instead of floating-point, to ensure reproducibility of the results across different computer hardware, and includes a real, Turing-complete programming language, following intense lobbying by Guy Steele.

In 1989, Donald Knuth released new versions of TeX and METAFONT. Despite his desire to keep the program stable, Knuth realised that 128 different characters for the text input were not enough to accommodate foreign languages; the main change in version 3.0 of TeX is thus the ability to work with 8-bit inputs, allowing 256 different characters in the text input.

Since version 3, TeX has used an idiosyncratic version numbering system, where updates have been indicated by adding an extra digit at the end of the decimal, so that the version number asymptotically approaches π. This is a reflection of the fact that TeX is now very stable, and only minor updates are anticipated. The current version of TeX is 3.1415926; it was last updated in March 2008. The design was frozen after version 3.0, and no new feature or fundamental change will be added, so all newer versions will contain only bug fixes. Even though Donald Knuth himself has suggested a few areas in which TeX could have been improved, he indicated that he firmly believes that having an unchanged system that will produce the same output now and in the future is more important than introducing new features. For this reason, he has stated that the "absolutely final change (to be made after my death)" will be to change the version number to π, at which point all remaining bugs will become features. Likewise, versions of METAFONT after 2.0 asymptotically approach e, and a similar change will be applied after Knuth's death.

However, since the source code of TeX is essentially in the public domain (see below), other programmers are allowed (and explicitly encouraged) to improve the system, but are required to use another name to distribute the modified TeX, meaning that the source code can still evolve. For example, the Omega project was developed after 1991, primarily to enhance TeX's multilingual typesetting abilities. Donald Knuth himself created "unofficial" modified versions, such as TeX-XeT, which allows a user to mix texts written in left-to-right and right-to-left writing systems in the same document.

Typesetting system

TeX commands commonly start with a backslash and are grouped with curly braces. However, almost all of TeX's syntactic properties can be changed on the fly which makes TeX input hard to parse by anything but TeX itself. TeX is a macro and token-based language: many commands, including most user-defined ones, are expanded on the fly until only unexpandable tokens remain which get executed. Expansion itself is practically side-effect free. Tail recursion of macros takes no memory, and if-then-else constructs are available. This makes TeX a Turing-complete language even at the expansion level.

The system can be divided into four levels: in the first, characters are read from the input file and assigned a category code (sometimes called "catcode", for short). Combinations of a backslash (really: any character of category zero) followed by letters (characters of category 11) or a single other character are replaced by a control sequence token. In this sense this stage is like lexical analysis, although it does not form numbers from digits. In the next stage, expandable control sequences (such as conditionals or defined macros) are replaced by their replacement text. The input for the third stage is then a stream of characters (including ones with special meaning) and unexpandable control sequences (typically assignments and visual commands). Here characters get assembled into a paragraph. TeX's paragraph breaking algorithm works by optimizing breakpoints over the whole paragraph. The fourth stage breaks the vertical list of lines and other material into pages.

The TeX system has precise knowledge of the sizes of all characters and symbols, and using this information, it computes the optimal arrangement of letters per line and lines per page. It then produces a DVI file ("DeVice Independent") containing the final locations of all characters. This dvi file can be printed directly given an appropriate printer driver, or it can be converted to other formats. Nowadays, PDFTeX is often used which bypasses DVI generation altogether.

The base TeX system understands about 300 commands, called primitives. However, these low-level commands are rarely used directly by users, and most functionality is provided by format files (predumped memory images of TeX after large macro collections have been loaded). Knuth's original default format, which adds about 600 commands, is Plain TeX. The most widely used format is LaTeX, originally developed by Leslie Lamport, which incorporates document styles for books, letters, slides, etc., and adds support for referencing and automatic numbering of sections and equations. Another widely used format, AMS-TeX, is produced by the American Mathematical Society, and provides many more user-friendly commands, which can be altered by journals to fit with their house style. Most of the features of AMS-TeX can be used in LaTeX by using the AMS "packages". This is then referred to as AMS-LaTeX. Other formats include ConTeXt, used primarily for desktop publishing and written mostly by Hans Hagen at Pragma.

How TeX is run

A sample page produced using TeX with the LaTeX macros.

A sample Hello world program in plain TeX is:

Hello, World
\bye          % marks the end of the file; not shown in the final output

This might be in a file myfile.tex, as .tex is a common file extension for plain TeX files.

By default, everything that follows a percent sign on a line is a comment, ignored by TeX. Running TeX on this file (for example, by typing tex myfile.tex in a command line interpreter, or by calling it from a graphical user interface) will create an output file called myfile.dvi, representing the content of the page in a device independent format ( DVI). A DVI file could then be either viewed on screen or converted to a suitable format for any of the various printers that a device driver existed for (printer support was generally not an operating system feature at the time that TeX was created).

Knuth has said that there is nothing inherent in TeX that requires DVI as the output format, and later versions of TeX, notably pdfTeX, XeTeX and LuaTeX, all support output directly to PDF.

Mathematical example

TeX provides a different text syntax specifically for mathematical formulae. For example, the Quadratic equation's formula appears as:

The quadratic formula is $-b \pm \sqrt{b^2 - 4ac} \over 2a$
\bye

and the output would resemble:

The formula is printed in a way a person would write by hand, or typeset the equation. In a document, entering mathematics mode is done by starting with a $ symbol, then entering a formula in TeX syntax, and closing again with another $ symbol. Knuth explained in jest that he chose the dollar sign to indicate the beginning and end of mathematical mode in plain TeX because typesetting mathematics was traditionally supposed to be expensive. Display mathematics (mathematics presented centered on a new line) is similar but uses $$ instead of a single $ symbol. For example, the above with the quadratic formula in display math:

The quadratic formula is $$-b \pm \sqrt{b^2 - 4ac} \over 2a$$
\bye

renders as

Novel aspects

The TeX software incorporates several aspects that were not available, or were of lower quality, in other typesetting programs at the time when TeX was released. Some of the innovations are based on interesting algorithms, and have led to several theses for Knuth's students. While some of these discoveries have now been incorporated into other typesetting programs, others, such as the rules for mathematical spacing, are still unique.

Mathematical spacing

Mathematical text typeset using TeX and the AMS Euler font.

Since the primary goal of the TeX language is high-quality typesetting for publishers of books, Knuth gave a lot of attention to the spacing rules for mathematical formulae. He took three bodies of work that he considered to be standards of excellence for mathematical typography: the books typeset by Addison-Wesley Publishing's house (the publisher of The Art of Computer Programming), in particular the work done by Hans Wolf and Joseph Louis Lagrange (thermodynamics innovation, +1856); editions of the mathematical journal Acta Mathematica dating from around 1910; and a copy of Indagationes Mathematicae, a Dutch mathematics journal. Knuth looked closely at these printed-papers to sort out and look for a set of rules for spacing calling it TeX. While TeX provides some basic rules and the tools needed to specify proper spacing, the exact parameters depend on the font used to typeset the formula. For example, the spacing for Knuth's Computer Modern fonts has been precisely fine-tuned over the years and is now set, but when other fonts, such as AMS Euler, were used by Knuth for the first time, new spacing parameters had to be defined.

Hyphenation and justification

In comparison with manual typesetting, the problem of justification is easy to solve with a digital system such as TeX, which, provided that good points for line breaking have been defined, can automatically spread the spaces between words to fill in the line. The problem is thus to find the set of breakpoints that will give the most pleasing result. Many line breaking algorithms use a first-fit approach, where the breakpoints for each line are determined one after the other, and no breakpoint is changed after it has been chosen. Such a system is not able to define a breakpoint depending on the effect that it will have on the following lines. In comparison, the total-fit line breaking algorithm used by TeX and developed by Donald Knuth and Michael Plass considers all the possible breakpoints in a paragraph, and finds the combination of line breaks that will produce the most globally pleasing arrangement.

Formally, the algorithm defines a value called badness associated with each possible line break; the badness is increased if the spaces on the line must stretch or shrink too much to make the line the correct width. Penalties are added if a breakpoint is particularly undesirable: for example, if a word must be hyphenated, if two lines in a row are hyphenated, or if a very loose line is immediately followed by a very tight line. The algorithm will then find the breakpoints that will minimize the sum of squares of the badness (including penalties) of the resulting lines. If the paragraph contains possible breakpoints, the number of situations that must be evaluated naively is . However, by using the method of dynamic programming, the complexity of the algorithm can be brought down to (see Big O notation). Further simplifications (for example, not testing extremely unlikely breakpoints such as a hyphenation in the first word of a paragraph) lead to an efficient algorithm whose running time is almost always of order . A similar algorithm is used to determine the best way to break paragraphs across two pages, in order to avoid widows or orphans (lines that appear alone on a page while the rest of the paragraph is on the following or preceding page). However, in general, a thesis by Michael Plass shows how the page breaking problem can be NP-complete because of the added complication of placing figures.

TeX's line breaking algorithm has been adopted by several other programs, such as Adobe InDesign (a desktop publishing application) and the GNU fmt Unix command line utility.

If no suitable line break can be found for a line, the system will try to hyphenate a word. The original version of TeX used a hyphenation algorithm based on a set of rules for the removal of prefixes and suffixes of words, and for deciding if it should insert a break between the two consonants in a pattern of the form vowel–consonant–consonant–vowel (which is possible most of the time). TeX82 introduced a new hyphenation algorithm, designed by Frank Liang in 1983, to assign priorities to breakpoints in letter groups. A list of hyphenation patterns is first generated automatically from a corpus of hyphenated words (a list of 50,000 words). If TeX must find the acceptable hyphenation positions in the word encyclopedia, for example, it will consider all the subwords of the extended word .encyclopedia., where . is a special marker to indicate the beginning or end of the word. The list of subwords include all the subwords of length 1 (., e, n, c, y, etc.), of length 2 (.e, en, nc, etc.), etc., up to the subword of length 14, which is the word itself, including the markers. TeX will then look into its list of hyphenation patterns, and find subwords for which it has calculated the desirability of hyphenation at each position. In the case of our word, 11 such patterns can be matched, namely ₁c₄l₄, ₁cy, ₁d₄i₃a, ₄edi, e₃dia, ₂i₁a, ope₅d, ₂p₂ed, ₃pedi, pedia₄, y₁c. For each position in the word, TeX will calculate the maximum value obtained among all matching patterns, yielding en₁cy₁c₄l₄o₃p₄e₅d₄i₃a₄. Finally, the acceptable positions are those indicated by an odd number, yielding the acceptable hyphenations en-cy-clo-pe-di-a. This system based on subwords allows the definition of very general patterns (such as ₂i₁a), with low indicative numbers (either odd or even), which can then be superseded by more specific patterns (such as ₁d₄i₃a) if necessary. These patterns find about 90% of the hyphens in the original dictionary; more importantly, they do not insert any spurious hyphen. In addition, a list of exceptions (words for which the patterns do not predict the correct hyphenation) are included with the Plain TeX format; additional ones can be specified by the user.

Metafont

Metafont, not strictly part of TeX, is a font description system which allows the designer to describe characters algorithmically. It uses Bézier curves in a fairly standard way to generate the actual characters to be displayed, but Knuth devotes lots of attention to the rasterizing problem on bitmapped displays. Another thesis, by John Hobby, further explores this problem of digitizing "brush trajectories". This term derives from the fact that Metafont describes characters as having been drawn by abstract brushes (and erasers).

It is commonly believed that TeX is based on bitmap fonts but, in fact, these programs know nothing about the fonts that they are using other than their dimensions. It is the responsibility of the device driver to appropriately handle fonts of other types, including PostScript Type 1, and TrueType. Computer Modern (commonly known as "the TeX font") is freely available in Type 1 format, as are the AMS math fonts. Users of TeX systems that output directly to PDF, such as pdfTeX, XeTeX or LuaTeX, generally never use Metafont output at all.

Macro language

TeX provides an unusual macro language; the definition of a macro not only includes a list of commands but also the syntax of the call. Macros are completely integrated with a full-scale interpreted compile-time language that also guides processing.

TeX's macro level of operation is lexical, but it is a built-in facility of TeX, that makes use of syntax interpretation. Comparing with most widely used lexical preprocessors like M4, it differs slightly, as the body of a macro gets tokenized at definition time, that is, it is not completely raw text. Except for a few very special cases, this gives the same behaviour.

The TeX macro language has been successfully used to extend TeX to, for instance, LaTeX and ConTeXt.

Development

The original source code for the current TeX software is written in WEB, a mixture of documentation written in TeX and a Pascal subset in order to ensure portability. For example, TeX does all of its dynamic allocation itself from fixed-size arrays and uses only fixed-point arithmetic for its internal calculations. As a result, TeX has been ported to almost all operating systems, usually by using the web2c program to convert the source code into C instead of directly compiling the Pascal code.

Knuth has kept a very detailed log of all the bugs he has corrected and changes he has made in the program since 1982; as of 2008, the list contains 427 entries, not including the version modification that should be done after his death as the final change in TeX. Donald Knuth offers monetary awards to people who find and report a bug in TeX. The award per bug started at $2.56 (one "hexadecimal dollar") and doubled every year until it was frozen at its current value of $327.68. Knuth, however, has lost relatively little money as there have been very few bugs claimed. In addition, recipients have been known to frame their check as proof that they found a bug in TeX rather than cashing it. Due to scammers finding scanned copies of his checks on the internet and using them to try to drain his bank account, Knuth no longer sends out real checks, but those who submit bug reports can get credit at The Bank of San Serriffe instead.

Packages

TeX is usually provided in the form of an easy-to-install bundle of TeX itself along with METAFONT and all the necessary fonts, documents formats, and utilities needed to use the typesetting system. On UNIX-compatible systems, including GNU/Linux and Apple Mac OS X, TeX is distributed in the form of the teTeX distribution and more recently the TeX Live distribution. On Microsoft Windows, there is the MiKTeX distribution (enhanced by proTeXt) and the MS Windows version of TeX Live.

Several document processing systems are based on TeX, notably jadeTeX, which uses TeX as a backend for printing from James Clark's DSSSL Engine, the Arbortext publishing system, and Texinfo, the GNU documentation processing system. TeX has been the official typesetting package for the GNU operating system since 1984.

XeTeX is a new TeX engine that supports Unicode. Originally making use of advanced Apple Mac OS X-specific font technologies, it now supports OpenType and is available on GNU/Linux and MS Windows.

Numerous extensions and companion programs for TeX exist, among them BibTeX for bibliographies (distributed with LaTeX), pdfTeX, which bypasses dvi and produces output in Adobe Systems' Portable Document Format, and Omega, which allows TeX to use the Unicode character set. Most TeX extensions are available for free from CTAN, the Comprehensive TeX Archive Network.

Editors

There are a variety of editors designed to work with TeX.

The TeXmacs text editor is a WYSIWYG scientific text editor that is inspired by both TeX and Emacs. It uses Knuth's fonts, and can generate TeX output.

LyX is a WYSIWYM document processor which runs on a variety of platforms including GNU/Linux, MS Windows (newer versions require MS Windows 2000 or later) or Apple Mac OS X (using a non-native Qt front-end).

TeXShop for Apple Mac OS X, TeXworks for GNU/Linux, and WinShell for MS Windows are similar tools and provide integrated development environment (IDE) for working with LaTeX or TeX. For KDE/ Qt, Kile provides such an IDE.

Texmaker is the Pure Qt equivalent of Kile, with a user interface that is nearly the same as Kile's.

GNU Emacs has various built-in and third party packages with support for TeX, the major one being AUCTeX.

For Vim there is the Vim-LaTeX Suite and Automatic TeX.

License

Donald Knuth has indicated several times that the source code of TeX has been placed into the "public domain," and he strongly encourages modifications or experimentations with this source code. However, since the code is still copyrighted, it is technically free/ open-source software but is not in the public domain in the legal sense. In particular, since Knuth highly values the reproducibility of the output of all versions of TeX, any changed version must not be called TeX, TeX, or anything confusingly similar. To enforce this rule, any implementation of the system must pass a test suite called the TRIP test before being allowed to be called TeX. The question of license is somewhat confused by the statements included at the beginning of the TeX source code, which indicate that "all rights are reserved. Copying of this file is authorized only if (...) you make absolutely no changes to your copy". However, this restriction should be interpreted as a prohibition to change the source code as long as the file is called tex.web. This interpretation is confirmed later in the source code when the TRIP test is mentioned ("If this program is changed, the resulting system should not be called 'TeX'").

The American Mathematical Society tried in the early 1980s to claim a trademark for TeX. This was rejected because at the time "TEX" (all caps) was registered by Honeywell for the "Text EXecutive" text processing system.

Use of TeX

In several technical fields, in particular, computer science, mathematics, engineering, and physics, TeX has become a de facto standard. Many thousands of books have been published using TeX, including books published by Addison-Wesley, Cambridge University Press, Elsevier, Oxford University Press and Springer. Numerous journals in these fields are produced using TeX or LaTeX, allowing authors to submit their raw manuscript written in TeX.

While many publications in other fields, including dictionaries and legal publications, have been produced using TeX, it has not been as successful as in more technical fields, because TeX was primarily designed for mathematics. When he designed TeX, Donald Knuth did not believe that a single typesetting system would fit everyone's needs; instead, he designed many hooks inside the program so that it would be possible to write extensions, and released the source code, hoping that publishers would design versions tailored to their needs. While such extensions have been created (including some by Knuth himself), most people have extended TeX only using macros and it has remained a system associated with technical typesetting.

It is possible to use TeX for automatic generation of sophisticated layout for XML data. The differences in syntax between the two description languages can be overcome with the help of TeXML. In the context of XML publication, TeX can thus be considered an alternative to XSL-FO.

Pronouncing and writing "TeX"

The name TeX is intended by its developer to be / ˈ t ɛ x /, with the final consonant of loch or Bach. The letters of the name are meant to represent the capital Greek letters tau, epsilon, and chi, as TeX is an abbreviation of τέχνη (ΤΕΧΝΗ – technē), Greek for both "art" and "craft", which is also the root word of technical. English speakers often pronounce it /ˈtɛk/, like the first syllable of technical.

Donald Knuth instructs that it be typeset with the "E" below the baseline and reduced spacing between the letters. This is done, as Knuth mentions in his TeXbook, to distinguish TeX from other system names such as TEX, the Text EXecutive processor (developed by Honeywell Information Systems). Fans like to proliferate names from the word "TeX"—such as TeXnician (user of TeX software), TeXhacker (TeX programmer), TeXmaster (competent TeX programmer), TeXhax, and TeXnique.

Community

TeX Users Group's Logo

Notable entities in the TeX community include the TeX Users Group, which publishes TUGboat and The PracTeX Journal, covering a wide range of topics in digital typography relevant to TeX. The Deutschsprachige Anwendervereinigung TeX is a large user group in Germany. The TeX Users Group was founded in 1980 for educational and scientific purposes, provides an organization for those who have an interest in typography and font design, and are users of the TeX typesetting system invented by Donald Knuth. The TeX Users Group represents the interests of TeX users worldwide. The TeX Users Group publishes the journal TUGboat three times per year.