Standard Performance Evaluation Corporation

From Wikipedia, the free encyclopedia

The Standard Performance Evaluation Corporation (SPEC) is a non-profit organization that aims to produce fair, impartial and meaningful benchmarks for computers. SPEC was founded in 1988 and their goal is to ensure that the marketplace has a fair and useful set of metrics to differentiate candidate systems. Its member organizations includes leading computer and software manufacturers. SPEC benchmarks are widely used today in evaluating the performance of computer systems; the results are published on the SPEC web site. Results are informally referred to as "SPECmarks", results or even just "SPEC".

The benchmarks aim to test "real-life" situations. SPECweb2005, for example, tests web server performance by performing various types of parallel HTTP requests, and SPEC CPU tests CPU performance by measuring the run time of several programs such as the compiler gcc and the chess program crafty. The various tasks are assigned weights based on their perceived importance; these weights are used to compute a single benchmark result in the end.

SPEC benchmarks are written in a platform neutral programming language (usually C or Fortran), and the interested parties may compile the code using whatever compiler they prefer for their platform, but may not change the code. Manufacturers have been known to optimize their compilers to improve performance of the various SPEC benchmarks.

In order to use a benchmark, a license has to be purchased from SPEC; the costs vary from test to test with a typical range from several hundred to several thousand dollars. This pay-for-license model might seem to be in violation of the GPL as the benchmarks include software such as GCC that is licensed by the GPL. However, the GPL does not require software to be distributed for free, only that recipients be allowed to redistribute any GPLed software that they receive; the license agreement for SPEC specifically exempts items that are under "licenses that require free distribution", and the files themselves are placed in a separate part of the overall software package.

1 Benchmarks
2 External links

[edit] Benchmarks

[edit] Current

SPEC CPU2006, combined performance of CPU, memory and compiler
- CINT2006 ("SPECint"), testing integer arithmetic, with programs such as compilers, interpreters, word processors, chess programs etc.
- CFP2006 ("SPECfp"), testing floating point performance, with physical simulations, 3D graphics, image processing, computational chemistry etc.
SPECweb2005, PHP and/or JSP performance.
SPECviewperf, performance of an OpenGL 3D graphics system, tested with various rendering tasks from real applications
SPECapc, performance of several 3D-intensive popular applications on a given system
SPEC OMP V3.1, for evaluating performance based on OpenMP (http://www.openmp.org) applications.
SPECjvm98, performance of a java client system running a Java virtual machine
SPECjAppServer2004, a multi-tier benchmark for measuring the performance of Java 2 Enterprise Edition (J2EE) technology-based application servers.
SPECjbb2005, evaluates the performance of server side Java by emulating a three-tier client/server system (with emphasis on the middle tier).
SPEC MAIL2001, performance of a mail server, testing SMTP and POP protocols
SPEC SFS97_R1, NFS file server throughput and response time

[edit] Future

SPECappPlatform, performance of web enterprise platforms (such as Java EE and .NET)
SPECimap2003, performance of an enterprise level mail server, testing SMTP and IMAP4 protocols
SPEC MPI, performance of MPI (Message Passing Interface) based applications