Standard Performance Evaluation Corporation

From Wikipedia, the free encyclopedia

The Standard Performance Evaluation Corporation (SPEC) is a non-profit organization that aims to produce fair, impartial and meaningful benchmarks for computers. SPEC was founded in 1988 and is financed by its member organizations which include all leading computer and software manufacturers. SPEC benchmarks are widely used today in evaluating the performance of computer systems; the results are published on the SPEC web site.

The benchmarks aim to test "real-life" situations. SPECweb2005, for example, tests web server performance by performing various types of parallel HTTP requests, and SPEC CPU tests CPU performance by measuring the run time of several programs such as the compiler gcc and the chess program crafty. The various tasks are assigned weights based on their perceived importance; these weights are used to compute a single benchmark result in the end.

SPEC benchmarks are written in a platform neutral programming language (usually C or Fortran), and the interested parties may compile the code using whatever compiler they prefer for their platform, but may not change the code. Manufacturers have been known to optimize their compilers to improve performance of the various SPEC benchmarks.

In order to use a benchmark, a license has to be purchased from SPEC; the costs vary from test to test with a typical range from several hundred to several thousand dollars. This pay-for-license model has been criticized as being in violation of the GPL as it includes software that is licensed by the GPL.

1 Benchmarks
2 External links

[edit] Benchmarks

[edit] Current

SPEC CPU2006, combined performance of CPU, memory and compiler
- CINT2006 ("SPECint"), testing integer arithmetic, with programs such as compilers, interpreters, word processors, chess programs etc.
- CFP2006 ("SPECfp"), testing floating point performance, with physical simulations, 3D graphics, image processing, computational chemistry etc.
SPECweb2005, PHP and/or JSP performance.
SPECviewperf, performance of an OpenGL 3D graphics system, tested with various rendering tasks from real applications
SPECapc, performance of several 3D-intensive popular applications on a given system
SPEC HPC2002, testing high end parallel computing systems with applications such as weather prediction and computational chemistry
SPEC OMP V3.1, for evaluating performance based on OpenMP (http://www.openmp.org) applications.
SPECjvm98, performance of a java client system running a Java virtual machine
SPECjAppServer2004, a multi-tier benchmark for measuring the performance of Java 2 Enterprise Edition (J2EE) technology-based application servers.
SPECjbb2005, evaluates the performance of server side Java by emulating a three-tier client/server system (with emphasis on the middle tier).
SPEC MAIL2001, performance of a mail server, testing SMTP and POP protocols
SPEC SFS97_R1, NFS file server throughput and response time

[edit] Future

SPECappPlatform, performance of web enterprise platforms (such as Java EE and .NET)
SPECimap2003, performance of an enterprise level mail server, testing SMTP and IMAP4 protocols
SPEC MPI, performance of MPI (Message Passing Interface) based applications