| Standard Performance Evaluation Corporation | |
|---|---|
| Formation | 1988 |
| Type | Not-for-profit |
| Headquarters | Gainesville, Virginia |
| Membership | Hardware & Software Vendors, Universities, Research Centers |
| Staff | 5 |
| Website | http://www.spec.org |
The Standard Performance Evaluation Corporation (SPEC) is a non-profit organization that aims to "produce, establish, maintain and endorse a standardized set" of performance benchmarks for computers.[1]
SPEC was founded in 1988.[2][3] SPEC benchmarks are widely used to evaluate the performance of computer systems; the test results are published on the SPEC website. Results are sometimes informally referred to as "SPECmarks" or just "SPEC".
Contents |
Membership allows:
The list of members is available on SPEC's membership page.
The benchmarks aim to test "real-life" situations. There are several benchmarks testing Java scenarios, from simple computation (SPECjbb) to a full system with Java EE, database, disk, and network (SPECjEnterprise). The SPECweb benchmarks test web server performance by performing various types of parallel HTTP requests.
The SPEC CPU suites test CPU performance by measuring the run time of several programs such as the compiler gcc, the chemistry program gamess, and the weather program WRF. The various tasks are equally weighted; no attempt is made to weight them based on their perceived importance. An overall score is based on a geometric mean.
SPEC benchmarks are written in a platform neutral programming language (usually C, Java or Fortran), and the interested parties may compile the code using whatever compiler they prefer for their platform, but may not change the code. Manufacturers have been known to optimize their compilers to improve performance of the various SPEC benchmarks. SPEC has rules that attempt to limit such optimizations.
In order to use a benchmark, a license has to be purchased from SPEC; the costs vary from test to test with a typical range from several hundred to several thousand dollars. This pay-for-license model might seem to be in violation of the GPL as the benchmarks include software such as GCC that is licensed by the GPL. However, the GPL does not require software to be distributed for free, only that recipients be allowed to redistribute any GPLed software that they receive; the license agreement for SPEC specifically exempts items that are under "licenses that require free distribution", and the files themselves are placed in a separate part of the overall software package.
SPEC attempts to create an environment where arguments are settled by appeal to notions of technical credibility, representativeness, or the "level playing field". SPEC representatives are typically engineers with expertise in the areas being benchmarked. Benchmarks include "run rules", which describe the conditions of measurement and documentation requirements. Results that are published on SPEC's website undergo a peer review by members' performance engineers.