|
In the context of software engineering, software quality refers to two related but distinct notions that exist wherever quality is defined in a business context:
Structural quality is evaluated through the analysis of the software inner structure, its source code, in effect how its architecture adheres to sound principles of software architecture. In contrast, functional quality is typically enforced and measured through software testing.
Historically, the structure, classification and terminology of attributes and metrics applicable to software quality management have been derived or extracted from the ISO 9126-3 and the subsequent ISO 25000:2005 quality model. Based on these models, the software structural quality characteristics have been clearly defined by the Consortium for IT Software Quality (CISQ), an independent organization founded by the Software Engineering Institute (SEI) at Carnegie Mellon University, and the Object Management Group (OMG).
CISQ has defined 5 major desirable characteristics needed for a piece of software to provide business value: Reliability, Efficiency, Security, Maintainability and (adequate) Size.
Software quality measurement is about quantifying to what extent a software or system rates along each of these five dimensions. An aggregated measure of software quality can be computed through a qualitative or a quantitative scoring scheme or a mix of both and then a weighting system reflecting the priorities. This view of software quality being positioned on a linear continuum has to be supplemented by the analysis of Critical Programming Errors that under specific circumstances can lead to catastrophic outages or performance degradations that make a given system unsuitable for use regardless of rating based on aggregated measurements.
"A science is as mature as its measurement tools," (Louis Pasteur in Ebert and al.l, p. 91) and software engineering has evolved to a level of maturity that makes it not only possible but also necessary to measure quality software for at least two reasons:
However, the distinction between measuring and improving software quality in an embedded system (with emphasis on risk management) and software quality in business software (with emphasis on cost and maintainability management) is becoming somewhat irrelevant. Embedded systems now often include a user interface and their designers are as much concerned with issues affecting usability and user productivity as their counterparts who focus on business applications. The latter are in turn looking at ERP or CRM system as a corporate nervous system whose uptime and performance are vital to the well-being of the enterprise. This convergence is most visible in mobile computing: a user who accesses an ERP application on their smartphone is depending on the quality of software across all types of software layers.
Both types of software now use multi-layered technology stacks and complex architecture so software quality analysis and measurement have to be managed in a comprehensive and consistent manner, decoupled from the software's ultimate purpose or use. In both cases, engineers and management need to be able to make rational decisions based on measurement and fact-based analysis in adherence to the precept "In God (we) trust. All others bring data". ((mis-)attributed to W. Edwards Deming and others).
Even though (as noted in the article on quality in business) "quality is a perceptual, conditional and somewhat subjective attribute and may be understood differently by different people," Software structural quality characteristics have been clearly defined by the Consortium for IT Software Quality (CISQ), an independent organization founded by the Software Engineering Institute (SEI) at Carnegie Mellon University <http://www.sei.cmu.edu>, and the Object Management Group (OMG) [1]. Under the guidance of Bill Curtis, co-author of the Capability Maturity Model framework and CISQ's first Director and Capers Jones, CISQ's Distinguished Advisor, CISQ has defined 5 major desirable characteristics of a piece of software needed to provide business value (see CISQ 2009 Executive Forums Report). In the House of Quality model, these are "Whats" that need to be achieved:
Software functional quality is defined as conformance to explicitly stated functional requirements, identified for example using Voice of the Customer analysis (part of the Design for Six Sigma toolkit and/or documented through use cases) and the level of satisfaction experienced by end-users. The later is referred as to as usability and is concerned with how intuitive and responsive the user interface is, how easy simple and complex operations can be performed, how useful error messages are. Typically, software testing practices and tools insure that a piece of software behaves in compliance with the original design, planned user experience and desired testability, i.e. a software's disposition to support acceptance criteria.
The dual structural/functional dimension of software quality is consistent with the model proposed in Steve McConnell's Code Complete which divides software characteristics into two pieces: internal and external quality characteristics. External quality characteristics are those parts of a product that face its users, where internal quality characteristics are those that do not.[5]
One of the challenges in defining quality is that "everyone feels they understand it" [6] and other definitions of software quality could be based on extending the various description of the concept of quality used in business (see a list of possible definition here.)
Dr. Tom DeMarco has proposed that "a product's quality is a function of how much it changes the world for the better.".[7] This can be interpreted as meaning that functional quality and user satisfaction, is more important than structural quality in determining software quality.
Another definition, coined by Gerald Weinberg in Quality Software Management: Systems Thinking, is "Quality is value to some person." This definition stresses that quality is inherently subjective - different people will experience the quality of the same software very differently. One strength of this definition is the questions it invites software teams to consider, such as "Who are the people we want to value our software?" and "What will be valuable to them?"
Although the concepts presented in this section are applicable to both Software Structural and Functional Quality, measurement of the latter is essentially performed through testing, see main article: Software Testing.
Software quality measurement is about quantifying to what extent a software or system possesses desirable characteristics. This can be performed through qualitative or quantitative means or a mix of both. In both cases, for each desirable characteristic, there are a set of measurable attributes the existence of which in a piece of software or system tend to be correlated and associated to this characteristic. For example, an attribute associated with portability is the number of target-dependent statements in a program. More precisely, using the Quality Function Deployment approach, these measurable attributes are the "Hows" that need to be enforced to enable the "whats" in the Software Quality definition above.
The structure, classification and terminology of attributes and metrics applicable to software quality management have been derived or extracted from the ISO 9126-3 and the subsequent ISO 25000:2005 quality model. The main focus is on internal structural quality. Subcategories have been created to handle specific areas like business application architecture and technical characteristics such as data access and manipulation or the notion of transactions.
The dependence tree between software quality characteristics and their measurable attributes is represented in the diagram on the right, where each of the 5 characteristics that matter for the user (right) or owner of the business system depends on measurable attributes (left):
Many of the existing software measures count structural elements of the application that result from parsing the source code such individual instructions (Park, 1992),[8] tokens (Halstead, 1977),[9] control structures (McCabe, 1976), and objects (Chidamber & Kemerer, 1994).[10]
Software quality measurement is about quantifying to what extent a software or system rate along these dimensions. The analysis can be performed using a qualitative, quantitative approach or a mix of both to provide an aggregate view (using for example weighted average(s) that reflect relative importance between the factor being measured).
This view of software quality on a linear continuum has to be supplemented by the identification of discrete Critical Programming Errors. These vulnerabilities may not fail a test case, but they are the result of bad practices that under specific circumstances can lead to catastrophic outages, performance degradations, security breaches, corrupted data, and myriad other problems (Nygard, 2007)[11] that makes a given system de facto unsuitable for use regardless of its rating based on aggregated measurements. A well known example of vulnerability is the Common Weakness Enumeration at http://cwe.mitre.org/ (Martin, 2001),[12] a repository of vulnerabilities in the source code that make applications exposed to security breaches.
The measurement of critical application characteristics involves measuring structural attributes of the application's architecture, coding, in-line documentation, as displayed in the picture above. Thus, each characteristic is affected by attributes at numerous levels of abstraction in the application and all of which must be included calculating the characteristic’s measure if it is to be a valuable predictor of quality outcomes that affect the business. The layered approach to calculating characteristic measures displayed in the figure above was first proposed by Boehm and his colleagues at TRW (Boehm, 1978)[13] and is the approach taken in the ISO 9126 and 25000 series standards. These attributes can be measured from the parsed results of a static analysis of the application source code. Even dynamic characteristics of applications such as reliability and performance efficiency have their causal roots in the static structure of the application.
Structural quality analysis and measurement is performed through the analysis of the source code, the architecture, software framework, database schema in relationship to principles and standards that together define the conceptual and logical architecture of a system. This is distinct from the basic, local, component-level code analysis typically performed by development tools which are mostly concerned with implementation considerations and are crucial during debugging and testing activities.
The root causes of poor reliability are found in a combination of non- compliance with good architectural and coding practices. This non-compliance can be detected by measuring the static quality attributes of an application. Assessing the static attributes underlying an application’s reliability provides an estimate of the level of business risk and the likelihood of potential application failures and defects the application will experience when placed in operation.
Assessing reliability requires checks of at least the following software engineering best practices and technical attributes:
|
|
Depending on the application architecture and the third-party components used (such as external libraries or frameworks), custom checks should be defined along the lines drawn by the above list of best practices to ensure a better assessment of the reliability of the delivered software.
As with Reliability, the causes of performance inefficiency are often found in violations of good architectural and coding practice which can be detected by measuring the static quality attributes of an application. These static attributes predict potential operational performance bottlenecks and future scalability problems, especially for applications requiring high execution speed for handling complex algorithms or huge volumes of data.
Assessing performance efficiency requires checking at least the following software engineering best practices and technical attributes:
Most security vulnerabilities result from poor coding and architectural practices such as SQL injection or cross-site scripting. These are well documented in lists maintained by CWE http://cwe.mitre.org/ (see below), and the SEI/Computer Emergency Center (CERT) at Carnegie Mellon University.
Assessing security requires at least checking the following software engineering best practices and technical attributes:
Maintainability includes concepts of modularity, understandability, changeability, testability, reusability, and transferability from one development team to another. These do not take the form of critical issues at the code level. Rather, poor maintainability is typically the result of thousands of minor violations with best practices in documentation, complexity avoidance strategy, and basic programming practices that make the difference between clean and easy-to-read code vs. unorganized and difficult-to-read code.[14]
Assessing maintainability requires checking the following software engineering best practices and technical attributes:
|
|
Measuring software size requires that the whole source code be correctly gathered, including database structure scripts, data manipulation source code, component headers, configuration files etc. There are essentially two types of software sizes to be measured, the technical size (footprint) and the functional size:
The Function Point Analysis sizing standard is supported by the International Function Point Users Group (IFPUG) (www.ifpug.org). It can be applied early in the software development life-cycle and it is not dependent on lines of code like the somewhat inaccurate Backfiring method. The method is technology agnostic and can be used for comparative analysis across organizations and across industries.
Since the inception of Function Point Analysis, several variations have evolved and the family of functional sizing techniques has broadened to include such sizing measures as COSMIC , NESMA, Use Case Points, FP Lite, Early and Quick FPs, and most recently Story Points. However, Function Points has a history of statistical accuracy, and has been used as a common unit of work measurement in numerous application development management (ADM) or outsourcing engagements, serving as the ‘currency’ by which services are delivered and performance is measured.
One common limitation to the Function Point methodology is that it is a manual process and therefore it can be labor intensive and costly in large scale initiatives such as application development or outsourcing engagements. This negative aspect of applying the methodology may be what motivated industry IT leaders to form the Consortium for IT Software Quality (www.it-cisq.org) focused on introducing a computable metrics standard for automating the measuring of software size while the IFPUG www.ifpug.org keep promoting a manual approach as most of its activity rely on FP counters certifications.
In November 2011, CISQ announced the availability of its first metric standard, Automated Function Points, to the CISQ membership, in CISQ Technical Report 2011-01 available at http://www.cisq.org/cisqwiki/images/a/a2/CISQ_Function_Point_Specification.pdf. These recommendations have been developed in OMG’s Request for Comment format and submitted to OMG’s process for standardization.
Critical Programming Errors are specific architectural and/or coding bad practices that result in the highest, immediate or long term, business disruption risk.
These are quite often technology-related and depend heavily on the context, business objectives and risks. Some may consider respect for naming conventions while others – those preparing the ground for a knowledge transfer for example – will consider it as absolutely critical.
Critical Programming Errors can also be classified per CISQ Characteristics. Basic example below:
|
|||||||||||||||||||||||||||||||