Core Image

From Wikipedia, the free encyclopedia

	Mac OS X Graphics Model Rendering QuickDraw • OpenGL • Quartz 2D • Core Image • Core Video Compositing Quartz Compositor

Core Image is an extensible pixel-accurate image processing framework for Mac OS X. It extends and compliments the Core Graphics framework with its plugin-based architecture for applying filters and effects. ^[1]

Contents 1 Overview 2 Pixel Accuracy 3 CPU Fallback vs. GPU Processing 4 Core Video 5 Compatible GPUs 5.1 ATI 5.2 Intel 5.3 NVIDIA 6 Apple applications that use Core Image 7 Image Units 8 References 9 External links

[edit] Overview

Core Image is designed to abstract the pixel-level image manipulation process, making it available to applications without requiring the addition of hardware-specific code. Core Image works by applying an Image Unit to the pixel data of a given image, resulting in the creation of a new image. Each image unit specifies a filter, transform, or other effect which can be applied to the original pixel data without modifying the original data. In this way, Image Units provide non-destructive image editing.^[2]

Like Photoshop filters, one Image Unit can be applied in conjunction with others. Core Image can assemble an instruction pipeline on the fly so that only one calculation needs to be applied to the pixel data to achieve the effect of applying multiple units. The result is that multiple Image Units can be applied to an image without requiring a significant increase in processing time. Core Image utilizes a just-in-time compiler for creating the code which performs the pixel operations associated with one or more Image Units. The code is then executed by either the CPU or graphics card's GPU, whichever can perform the caclulation faster.^[3]

[edit] Pixel Accuracy

All pixel processing provided by an Image Unit is performed in a pre-multiplied alpha (RGBA) color space, storing four color channels: red, green, blue, and transparency (alpha). Each color channel is represented by a 32-bit, floating point number. This provides exceptional color depth, far greater than can be perceived by the human eye, as each pixel is represented by a 128-bit vector (four 32-bit color channels). The floating-point calculation model also provides exceptional performance, especially for color spaces of lower bit-depth. This is useful when processing multiple frames of video data.^[4][5]

[edit] CPU Fallback vs. GPU Processing

Core Image, being only for two-dimensional graphics, uses only the pixel shaders of the GPU, not the vertex shaders. This means that the more pixel shader pipelines a GPU has, the better the performance will be. Other considerations are VRAM bandwidth, memory bus width, and GPU clock speed.

However, if a suitable graphics processor is not available, Core Image will fall back to the next best option: using SSE3+ on the Intel Core and Intel Core 2 processors, using the AltiVec vector-processing abilities of the G5 and G4 processors, or plain scalar code on a G3. Core Image will automatically adopt the fastest approach for a given machine architecture. However, realistically, real-time processing requires a compatible GPU.

When the CPU-fallback is used (as recommended by Apple for final rendering, most likely because VRAM readback over AGP is very slow), Core Image performance scales almost linearly with the number of CPUs and their clock speed.

Core Image has further implications than simple GUI enhancements, such as the ripple effect generated by Dashboard. Developers can write their own filters and package them in bundles called Image Units. However, unlike a traditional filter that manipulates the source image, Core Image performs a manipulation as an overlay — maintaining the original image. This is called "non-destructive" manipulation. The result is the ability to do highly complex and layered image manipulations with little or no loss in quality. Image units can be tested with the Core Image Fun House application.

[edit] Core Video

Running alongside Core Image is Core Video. In contrast to Core Image's Objective-C API, Core Video is a pure C API that follows Core Foundation conventions. Transition effects work like filters, combining two or more images to form a result image, and accepting a time value (0 < t < 1) as a parameter. By driving an animation using a timer, a transition from one image to another is easily implemented.

Contrary to a common assumption, Core Video and Core Image are functionally independent. However, there is a synergy across Core Video, Core Image and QuickTime 7, which essentially allows on-the-fly per-frame image processing on QuickTime movies.

Both technologies depend heavily on the system's graphics card to attain a high level of performance, although Core Video can quite successfully be used without a GPU meeting Core Image's requirements.

[edit] Compatible GPUs

Any Macintosh-compatible GPU that supports at least second-generation shaders, more specifically the ARB_fragment_program OpenGL extension, can also do Core Image. The same requirements are used by Quartz Extreme.

The following GPUs are supported (as of March 2006):

[edit] ATI

• ATI Radeon X1900 XT
• ATI (Mobility) Radeon X1600 XT
• ATI (Mobility) Radeon X1600
• ATI Radeon X600 Pro
• ATI Radeon X800 XT
• ATI Radeon 9800 XT
• ATI Radeon 9800 Pro
• ATI Radeon 9700 Pro
• ATI Radeon 9600 XT
• ATI Radeon 9600 Pro
• ATI Radeon 9650
• ATI Radeon 9600
• ATI Mobility Radeon 9700
• ATI Mobility Radeon 9600 (only 64MB version)
• ATI Mobility Radeon 9550

[edit] Intel

• Intel GMA 900 (Developer Transition Kit)
• Intel GMA 950

[edit] NVIDIA

• NVIDIA Quadro FX 4500 DDL
• NVIDIA GeForce 7800 GT
• NVIDIA GeForce 7800 GS (flashed PC-Card)
• NVIDIA GeForce 7300 GT
• NVIDIA GeForce 6800 Ultra DDL
• NVIDIA GeForce 6800 GT DDL
• NVIDIA GeForce 6600 LE
• NVIDIA GeForce 6600 GT (flashed PC-Card)
• NVIDIA GeForce 6600
• NVIDIA GeForce 6200 (flashed PC-Card)
• NVIDIA GeForce FX 5200 Ultra
• NVIDIA GeForce FX 5200 Go
• NVIDIA GeForce FX 5200

Note: The GeForce FX 5200 is only barely suited for Core Image due to poor pixel shader performance. With most Macs, the CPU is faster than the 5200 in rendering, so Apple disabled Core Image support for it and also did not support the 5200 in Aperture.

[edit] Apple applications that use Core Image

Core Image is part of the operating system itself, and can be implemented in any application. The CPU-fallback approach ensures compatibility on any system. It is not available on any version of Mac OS X prior to 10.4.

Apple ships a program called Quartz Composer with its Developer Tools. Its original maker is Pierre-Olivier Latour, who was later hired by Apple. Quartz Composer is basically Version 2 of his program Pixelshox, which is not supported any more, but which also runs on earlier versions of Mac OS X. Both Pixelshox and Quartz Composer support an approach similar to Apple's compositing-program Shake which can generate OpenGL-based scenes by linking various building blocks with nodes, feeding the output from one module to another. A multitude of data types (e.g. Boolean, numeric, image, video, RGB) and inputs (e.g. DV, image, video, generators, mouse, RSS-feed) are supported, and in- and outputs can be mapped to programs in Xcode. People without any programming knowledge can build complex scenes by just linking various blocks in the right way.

Unlike Pixelshox, Quartz Composer now also supports Core Image Filters (delimited by a dot in front of the Module). Since Quartz Composer can export the scene as a QuickTime 7 movie, developers can embed these, and thus be able to use Core Image in web pages. Since Quartz Composer only exports the scene and its assets to QuickTime and not rendered frames of video, the resulting file is very small. This means, though, that the movie requires Core Image, so it will not work on Windows, nor on versions of Mac OS X prior to Mac OS X 10.4, even with QuickTime 7 installed.

Also in the Developer Tools is Core Image Fun House, similar to another Core Image-based program called iMaginator. Outside of the Developer Tools, Dashboard uses Core Image for the Water Ripple effect, Apple's photography software Aperture uses it for other things, and some of iMovie's filters use Core Video.

Apple's title generator Motion uses similar GPU-based-rendering techniques, but does not (yet) use Core Image, using its own frameworks instead. This is done for Mac OS X v10.3 compatibility.

[edit] Image Units

• Median, Gaussian, Motion and Zoom blurs
• Noise Reduction
• Full color, hue, temperature, white point and saturation control
• Pinch, Hole, Dump, Displacement, Glass, torus Lens, Twirl, Vortex, Circle Splash and Circular warp distortions
• Several generator filters including Star Shine, Sunbeams, Checkerboard and Lenticular Halo
• Color blends: color burn, darken, difference, exclusion, hard light, hue, lighten, luminosity, multiply, overlay, saturation, screen, soft light
• Cropping
• Scaling, rotation, shearing and translation using Affine transformations
• Several halftone filters including CMYK, dot, hatched and line
• Deconvolution
• Transition effects such as Swipe, Flash, Page Curl, Copy Machine, Disintegrate with Mask, Dissolve

[edit] References

[edit] External links

• Core Image overview on Apple's web site
• Core Image kernels: Some examples for beginners