Multilinear principal component analysis

Multilinear principal component analysis (MPCA) [1][2] [3] [4] is a mathematical procedure that uses multiple orthogonal transformations to convert a set of multidimensional objects into another set of multidimensional objects of lower dimensions. There is one orthogonal (linear) transformation for each dimension (mode); hence multilinear. This transformation aims to capture as high a variance as possible, accounting for as much of the variability in the data as possible, subject to the constraint of mode-wise orthogonality.

MPCA is a multilinear extension of principal component analysis (PCA). The major difference is that PCA needs to reshape a multidimensional object into a vector, while MPCA operates directly on multidimensional objects through mode-wise processing. For example, for 100x100 images, PCA operates on vectors of 10000x1 while MPCA operates on vectors of 100x1 in two modes. For the same amount of dimension reduction, PCA needs to estimate 49*(10000/(100*2)-1) times more parameters than MPCA. Thus, MPCA is more efficient and better conditioned in practice.

MPCA is a basic algorithm for dimension reduction via multilinear subspace learning. In wider scope, it belongs to tensor-based computation. Its origin can be traced back to the Tucker decomposition[5] in 1960s and it is closely related to higher-order singular value decomposition,[6] (HOSVD) and to the best rank-(R1, R2, ..., RN ) approximation of higher-order tensors.[7]

The algorithm

MPCA performs feature extraction by determining a multilinear projection that captures most of the original tensorial input variations. As in PCA, MPCA works on centered data. The MPCA solution follows the alternating least square (ALS) approach.[8] Thus, is iterative in nature and it proceeds by decomposing the original problem to a series of multiple projection subproblems. Each subproblem is a classical PCA problem, which can be easily solved.

It should be noted that while PCA with orthogonal transformations produces uncorrelated features/variables, this is not the case for MPCA. Due to the nature of tensor-to-tensor transformation, MPCA features are not uncorrelated in general although the transformation in each mode is orthogonal.[9] In contrast, the uncorrelated MPCA (UMPCA) generates uncorrelated multilinear features.[9]

Feature selection

MPCA produces tensorial features. For conventional usage, vectorial features are often preferred. For example most classifiers in the literature takes vectors as input. On the other hand, as there are correlations among MPCA features, a further selection process often improves the performance. Supervised (discriminative) MPCA feature selection is used in object recognition[10] while unsupervised MPCA feature selection is employed in visualization task.[11]

Extensions

Various extensions of MPCA have been developed:[12]

Resources

References

  1. ↑ M. A. O. Vasilescu, D. Terzopoulos (2002) "Multilinear Analysis of Image Ensembles: TensorFaces", Proc. 7th European Conference on Computer Vision (ECCV'02), Copenhagen, Denmark, May, 2002
  2. ↑ M. A. O. Vasilescu, D. Terzopoulos (2003) "Multilinear Subspace Analysis of Image Ensembles", "Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’03), Madison, WI, June, 2003"
  3. ↑ M. Alex O. Vasilescu (2002) "Human Motion Signatures: Analysis, Synthesis, Recognition", "Proceedings of the International Conference on Pattern Recognition (ICPR’02)", Quebec City, Canada, August, 2002
  4. ↑ H. Lu, K. N. Plataniotis, and A. N. Venetsanopoulos, (2008) "MPCA: Multilinear principal component analysis of tensor objects", IEEE Trans. Neural Netw., 19 (1), 18–39
  5. ↑ Tucker, Ledyard R (September 1966). "Some mathematical notes on three-mode factor analysis". Psychometrika 31 (3): 279–311. doi:10.1007/BF02289464.
  6. ↑ L.D. Lathauwer, B.D. Moor, J. Vandewalle (2000) "A multilinear singular value decomposition", SIAM Journal of Matrix Analysis and Applications, 21 (4), 1253–1278
  7. ↑ L. D. Lathauwer, B. D. Moor, J. Vandewalle (2000) "On the best rank-1 and rank-(R1, R2, ..., RN ) approximation of higher-order tensors", SIAM Journal of Matrix Analysis and Applications 21 (4), 1324–1342.
  8. ↑ P. M. Kroonenberg and J. de Leeuw, Principal component analysis of three-mode data by means of alternating least squares algorithms, Psychometrika, 45 (1980), pp. 69–97.
  9. ↑ 9.0 9.1 9.2 H. Lu, K. N. Plataniotis, and A. N. Venetsanopoulos, "Uncorrelated multilinear principal component analysis for unsupervised multilinear subspace learning," IEEE Trans. Neural Netw., vol. 20, no. 11, pp. 1820–1836, Nov. 2009.
  10. ↑ , M. A. O. Vasilescu, D. Terzopoulos (2003) "Multilinear Subspace Analysis of Image Ensembles", "Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’03), Madison, WI, June, 2003"
  11. ↑ H. Lu, H.-L. Eng, M. Thida, and K.N. Plataniotis, "Visualization and Clustering of Crowd Video Content in MPCA Subspace," in Proceedings of the 19th ACM Conference on Information and Knowledge Management (CIKM 2010) , Toronto, ON, Canada, October, 2010.
  12. ↑ Lu, Haiping; Plataniotis, K.N.; Venetsanopoulos, A.N. (2011). "A Survey of Multilinear Subspace Learning for Tensor Data". Pattern Recognition 44 (7): 1540–1551. doi:10.1016/j.patcog.2011.01.004.
  13. ↑ H. Lu, K. N. Plataniotis and A. N. Venetsanopoulos, "Boosting Discriminant Learners for Gait Recognition using MPCA Features", EURASIP Journal on Image and Video Processing, Volume 2009, Article ID 713183, 11 pages, 2009. doi:10.1155/2009/713183.
  14. ↑ Y. Panagakis, C. Kotropoulos, G. R. Arce, "Non-negative multilinear principal component analysis of auditory temporal modulations for music genre classification", IEEE Trans. on Audio, Speech, and Language Processing, vol. 18, no. 3, pp. 576–588, 2010.
  15. ↑ K. Inoue, K. Hara, K. Urahama, "Robust multilinear principal component analysis", Proc. IEEE Conference on Computer Vision, 2009, pp. 591–597.