HT2011, 10 ECTS
Recognizing patterns in nature, numbers, geometry, and the world around us forms the basis of discovery in mathematics and the sciences. The human brain from the time of its inception is in ever continual learning, constantly striving to sort out and simplify complex input upon which it builds generalizations and innately attempts to find structure. Understanding the nature of such a process and how it takes place in the mind is in itself a difficult task. Humans are naturally adept at integrating and assimilating information and at recognizing patterns intuitively whether in human behavior, identifying familiar objects, or recognizing other people. However, the process through which we achieve such recognition remains almost entirely hidden from us, and there is only little we know today of how our own brain functions and how it is built for pattern recognition tasks. Despite this reality, great applications still arise from our current knowledge and from extending our ability in pattern recognition to machines enabling them to mimic how humans perform such tasks. On the other hand, though lacking in cognition computers provide us with a greater ability than our own in deriving detailed features from objects that may otherwise not be apparent to a human observer, a fact that may prove vital in achieving correct classification. The understanding of how we may program machines to carry out pattern recognition tasks on their own and assist humans in decision making processes constitutes the basis of this course.
The course is intended to provide the student with an extensive and thorough insight into pattern recognition.
The main topics covered are supervised and unsupervised classification methods, regression, density estimation, and dimensionality reduction.
The course includes a treatment of the following subjects:
The topics will be presented theoretically through lectures and discussions and practically through lab sessions and project work.
In completing the course, the student shall be able to:
The course will include six lab sessions requiring programming in Matlab. Each lab session is generally assigned a two week deadline. Note that the labs are a compulsory part of the course. You are encouraged to work in groups of two during the labs, however you may also choose to work individually. You will be provided assistance throughout the lab periods, and discussions among students and groups are highly recommended.
For the project assignment you are encouraged to work in groups of two or three. You have to choose your own project in either one of the following main topics: unsupervised classification, supervised classification, or regression.
You will have to adapt the lab work in order to solve more challenging problems. There will be some project suggestions, however in any case the datasets you choose for the project have to be approved a priori.
All lectures will be held at the Centre for Image Analysis (CBA), Polacksbacken 2, Uppsala.
Lecture 1 : Introduction to Pattern Recognition
Thursday, 27 Oct, 10.15-12.00 room: 2115 lecturer: J,Azar Main idea, supervised classification, clustering, and regression. Feature space representation, terminology, decision functions & generalization. Multivariate data visualization. (Appendix D, 1.1-1.4).
Lecture 2 : Mathematical Background
Friday, 28 Oct, 10:15-12:00 room: 2115 lecturer: J,Azar Measurements and features in regard to image processing. Revision of basic probability & statistics: joint probabilities, central limit theorem, Bayes’ theorem, covariance, independence and correlation. Lagrange multipliers. (Appendix E)
Lecture 3 : Fundamental Concepts
Tuesday, 1 Nov, 10:15-12:00 room: 2115 lecturer: J,Azar Bayesian approach to pattern recognition. Density estimation: parametric methods, multivariate Gaussian, data whitening. Non-parametric methods (k-NN, Parzen), curse of dimensionality, maximum likelihood, cross-validation. (2.2, 3.1-3.3, 3.5)
Thursday, 3 Nov, 13:15-17:00 room: 2315D lecturer: J,Azar Data visualization, central limit theorem, multivariate normal distribution, data whitening, non-parametric density estimation: Parzen, nearest neighbor.
Lecture 4 : Feature Selection
Monday, 7 Nov, 10:15-12:00 room: 2115 lecturer: J,Azar Search algorithms, branch & bound, scatter matrices, criteria functions. Feature selection by global optimization: (meta)-heuristic methods: genetic algorithms, simulated annealing. (9.1-9.2)
Lecture 5 : Feature Extraction
Tuesday, 8 Nov, 13:15-15:00 room: 2115 lecturer: J,Azar Linear feature extraction: PCA, LDA-Fisher mapping. Nonlinear feature extraction: overview, multidimensional scaling, dissimilarity-based classifiers & embedding. (9.3-9.4, Appendix A)
Thursday, 10 Nov, 13:15-17:00 room: 2315D lecturer: J,Azar Forward selection, backward selection, take l-add-r selection, branch & bound, genetic algorithms. PCA, Fisher mapping, nonlinear feature extraction, multidimensional scaling, dissimilarity representation. Unsupervised Classification
Lecture 6 : Clustering
Monday, 14 Nov, 15:15-17:00 room: 2115 lecturer: J,Azar Unsupervised learning, hierarchical clustering, k-means, fuzzy c-means, mean shift algorithm. Gaussian mixture model, expectation-maximization algorithm, self-organizing maps. (10.1-10.5, 2.3)
Lecture 7 : Cluster Validation
Tuesday, 15 Nov, 13:15-15:00 room: 2115 lecturer: J,Azar Cluster validation, number of clusters, distortion measures, Davies-Bouldin index, other assessment criteria. Novelty detection, ROC curve. (10.6-10.10, 8.2.3)
Thursday, 17 Nov, 13:15-17:00 room: 2315D lecturer: J,Azar Hierarchical clustering, k-means, fuzzy c-means, Gaussian mixture model, expectation-maximization, Davies-Bouldin index, self-organizing maps. Supervised Classification I
Lecture 8 : Bayesian Classifiers
Monday, 21 Nov, 10:15-12:00 room: 2115 lecturer: J,Azar Bayes decision theory, Bayes classifier, Bayes error & risk, logistic classifier. Parzen classifier, k-NN classifier, proportional classifier. (3.3, 3.5, 4.2.1, 4.4)
Lecture 9 : Bayesian Normal based Classifiers/Discriminant Analysis
Tuesday, 22 Nov, 10:15-12:00 room: 2115 lecturer: J,Azar Quadratic discriminant classifier, linear discriminant classifier, nearest mean classifier. Fisher classifier, classification confidence & rejection. (2.2, 4.2.3, 4.3.9)
Friday, 25 Nov, 13:15-17:00 room: 2315D lecturer: J,Azar Implementation of Bayesian classifier, Parzen classifier, k-NN classifier, logistic classifier, quadratic/linear/nearest-mean classifiers, and Fisher classifier. Curse of dimensionality. Regression
Lecture 10 : Linear Regression
Monday, 28 Nov, 10:15-12:00 room: 2115 lecturer: J,Azar Bayesian regression. MMSE estimator, MAP estimator, ML estimator. Model evaluation, quality of regression.
Lecture 11 : Nonlinear & Multidimensional Regression
Monday, 28 Nov, 15:15-17:00 room: 2115 lecturer: J,Azar Nonlinear regression, kernel smoothing, local regression, backfitting algorithm.
Multidimensional regression: confidence bounds, model regularization: ridge regression, Least-Absolute-Shrinkage-&-Selection-Operator.
Wednesday, 30 Nov, 13:15-17:00 room: 2315D lecturer: J,Azar Linear regression, MMSE, MAP, MLE, quality measures. Nonlinear regression: kernel smoothing/local weighted regression. Supervised Classification II
Lecture 12 : Support Vector Machines
Tuesday, 6 Dec, 13:15-15:00 room: 2115 lecturer: J,Azar Support vector classifier: (non)-separablity, slack variable, (non)-linearity, kernel trick, multiclass problems, control parameters. Classifier complexity, VC-dimension. (4.2.5, 5.4, 4.3, 11.6)
Lecture 13 : Artificial Neural Networks
Thursday, 8 Dec, 10:15-12:00 room: 2115 lecturer: J,Azar Classification: perceptron, multi-layer perceptron, backpropagation training, decision functions. Autoregressive ANN, radial basis function ANN. Use in regression & feature extraction. (4.2.2, 6.2)
Lecture 14 : Combining Classifiers
Friday, 9 Dec, 10:15-12:00 room: 2115 lecturer: J,Azar Ensemble classification: fixed rules, trained combiners. Improving classifier performance: bootstrap aggregating, adaptive resampling and combining, boosting, and cloning approach. (8.4)
Monday, 12 Dec, 13:15-17:00 room: 2315D lecturer: J,Azar SVM, ANN, ensemble classification, complexity: bias-variance trade-off, improving performance (implement either boosting or cloning).
Lecture 15 : Review
Thursday, 15 Dec, 10:15-12:00 room: 2115 lecturer: J,Azar Course review/philosophical approach to principles of pattern recognition.
Information concerning the final exam.
Tuesday, 20 Dec, 13:15-17:00 room: 2115
Thursday, 12 Jan, 13:15-17:00 room: 2115