Publications
SUBMITTED
Parisotto, Simone; Vitoria, Patricia; Ballester, Coloma; Bugeau, Aurélie; Reynolds, Suzanne; Schönlieb, Carola-Bibiane: The Art of Inpainting: A Monograph on Mathematical Methods for the Virtual Restoration of Illuminated Manuscripts. (submitted), 2020. (Type: Book | Abstract)@book{ParVitBalBugReySch2020,This book is intended as the vademecum guide for the inpainting problem, referred to as the art of filling in missing imaging data so that the new information appears natural to the human eye. In this monograph we present different approaches in mathematics and computer vision: from local methods based on partial differential equations to non-local variational methods based on sophisticated copy-and-paste strategies of image patches, to deep learning approaches. |
Parisotto, Simone; Launaro, Alessandro; Leone, Ninetta; Schönlieb, Carola-Bibiane: Unsupervised clustering of Roman pottery profiles from their SSAE representation. In: (submitted, ArXiv: 2006.03156), 2020. (Type: Inproceedings | Abstract | Links)@inproceedings{Parisotto2020d,In this paper we introduce the ROman COmmonware POTtery (ROCOPOT) database, which comprises of more than 2000 black and white imaging profiles of pottery shapes extracted from 11 Roman catalogues and related to different excavation sites. The partiality and the handcrafted variance of the shape fragments within this new database make their unsupervised clustering a very challenging problem: profile similarities are thus explored via the hierarchical clustering of non-linear features learned in the latent representation space of a stacked sparse autoencoder (SSAE) network, unveiling new profile matches. Results are commented both from a mathematical and archaeological perspective so as to unlock new research directions in the respective communities. |
Burger, Martin; Korolev, Yury; Parisotto, Simone; Schönlieb, Carola-Bibiane: Total Variation Regularisation with Spatially Variable Lipschitz Constraints. In: (submitted, ArXiv: 1912.02768), 2019. (Type: Journal Article | Abstract | Links)@article{BurKorParSch2019,We introduce a first order Total Variation type regulariser that decomposes a function into a part with a given Lipschitz constant (which is also allowed to vary spatially) and a jump part. The kernel of this regulariser contains all functions whose Lipschitz constant does not exceed a given value, hence by locally adjusting this value one can determine how much variation is the reconstruction allowed to have. We prove regularising properties of this functional, study its connections to other Total Variation type regularisers and propose a primal dual optimisation scheme. Our numerical experiments demonstrate that the proposed first order regulariser can achieve reconstruction quality similar to that of second order regularisers such as Total Generalised Variation, while requiring significantly less computational time. |
PUBLISHED, PEER REVIEWED
Journal Articles |
Parisotto, Simone; Lellmann, Jan; Masnou, Simon; Schönlieb, Carola-Bibiane: Higher-order total directional variation: Imaging Applications. In: SIAM Journal on Imaging Sciences, 13 (4), pp. 2063–2104, 2020. (Type: Journal Article | Abstract | Links)@article{ParLelMasSch2020,We introduce a class of higher-order anisotropic total variation regularizers, which are defined for possibly inhomogeneous, smooth elliptic anisotropies, that extends the total generalized variation regularizer and its variants. We propose a primal-dual hybrid gradient approach to approximating numerically the associated gradient flow. This choice of regularizers allows us to preserve and enhance intrinsic anisotropic features in images. This is illustrated on various examples from different imaging applications: image denoising, wavelet-based image zooming, and reconstruction of surfaces from scattered height measurements. |
Parisotto, Simone; Calatroni, Luca; Bugeau, Aurélie; Papadakis, Nicolas; Schönlieb, Carola-Bibiane: Variational Osmosis for Non-linear Image Fusion. In: IEEE Transactions on Image Processing, 2020. (Type: Journal Article | Abstract | Links)@article{ParCalBugPapSch2020,We propose a new variational model for non-linear image fusion. Our approach is based on the use of an osmosis energy term related to the one studied in Vogel et al. [44] and Weickert et al. [45]. The minimization of the proposed non-convex energy realizes visually plausible image data fusion, invariant to multiplicative brightness changes. On the practical side, it requires minimal supervision and parameter tuning and can encode prior information on the structure of the images to be fused. For the numerical solution of the proposed model, we develop a primal-dual algorithm and we apply the resulting minimization scheme to solve multi-modal face fusion, color transfer and cultural heritage conservation problems. Visual and quantitative comparisons to state-of-the-art approaches prove the out-performance and the flexibility of our method. |
Parisotto, Simone; Masnou, Simon; Schönlieb, Carola-Bibiane: Higher-order Total Directional Variation: Analysis. In: SIAM Journal on Imaging Sciences, 13 (1), pp. 474–496, 2020. (Type: Journal Article | Abstract | Links)@article{ParMasSch2020,We analyze a new notion of total anisotropic higher-order variation which, differently from total generalized variation in [K. Bredies, K. Kunisch, and T. Pock, SIAM J. Imaging Sci., 3 (2010), pp. 492–526], quantifies for possibly nonsymmetric tensor fields their variations at arbitrary order weighted by possibly inhomogeneous, smooth elliptic anisotropies. We prove some properties of this total variation and of the associated spaces of tensors with finite variations. We show the existence of solutions to a related regularity-fidelity optimization problem. We also prove a decomposition formula which appears to be helpful for the design of numerical schemes, as shown in a companion paper, where several applications to image processing are studied. |
Parisotto, Simone; Calatroni, Luca; Caliari, Marco; Schönlieb, Carola-Bibiane; Weickert, Joachim: Anisotropic osmosis filtering for shadow removal in images. In: Inverse Problems, 35 (5), 2019. (Type: Journal Article | Abstract | Links)@article{ParCalCalSchWei2019,We present an anisotropic extension of the isotropic osmosis model that has been introduced by Weickert et al (2013 Energy Minimization Methods in Computer Vision and Pattern Recognition (Berlin: Springer)) for visual computing applications, and we adapt it specifically to shadow removal applications. We show that in the integrable setting, linear anisotropic osmosis minimises an energy that involves a suitable quadratic form which models local directional structures. In our shadow removal applications we estimate the local structure via a modified tensor voting approach (Moreno et al 2012 New Developments in the Visualization and Processing of Tensor Fields (Berlin: Springer)) and use this information within an anisotropic diffusion inpainting that resembles edge-enhancing anisotropic diffusion inpainting (Galić et al 2008 J. Math. Imaging Vis. 31 255–69; Weickert and Welk 2006 Visualization and Processing of Tensor Fields (Berlin: Springer)). Our numerical scheme combines the nonnegativity preserving stencil of Fehrenbach and Mirebeau (2014 J. Math. Imaging Vis. 49 123–47) with an exact time stepping based on highly accurate polynomial approximations of the matrix exponential. The resulting anisotropic model is tested on several synthetic and natural images corrupted by constant shadows. We show that it outperforms isotropic osmosis, since it does not suffer from blurring artefacts at the shadow boundaries. |
Calatroni, Luca; Marie d’Autume and, Rob Hocking; Panayotova, Stella; Parisotto, Simone; Ricciardi, Paola; Schönlieb, Carola‑Bibiane: Unveiling the invisible: mathematical methods for restoring and interpreting illuminated manuscripts. In: Heritage Science, 6 (1), pp. 56, 2018. (Type: Journal Article | Abstract | Links)@article{CalAutHocPanParRicSch2018,The last 50 years have seen an impressive development of mathematical methods for the analysis and processing of digital images, mostly in the context of photography, biomedical imaging and various forms of engineering. The arts have been mostly overlooked in this process, apart from a few exceptional works in the last 10 years. With the rapid emergence of digitisation in the arts, however, the arts domain is becoming increasingly receptive to digital image processing methods and the importance of paying attention to this therefore increases. In this paper we discuss a range of mathematical methods for digital image restoration and digital visualisation for illuminated manuscripts. The latter provide an interesting opportunity for digital manipulation because they traditionally remain physically untouched. At the same time they also serve as an example for the possibilities mathematics and digital restoration offer as a generic and objective toolkit for the arts. |
Daffara, Claudia; Parisotto, Simone; Ambrosini, Dario: A multipurpose, dual-mode imaging in the MWIR range for artwork diagnostic: a systematic approach. In: Optics and Lasers in Engineering, 2017. (Type: Journal Article | Abstract | Links)@article{DafParAmb2017,We present a multi-purpose, dual-mode imaging method in the Mid-Wavelength Infrared (MWIR) range (from 3 µm to 5 µm) for a more efficient nondestructive analysis of artworks. Using a setup based on a MWIR thermal camera and multiple radiation sources, two radiometric image datasets are acquired in different acquisition modalities, the image in quasi-reflectance mode (TQR) and the thermal sequence in emission mode. Here, the advantages are: the complementarity of the information; the use of the quasi-reflectance map for calculating the emissivity map; the use of TQR map for a referentiation to the visible of the thermographic images. The concept of the method is presented, the practical feasibility is demonstrated through a custom imaging setup, the potentiality for the nondestructive analysis is shown on a notable application to cultural heritage. The method has been used as experimental tool in support of the restoration of the mural painting “Monocromo” by Leonardo da Vinci. Feedback from the operators and a comparison with some conventional diagnostic techniques is also given to underline the novelty and potentiality of the method. |
Inproceedings |
Leone, Ninetta; Parisotto, Simone; Targonska-Hadzibabic, Kasia; Bucklow, Spike; Launaro, Alessandro; Reynolds, Suzanne; Schönlieb, Carola-Bibiane: Art Speaks Maths, Maths Speaks Art. In: Yackel, Carolyn; Bosch, Robert; Torrence, Eve; Fenyvesi, Kristòf (Ed.): Proceedings of Bridges 2020: Mathematics, Art, Music, Architecture, Education, Culture, pp. 481-484, Tessellations Publishing, 2021, ISBN: 978-1-938664-36-6. (Type: Inproceedings | Abstract | Links)@inproceedings{LeoParTHaBucLauReySch2021,Our interdisciplinary team Mathematics for Applications in Cultural Heritage (MACH) aims to use mathematical research for the benefit of the arts and humanities. Our ultimate goal is to create user-friendly software toolkits for artists, art conservators and archaeologists. In order for their underlying mathematical engines and functionality to be optimised for the needs of the end users, we pursue an iterative approach based on a continuous communication between the mathematicians and the cultural-heritage members of our team. Our paper illustrates how maths can speak art, but only if first art speaks maths. |
Daffara, Claudia; Parisotto, Simone; Mazzocato, Sara; Mariotti, Paola Ilaria; Ambrosini, Dario: Thermal imaging in the 3-5 micron range for precise localization of defects: application on frescoes at the Sforza Castle. In: Groves, Roger; Liang, Haida (Ed.): Optics for Arts, Architecture, and Archaeology VIII, SPIE, 2021. (Type: Inproceedings | Links)@inproceedings{Daffara2021, |
Parisotto, Simone; Schönlieb, Carola-Bibiane: Total Directional Variation for Video Denoising. In: Burger M. Lellmann J., Modersitzki J. (Ed.): Scale Space and Variational Methods in Computer Vision (SSVM 2019), Lecture Notes in Computer Science, Springer, 2019. (Type: Inproceedings | Abstract | Links)@inproceedings{ParSch2019,In this paper we propose a variational approach for video denoising, based on a total directional variation (TDV) regulariser proposed in Parisotto et al. [20, 21] for image denoising and interpolation. In the TDV regulariser, the underlying image structure is encoded by means of weighted derivatives so as to enhance the anisotropic structures in images, e.g. stripes or curves with a dominant local directionality. For the extension of TDV to video denoising, the space-time structure is captured by the volumetric structure tensor guiding the smoothing process. We discuss this and present our whole video denoising workflow. The numerical results are compared with some state-of-the-art video denoising methods. |
Parisotto, Simone; Calatroni, Luca; Daffara, Claudia: Digital Cultural Heritage Imaging via Osmosis Filtering. In: Mansouri, Alamin; Moataz, Abderrahim El; Nouboud, Fathallah; Mammass, Driss (Ed.): ICISP 2018: Image and Signal Processing, Lecture Notes in Computer Science, pp. 407-415, Springer, 2018. (Type: Inproceedings | Abstract | Links)@inproceedings{ParCalDaf2018,In Cultural Heritage (CH) imaging, data acquired within different spectral regions are often used to inspect surface and sub-surface features. Due to the experimental setup, these images may suffer from intensity inhomogeneities, which may prevent conservators from distinguishing the physical properties of the object under restoration. Furthermore, in multi-modal imaging, the transfer of information between one modality to another is often used to integrate image contents. In this paper, we apply the image osmosis model proposed in [4,10,12] to solve correct these problems arising when diagnostic CH imaging techniques based on reflectance, emission and fluorescence mode in the optical and thermal range are used. For an efficient computation, we use stable operator splitting techniques to solve the discretised model. We test our methods on real artwork datasets: the thermal measurements of the mural painting “Monocromo” by Leonardo Da Vinci, the UV-VIS-IR imaging of an ancient Russian icon and the Archimedes Palimpsest dataset. |
Parisotto, Simone; Calatroni, Luca; Daffara, Claudia: Mathematical osmosis imaging for multi-modal and multi-spectral applications in Cultural Heritage conservation. In: Image Processing for Art Investigation, Ghent, 2018. (Type: Inproceedings | Abstract | Links)@inproceedings{ParCalDaf2018b,In this work we present a dual-mode mid-infrared workflow [6], for detecting sub-superficial mural damages in frescoes artworks. Due to the large nature of frescoes, multiple thermal images are recorded. Thus, the experimental setup may introduce measurements errors, seen as inter-frame changes in the image contrast, after mosaicking. An approach to lowering errors is to post-process the mosaic [10] via osmosis partial differential equation (PDE) [12, 13], which preserves details, mass and balance the lights: efficient numerical study for osmosis on large images is proposed [2, 11], based on operator splitting [8]. Our range of Cultural Heritage applications include the detection of sub-superficial voids in Monocromo (L. Da Vinci, Castello Sforzesco, Milan) [5], the light-balance for multi-spectral imaging and the data integration on the Archimedes Palimpsest [10]. |
Calatroni, Luca; Estatico, Claudio; Garibaldi, Nicola; Parisotto, Simone: Alternating Direction Implicit (ADI) schemes for a PDE-based image osmosis model. In: Journal of Physics: Conference Series, IOP publishing, 2017. (Type: Inproceedings | Abstract | Links)@inproceedings{CalEstGarPar2017,We consider Alternating Direction Implicit (ADI) splitting schemes to compute efficiently the numerical solution of the PDE osmosis model considered by Weickert et al. in [10] for several imaging applications. The discretised scheme is shown to preserve analogous properties to the continuous model. The dimensional splitting strategy traduces numerically into the solution of simple tridiagonal systems for which standard matrix factorisation techniques can be used to improve upon the performance of classical implicit methods, even for large time steps. Applications to the shadow removal problem are presented. |
Daffara, Claudia; Parisotto, Simone; Mariotti, Paola Ilaria: Mid-infrared thermal imaging for an effective mapping of surface materials and sub-surface detachments in mural paintings: integration of thermography and thermal quasi-reflectography. In: Optics for Arts, Architecture, and Archaeology, International Society for Optics and Photonics 2015. (Type: Inproceedings | Abstract | Links)@inproceedings{DafParMar2015,Cultural Heritage is discovering how precious is thermal analysis as a tool to improve the restoration, thanks to its ability to inspect hidden details. In this work a novel dual mode imaging approach, based on the integration of thermography and thermal quasi-reflectography (TQR) in the mid-IR is demonstrated for an effective mapping of surface materials and of sub-surface detachments in mural painting. The tool was validated through a unique application: the “Monocromo” by Leonardo da Vinci in Italy. The dual mode acquisition provided two spatially aligned dataset: the TQR image and the thermal sequence. Main steps of the workflow included: 1) TQR analysis to map surface features and 2) to estimate the emissivity; 3) projection of the TQR frame on reference orthophoto and TQR mosaicking; 4) thermography analysis to map detachments; 5) use TQR to solve spatial referencing and mosaicking for the thermal-processed frames. Referencing of thermal images in the visible is a difficult aspect of the thermography technique that the dual mode approach allows to solve in effective way. We finally obtained the TQR and the thermal maps spatially referenced to the mural painting, thus providing the restorer a valuable tool for the restoration of the detachments. |
CHAPTERS IN BOOKS
Daffara, Claudia; Parisotto, Simone; Mariotti, Paola Ilaria: Infrared Analysis and ‘Thermal Quasi-Reflectography’. In: Leonardo Da Vinci – The Sala delle Asse of the Sforza Castle, pp. 186-199, Silvana Editrice, 2017, ISBN: 9788836636778. (Type: Book Chapter | Links)@inbook{ParisottoLeonardoBook2017, |
THESES
Parisotto, Simone: Anisotropic variational models and PDEs for inverse imaging problems. University of Cambridge, 2019. (Type: PhD Thesis | Abstract | Links)@phdthesis{Parisotto2019,Supervisor: Prof Carola-Bibiane Schönlieb (University of Cambridge) Co-supervisor: Prof Simon Masnou (Université Lyon 1) In this thesis we study new anisotropic variational regularisers and partial differential equations (PDEs) for solving inverse imaging problems that arise in a variety of real-world applications. Firstly, we introduce a new anisotropic higher-order total directional variation regulariser. We describe both the theoretical and the numerical details for its use within a variational formulation for solving inverse problems and give examples for the reconstruction of noisy images and videos, image zooming and the interpolation of scattered surface data. Secondly, we focus on a non-symmetric drift-diffusion equation, called osmosis. We propose an efficient numerical implementation of the osmosis equation, based on alternate directions and operator splitting techniques. We study their scale-space properties and show their efficiency in processing large images. Moreover, we generalise the osmosis equation to accommodate suitable directional information: this modification turns out to be useful to correct for the well-known blurring artefacts the original osmosis model introduces when applied to shadow removal in images. Last but not least, we explore applications of variational models and PDEs to cultural heritage conservation. We develop a new non-invasive technique that uses multi-modal imaging for detecting sub-superficial defects in fresco walls at sub-millimetre precision. We correct light-inhomogeneities in these imaging measurements that are due to measurement errors via osmosis filtering, in particular making use of the efficient computational schemes that we introduced before for dealing with the large-scale nature of these measurements. Finally, we propose a semi-supervised workflow for the detection and inpainting of defects in damaged illuminated manuscripts. Keywords: Total directional variation, anisotropic diffusion, osmosis filter, cultural heritage conservation, primal-dual hybrid gradient, dimensional splitting, inverse problems, image denoising, video denoising, image zooming, surface interpolation, digital elevation maps, shadow removal, thermal quasi-reflectography, non-destructive imaging, dual-mode mid-infrared imaging, inpainting, illuminated manuscripts. |
Parisotto, Simone: Variational Methods in Image Processing for Inpainting and Shadow Removal. University of Verona, Strada Le Grazie, 15, 2014. (Type: Masters Thesis | Abstract | Links)@mastersthesis{ParisottoMasterThesis2014,Advisor: Prof Giandomenico Orlandi (University of Verona) Second Advisor: Prof Simon Masnou (Université Lyon 1) This work deals with variational and PDE methods for computer vision applications such as inpainting, shadow removal and other image processing tasks. Inpainting (as well as image restoration) is a wide and open research field: when a corrupted image is given, typically with a hole where data are missing, one wants to get a credible retouching based on the information surrounding the hole itself. This problem can be attacked with geometric or exemplar-based methods: the former consider the propagation of level lines according to the curvature c, as e.g. [Masnou and Morel(1998), Chan et al.(2002)]; the latter, presented for instance in the very recent work [Arias et al.(2011)], extends the texture-oriented approach by [Efros and Leung(1999)] minimizing a functional whose core is a patch-comparison metric distance. One of these metrics, called Non local Poisson, is associated to a diffusion-transport Euler-Lagrange equation and provides good results when the illumination changes within the inpainting domain: the same equation arises in contexts dealing with lightings, such as the Shadow Removal problem, where the aim is to recover the information underlying shadow areas. The Shadow Removal problem can be modelled as in [Weickert et al.(2013)] by means of a drift-diffusion equation, and solved, as in [Vogel et al.(2013)] with the iterative BiCGStab solver. In our work, we aimed to test other numerical methods such as Exponential Integrators, which are exact in the time discretization domain, and Fourier-based collocation methods. For the sake of completeness, we propose here also some variants to the standard BiCGStab algorithm in order to adapt the timesteps to the number of iterations expected from the iterative solver. This is useful when the precision of the solution must be seriously considered while Exponential Integrators result too much slow. Our contribution shows a very strong speed up in the computation time (despite of a visually negligible Gibbs phenomenon when using the Fourier approach). This thesis is organized as follows: the first three chapters deal with the necessary theoretical background concerning basic notions and main results of Geometric Measure Theory, Finite Perimeter Sets and Functions of Bounded Variation (BV for short). In Chapter 4, we discuss the Inpainting Problem in a BV context with both previously cited approaches. Finally, in Chapter 5, we describe the connection between the Inpainting and the Shadow Removal Problem by completing our dissertation with some numerical experiments. |
Parisotto, Simone: Nonequispaced Fast Fourier Transform and Applications. 2010. (Type: Masters Thesis | Abstract | Links)@mastersthesis{ParisottoBachelorTheses2010,Advisor: Prof. Marco Caliari This thesis concerns the Discrete Fourier Transform (DFT) and its implementation and approximation in MATLAB. In order to evaluate the expression above at a set of N nodes we can use: - the Fast Fourier Transform (FFT), cost O(N logN), if the nodes are equispaced; - the Matrix Fourier Transform (MFT), cost O(N^2), , if the nodes are equispaced; – the recent Nonequispaced Fast Fourier Transform (NFFT), cost O(N logN + mN); m<<N, if the nodes are not equispaced. We will call the algorithms by their acronyms, referring to the transformation from the space domain to the frequency domain, or adding an I (Inverse) to their acronyms, referring to the transformation from the frequency domain to the space domain. The aims of this thesis are: – showing that FFT is more accurate and less expensive than MFT if the number of coefficients is larger than a certain number; – showing that NFFT is less expensive than MFT, only slightly less accurate than MFT and FFT and it can be used for a set of nonequispaced nodes; -showing how NFFT can be used in solving hyperbolic Partial Differential Equation with a periodic transport coefficient. |