publications

2023

1. JMLR

   Neural Operator: Learning Maps Between Function Spaces With Applications to PDEs

   Kovachki, Nikola B, Li, Zongyi, Liu, Burigede, Azizzadenesheli, Kamyar, Bhattacharya, Kaushik, Stuart, Andrew M, and Anandkumar, Anima

   Journal of Machine Learning 2023

   Abs Bib HTML PDF

   The classical development of neural networks has primarily focused on learning mappings between finite dimensional Euclidean spaces or finite sets. We propose a generalization of neural networks to learn operators, termed neural operators, that map between infinite dimensional function spaces. We formulate the neural operator as a composition of linear integral operators and nonlinear activation functions. We prove a universal approximation theorem for our proposed neural operator, showing that it can approximate any given nonlinear continuous operator. The proposed neural operators are also discretization-invariant, i.e., they share the same model parameters among different discretization of the underlying function spaces. Furthermore, we introduce four classes of efficient parameterization, viz., graph neural operators, multi-pole graph neural operators, low-rank neural operators, and Fourier neural operators. An important application for neural operators is learning surrogate maps for the solution operators of partial differential equations (PDEs). We consider standard PDEs such as the Burgers, Darcy subsurface flow, and the Navier-Stokes equations, and show that the proposed neural operators have superior performance compared to existing machine learning based methodologies, while being several orders of magnitude faster than conventional PDE solvers.

   @article{kovachki2023neural,
     abbr = {JMLR},
     bibtex_show = {true},
     title = {Neural Operator: Learning Maps Between Function Spaces With Applications to PDEs},
     author = {Kovachki, Nikola B and Li, Zongyi and Liu, Burigede and Azizzadenesheli, Kamyar and Bhattacharya, Kaushik and Stuart, Andrew M and Anandkumar, Anima},
     journal = {Journal of Machine Learning},
     volume = {24},
     year = {2023},
     number = {89},
     html = {https://www.jmlr.org/papers/v24/21-1524.html},
     pdf = {https://www.jmlr.org/papers/volume24/21-1524/21-1524.pdf},
     selected = {true}
   }

2. arXiv

   An Approximation Theory Framework for Measure-Transport Sampling Algorithms

   Baptista, Ricardo, Hosseini, Bamdad, Kovachki, Nikola B, Marzouk, Youssef M, and Sagiv, Amir

   CoRR 2023

   Abs Bib HTML PDF

   This article presents a general approximation-theoretic framework to analyze measure transport algorithms for probabilistic modeling. A primary motivating application for such algorithms is sampling – a central task in statistical inference and generative modeling. We provide a priori error estimates in the continuum limit, i.e., when the measures (or their densities) are given, but when the transport map is discretized or approximated using a finite-dimensional function space. Our analysis relies on the regularity theory of transport maps and on classical approximation theory for high-dimensional functions. A third element of our analysis, which is of independent interest, is the development of new stability estimates that relate the distance between two maps to the distance (or divergence) between the pushforward measures they define. We present a series of applications of our framework, where quantitative convergence rates are obtained for practical problems using Wasserstein metrics, maximum mean discrepancy, and Kullback–Leibler divergence. Specialized rates for approximations of the popular triangular Knothe-Rosenblatt maps are obtained, followed by numerical experiments that demonstrate and extend our theory.

   @article{baptista2023anapproximation,
     abbr = {arXiv},
     bibtex_show = {true},
     author = {Baptista, Ricardo and Hosseini, Bamdad and Kovachki, Nikola B and Marzouk, Youssef M and Sagiv, Amir},
     title = {An Approximation Theory Framework for Measure-Transport Sampling Algorithms},
     journal = {CoRR},
     volume = {abs/2302.13965},
     year = {2023},
     html = {https://arxiv.org/abs/2302.13965},
     pdf = {https://arxiv.org/pdf/2302.13965.pdf},
     selected = {true}
   }

3. arXiv

   Geometry-Informed Neural Operator for Large-Scale 3D PDEs

   Li, Zongyi, Kovachki, Nikola B, Choy, Christopher, Li, Boyi, Kossaifi, Jean, Otta, Shourya P, Nabian, Mohammad A, Stadler, Maximilian, Hundt, Christian, Azizzadenesheli, Kamyar, and Anandkumar, Anima

   CoRR 2023

   Abs Bib HTML PDF

   We propose the geometry-informed neural operator (GINO), a highly efficient approach to learning the solution operator of large-scale partial differential equations with varying geometries. GINO uses a signed distance function and point-cloud representations of the input shape and neural operators based on graph and Fourier architectures to learn the solution operator. The graph neural operator handles irregular grids and transforms them into and from regular latent grids on which Fourier neural operator can be efficiently applied. GINO is discretization-convergent, meaning the trained model can be applied to arbitrary discretization of the continuous domain and it converges to the continuum operator as the discretization is refined. To empirically validate the performance of our method on large-scale simulation, we generate the industry-standard aerodynamics dataset of 3D vehicle geometries with Reynolds numbers as high as five million. For this large-scale 3D fluid simulation, numerical methods are expensive to compute surface pressure. We successfully trained GINO to predict the pressure on car surfaces using only five hundred data points. The cost-accuracy experiments show a 26,000× speed-up compared to optimized GPU-based computational fluid dynamics (CFD) simulators on computing the drag coefficient. When tested on new combinations of geometries and boundary conditions (inlet velocities), GINO obtains a one-fourth reduction in error rate compared to deep neural network approaches.

   @article{li2023geometry-informed,
     abbr = {arXiv},
     bibtex_show = {true},
     author = {Li, Zongyi and Kovachki, Nikola B and Choy, Christopher and Li, Boyi and Kossaifi, Jean and Otta, Shourya P and Nabian, Mohammad A and Stadler, Maximilian and Hundt, Christian and Azizzadenesheli, Kamyar and Anandkumar, Anima},
     title = {Geometry-Informed Neural Operator for Large-Scale 3D PDEs},
     journal = {CoRR},
     volume = {abs/2309.00583},
     year = {2023},
     html = {https://arxiv.org/abs/2309.00583},
     pdf = {https://arxiv.org/pdf/2309.00583.pdf}
   }

4. arXiv

   Learning Homogenization for Elliptic Operators

   Bhattacharya, Kaushik, Kovachki, Nikola B, Rajan, Akilla, and Trautner, Margaret

   CoRR 2023

   Abs Bib HTML PDF

   Multiscale partial differential equations (PDEs) arise in various applications, and several schemes have been developed to solve them efficiently. Homogenization theory is a powerful methodology that eliminates the small-scale dependence, resulting in simplified equations that are computationally tractable. In the field of continuum mechanics, homogenization is crucial for deriving constitutive laws that incorporate microscale physics in order to formulate balance laws for the macroscopic quantities of interest. However, obtaining homogenized constitutive laws is often challenging as they do not in general have an analytic form and can exhibit phenomena not present on the microscale. In response, data-driven learning of the constitutive law has been proposed as appropriate for this task. However, a major challenge in data-driven learning approaches for this problem has remained unexplored: the impact of discontinuities and corner interfaces in the underlying material. These discontinuities in the coefficients affect the smoothness of the solutions of the underlying equations. Given the prevalence of discontinuous materials in continuum mechanics applications, it is important to address the challenge of learning in this context; in particular to develop underpinning theory to establish the reliability of data-driven methods in this scientific domain. The paper addresses this unexplored challenge by investigating the learnability of homogenized constitutive laws for elliptic operators in the presence of such complexities. Approximation theory is presented, and numerical experiments are performed which validate the theory for the solution operator defined by the cell-problem arising in homogenization for elliptic PDEs.

   @article{bhattacharya2023learning,
     abbr = {arXiv},
     bibtex_show = {true},
     title = {Learning Homogenization for Elliptic Operators},
     author = {Bhattacharya, Kaushik and Kovachki, Nikola B and Rajan, Akilla and Trautner, Margaret},
     journal = {CoRR},
     volume = {abs/2306.12006},
     year = {2023},
     html = {https://arxiv.org/abs/2306.12006},
     pdf = {https://arxiv.org/pdf/2306.12006.pdf}
   }

5. SIAM UQ

   Convergence Rates for Learning Linear Operators from Noisy Data

   Hoop, Maarten V, Kovachki, Nikola B, Nelsen, Nicholas H, and Stuart, Andrew M

   SIAM Journal on Uncertainty Quantification 2023

   Abs Bib HTML PDF

   This paper studies the learning of linear operators between infinite-dimensional Hilbert spaces. The training data comprises pairs of random input vectors in a Hilbert space and their noisy images under an unknown self-adjoint linear operator. Assuming that the operator is diagonalizable in a known basis, this work solves the equivalent inverse problem of estimating the operator’s eigenvalues given the data. Adopting a Bayesian approach, the theoretical analysis establishes posterior contraction rates in the infinite data limit with Gaussian priors that are not directly linked to the forward map of the inverse problem. The main results also include learning-theoretic generalization error guarantees for a wide range of distribution shifts. These convergence rates quantify the effects of data smoothness and true eigenvalue decay or growth, for compact or unbounded operators, respectively, on sample complexity. Numerical evidence supports the theory in diagonal and non-diagonal settings.

   @article{de2023convergence,
     abbr = {SIAM UQ},
     bibtex_show = {true},
     title = {Convergence Rates for Learning Linear Operators from Noisy Data},
     author = {de Hoop, Maarten V and Kovachki, Nikola B and Nelsen, Nicholas H and Stuart, Andrew M},
     journal = {SIAM Journal on Uncertainty Quantification},
     volume = {11},
     number = {2},
     year = {2023},
     html = {https://epubs.siam.org/doi/10.1137/21M1442942},
     pdf = {https://arxiv.org/pdf/2108.12515.pdf}
   }

6. arXiv

   Conditional Sampling with Monotone GANs: from Generative Models to Likelihood-Free Inference

   Baptista, Ricardo, Hosseini, Bamdad, Kovachki, Nikola B, and Marzouk, Youssef M

   CoRR 2023

   Abs Bib HTML PDF

   We present a novel framework for conditional sampling of probability measures, using block triangular transport maps. We develop the theoretical foundations of block triangular transport in a Banach space setting, establishing general conditions under which conditional sampling can be achieved and drawing connections between monotone block triangular maps and optimal transport. Based on this theory, we then introduce a computational approach, called monotone generative adversarial networks (M-GANs), to learn suitable block triangular maps. Our algorithm uses only samples from the underlying joint probability measure and is hence likelihood-free. Numerical experiments with M-GAN demonstrate accurate sampling of conditional measures in synthetic examples, Bayesian inverse problems involving ordinary and partial differential equations, and probabilistic image in-painting.

   @article{baptista2023conditional,
     abbr = {arXiv},
     bibtex_show = {true},
     title = {Conditional Sampling with Monotone GANs: from Generative Models to Likelihood-Free Inference},
     author = {Baptista, Ricardo and Hosseini, Bamdad and Kovachki, Nikola B and Marzouk, Youssef M},
     journal = {CoRR},
     volume = {abs/2006.06755},
     year = {2023},
     html = {https://arxiv.org/abs/2006.06755},
     pdf = {https://arxiv.org/pdf/2006.06755.pdf}
   }

7. arXiv

   Score-based Diffusion Models in Function Space

   Lim, Jae H, Kovachki, Nikola B, Baptista, Ricardo, Beckham, Christopher, Azizzadenesheli, Kamyar, Kossaifi, Jean, Voleti, Vikram, Song, Jiaming, Kreis, Karsten, Kautz, Jan, Pal, Christopher, Vahdat, Arash, and Anandkumar, Anima

   CoRR 2023

   Abs Bib HTML PDF

   Diffusion models have recently emerged as a powerful framework for generative modeling. They consist of a forward process that perturbs input data with Gaussian white noise and a reverse process that learns a score function to generate samples by denoising. Despite their tremendous success, they are mostly formulated on finite-dimensional spaces, e.g. Euclidean, limiting their applications to many domains where the data has a functional form such as in scientific computing and 3D geometric data analysis. In this work, we introduce a mathematically rigorous framework called Denoising Diffusion Operators (DDOs) for training diffusion models in function space. In DDOs, the forward process perturbs input functions gradually using a Gaussian process. The generative process is formulated by integrating a function-valued Langevin dynamic. Our approach requires an appropriate notion of the score for the perturbed data distribution, which we obtain by generalizing denoising score matching to function spaces that can be infinite-dimensional. We show that the corresponding discretized algorithm generates accurate samples at a fixed cost that is independent of the data resolution. We theoretically and numerically verify the applicability of our approach on a set of problems, including generating solutions to the Navier-Stokes equation viewed as the push-forward distribution of forcings from a Gaussian Random Field (GRF).

   @article{lim2023score-based,
     abbr = {arXiv},
     bibtex_show = {true},
     title = {Score-based Diffusion Models in Function Space},
     author = {Lim, Jae H and Kovachki, Nikola B and Baptista, Ricardo and Beckham, Christopher and Azizzadenesheli, Kamyar and Kossaifi, Jean and Voleti, Vikram and Song, Jiaming and Kreis, Karsten and Kautz, Jan and Pal, Christopher and Vahdat, Arash and Anandkumar, Anima},
     journal = {CoRR},
     volume = {abs/2302.07400},
     year = {2023},
     html = {https://arxiv.org/abs/2302.07400},
     pdf = {https://arxiv.org/pdf/2302.07400.pdf}
   }

8. arXiv

   Multi-Grid Tensorized Fourier Neural Operator for High-Resolution PDEs

   Kossaifi, Jean, Kovachki, Nikola B, Azizzadenesheli, Kamyar, and Anandkumar, Anima

   CoRR 2023

   Abs Bib HTML PDF

   Memory complexity and data scarcity have so far prohibited learning solution operators of partial differential equations (PDEs) at high resolutions. We address these limitations by introducing a new data efficient and highly parallelizable operator learning approach with reduced memory requirement and better generalization, called multi-grid tensorized neural operator (MG-TFNO). MG-TFNO scales to large resolutions by leveraging local and global structures of full-scale, real-world phenomena, through a decomposition of both the input domain and the operator’s parameter space. Our contributions are threefold: i) we enable parallelization over input samples with a novel multi-grid-based domain decomposition, ii) we represent the parameters of the model in a high-order latent subspace of the Fourier domain, through a global tensor factorization, resulting in an extreme reduction in the number of parameters and improved generalization, and iii) we propose architectural improvements to the backbone FNO. Our approach can be used in any operator learning setting. We demonstrate superior performance on the turbulent Navier-Stokes equations where we achieve less than half the error with over 150x compression. The tensorization combined with the domain decomposition, yields over 150x reduction in the number of parameters and 7x reduction in the domain size without losses in accuracy, while slightly enabling parallelism.

   @article{kossaifi2023multi-grid,
     abbr = {arXiv},
     bibtex_show = {true},
     title = {Multi-Grid Tensorized Fourier Neural Operator for High-Resolution PDEs},
     author = {Kossaifi, Jean and Kovachki, Nikola B and Azizzadenesheli, Kamyar and Anandkumar, Anima},
     journal = {CoRR},
     volume = {abs/2310.00120},
     year = {2023},
     html = {https://arxiv.org/abs/2310.00120},
     pdf = {https://arxiv.org/pdf/2310.00120.pdf}
   }

9. arXiv

   Neural Operators for Accelerating Scientific Simulations and Design

   Azizzadenesheli, Kamyar, Kovachki, Nikola B, Li, Zongyi, Liu-Schiaffini, Miguel, Kossaifi, Jean, and Anandkumar, Anima

   CoRR 2023

   Abs Bib HTML PDF

   Scientific discovery and engineering design are currently limited by the time and cost of physical experiments, selected mostly through trial-and-error and intuition that require deep domain expertise. Numerical simulations present an alternative to physical experiments but are usually infeasible for complex real-world domains due to the computational requirements of existing numerical methods. Artificial intelligence (AI) presents a potential paradigm shift by developing fast data-driven surrogate models. In particular, an AI framework, known as neural operators, presents a principled framework for learning mappings between functions defined on continuous domains, e.g., spatiotemporal processes and partial differential equations (PDE). They can extrapolate and predict solutions at new locations unseen during training, i.e., perform zero-shot super-resolution. Neural operators can augment or even replace existing simulators in many applications, such as computational fluid dynamics, weather forecasting, and material modeling, while being 4-5 orders of magnitude faster. Further, neural operators can be integrated with physics and other domain constraints enforced at finer resolutions to obtain high-fidelity solutions and good generalization. Since neural operators are differentiable, they can directly optimize parameters for inverse design and other inverse problems. We believe that neural operators present a transformative approach to simulation and design, enabling rapid research and development.

   @article{azizzadenesheli2023neural,
     abbr = {arXiv},
     bibtex_show = {true},
     author = {Azizzadenesheli, Kamyar and Kovachki, Nikola B and Li, Zongyi and Liu-Schiaffini, Miguel and Kossaifi, Jean and Anandkumar, Anima},
     title = {Neural Operators for Accelerating Scientific Simulations and Design},
     journal = {CoRR},
     volume = {abs/2003.03485},
     year = {2023},
     html = {https://arxiv.org/abs/2309.15325},
     pdf = {https://arxiv.org/pdf/2309.15325.pdf}
   }

10. arXiv

    Tipping Point Forecasting in Non-Stationary Dynamics on Function Spaces

    Liu-Schiaffini, Miguel, Singer, Clare E, Kovachki, Nikola B, Schneider, Tapio, Azizzadenesheli, Kamyar, and Anandkumar, Anima

    CoRR 2023

    Abs Bib HTML PDF

    Tipping points are abrupt, drastic, and often irreversible changes in the evolution of non-stationary and chaotic dynamical systems. For instance, increased greenhouse gas concentrations are predicted to lead to drastic decreases in low cloud cover, referred to as a climatological tipping point. In this paper, we learn the evolution of such non-stationary dynamical systems using a novel recurrent neural operator (RNO), which learns mappings between function spaces. After training RNO on only the pre-tipping dynamics, we employ it to detect future tipping points using an uncertainty-based approach. In particular, we propose a conformal prediction framework to forecast tipping points by monitoring deviations from physics constraints (such as conserved quantities and partial differential equations), enabling forecasting of these abrupt changes along with a rigorous measure of uncertainty. We illustrate our proposed methodology on non-stationary ordinary and partial differential equations, such as the Lorenz-63 and Kuramoto-Sivashinsky equations. We also apply our methods to forecast a climate tipping point in stratocumulus cloud cover. In our experiments, we demonstrate that even partial or approximate physics constraints can be used to accurately forecast future tipping points.

    @article{liu-schiaffini2023tipping,
      abbr = {arXiv},
      bibtex_show = {true},
      author = {Liu-Schiaffini, Miguel and Singer, Clare E and Kovachki, Nikola B and Schneider, Tapio and Azizzadenesheli, Kamyar and Anandkumar, Anima},
      title = {Tipping Point Forecasting in Non-Stationary Dynamics on Function Spaces},
      journal = {CoRR},
      volume = {abs/2308.08794},
      year = {2023},
      html = {https://arxiv.org/abs/2308.08794},
      pdf = {https://arxiv.org/pdf/2308.08794.pdf}
    }

2022

1. JMPS

   A Learning-based Multiscale Method and its Application to Inelastic Impact Problems

   Liu, Burigede, Kovachki, Nikola B, Li, Zongyi, Azizzadenesheli, Kamyar, Anandkumar, Anima, Stuart, Andrew M, and Bhattacharya, Kaushik

   Journal of the Mechanics and Physics of Solids 2022

   Abs Bib HTML PDF

   The macroscopic properties of materials that we observe and exploit in engineering application result from complex interactions between physics at multiple length and time scales: electronic, atomistic, defects, domains etc. Multiscale modeling seeks to understand these interactions by exploiting the inherent hierarchy where the behavior at a coarser scale regulates and averages the behavior at a finer scale. This requires the repeated solution of computationally expensive finer-scale models, and often a priori knowledge of those aspects of the finer-scale behavior that affect the coarser scale (order parameters, state variables, descriptors, etc.). We address this challenge in a two-scale setting where we learn the fine-scale behavior from off-line calculations and then use the learnt behavior directly in coarse scale calculations. The approach builds on the recent success of deep neural networks by combining their approximation power in high dimensions with ideas from model reduction. It results in a neural network approximation that has high fidelity, is computationally inexpensive, is independent of the need for a priori knowledge, and can be used directly in the coarse scale calculations. We demonstrate the approach on problems involving the impact of magnesium, a promising light-weight structural and protective material.

   @article{liu2022alearningbased,
     abbr = {JMPS},
     bibtex_show = {true},
     title = {A Learning-based Multiscale Method and its Application to Inelastic Impact Problems},
     journal = {Journal of the Mechanics and Physics of Solids},
     volume = {158},
     year = {2022},
     html = {https://www.sciencedirect.com/science/article/pii/S0022509621002982},
     pdf = {https://arxiv.org/pdf/2102.07256.pdf},
     author = {Liu, Burigede and Kovachki, Nikola B and Li, Zongyi and Azizzadenesheli, Kamyar and Anandkumar, Anima and Stuart, Andrew M and Bhattacharya, Kaushik},
     selected = {true}
   }

2. MM

   Multiscale Modeling of Materials: Computing, Data Science, Uncertainty and Goal-oriented Optimization

   Kovachki, Nikola B, Liu, Burigede, Sun, Xingsheng, Zhou, Hao, Bhattacharya, Kaushik, Ortiz, Michael, and Stuart, Andrew M

   Mechanics of Materials 2022

   Abs Bib HTML PDF

   The recent decades have seen various attempts at accelerating the process of developing materials targeted towards specific applications. The performance required for a particular application leads to the choice of a particular material system whose properties are optimized by manipulating its underlying microstructure through processing. The specific configuration of the structure is then designed by characterizing the material in detail, and using this characterization along with physical principles in system level simulations and optimization. These have been advanced by multiscale modeling of materials, high-throughput experimentations, materials data-bases, topology optimization and other ideas. Still, developing materials for extreme applications involving large deformation, high strain rates and high temperatures remains a challenge. This article reviews a number of recent methods that advance the goal of designing materials targeted by specific applications.

   @article{kovachki2022multiscale,
     abbr = {MM},
     bibtex_show = {true},
     title = {Multiscale Modeling of Materials: Computing, Data Science, Uncertainty and Goal-oriented Optimization},
     author = {Kovachki, Nikola B and Liu, Burigede and Sun, Xingsheng and Zhou, Hao and Bhattacharya, Kaushik and Ortiz, Michael and Stuart, Andrew M},
     journal = {Mechanics of Materials},
     volume = {165},
     year = {2022},
     html = {https://www.sciencedirect.com/science/article/abs/pii/S0167663621003689},
     pdf = {https://arxiv.org/pdf/2104.05918.pdf}
   }

3. NeurIPS

   Learning Dissipative Dynamics in Chaotic Systems

   Li, Zongyi, Kovachki, Nikola B, Azizzadenesheli, Kamyar, Liu, Burigede, Bhattacharya, Kaushik, Stuart, Andrew M, and Anandkumar, Anima

   In Advances in Neural Information Processing Systems (NeurIPS) 2022

   Abs Bib HTML PDF

   Chaotic systems are notoriously challenging to predict because of their sensitivity to perturbations and errors due to time stepping. Despite this unpredictable behavior, for many dissipative systems the statistics of the long term trajectories are governed by an invariant measure supported on a set, known as the global attractor; for many problems this set is finite dimensional, even if the state space is infinite dimensional. For Markovian systems, the statistical properties of long-term trajectories are uniquely determined by the solution operator that maps the evolution of the system over arbitrary positive time increments. In this work, we propose a machine learning framework to learn the underlying solution operator for dissipative chaotic systems, showing that the resulting learned operator accurately captures short-time trajectories and long-time statistical behavior. Using this framework, we are able to predict various statistics of the invariant measure for the turbulent Kolmogorov Flow dynamics with Reynolds numbers up to 5000.

   @inproceedings{li2022learning,
     abbr = {NeurIPS},
     bibtex_show = {true},
     title = {Learning Dissipative Dynamics in Chaotic Systems},
     author = {Li, Zongyi and Kovachki, Nikola B and Azizzadenesheli, Kamyar and Liu, Burigede and Bhattacharya, Kaushik and Stuart, Andrew M and Anandkumar, Anima},
     booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
     volume = {36},
     year = {2022},
     html = {https://proceedings.neurips.cc/paper_files/paper/2022/hash/6ad68277e27b42c60ac228c9859fc1a2-Abstract-Conference.html},
     pdf = {https://proceedings.neurips.cc/paper_files/paper/2022/file/6ad68277e27b42c60ac228c9859fc1a2-Paper-Conference.pdf}
   }

2021

1. ICLR

   Fourier Neural Operator for Parametric Partial Differential Equations

   Li, Zongyi, Kovachki, Nikola B, Azizzadenesheli, Kamyar, Liu, Burigede, Bhattacharya, Kaushik, Stuart, Andrew M, and Anandkumar, Anima

   In International Conference on Learning Representations (ICLR) 2021

   Abs Bib HTML PDF

   The classical development of neural networks has primarily focused on learning mappings between finite-dimensional Euclidean spaces. Recently, this has been generalized to neural operators that learn mappings between function spaces. For partial differential equations (PDEs), neural operators directly learn the mapping from any functional parametric dependence to the solution. Thus, they learn an entire family of PDEs, in contrast to classical methods which solve one instance of the equation. In this work, we formulate a new neural operator by parameterizing the integral kernel directly in Fourier space, allowing for an expressive and efficient architecture. We perform experiments on Burgers’ equation, Darcy flow, and Navier-Stokes equation. The Fourier neural operator is the first ML-based method to successfully model turbulent flows with zero-shot super-resolution. It is up to three orders of magnitude faster compared to traditional PDE solvers. Additionally, it achieves superior accuracy compared to previous learning-based solvers under fixed resolution.

   @inproceedings{li2021fourier,
     abbr = {ICLR},
     bibtex_show = {true},
     author = {Li, Zongyi and Kovachki, Nikola B and Azizzadenesheli, Kamyar and Liu, Burigede and Bhattacharya, Kaushik and Stuart, Andrew M and Anandkumar, Anima},
     title = {Fourier Neural Operator for Parametric Partial Differential Equations},
     booktitle = {International Conference on Learning Representations (ICLR)},
     volume = {9},
     publisher = {OpenReview.net},
     year = {2021},
     pdf = {https://openreview.net/pdf?id=c8P9NQVtmnO},
     html = {https://iclr.cc/virtual/2021/poster/3281},
     selected = {true}
   }

2. SMAI-JCM

   Model Reduction and Neural Networks for Parametric PDEs

   Bhattacharya, Kaushik, Hosseini, Bamdad, Kovachki, Nikola B, and Stuart, Andrew M

   The SMAI journal of computational mathematics 2021

   Abs Bib HTML PDF

   We develop a general framework for data-driven approximation of input-output maps between infinite-dimensional spaces. The proposed approach is motivated by the recent successes of neural networks and deep learning, in combination with ideas from model reduction. This combination results in a neural network approximation which, in principle, is defined on infinite-dimensional spaces and, in practice, is robust to the dimension of finite-dimensional approximations of these spaces required for computation. For a class of input-output maps, and suitably chosen probability measures on the inputs, we prove convergence of the proposed approximation methodology. We also include numerical experiments which demonstrate the effectiveness of the method, showing convergence and robustness of the approximation scheme with respect to the size of the discretization, and compare it with existing algorithms from the literature; our examples include the mapping from coefficient to solution in a divergence form elliptic partial differential equation (PDE) problem, and the solution operator for viscous Burgers’ equation.

   @article{bhattacharya2021model,
     abbr = {SMAI-JCM},
     bibtex_show = {true},
     author = {Bhattacharya, Kaushik and Hosseini, Bamdad and Kovachki, Nikola B and Stuart, Andrew M},
     title = {Model Reduction and Neural Networks for Parametric PDEs},
     journal = {The SMAI journal of computational mathematics},
     publisher = {Soci\'et\'e de Math\'ematiques Appliqu\'ees et Industrielles},
     volume = {7},
     year = {2021},
     html = {https://smai-jcm.centre-mersenne.org/articles/10.5802/smai-jcm.74/},
     pdf = {https://smai-jcm.centre-mersenne.org/item/10.5802/smai-jcm.74.pdf},
     selected = {true}
   }

3. arXiv

   Physics-Informed Neural Operator for Learning Partial Differential Equations

   Li, Zongyi, Zheng, Hongkai, Kovachki, Nikola B, Jin, David, Chen, Haoxuan, Liu, Burigede, Azizzadenesheli, Kamyar, and Anandkumar, Anima

   CoRR 2021

   Abs Bib HTML PDF

   In this paper, we propose physics-informed neural operators (PINO) that combine training data and physics constraints to learn the solution operator of a given family of parametric Partial Differential Equations (PDE). PINO is the first hybrid approach incorporating data and PDE constraints at different resolutions to learn the operator. Specifically, in PINO, we combine coarse-resolution training data with PDE constraints imposed at a higher resolution. The resulting PINO model can accurately approximate the ground-truth solution operator for many popular PDE families and shows no degradation in accuracy even under zero-shot super-resolution, i.e., being able to predict beyond the resolution of training data. PINO uses the Fourier neural operator (FNO) framework that is guaranteed to be a universal approximator for any continuous operator and discretization-convergent in the limit of mesh refinement. By adding PDE constraints to FNO at a higher resolution, we obtain a high-fidelity reconstruction of the ground-truth operator. Moreover, PINO succeeds in settings where no training data is available and only PDE constraints are imposed, while previous approaches, such as the Physics-Informed Neural Network (PINN), fail due to optimization challenges, e.g., in multi-scale dynamic systems such as Kolmogorov flows.

   @article{li2021physicsinformed,
     abbr = {arXiv},
     bibtex_show = {true},
     title = {Physics-Informed Neural Operator for Learning Partial Differential Equations},
     author = {Li, Zongyi and Zheng, Hongkai and Kovachki, Nikola B and Jin, David and Chen, Haoxuan and Liu, Burigede and Azizzadenesheli, Kamyar and Anandkumar, Anima},
     journal = {CoRR},
     volume = {abs/2111.03794},
     year = {2021},
     html = {https://arxiv.org/abs/2111.03794},
     pdf = {https://arxiv.org/pdf/2111.03794.pdf}
   }

4. JMLR

   On Universal Approximation and Error Bounds for Fourier Neural Operators

   Kovachki, Nikola B, Lanthaler, Samuel, and Mishra, Siddhartha

   Journal of Machine Learning Research 2021

   Abs Bib HTML PDF

   Fourier neural operators (FNOs) have recently been proposed as an effective framework for learning operators that map between infinite-dimensional spaces. We prove that FNOs are universal, in the sense that they can approximate any continuous operator to desired accuracy. Moreover, we suggest a mechanism by which FNOs can approximate operators associated with PDEs efficiently. Explicit error bounds are derived to show that the size of the FNO, approximating operators associated with a Darcy type elliptic PDE and with the incompressible Navier-Stokes equations of fluid dynamics, only increases sub (log)-linearly in terms of the reciprocal of the error. Thus, FNOs are shown to efficiently approximate operators arising in a large class of PDEs.

   @article{kovachki2021onuniversal,
     abbr = {JMLR},
     bibtex_show = {true},
     title = {On Universal Approximation and Error Bounds for Fourier Neural Operators},
     author = {Kovachki, Nikola B and Lanthaler, Samuel and Mishra, Siddhartha},
     journal = {Journal of Machine Learning Research},
     year = {2021},
     volume = {22},
     number = {290},
     html = {https://jmlr.org/papers/v22/21-0806.html},
     pdf = {https://jmlr.org/papers/volume22/21-0806/21-0806.pdf}
   }

5. JMLR

   Continuous Time Analysis of Momentum Methods

   Kovachki, Nikola B, and Stuart, Andrew M

   Journal of Machine Learning Research 2021

   Abs Bib HTML PDF

   Gradient descent-based optimization methods underpin the parameter training of neural networks, and hence comprise a significant component in the impressive test results found in a number of applications. Introducing stochasticity is key to their success in practical problems, and there is some understanding of the role of stochastic gradient descent in this context. Momentum modifications of gradient descent such as Polyak’s Heavy Ball method (HB) and Nesterov’s method of accelerated gradients (NAG), are also widely adopted. In this work our focus is on understanding the role of momentum in the training of neural networks, concentrating on the common situation in which the momentum contribution is fixed at each step of the algorithm. To expose the ideas simply we work in the deterministic setting. Our approach is to derive continuous time approximations of the discrete algorithms; these continuous time approximations provide insights into the mechanisms at play within the discrete algorithms. We prove three such approximations. Firstly we show that standard implementations of fixed momentum methods approximate a time-rescaled gradient descent flow, asymptotically as the learning rate shrinks to zero; this result does not distinguish momentum methods from pure gradient descent, in the limit of vanishing learning rate. We then proceed to prove two results aimed at understanding the observed practical advantages of fixed momentum methods over gradient descent, when implemented in the non-asymptotic regime with fixed small, but non-zero, learning rate. We achieve this by proving approximations to continuous time limits in which the small but fixed learning rate appears as a parameter; this is known as the method of modified equations in the numerical analysis literature, recently rediscovered as the high resolution ODE approximation in the machine learning context. In our second result we show that the momentum method is approximated by a continuous time gradient flow, with an additional momentum-dependent second order time-derivative correction, proportional to the learning rate; this may be used to explain the stabilizing effect of momentum algorithms in their transient phase. Furthermore in a third result we show that the momentum methods admit an exponentially attractive invariant manifold on which the dynamics reduces, approximately, to a gradient flow with respect to a modified loss function, equal to the original loss function plus a small perturbation proportional to the learning rate; this small correction provides convexification of the loss function and encodes additional robustness present in momentum methods, beyond the transient phase.

   @article{kovachki2021continuous,
     abbr = {JMLR},
     bibtex_show = {true},
     author = {Kovachki, Nikola B and Stuart, Andrew M},
     title = {Continuous Time Analysis of Momentum Methods},
     journal = {Journal of Machine Learning Research},
     year = {2021},
     volume = {22},
     number = {17},
     html = {https://www.jmlr.org/papers/v22/19-466.html},
     pdf = {https://www.jmlr.org/papers/volume22/19-466/19-466.pdf}
   }

2020

1. NeurIPS

   Multipole Graph Neural Operator for Parametric Partial Differential Equations

   Li, Zongyi, Kovachki, Nikola B, Azizzadenesheli, Kamyar, Liu, Burigede, Stuart, Andrew M, Bhattacharya, Kaushik, and Anandkumar, Anima

   In Advances in Neural Information Processing Systems (NeurIPS) 2020

   Abs Bib HTML PDF

   One of the main challenges in using deep learning-based methods for simulating physical systems and solving partial differential equations (PDEs) is formulating physics-based data in the desired structure for neural networks. Graph neural networks (GNNs) have gained popularity in this area since graphs offer a natural way of modeling particle interactions and provide a clear way of discretizing the continuum models. However, the graphs constructed for approximating such tasks usually ignore long-range interactions due to unfavorable scaling of the computational complexity with respect to the number of nodes. The errors due to these approximations scale with the discretization of the system, thereby not allowing for generalization under mesh-refinement. Inspired by the classical multipole methods, we purpose a novel multi-level graph neural network framework that captures interaction at all ranges with only linear complexity. Our multi-level formulation is equivalent to recursively adding inducing points to the kernel matrix, unifying GNNs with multi-resolution matrix factorization of the kernel. Experiments confirm our multi-graph network learns discretization-invariant solution operators to PDEs and can be evaluated in linear time.

   @inproceedings{li2020multipole,
     abbr = {NeurIPS},
     bibtex_show = {true},
     author = {Li, Zongyi and Kovachki, Nikola B and Azizzadenesheli, Kamyar and Liu, Burigede and Stuart, Andrew M and Bhattacharya, Kaushik and Anandkumar, Anima},
     booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
     title = {Multipole Graph Neural Operator for Parametric Partial Differential Equations},
     pdf = {https://proceedings.neurips.cc/paper/2020/file/4b21cf96d4cf612f239a6c322b10c8fe-Paper.pdf},
     html = {https://papers.nips.cc/paper/2020/hash/4b21cf96d4cf612f239a6c322b10c8fe-Abstract.html},
     volume = {33},
     year = {2020}
   }

2. arXiv

   Neural Operator: Graph Kernel Network for Partial Differential Equations

   Li, Zongyi, Kovachki, Nikola B, Azizzadenesheli, Kamyar, Liu, Burigede, Stuart, Andrew M, Bhattacharya, Kaushik, and Anandkumar, Anima

   CoRR 2020

   Abs Bib HTML PDF

   The classical development of neural networks has been primarily for mappings between a finite-dimensional Euclidean space and a set of classes, or between two finite-dimensional Euclidean spaces. The purpose of this work is to generalize neural networks so that they can learn mappings between infinite-dimensional spaces (operators). The key innovation in our work is that a single set of network parameters, within a carefully designed network architecture, may be used to describe mappings between infinite-dimensional spaces and between different finite-dimensional approximations of those spaces. We formulate approximation of the infinite-dimensional mapping by composing nonlinear activation functions and a class of integral operators. The kernel integration is computed by message passing on graph networks. This approach has substantial practical consequences which we will illustrate in the context of mappings between input data to partial differential equations (PDEs) and their solutions. In this context, such learned networks can generalize among different approximation methods for the PDE (such as finite difference or finite element methods) and among approximations corresponding to different underlying levels of resolution and discretization. Experiments confirm that the proposed graph kernel network does have the desired properties and show competitive performance compared to the state of the art solvers.

   @article{li2020neural,
     abbr = {arXiv},
     bibtex_show = {true},
     author = {Li, Zongyi and Kovachki, Nikola B and Azizzadenesheli, Kamyar and Liu, Burigede and Stuart, Andrew M and Bhattacharya, Kaushik and Anandkumar, Anima},
     title = {Neural Operator: Graph Kernel Network for Partial Differential Equations},
     journal = {CoRR},
     volume = {abs/2003.03485},
     year = {2020},
     html = {https://arxiv.org/abs/2003.03485},
     pdf = {https://arxiv.org/pdf/2003.03485.pdf}
   }

2019

1. JCTC

   Regression Clustering for Improved Accuracy and Training Costs with Molecular-Orbital-Based Machine Learning

   Cheng, Lixue, Kovachki, Nikola B, Welborn, Matthew, and Miller, Thomas F

   Journal of Chemical Theory and Computation 2019

   Abs Bib HTML PDF

   Machine learning (ML) in the representation of molecular-orbital-based (MOB) features has been shown to be an accurate and transferable approach to the prediction of post-Hartree–Fock correlation energies. Previous applications of MOB-ML employed Gaussian Process Regression (GPR), which provides good prediction accuracy with small training sets; however, the cost of GPR training scales cubically with the amount of data and becomes a computational bottleneck for large training sets. In the current work, we address this problem by introducing a clustering/regression/classification implementation of MOB-ML. In the first step, regression clustering (RC) is used to partition the training data to best fit an ensemble of linear regression (LR) models; in the second step, each cluster is regressed independently, using either LR or GPR; and in the third step, a random forest classifier (RFC) is trained for the prediction of cluster assignments based on MOB feature values. Upon inspection, RC is found to recapitulate chemically intuitive groupings of the frontier molecular orbitals, and the combined RC/LR/RFC and RC/GPR/RFC implementations of MOB-ML are found to provide good prediction accuracy with greatly reduced wall-clock training times. For a data set of thermalized (350 K) geometries of 7211 organic molecules of up to seven heavy atoms (QM7b-T), both RC/LR/RFC and RC/GPR/RFC reach chemical accuracy (1 kcal/mol prediction error) with only 300 training molecules, while providing 35000-fold and 4500-fold reductions in the wall-clock training time, respectively, compared to MOB-ML without clustering. The resulting models are also demonstrated to retain transferability for the prediction of large-molecule energies with only small-molecule training data. Finally, it is shown that capping the number of training data points per cluster leads to further improvements in prediction accuracy with negligible increases in wall-clock training time.

   @article{cheng2019regression,
     abbr = {JCTC},
     bibtex_show = {true},
     author = {Cheng, Lixue and Kovachki, Nikola B and Welborn, Matthew and Miller, Thomas F},
     title = {Regression Clustering for Improved Accuracy and Training Costs with Molecular-Orbital-Based Machine Learning},
     journal = {Journal of Chemical Theory and Computation},
     volume = {15},
     number = {12},
     year = {2019},
     html = {https://pubs.acs.org/doi/abs/10.1021/acs.jctc.9b00884},
     pdf = {regression_clustering.pdf}
   }

2. IP

   Ensemble Kalman Inversion: a Derivative-free Technique for Machine Learning Tasks

   Kovachki, Nikola B, and Stuart, Andrew M

   Inverse Problems 2019

   Abs Bib HTML PDF

   The standard probabilistic perspective on machine learning gives rise to empirical risk-minimization tasks that are frequently solved by stochastic gradient descent (SGD) and variants thereof. We present a formulation of these tasks as classical inverse or filtering problems and, furthermore, we propose an efficient, gradient-free algorithm for finding a solution to these problems using ensemble Kalman inversion (EKI). The method is inherently parallelizable and is applicable to problems with non-differentiable loss functions, for which back-propagation is not possible. Applications of our approach include offline and online supervised learning with deep neural networks, as well as graph-based semi-supervised learning. The essence of the EKI procedure is an ensemble based approximate gradient descent in which derivatives are replaced by differences from within the ensemble. We suggest several modifications to the basic method, derived from empirically successful heuristics developed in the context of SGD. Numerical results demonstrate wide applicability and robustness of the proposed algorithm.

   @article{kovachki2019ensemble,
     abbr = {IP},
     bibtex_show = {true},
     year = {2019},
     publisher = {IOP Publishing},
     volume = {35},
     number = {9},
     journal = {Inverse Problems},
     author = {Kovachki, Nikola B and Stuart, Andrew M},
     title = {Ensemble Kalman Inversion: a Derivative-free Technique for Machine Learning Tasks},
     html = {https://iopscience.iop.org/article/10.1088/1361-6420/ab1c3a},
     pdf = {https://arxiv.org/pdf/1808.03620.pdf}
   }