{"title":"A tutorial on automatic differentiation with complex numbers","authors":"Nicholas Krämer","doi":"arxiv-2409.06752","DOIUrl":null,"url":null,"abstract":"Automatic differentiation is everywhere, but there exists only minimal\ndocumentation of how it works in complex arithmetic beyond stating \"derivatives\nin $\\mathbb{C}^d$\" $\\cong$ \"derivatives in $\\mathbb{R}^{2d}$\" and, at best,\nshallow references to Wirtinger calculus. Unfortunately, the equivalence\n$\\mathbb{C}^d \\cong \\mathbb{R}^{2d}$ becomes insufficient as soon as we need to\nderive custom gradient rules, e.g., to avoid differentiating \"through\"\nexpensive linear algebra functions or differential equation simulators. To\ncombat such a lack of documentation, this article surveys forward- and\nreverse-mode automatic differentiation with complex numbers, covering topics\nsuch as Wirtinger derivatives, a modified chain rule, and different gradient\nconventions while explicitly avoiding holomorphicity and the Cauchy--Riemann\nequations (which would be far too restrictive). To be precise, we will derive,\nexplain, and implement a complex version of Jacobian-vector and vector-Jacobian\nproducts almost entirely with linear algebra without relying on complex\nanalysis or differential geometry. This tutorial is a call to action, for users\nand developers alike, to take complex values seriously when implementing custom\ngradient propagation rules -- the manuscript explains how.","PeriodicalId":501162,"journal":{"name":"arXiv - MATH - Numerical Analysis","volume":"171 1","pages":""},"PeriodicalIF":0.0000,"publicationDate":"2024-09-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - MATH - Numerical Analysis","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2409.06752","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0
Abstract
Automatic differentiation is everywhere, but there exists only minimal
documentation of how it works in complex arithmetic beyond stating "derivatives
in $\mathbb{C}^d$" $\cong$ "derivatives in $\mathbb{R}^{2d}$" and, at best,
shallow references to Wirtinger calculus. Unfortunately, the equivalence
$\mathbb{C}^d \cong \mathbb{R}^{2d}$ becomes insufficient as soon as we need to
derive custom gradient rules, e.g., to avoid differentiating "through"
expensive linear algebra functions or differential equation simulators. To
combat such a lack of documentation, this article surveys forward- and
reverse-mode automatic differentiation with complex numbers, covering topics
such as Wirtinger derivatives, a modified chain rule, and different gradient
conventions while explicitly avoiding holomorphicity and the Cauchy--Riemann
equations (which would be far too restrictive). To be precise, we will derive,
explain, and implement a complex version of Jacobian-vector and vector-Jacobian
products almost entirely with linear algebra without relying on complex
analysis or differential geometry. This tutorial is a call to action, for users
and developers alike, to take complex values seriously when implementing custom
gradient propagation rules -- the manuscript explains how.