Automatic Differentiation in Modern Machine Learning: Mathematical Foundations, Algorithmic Modes, Implementation Strategies, and Contemporary Challenges: A Comprehensive Review Article
DOI:
https://doi.org/10.61166/interkoneksi.v4i2.99Keywords:
automatic differentiation, algorithmic differentiation, backpropagation, computational graph, Jacobian–vector product, probabilistic programmingAbstract
Automatic differentiation (AD) is the algorithmic backbone of nearly every gradient-based method used in contemporary machine learning, ranging from stochastic gradient descent for deep neural networks and Adam-style adaptive optimizers, through gradient-based Markov chain Monte Carlo and variational inference in probabilistic programs, to end-to-end differentiable simulation in scientific machine learning. AD evaluates derivatives of numerical functions defined by computer programs to floating-point precision, and, in its reverse mode, at a computational cost that is bounded by a small constant multiple of the cost of evaluating the function itself, independently of the number of inputs. This review synthesizes the theoretical foundations, algorithmic modes, and implementation strategies of AD; positions AD unambiguously against the two techniques with which it is most frequently confused — namely, numerical (finite-difference) differentiation, which is cheap but suffers from truncation and round-off error, and symbolic differentiation, which is exact but subject to expression swell for programs with control flow; and formalises the chain rule as the unifying mechanism that both the forward and reverse accumulation modes exploit. The complexity trade-offs between the two modes are analysed with reference to the shape of the Jacobian to be evaluated, and their combination for Hessian–vector products, mixed-mode, and higher-order derivatives is discussed. The historical convergence of reverse-mode AD with the backpropagation algorithm is traced from its independent origins in numerical rounding-error analysis (Linnainmaa, 1970), control theory (Werbos, 1974), general-purpose gradient compilation (Speelpenning, 1980), and connectionist learning (Rumelhart et al., 1986). The article further surveys the three principal implementation strategies — operator overloading, source-to-source transformation, and tracing with just-in-time compilation — as realised in the frameworks that dominate current practice, TensorFlow (Abadi et al., 2016), PyTorch (Paszke et al., 2019), and JAX (Bradbury et al., 2018). It closes with a discussion of open problems: correctness at points of non-differentiability, memory-efficient reverse mode through checkpointing, sound differentiation of stochastic and probabilistic programs, and formal verification of AD in compiler-embedded settings. The article integrates and updates prior treatments (Griewank & Walther, 2008; Baydin et al., 2018; Margossian, 2019) and reports on 2022–2024 developments in Enzyme, JAX, and differentiable programming for differential equations (Hückelheim et al., 2023; Sapienza et al., 2024).
Downloads
References
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., Kudlur, M., Levenberg, J., Monga, R., Moore, S., Murray, D. G., Steiner, B., Tucker, P., Vasudevan, V., Warden, P., … Zheng, X. (2016). TensorFlow: A system for large-scale machine learning. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’16) (pp. 265–283). USENIX Association.
Baydin, A. G., Pearlmutter, B. A., Radul, A. A., & Siskind, J. M. (2018). Automatic differentiation in machine learning: A survey. Journal of Machine Learning Research, 18(153), 1–43. https://www.jmlr.org/papers/volume18/17-468/17-468.pdf
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., & Zhang, Q. (2018). JAX: Composable transformations of Python+NumPy programs [Computer software]. Google Research. https://github.com/google/jax
Chen, R. T. Q., Rubanova, Y., Bettencourt, J., & Duvenaud, D. (2018). Neural ordinary differential equations. In Advances in Neural Information Processing Systems 31 (pp. 6571–6583). Curran Associates.
Griewank, A., & Walther, A. (2000). Algorithm 799: Revolve — An implementation of checkpointing for the reverse or adjoint mode of computational differentiation. ACM Transactions on Mathematical Software, 26(1), 19–45. https://doi.org/10.1145/347837.347846
Griewank, A., & Walther, A. (2008). Evaluating derivatives: Principles and techniques of algorithmic differentiation (2nd ed.). Society for Industrial and Applied Mathematics. https://doi.org/10.1137/1.9780898717761
Hückelheim, J., Menon, H., Moses, W., Christianson, B., Hovland, P., & Hascoët, L. (2023). Understanding automatic differentiation pitfalls. arXiv. https://arxiv.org/abs/2305.07546
Innes, M. (2018). Don’t unroll adjoint: Differentiating SSA-form programs. arXiv. https://arxiv.org/abs/1810.07951
Innes, M., Edelman, A., Fischer, K., Rackauckas, C., Saba, E., Shah, V. B., & Tebbutt, W. (2019). A differentiable programming system to bridge machine learning and scientific computing. arXiv. https://arxiv.org/abs/1907.07587
Laurel, J., Yang, R., Ugare, S., Nagel, R., Singh, G., & Misailovic, S. (2022). A general construction for abstract interpretation of higher-order automatic differentiation. Proceedings of the ACM on Programming Languages, 6(OOPSLA2), Article 174. https://doi.org/10.1145/3563324
Lew, A. K., Huot, M., Staton, S., & Mansinghka, V. K. (2023). ADEV: Sound automatic differentiation of expected values of probabilistic programs. Proceedings of the ACM on Programming Languages, 7(POPL), Article 121. https://doi.org/10.1145/3571198
Linnainmaa, S. (1970). The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors [Master’s thesis, University of Helsinki].
Linnainmaa, S. (1976). Taylor expansion of the accumulated rounding error. BIT Numerical Mathematics, 16(2), 146–160. https://doi.org/10.1007/BF01931367
Margossian, C. C. (2019). A review of automatic differentiation and its efficient implementation. WIREs Data Mining and Knowledge Discovery, 9(4), Article e1305. https://doi.org/10.1002/widm.1305
Martins, J. R. R. A., Sturdza, P., & Alonso, J. J. (2003). The complex-step derivative approximation. ACM Transactions on Mathematical Software, 29(3), 245–262. https://doi.org/10.1145/838250.838251
Moses, W., & Churavy, V. (2020). Instead of rewriting foreign code for machine learning, automatically synthesize fast gradients. In Advances in Neural Information Processing Systems 33 (pp. 12472–12485). Curran Associates.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., … Chintala, S. (2019). PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 (pp. 8024–8035). Curran Associates.
Pearlmutter, B. A. (1994). Fast exact multiplication by the Hessian. Neural Computation, 6(1), 147–160. https://doi.org/10.1162/neco.1994.6.1.147
Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019). Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378, 686–707. https://doi.org/10.1016/j.jcp.2018.10.045
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323(6088), 533–536. https://doi.org/10.1038/323533a0
Sapienza, F., Bolibar, J., Schäfer, F., Groenke, B., Pal, A., Boussange, V., Heimbach, P., & Pérez-Carrasco, R. (2024). Differentiable programming for differential equations: A review. arXiv. https://arxiv.org/abs/2406.09699
Speelpenning, B. (1980). Compiling fast partial derivatives of functions given by algorithms [Doctoral dissertation, University of Illinois at Urbana-Champaign]. Department of Computer Science Technical Report UIUCDCS-R-80-1002.
Van Merriënboer, B., Breuleux, O., Bergeron, A., & Lamblin, P. (2018). Tangent: Automatic differentiation using source-code transformation for dynamically typed array programming. In Advances in Neural Information Processing Systems 31 (pp. 6256–6265). Curran Associates.
Werbos, P. J. (1974). Beyond regression: New tools for prediction and analysis in the behavioral sciences [Doctoral dissertation, Harvard University].
Werbos, P. J. (1990). Backpropagation through time: What it does and how to do it. Proceedings of the IEEE, 78(10), 1550–1560. https://doi.org/10.1109/5.58337
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Mohammad Qasam Adil, Mirwais Ghafare
This work is licensed under a Creative Commons Attribution 4.0 International License.
