💭 My experience as an Applied Scientist Intern at Amazon
Published:
Posts by Tags
Amazon
Bayesian inference
♾️ Infinite Widths Part I: Neural Networks as Gaussian Processes
Published:
This is the first post of a short series on the infinite-width limits of deep neural networks (DNNs). We start by reviewing the correspondence between neural networks and Gaussian Processes (GPs).
Bayesian neural networks
♾️ Infinite Widths Part I: Neural Networks as Gaussian Processes
Published:
This is the first post of a short series on the infinite-width limits of deep neural networks (DNNs). We start by reviewing the correspondence between neural networks and Gaussian Processes (GPs).
Fisher information
Thermodynamic Natural Gradient Descent
Published:
I recently came across this paper Thermodynamic Natural Gradient Descent by Normal Computing. I found it very interesting, so below is my brief take on it.
Predictive Coding as a 2nd-Order Method
Published:
📖 TL;DR: Predictive coding implicitly performs a 2nd-order weight update via 1st-order (gradient) updates on neurons that in some cases allow it to converge faster than backpropagation with standard stochastic gradient descent.
Gaussian processes
♾️ Infinite Widths Part I: Neural Networks as Gaussian Processes
Published:
This is the first post of a short series on the infinite-width limits of deep neural networks (DNNs). We start by reviewing the correspondence between neural networks and Gaussian Processes (GPs).
KAN
KANs Made Simple
Published:
Confused about the recent KAN: Kolmogorov-Arnold Networks? I was too, so here’s a minimal explanation that makes it easy to see the difference between KANs and multi-layer perceptrons (MLPs).
Kolmogorov-Arnold networks
KANs Made Simple
Published:
Confused about the recent KAN: Kolmogorov-Arnold Networks? I was too, so here’s a minimal explanation that makes it easy to see the difference between KANs and multi-layer perceptrons (MLPs).
Kolmogorov-Arnold representation theorem
KANs Made Simple
Published:
Confused about the recent KAN: Kolmogorov-Arnold Networks? I was too, so here’s a minimal explanation that makes it easy to see the difference between KANs and multi-layer perceptrons (MLPs).
Normal Computing
Thermodynamic Natural Gradient Descent
Published:
I recently came across this paper Thermodynamic Natural Gradient Descent by Normal Computing. I found it very interesting, so below is my brief take on it.
PhD
PhD Reflections
Published:
Having recently submitted my PhD thesis, I’ve been thinking a lot about my PhD experience. Here I would like to share some reflections. Needless to say that this is my own, biased experience, and PhDs can vary greatly depending on the field, lab, supervisor, etc.
applied scientist
backpropagation
Can We Scale Predictive Coding? or Why the Brain Might Be Much Wider Than Deep
Published:
📖 TL;DR: The gradients computed by predictive coding converge to backpropagation’s for much wider than deep networks (like the brain), under stable parameterisations.
Scaling Predictive Coding to 100+ Layer Networks
Published:
📖 TL;DR: We introduce \(\mu\)PC, a reparameterisation of predictive coding networks that enables stable training of 100+ layer ResNets on simple tasks with zero-shot hyperparameter transfer.
⛰️ The Energy Landscape of Predictive Coding Networks
Published:
📖 TL;DR: Predictive coding makes the loss landscape of feedforward neural networks more benign and robust to vanishing gradients.
Predictive Coding as a 2nd-Order Method
Published:
📖 TL;DR: Predictive coding implicitly performs a 2nd-order weight update via 1st-order (gradient) updates on neurons that in some cases allow it to converge faster than backpropagation with standard stochastic gradient descent.
central limit theorem
♾️ Infinite Widths Part I: Neural Networks as Gaussian Processes
Published:
This is the first post of a short series on the infinite-width limits of deep neural networks (DNNs). We start by reviewing the correspondence between neural networks and Gaussian Processes (GPs).
deep information propagation
♾️ Infinite Widths Part I: Neural Networks as Gaussian Processes
Published:
This is the first post of a short series on the infinite-width limits of deep neural networks (DNNs). We start by reviewing the correspondence between neural networks and Gaussian Processes (GPs).
deep neural networks
PhD Reflections
Published:
Having recently submitted my PhD thesis, I’ve been thinking a lot about my PhD experience. Here I would like to share some reflections. Needless to say that this is my own, biased experience, and PhDs can vary greatly depending on the field, lab, supervisor, etc.
Scaling Predictive Coding to 100+ Layer Networks
Published:
📖 TL;DR: We introduce \(\mu\)PC, a reparameterisation of predictive coding networks that enables stable training of 100+ layer ResNets on simple tasks with zero-shot hyperparameter transfer.
♾️ Infinite Widths (& Depths) Part III: The Maximal Update Parameterisation (\(\mu\)P)
Published:
This is the third and last post of a short series on the infinite-width limits of deep neural networks (DNNs). In Part I, we showed that the output of a random network becomes Gaussian distributed in the infinite-width limit. Part II went beyond initialisation and showed that infinitely wide nets trained with GD are basically kernel methods.
♾️ Infinite Widths Part II: The Neural Tangent Kernel
Published:
This is the second post of a short series on the infinite-width limits of deep neural networks (DNNs). Previously, we reviewed the correspondence between neural networks and Gaussian Processes (NNGP), showing that, as the number neurons in the hidden layers grows to infinity, the output of a random network becomes Gaussian distributed.
♾️ Infinite Widths Part I: Neural Networks as Gaussian Processes
Published:
This is the first post of a short series on the infinite-width limits of deep neural networks (DNNs). We start by reviewing the correspondence between neural networks and Gaussian Processes (GPs).
KANs Made Simple
Published:
Confused about the recent KAN: Kolmogorov-Arnold Networks? I was too, so here’s a minimal explanation that makes it easy to see the difference between KANs and multi-layer perceptrons (MLPs).
⛰️ The Energy Landscape of Predictive Coding Networks
Published:
📖 TL;DR: Predictive coding makes the loss landscape of feedforward neural networks more benign and robust to vanishing gradients.
Predictive Coding as a 2nd-Order Method
Published:
📖 TL;DR: Predictive coding implicitly performs a 2nd-order weight update via 1st-order (gradient) updates on neurons that in some cases allow it to converge faster than backpropagation with standard stochastic gradient descent.
depth-mup
Scaling Predictive Coding to 100+ Layer Networks
Published:
📖 TL;DR: We introduce \(\mu\)PC, a reparameterisation of predictive coding networks that enables stable training of 100+ layer ResNets on simple tasks with zero-shot hyperparameter transfer.
dynamical mean field theory
Can We Scale Predictive Coding? or Why the Brain Might Be Much Wider Than Deep
Published:
📖 TL;DR: The gradients computed by predictive coding converge to backpropagation’s for much wider than deep networks (like the brain), under stable parameterisations.
♾️ Infinite Widths (& Depths) Part III: The Maximal Update Parameterisation (\(\mu\)P)
Published:
This is the third and last post of a short series on the infinite-width limits of deep neural networks (DNNs). In Part I, we showed that the output of a random network becomes Gaussian distributed in the infinite-width limit. Part II went beyond initialisation and showed that infinitely wide nets trained with GD are basically kernel methods.
energy-based models
Energy-based Transformers
Published:
📖 TL;DR: Energy-based Transformers (EBTs) learn a scalar energy function parameterised by a transformer. Empirically, EBTs show promising scaling and reasoning properties on both language and vision tasks.
energy-based transformers
Energy-based Transformers
Published:
📖 TL;DR: Energy-based Transformers (EBTs) learn a scalar energy function parameterised by a transformer. Empirically, EBTs show promising scaling and reasoning properties on both language and vision tasks.
feature learning
♾️ Infinite Widths (& Depths) Part III: The Maximal Update Parameterisation (\(\mu\)P)
Published:
This is the third and last post of a short series on the infinite-width limits of deep neural networks (DNNs). In Part I, we showed that the output of a random network becomes Gaussian distributed in the infinite-width limit. Part II went beyond initialisation and showed that infinitely wide nets trained with GD are basically kernel methods.
gradient descent
⛰️ The Energy Landscape of Predictive Coding Networks
Published:
📖 TL;DR: Predictive coding makes the loss landscape of feedforward neural networks more benign and robust to vanishing gradients.
hyperparameter transfer
Can We Scale Predictive Coding? or Why the Brain Might Be Much Wider Than Deep
Published:
📖 TL;DR: The gradients computed by predictive coding converge to backpropagation’s for much wider than deep networks (like the brain), under stable parameterisations.
Scaling Predictive Coding to 100+ Layer Networks
Published:
📖 TL;DR: We introduce \(\mu\)PC, a reparameterisation of predictive coding networks that enables stable training of 100+ layer ResNets on simple tasks with zero-shot hyperparameter transfer.
♾️ Infinite Widths (& Depths) Part III: The Maximal Update Parameterisation (\(\mu\)P)
Published:
This is the third and last post of a short series on the infinite-width limits of deep neural networks (DNNs). In Part I, we showed that the output of a random network becomes Gaussian distributed in the infinite-width limit. Part II went beyond initialisation and showed that infinitely wide nets trained with GD are basically kernel methods.
implicit gradient descent dynamics
In-Context Learning Demystified?
Published:
📖 TL;DR: a transformer block implicitly uses the input context to modify its MLP weights.
in-context learning
In-Context Learning Demystified?
Published:
📖 TL;DR: a transformer block implicitly uses the input context to modify its MLP weights.
industry
inference as optimisation
Energy-based Transformers
Published:
📖 TL;DR: Energy-based Transformers (EBTs) learn a scalar energy function parameterised by a transformer. Empirically, EBTs show promising scaling and reasoning properties on both language and vision tasks.
inference learning
Scaling Predictive Coding to 100+ Layer Networks
Published:
📖 TL;DR: We introduce \(\mu\)PC, a reparameterisation of predictive coding networks that enables stable training of 100+ layer ResNets on simple tasks with zero-shot hyperparameter transfer.
⛰️ The Energy Landscape of Predictive Coding Networks
Published:
📖 TL;DR: Predictive coding makes the loss landscape of feedforward neural networks more benign and robust to vanishing gradients.
Predictive Coding as a 2nd-Order Method
Published:
📖 TL;DR: Predictive coding implicitly performs a 2nd-order weight update via 1st-order (gradient) updates on neurons that in some cases allow it to converge faster than backpropagation with standard stochastic gradient descent.
infinite depth
Can We Scale Predictive Coding? or Why the Brain Might Be Much Wider Than Deep
Published:
📖 TL;DR: The gradients computed by predictive coding converge to backpropagation’s for much wider than deep networks (like the brain), under stable parameterisations.
infinite width
Can We Scale Predictive Coding? or Why the Brain Might Be Much Wider Than Deep
Published:
📖 TL;DR: The gradients computed by predictive coding converge to backpropagation’s for much wider than deep networks (like the brain), under stable parameterisations.
infinite width limit
♾️ Infinite Widths (& Depths) Part III: The Maximal Update Parameterisation (\(\mu\)P)
Published:
This is the third and last post of a short series on the infinite-width limits of deep neural networks (DNNs). In Part I, we showed that the output of a random network becomes Gaussian distributed in the infinite-width limit. Part II went beyond initialisation and showed that infinitely wide nets trained with GD are basically kernel methods.
♾️ Infinite Widths Part II: The Neural Tangent Kernel
Published:
This is the second post of a short series on the infinite-width limits of deep neural networks (DNNs). Previously, we reviewed the correspondence between neural networks and Gaussian Processes (NNGP), showing that, as the number neurons in the hidden layers grows to infinity, the output of a random network becomes Gaussian distributed.
♾️ Infinite Widths Part I: Neural Networks as Gaussian Processes
Published:
This is the first post of a short series on the infinite-width limits of deep neural networks (DNNs). We start by reviewing the correspondence between neural networks and Gaussian Processes (GPs).
internship
interpretability
KANs Made Simple
Published:
Confused about the recent KAN: Kolmogorov-Arnold Networks? I was too, so here’s a minimal explanation that makes it easy to see the difference between KANs and multi-layer perceptrons (MLPs).
kernel methods
♾️ Infinite Widths Part II: The Neural Tangent Kernel
Published:
This is the second post of a short series on the infinite-width limits of deep neural networks (DNNs). Previously, we reviewed the correspondence between neural networks and Gaussian Processes (NNGP), showing that, as the number neurons in the hidden layers grows to infinity, the output of a random network becomes Gaussian distributed.
large language models
In-Context Learning Demystified?
Published:
📖 TL;DR: a transformer block implicitly uses the input context to modify its MLP weights.
lazy learning
♾️ Infinite Widths Part II: The Neural Tangent Kernel
Published:
This is the second post of a short series on the infinite-width limits of deep neural networks (DNNs). Previously, we reviewed the correspondence between neural networks and Gaussian Processes (NNGP), showing that, as the number neurons in the hidden layers grows to infinity, the output of a random network becomes Gaussian distributed.
linear regime
♾️ Infinite Widths Part II: The Neural Tangent Kernel
Published:
This is the second post of a short series on the infinite-width limits of deep neural networks (DNNs). Previously, we reviewed the correspondence between neural networks and Gaussian Processes (NNGP), showing that, as the number neurons in the hidden layers grows to infinity, the output of a random network becomes Gaussian distributed.
local learning
Can We Scale Predictive Coding? or Why the Brain Might Be Much Wider Than Deep
Published:
📖 TL;DR: The gradients computed by predictive coding converge to backpropagation’s for much wider than deep networks (like the brain), under stable parameterisations.
Scaling Predictive Coding to 100+ Layer Networks
Published:
📖 TL;DR: We introduce \(\mu\)PC, a reparameterisation of predictive coding networks that enables stable training of 100+ layer ResNets on simple tasks with zero-shot hyperparameter transfer.
⛰️ The Energy Landscape of Predictive Coding Networks
Published:
📖 TL;DR: Predictive coding makes the loss landscape of feedforward neural networks more benign and robust to vanishing gradients.
Predictive Coding as a 2nd-Order Method
Published:
📖 TL;DR: Predictive coding implicitly performs a 2nd-order weight update via 1st-order (gradient) updates on neurons that in some cases allow it to converge faster than backpropagation with standard stochastic gradient descent.
loss landscape
⛰️ The Energy Landscape of Predictive Coding Networks
Published:
📖 TL;DR: Predictive coding makes the loss landscape of feedforward neural networks more benign and robust to vanishing gradients.
machine learning
Thermodynamic Natural Gradient Descent
Published:
I recently came across this paper Thermodynamic Natural Gradient Descent by Normal Computing. I found it very interesting, so below is my brief take on it.
maximal update parameterisation
Can We Scale Predictive Coding? or Why the Brain Might Be Much Wider Than Deep
Published:
📖 TL;DR: The gradients computed by predictive coding converge to backpropagation’s for much wider than deep networks (like the brain), under stable parameterisations.
Scaling Predictive Coding to 100+ Layer Networks
Published:
📖 TL;DR: We introduce \(\mu\)PC, a reparameterisation of predictive coding networks that enables stable training of 100+ layer ResNets on simple tasks with zero-shot hyperparameter transfer.
♾️ Infinite Widths (& Depths) Part III: The Maximal Update Parameterisation (\(\mu\)P)
Published:
This is the third and last post of a short series on the infinite-width limits of deep neural networks (DNNs). In Part I, we showed that the output of a random network becomes Gaussian distributed in the infinite-width limit. Part II went beyond initialisation and showed that infinitely wide nets trained with GD are basically kernel methods.
muP
Can We Scale Predictive Coding? or Why the Brain Might Be Much Wider Than Deep
Published:
📖 TL;DR: The gradients computed by predictive coding converge to backpropagation’s for much wider than deep networks (like the brain), under stable parameterisations.
multi-layer perceptrons
KANs Made Simple
Published:
Confused about the recent KAN: Kolmogorov-Arnold Networks? I was too, so here’s a minimal explanation that makes it easy to see the difference between KANs and multi-layer perceptrons (MLPs).
mup
Scaling Predictive Coding to 100+ Layer Networks
Published:
📖 TL;DR: We introduce \(\mu\)PC, a reparameterisation of predictive coding networks that enables stable training of 100+ layer ResNets on simple tasks with zero-shot hyperparameter transfer.
♾️ Infinite Widths (& Depths) Part III: The Maximal Update Parameterisation (\(\mu\)P)
Published:
This is the third and last post of a short series on the infinite-width limits of deep neural networks (DNNs). In Part I, we showed that the output of a random network becomes Gaussian distributed in the infinite-width limit. Part II went beyond initialisation and showed that infinitely wide nets trained with GD are basically kernel methods.
natural gradient descent
Thermodynamic Natural Gradient Descent
Published:
I recently came across this paper Thermodynamic Natural Gradient Descent by Normal Computing. I found it very interesting, so below is my brief take on it.
neural scaling laws
KANs Made Simple
Published:
Confused about the recent KAN: Kolmogorov-Arnold Networks? I was too, so here’s a minimal explanation that makes it easy to see the difference between KANs and multi-layer perceptrons (MLPs).
neural tangent kernel
♾️ Infinite Widths (& Depths) Part III: The Maximal Update Parameterisation (\(\mu\)P)
Published:
This is the third and last post of a short series on the infinite-width limits of deep neural networks (DNNs). In Part I, we showed that the output of a random network becomes Gaussian distributed in the infinite-width limit. Part II went beyond initialisation and showed that infinitely wide nets trained with GD are basically kernel methods.
♾️ Infinite Widths Part II: The Neural Tangent Kernel
Published:
This is the second post of a short series on the infinite-width limits of deep neural networks (DNNs). Previously, we reviewed the correspondence between neural networks and Gaussian Processes (NNGP), showing that, as the number neurons in the hidden layers grows to infinity, the output of a random network becomes Gaussian distributed.
optimisation theory
Scaling Predictive Coding to 100+ Layer Networks
Published:
📖 TL;DR: We introduce \(\mu\)PC, a reparameterisation of predictive coding networks that enables stable training of 100+ layer ResNets on simple tasks with zero-shot hyperparameter transfer.
♾️ Infinite Widths (& Depths) Part III: The Maximal Update Parameterisation (\(\mu\)P)
Published:
This is the third and last post of a short series on the infinite-width limits of deep neural networks (DNNs). In Part I, we showed that the output of a random network becomes Gaussian distributed in the infinite-width limit. Part II went beyond initialisation and showed that infinitely wide nets trained with GD are basically kernel methods.
predictive coding
Can We Scale Predictive Coding? or Why the Brain Might Be Much Wider Than Deep
Published:
📖 TL;DR: The gradients computed by predictive coding converge to backpropagation’s for much wider than deep networks (like the brain), under stable parameterisations.
PhD Reflections
Published:
Having recently submitted my PhD thesis, I’ve been thinking a lot about my PhD experience. Here I would like to share some reflections. Needless to say that this is my own, biased experience, and PhDs can vary greatly depending on the field, lab, supervisor, etc.
Scaling Predictive Coding to 100+ Layer Networks
Published:
📖 TL;DR: We introduce \(\mu\)PC, a reparameterisation of predictive coding networks that enables stable training of 100+ layer ResNets on simple tasks with zero-shot hyperparameter transfer.
⛰️ The Energy Landscape of Predictive Coding Networks
Published:
📖 TL;DR: Predictive coding makes the loss landscape of feedforward neural networks more benign and robust to vanishing gradients.
Predictive Coding as a 2nd-Order Method
Published:
📖 TL;DR: Predictive coding implicitly performs a 2nd-order weight update via 1st-order (gradient) updates on neurons that in some cases allow it to converge faster than backpropagation with standard stochastic gradient descent.
reflections
PhD Reflections
Published:
Having recently submitted my PhD thesis, I’ve been thinking a lot about my PhD experience. Here I would like to share some reflections. Needless to say that this is my own, biased experience, and PhDs can vary greatly depending on the field, lab, supervisor, etc.
research
PhD Reflections
Published:
Having recently submitted my PhD thesis, I’ve been thinking a lot about my PhD experience. Here I would like to share some reflections. Needless to say that this is my own, biased experience, and PhDs can vary greatly depending on the field, lab, supervisor, etc.
rich regime
♾️ Infinite Widths (& Depths) Part III: The Maximal Update Parameterisation (\(\mu\)P)
Published:
This is the third and last post of a short series on the infinite-width limits of deep neural networks (DNNs). In Part I, we showed that the output of a random network becomes Gaussian distributed in the infinite-width limit. Part II went beyond initialisation and showed that infinitely wide nets trained with GD are basically kernel methods.
saddle points
⛰️ The Energy Landscape of Predictive Coding Networks
Published:
📖 TL;DR: Predictive coding makes the loss landscape of feedforward neural networks more benign and robust to vanishing gradients.
saddles
Predictive Coding as a 2nd-Order Method
Published:
📖 TL;DR: Predictive coding implicitly performs a 2nd-order weight update via 1st-order (gradient) updates on neurons that in some cases allow it to converge faster than backpropagation with standard stochastic gradient descent.
second-order method
Predictive Coding as a 2nd-Order Method
Published:
📖 TL;DR: Predictive coding implicitly performs a 2nd-order weight update via 1st-order (gradient) updates on neurons that in some cases allow it to converge faster than backpropagation with standard stochastic gradient descent.
second-order methods
Thermodynamic Natural Gradient Descent
Published:
I recently came across this paper Thermodynamic Natural Gradient Descent by Normal Computing. I found it very interesting, so below is my brief take on it.
splines
KANs Made Simple
Published:
Confused about the recent KAN: Kolmogorov-Arnold Networks? I was too, so here’s a minimal explanation that makes it easy to see the difference between KANs and multi-layer perceptrons (MLPs).
system-2 thinking
Energy-based Transformers
Published:
📖 TL;DR: Energy-based Transformers (EBTs) learn a scalar energy function parameterised by a transformer. Empirically, EBTs show promising scaling and reasoning properties on both language and vision tasks.
tensor programs
♾️ Infinite Widths (& Depths) Part III: The Maximal Update Parameterisation (\(\mu\)P)
Published:
This is the third and last post of a short series on the infinite-width limits of deep neural networks (DNNs). In Part I, we showed that the output of a random network becomes Gaussian distributed in the infinite-width limit. Part II went beyond initialisation and showed that infinitely wide nets trained with GD are basically kernel methods.
thermodynamic AI
Thermodynamic Natural Gradient Descent
Published:
I recently came across this paper Thermodynamic Natural Gradient Descent by Normal Computing. I found it very interesting, so below is my brief take on it.
transformers
In-Context Learning Demystified?
Published:
📖 TL;DR: a transformer block implicitly uses the input context to modify its MLP weights.
Energy-based Transformers
Published:
📖 TL;DR: Energy-based Transformers (EBTs) learn a scalar energy function parameterised by a transformer. Empirically, EBTs show promising scaling and reasoning properties on both language and vision tasks.
trust region
Predictive Coding as a 2nd-Order Method
Published:
📖 TL;DR: Predictive coding implicitly performs a 2nd-order weight update via 1st-order (gradient) updates on neurons that in some cases allow it to converge faster than backpropagation with standard stochastic gradient descent.
vanishing gradients
⛰️ The Energy Landscape of Predictive Coding Networks
Published:
📖 TL;DR: Predictive coding makes the loss landscape of feedforward neural networks more benign and robust to vanishing gradients.