Francesco Innocenti

Francesco Innocenti

PhD student in Theoretical Neuroscience

Posts by Tags

Amazon

Applied Scientist

Backpropagation

🧠 Predictive Coding as a 2nd-Order Method

10 minute read

Published:

📖 TL;DR: Predictive coding implicitly performs a 2nd-order weight update via 1st-order (gradient) updates on neurons that in some cases allow it to converge faster than backpropagation with standard stochastic gradient descent.

Bayesian Inference

Bayesian Neural Networks

Central Limit Theorem

Deep Information Propagation

Deep Neural Networks

♾️ Infinite Widths Part II: The Neural Tangent Kernel

6 minute read

Published:

This is the second post of a short series on the infinite-width limits of deep neural networks (DNNs). Previously, we reviewed the correspondence between neural networks and Gaussian Processes (GPs), basically finding that, as the number neurons in the hidden layers grows to infinity, the output of a random network becomes Gaussian distributed.

KANs Made Simple

2 minute read

Published:

🤔 Confused about the recent KAN: Kolmogorov-Arnold Networks? I was too, so here’s a minimal explanation that makes it easy to see the difference between KANs and multi-layer perceptrons (MLPs).

🧠 Predictive Coding as a 2nd-Order Method

10 minute read

Published:

📖 TL;DR: Predictive coding implicitly performs a 2nd-order weight update via 1st-order (gradient) updates on neurons that in some cases allow it to converge faster than backpropagation with standard stochastic gradient descent.

Fisher Information

🧠 Predictive Coding as a 2nd-Order Method

10 minute read

Published:

📖 TL;DR: Predictive coding implicitly performs a 2nd-order weight update via 1st-order (gradient) updates on neurons that in some cases allow it to converge faster than backpropagation with standard stochastic gradient descent.

Gaussian Processes

Gradient Descent

Industry

Inference Learning

🧠 Predictive Coding as a 2nd-Order Method

10 minute read

Published:

📖 TL;DR: Predictive coding implicitly performs a 2nd-order weight update via 1st-order (gradient) updates on neurons that in some cases allow it to converge faster than backpropagation with standard stochastic gradient descent.

Infinite Width Limit

♾️ Infinite Widths Part II: The Neural Tangent Kernel

6 minute read

Published:

This is the second post of a short series on the infinite-width limits of deep neural networks (DNNs). Previously, we reviewed the correspondence between neural networks and Gaussian Processes (GPs), basically finding that, as the number neurons in the hidden layers grows to infinity, the output of a random network becomes Gaussian distributed.

Internship

Interpretability

KANs Made Simple

2 minute read

Published:

🤔 Confused about the recent KAN: Kolmogorov-Arnold Networks? I was too, so here’s a minimal explanation that makes it easy to see the difference between KANs and multi-layer perceptrons (MLPs).

KAN

KANs Made Simple

2 minute read

Published:

🤔 Confused about the recent KAN: Kolmogorov-Arnold Networks? I was too, so here’s a minimal explanation that makes it easy to see the difference between KANs and multi-layer perceptrons (MLPs).

Kernel Methods

♾️ Infinite Widths Part II: The Neural Tangent Kernel

6 minute read

Published:

This is the second post of a short series on the infinite-width limits of deep neural networks (DNNs). Previously, we reviewed the correspondence between neural networks and Gaussian Processes (GPs), basically finding that, as the number neurons in the hidden layers grows to infinity, the output of a random network becomes Gaussian distributed.

Kolmogorov-Arnold Networks

KANs Made Simple

2 minute read

Published:

🤔 Confused about the recent KAN: Kolmogorov-Arnold Networks? I was too, so here’s a minimal explanation that makes it easy to see the difference between KANs and multi-layer perceptrons (MLPs).

Kolmogorov-Arnold Representation Theorem

KANs Made Simple

2 minute read

Published:

🤔 Confused about the recent KAN: Kolmogorov-Arnold Networks? I was too, so here’s a minimal explanation that makes it easy to see the difference between KANs and multi-layer perceptrons (MLPs).

Lazy Learning

♾️ Infinite Widths Part II: The Neural Tangent Kernel

6 minute read

Published:

This is the second post of a short series on the infinite-width limits of deep neural networks (DNNs). Previously, we reviewed the correspondence between neural networks and Gaussian Processes (GPs), basically finding that, as the number neurons in the hidden layers grows to infinity, the output of a random network becomes Gaussian distributed.

Linear Regime

♾️ Infinite Widths Part II: The Neural Tangent Kernel

6 minute read

Published:

This is the second post of a short series on the infinite-width limits of deep neural networks (DNNs). Previously, we reviewed the correspondence between neural networks and Gaussian Processes (GPs), basically finding that, as the number neurons in the hidden layers grows to infinity, the output of a random network becomes Gaussian distributed.

Local Learning

🧠 Predictive Coding as a 2nd-Order Method

10 minute read

Published:

📖 TL;DR: Predictive coding implicitly performs a 2nd-order weight update via 1st-order (gradient) updates on neurons that in some cases allow it to converge faster than backpropagation with standard stochastic gradient descent.

Loss Landscape

Machine Learning

Multi-layer Perceptrons

KANs Made Simple

2 minute read

Published:

🤔 Confused about the recent KAN: Kolmogorov-Arnold Networks? I was too, so here’s a minimal explanation that makes it easy to see the difference between KANs and multi-layer perceptrons (MLPs).

Natural Gradient Descent

Neural Scaling Laws

KANs Made Simple

2 minute read

Published:

🤔 Confused about the recent KAN: Kolmogorov-Arnold Networks? I was too, so here’s a minimal explanation that makes it easy to see the difference between KANs and multi-layer perceptrons (MLPs).

Neural Tangent Kernel

♾️ Infinite Widths Part II: The Neural Tangent Kernel

6 minute read

Published:

This is the second post of a short series on the infinite-width limits of deep neural networks (DNNs). Previously, we reviewed the correspondence between neural networks and Gaussian Processes (GPs), basically finding that, as the number neurons in the hidden layers grows to infinity, the output of a random network becomes Gaussian distributed.

Normal Computing

PhD

Predictive Coding

🧠 Predictive Coding as a 2nd-Order Method

10 minute read

Published:

📖 TL;DR: Predictive coding implicitly performs a 2nd-order weight update via 1st-order (gradient) updates on neurons that in some cases allow it to converge faster than backpropagation with standard stochastic gradient descent.

Saddle Points

Saddles

🧠 Predictive Coding as a 2nd-Order Method

10 minute read

Published:

📖 TL;DR: Predictive coding implicitly performs a 2nd-order weight update via 1st-order (gradient) updates on neurons that in some cases allow it to converge faster than backpropagation with standard stochastic gradient descent.

Second-Order

🧠 Predictive Coding as a 2nd-Order Method

10 minute read

Published:

📖 TL;DR: Predictive coding implicitly performs a 2nd-order weight update via 1st-order (gradient) updates on neurons that in some cases allow it to converge faster than backpropagation with standard stochastic gradient descent.

Second-order Methods

Splines

KANs Made Simple

2 minute read

Published:

🤔 Confused about the recent KAN: Kolmogorov-Arnold Networks? I was too, so here’s a minimal explanation that makes it easy to see the difference between KANs and multi-layer perceptrons (MLPs).

Thermodynamic AI

Trust Region

🧠 Predictive Coding as a 2nd-Order Method

10 minute read

Published:

📖 TL;DR: Predictive coding implicitly performs a 2nd-order weight update via 1st-order (gradient) updates on neurons that in some cases allow it to converge faster than backpropagation with standard stochastic gradient descent.

Vanishing Gradients