In a blog post published on July 15, 2026, by Zhengdao Chen, a Research Scientist at Google Research, a new study attempts to decode the source of creativity in diffusion models. In the research, titled "On the Interpolation Effect of Score Smoothing in Diffusion Models" and presented at the ICLR 2026 conference, the researcher demonstrates that the creative capability of diffusion models—meaning their ability to generate novel data rather than merely memorizing the dataset on which they were trained—is a mathematical consequence of the way neural networks learn a "smoothed" version of the score function. This smoothing drives the model to interpolate between training data points along the hidden data manifold.
Understanding the Denoising Process and Its Impact on Model Memory
Training a diffusion model begins by taking a set of real data samples from the training set—for example, photos of cats—and intentionally corrupting them with noise until they become completely unrecognizable. The model is then trained to reverse this corruption process step-by-step so that it can reconstruct a realistic-looking image from pure noise alone. This reconstruction process is referred to as "denoising."
According to Chen, if the model learns to perform this denoising process perfectly based solely on its training samples, it is expected to produce exact copies of these samples when deployed in practice. This behavior is known as "memorization." Under such circumstances, the model functions merely as an information retrieval tool rather than a creative engine capable of generating entirely novel outputs.
In practice, however, diffusion models usually do more than just memorize; they generalize to produce novel data samples. To understand how diffusion models actually denoise data in practice, the researcher suggests imagining random noise as a cloud of gas particles scattered across a room, where a "force field" pulls each particle in a specific direction until they form a meaningful shape. In a diffusion model, the moving particles are individual data points undergoing a denoising process. The "force field" is the score function (SF), which is learned from the training data and dictates where the particles should flow at any given moment. If the model relies on a score function learned perfectly from the training data, the force field will drive the particles into positions that replicate the training data points exactly, leading to memorization.
Creativity in One Dimension and the Impact of Score Smoothing
The Google study discovered that the creativity of diffusion models actually stems from the approximate nature of how neural networks typically learn: imperfect training resulting from regularization naturally leads to a slight blurring of the learned score function, a process termed "score smoothing." This blurring causes the denoising process to generate data that interpolates—meaning it falls in the gaps between training points—thereby producing novel and plausible data samples.
To demonstrate this, let us examine a one-dimensional world containing only two training data points: plus one (+1) and minus one (-1). In the late stages of the denoising process, the "perfect" score function resembles a curvy gray line showing a steep change in sign exactly halfway between the two points, which indicates a rapid change in the direction of pull near point zero. In other words, the entire space is divided almost sharply into two, where particles on the left side are pulled toward minus one, and particles on the right side are pulled toward plus one. Ultimately, every particle converges to one of the two training data points, and memorization occurs.
In practice, diffusion models do not have access to the "perfect" score function, and instead use an approximate version learned by a neural network. Due to the regularization effect of weight decay during training, neural networks struggle to learn functions containing such sharp cliffs. Instead, they tend to learn smoother versions of the perfect score function, softening the steep drop into a gentler slope.
The researchers set up an experiment where they trained two-layer ReLU neural networks to fit the score function in this one-dimensional example. The parameters of the neural networks were optimized using the popular AdamW algorithm under varying degrees of weight decay (WD). The experiment showed that the stronger the weight decay, the smoother the learned score function is in the middle region. This means that particles in this area flow more slowly than before, and will ultimately stop within the "interpolation zone" between the two training data points, which produces the interpolation effect. The researchers quantify this relationship in their paper by combining the function-space theory of neural network regularization with the mathematics of denoising. Additionally, the experiments demonstrated that even without explicit regularization strategies like weight decay, score smoothing can arise from the implicit regularization effect present in neural networks trained by gradient-based algorithms.
Score Smoothing and Manifold Recovery in Multi-Dimensional Spaces
In the real world, complex data such as high-resolution images exist in multi-dimensional pixel spaces rather than a simple one-dimensional world. The vast majority of this space is simply random noise that is meaningless to the human eye. Only a very small fraction of the data points in this space correspond to recognizable images, and these exist within what is known as the "data manifold," which resembles a sheet folded inside a larger space.
The shape and location of the data manifold are not known to the model in advance. Therefore, image generation can be considered a task of "manifold recovery," where the model must infer what the hidden data manifold looks like based on a finite number of training data points sampled from it, and then generate new points on the manifold that correspond to novel and meaningful images.
It turns out that score smoothing is critical for diffusion models to achieve this. In multi-dimensional settings, the effect of score smoothing manifests in a direction-dependent manner. Along directions that are parallel (or "tangential") to the hidden data manifold, score smoothing produces a slowing-down effect similar to that in the one-dimensional scenario. In contrast, along directions pointing directly toward the manifold, the "perfect" score function is already relatively smooth (in fact, a simple straight line if the manifold is flat), and further smoothing does not significantly affect it.
As a result, instead of applying brakes to the particles' flow in all directions (which would prevent them from reaching the manifold and cause the final images to be blurry), score smoothing does not slow down their movement toward the manifold, but only reduces their tendency to collapse toward the original training data along the tangential directions. In this way, the model achieves a balance between quality and novelty: the generated images are both realistic-looking (because the particles successfully reached the meaningful data manifold) and new (because they settled in the empty spaces between the original training data points).
Conclusions and Implications for Future Research and Drug Discovery
The implications of Google's research show that what we call the "creativity" of diffusion models may actually be a predictable mathematical consequence. Because neural networks cannot learn perfectly sharp functions, they create bridges that interpolate between known data.
In fields like image generation or drug discovery, this means that the diffusion model does not merely memorize two different cat images or two drug molecules shown to it; it explores the space around them to suggest a third, entirely new image or a novel molecular configuration that combines traces of both original examples.
Chen notes that this work represents only an initial effort to clarify this mechanism, and it remains to be seen what happens when data distributions or neural network architectures become more complex. However, by demonstrating that this behavior is fundamentally rooted in how neural networks learn, scientists can begin to intentionally build models that are better "interpolators," ensuring they remain creative engines while avoiding the pitfalls of blind memorization.
The researchers have publicly released the code for the numerical experiments used to generate the figures in their paper, allowing the scientific community to continue investigating this topic collaboratively.