On Complexity and Entropy

This post was inspired by reading Scott Aaronson’s blog post The First Law of Complexodynamics, which I discovered on this list of 30 readings recommended by Ilya Sutskever for understanding modern AI systems. (I don’t know if Ilya actually recommended these, but the list is interesting so I’m reading through them.)

---

The question Scott attempts to answer, originally posed by Sean Carroll in a talk

why does “complexity” or “interestingness” of physical systems seem to increase with time and then hit a maximum and decrease, in contrast to the entropy, which of course increases monotonically?

(does that sound like aging of the human mind to anyone? Interestingness peaks around 40s-50s and declines for most people, but there are exceptions - like John Williams and Isaac Asimov. More examples from an earlier post I wrote

Assumptions made by Scott and Sean

• at initial time the entropy is zero

• “entropy presumably gives an upper bound on complexity”

Here’s a working link to the paper on sophistication linked in Scott’s post.

---

It seems necessary to define entropy to proceed here, so I came up with the following (work-in-progress) definition:

The presence, in a given system, of entities, that inhibit the interactions of a smaller group of entities that belong together and exist for a purpose, by simply being present in large numbers and/or colliding/interacting with the entities of the smaller group and thereby reducing the number/duration of meaningful interactions that smaller group can have.

I think this applies to systems of particles, thoughts (in your mind), social systems - large networks like Twitter and also meetings, and biological systems like cells.

Some time ago, I realised that the best people in any field are those who are consistently interesting. Every interaction they have with the world seems unique. They don't repeat themselves in their writing (not entirely, there may be some references back to previous ideas or opinions). Every piece is different. Similarly, when they speak, it's often repetition of some things, but with differences. Isaac Asimov could speak without preparation on any topic in front of any audience, and no two of his talks were the same.

It seems that this is also what a truly complex system is like - where interactions between the system and the outside world will all seem unique. Which brings up the questions of LLMs: Is every interaction with an LLM unique?

The labs have managed to make the systems partially deterministic, in that if you have a specific seed, temperature, and given set of inputs, the output is going to be the same. They've been able to achieve consistency of output, which means the system is probably not so complex or can be constrained to have limited complexity?

(Sidenote - I find Google’s Gemini less consistent in its output in the API than Claude or GPT. That’s annoying as a developer, but I can’t say if it’s a better or worse LLM compared to the others.)

The idea of novelty as a characteristic of complexity is also discussed in this fun video from the physicist Sabine Hossenfelder (you should subscribe to her Youtube channel!)

The central idea in Sabine’s video is our expectation of surprise from a complex system. We’ve been unable to define mathematically or in precise scientific terms what we mean by the term ‘complexity’, but ‘we know it when we see it’. And our expectation of surprise, our inability to predict exactly what it will do seem to be some common features. The idea of emergent behaviour - again something that’s much talked about for LLMs - comes up.

Biological systems - cells, tissues, entire organisms, but even a single cell organelle like the mitochondria - are more complex than anything else we know. If I try to define complexity by observing what’s happening in a biological system, it would have the following features

• simultaneity of a large number of interactions - there’s parallel processing

• speed and scale of interactions - the number of individual reactions at any given moment is staggering

• feedback loops - genes switch on and off, reactions become faster or slower all based on what’s happening in the cell environment

• diversity - we’ve categorized biomolecules into lipids, carbohydrates, nucleic acids and proteins broadly, but the variety in even a single cell organelle is immense

• structure determines function - this is a central idea in biology (and why half of the 2024 Nobel Prize in Chemistry has been awarded for a machine learning model that can determine the structure of a protein from a sequence)

• redundancy - there are multiple pathways to ensure critical functions are not disrupted. For instance, there are multiple ways to make glucose from different substrates - amino acids, fatty acids, other sugars.

• repetition, with a difference - the individual units can be described as aggregates of the same basic components (whether subatomic particles, or individual biomolecules), but the resulting units differ and their behaviours differ

• a common, overarching goal - ‘to survive’ (whether as an individual, or through creating more of its kind) is the one for biological systems

Ultimately, Scott leaves us with this question

Can we show that for such a system, the complextropy becomes large at intermediate times (intuitively, because of the need to specify the irregular boundaries between the regions of all-black pixels, all-white pixels, and mixed black-and-white pixels)?

But I’m don’t entirely believe in the premise that complexity in a system rises then falls, and entropy rises monotonically. That seems to be the default behaviour, but I see exceptions. One example is that of human minds that don’t seem to atrophy with age. Another is cancer cells - they proliferate by hacking the normal cell cycle, inhibiting the normal mechanism of cell death and getting the cell to run in an infinite growth and reproduction loop. Their growth is limited not by their own degradation (= rising entropy), but by the death of the body they reside in.

Finally, I’m reminded of Isaac Asimov’s famous short story The Last Question here, which revolves around entropy. My takeaway from the story is that creation is the only event that reduces entropy, that demands that chaos order itself. Not just creation of life, but of any work of art or literature, rockets, cellphones, bottles. Creation demands humans and matter to put every other concern on the backfoot, ignore competing forces, and mold amorphous sand into marvelous intelligence. If you can learn to reduce entropy on demand, the original question in this blog post feels superfluous.

---

If you have received this post via email, you can reply with comments/feedback.

You can also say hi on Twitter and LinkedIn.

Thanks for reading, have a nice day!

---