Modern ML is advancing rapidly. It can be used to translate languages, generate computer programs, and produce reasonably-written text. More advances are likely in the near future. More generally, the future of computerized optimization systems is progressing rapidly and is of great importance.

I am very interested in understanding the effects of such systems on the future of humanity, and the future community of beings more generally. Unfortunately, our current language on these topics gets in the way of such understanding. We constantly make analogies to humans and to fictional entities in this discussion, which gets in the way of seeing such systems as they truly are. Our conversations often focus on how human or how concious such a system is, even when that’s not really what we’re interested in.

Specifically, we constantly use language that encourages us to anthropomorphize these systems. We call them “Artificial intelligences”, we call the field “Machine learning”. We refer to systems knowing about topics or wanting to progress towards a goal. While these terms are fine for colloquial speech, I think they get in the way of deep technical understanding.

In this sequence of posts, I will present a new set of terms for discussing this topic, chosen to not be anthropomorphic. Hopefully, this will make it easier to think clearly about these subjects in a technical context.

My goal is for this sequence of posts to be accessible and helpful both for people who have never encountered this topic before, and for people who are already steeped in the subject.

Optimization systems

To start with, we need a name for the topic. I don’t want to use standard names like “Artificial Intelligence” or “Machine Learning”. I will therefore build up the concept step by step, building towards a clear concept of our topic of interest.

Let’s start with the idea of a “optimization system”. An optimization system is something that tends to progress towards some objective. This is a very general concept.

For instance, rocks tend to roll down hills, so a rock can be described as an optimization system, with the objective of reaching a lower point of terrain. Animals tend to find and eat food, so animals can be considered as an optimization system for the objective of find and eating food. Evolution can be considered an optimization system towards finding living beings that thrive in their environments. People can be considered optimization systems towards a variety of objectives. A chess algorithm can be considered an optimization system towards the objective of finding strong chess moves. A language model can be considered an optimization system towards the objective of producing human-like text.

Note that optimization systems may or may not be designed to perform such optimization. No one designed a rock to roll down a hill, or evolution to evolve, while chess algorithms and language models were designed. They are all optimization systems.

The concept of optimization systems is a good starting point, but it is very general. It needs to be refined to a more targeted category for us to think about.

Kinds of optimization systems

Optimization systems vary along two important axes: flexibility and effectiveness.

By flexibility, I refer to an optimization system being able to optimize towards a wide variety of objectives, given the right conditions. Some optimization systems are inflexible, in that they can only optimize a narrow set of objectives. Rocks optimize towards reaching low terrain, and making things flat. Chess programs optimize towards generating good chess moves and winning chess games.

Other optimization systems are highly flexible. People optimize for all sorts of things, all of the time. Language models, given appropriate prompts, can optimize a wide variety of tasks, including ones that people used to design custom systems for each task.

Of course, this is a sliding scale, not a hard division into categories. It makes more sense to say that one optimization system is more flexible than another, rather than use absolute terms.

By effectiveness, I refer to how far the optimization system moves towards an objective, and how reliably it does so. Rocks often move somewhat downhill, but they rarely make it to the lowest possible terrain. By contrast, advanced chess programs output extremely good chess moves, in all but the most pathological positions.

Note that effectiveness is best compared between two optimization systems on the same objective. People are better than rocks at finding the lowest possible terrain (e.g. sea level or death valley). Chess programs are better than people at finding the best chess move in a position.

Topic of interest

Now we are ready to identify our topic of interest: Optimization systems that are

  • Quite flexible

  • Quite effective

  • Non-human

For short, I’ll call these “FEOSes”: Flexible and effective optimization systems. Until recently, FEOSes were hypothetical. Recently, with the rise of large language models such as the GPT family of algorithms, FEOSes have started to be realized.

Next time

In the next post, I’ll go into FEOS feedback mechanism, where the objective that a FEOS optimizes is related to improving the capabilities of future systems. Feedback is the primary reason why I’m interested in this topic.