Note on Large Language Models

LLMs have a certain analogy to compression. From the training data D we obtain a LLM T(D) which is supposed to contain (or "extract") the essential "information" or "statistical patterns" present in D. T(D) is much smaller than D. It is speculated that Claude models are trained on D of the size of a petabyte and that the models themselves range from 150 to 500 GB. The response to a given prompt is analogous to decompression. Supposedly T(D) can "generate" an approximation of all the information originally contained in D. Some questions:

1. Is it not true that the passage D -> T(D) is not lossless, that important information present in D is lost in T(D) and cannot be recovered by it?
2. Is there any way to study T(D) as a mathematical object, detect its structure and geometry? And to study likewise the correspondence between D and T(D)? If there are limitations to doing this are they practical or theoretical?
3. There is an analogy between passing from D to T(D) and passing from general to countable models of ZF set theory (which exist by the downward Löwenheim-Skolem theorems)?
4. Is there not some analogy between forcing using countable models and generic sets and the process of training to generate T(D)? In both cases there is pattern generalization from fragmentary data.
5. Is there any structural correspondence between the structure of T(D) and structures found in the world (not counting neurological analogues of MLPs)?
6. Can we construct toy universes, toy languages and toy training data and study how D -> T(D) works in this simplified idealized scenario to gain more insight regarding real world LLMs?
7. Do LLMs express an essentially emergent phenomenon in which hardware capabilities are a crucial factor? Can we formalize rigorously such a concept of emergent phenomenon or capability?
8. But most importantly LLMs are linear statistical predictors (next token predictors) and they are trained as such. We need to formalize clearly what LLMs are supposed to do in the first place. Suppose we have a (first-order) model M that represents the world. We want our LLM T to be able to deal with a good degree of approximation with the theory of M, Th(M). We are given a finite large set L of first-order formulae with probabilities of their belonging to Th(M). A transformation is applied to L to obtain the object T which is able to include the reliable part of L in Th(M) and to extrapolate to other elements of Th(M). Is this to be understood as both logical and statistical inference?
9. A LLM is just a finite state automaton. But recursively axiomatizable theories are in general not recursive. Can we can construct a theory T such that for any finite subset L of T all LLMs trained on L will err to an arbitrarily with regards to infinitely many sentences of T. We define metrics on expressions, that's the key.
10. And most importantly: are LLMs analogous to syntactic (and algebraic) models used in logic and category theory?  Or the training data is like the a poset P with the dense topology and the LLM is like the topos of sheaves over this site?

11. Do LLMs function essentially by analogy, metaphor, induction and extrapolation? This is an old idea in AI. 

Very short note on formal concepts in science

From whence do we get the impression that some mathematical models are closer to physical and spatio-temporal intuition,  more down to earth and intimately tied to concrete applications, while others hover close to the heights of allegedly less useful  'abstract-nonsense' ? The real numbers and differential equations, these are seen as tied to dynamic-geometric intuition and of vast applicability and interest in engineering and science.  But abstract algebra and category theory appear to have no direct relevance to applicable mathematical models or to their kind of concrete geometric-dynamic intuition. We present here a few short speculations.

Perhaps the abstract models do capture fundamental levels of reality (maybe even more fundamental than the so-called concrete spatio-temporal ones) and what is required is first of all the methodic development of a special kind of intuition or cognition to grasp the planes of reality which they model (maybe something like Goethe's method is called for?). And then further models are required which can effect a mediation or transfer between these two levels, or at least allow a continuity or gradual deformation between concrete spatio-temporal reality and higher levels of reality.

A beautiful illustration of such a meditation and construction is furnished by the double fibration in Penrose's twistor theory which allows a mediation and transfer between physically significant objects on Minkowski space and abstract cohomological objects on the complex algebraic variety $\mathbb{P}^3$.This operation allows solutions to conformally invariant differential equations on spacetime (e.g., Maxwell's equations, Yang-Mills, or linearized gravity) to be identified exactly with Cech or Dolbeault cohomology classes on specific regions of twistor space. This appears to have been Penrose's attempt to give a geometric (topological) interpretation of quantum non-locality. 

Another illustration is the theory of $\mathcal{D}$-modules which presents a mediation between derived categories (and monoidal categories) and concrete models of systems of partial differential equations.

The idea of Kant's schematism of the pure concepts of the understanding can be interpreted as finding the mediation between logic and geometry (including mathematical physics). This is of immense contemporary significance as we find versions of Kantian schematism in topos theory, homotopy type theory and in areas involving monoidal categories in algebra, geometry and physics and in linear logic and computer science. 

There is also apparently a mediation and transition between the theory of bifurcations of smooth vector fields, fundamental  and ubiquitous in concrete applied mathematics,  and monoidal category and operad theory. See also Physics, Topology, Logic and Computation: A Rosetta Stone by John C. Baez and Mike Stay.

An objection can be raised that our concrete allegedly intuitive mathematical models are themselves quite abstract but are only perceived as immediate and intuitive in our present cultural context by a deliberate forgetting of the complexity of their cognitive-historical past and genesis. For instance real numbers are constructed from the rationals by quite abstract, cardinality increasing, procedures involving equivalence classes and identification. And the same goes for the concepts of continuity, differentiability and so forth. Surely the ancient Greeks had other forms of intuitive perception and concreteness in their geometry, physics and engineering. We answer this objection by our arguments that the fundamental dynamic and geometric-topological intuitions of modern science were in fact identical to those of ancient Greece. It is the modern foundations in terms of Dedekind cuts and $\epsilon$s and $\delta$s that can be critiqued from a philosophical and logical point of view and alternative foundations (locales, realizability topoi, synthetic differential geometry, homotopy type theory) can be defended which are also more aligned to the theory of space and change found for instance in Aristotle's Physics.