A thesis on generative AI

We present to following thesis on generative AI.  That to understand fully the essence of generative AI we must first understand the socio-cultural and philosophical essence of the advent of the internet.  Generative AI is only possible not merely because of raw hardware capabilities and certain neural net architectures and training processes, but on a more fundamental level because of the nature of the large data sets employed. Generative AI is only possible because of the previous availability of a very special kind of massive data set - precisely the kind of data sets provided by the internet. Generative AI is simply the natural further development of the same factors at work in the genesis of the internet.  So our main task is to describe the nature of internet data as well as its social, psychological, linguistic and cultural correlates.  This is a difficult task. One should consider and evaluate the following aspects: the democratization of knowledge and expertise (what certain authors once called "the cult of amateurism"), the lessening of the both the physical and semantic volume of content - flattening of layers of semantic depth, connotation, subtext and extra-textual reference, a  massive atomization and repetition with small variants of certain templates, memes and "virality", propagation and reproduction of persistent psycho-linguistic patterns throughout the whole body of the data - all which structurally contribute to forming the kind of atomized, grid-like, decentralized, self-entangled and self-connected  massive data set which is precisely what allows the generative AI technology to produce LLMs which perform as they do. While traditional cultural-linguistic data involved relatively few authors with massive linguistic content, the internet data involves a massive amount of authors with relatively scarce linguistic content each.  Humanity is caught itself in the textual universe of the internet.  The internet data set can be described as an enmeshed chaos of micro cognitive-linguistic units - which we can compare to the set P of partial information used in forcing. If we wish to combat generative AI we need to get at the root of the whole historical-cultural process involved in the genesis of the internet. We need to instigate a revolution regarding authorship, epistemic authority and the nature of the written text. The semantic web project should be revived. Curiously enough the Pali canon is an very early example of large textual data which exhibits some of the characteristics of internet data.

If the internet era was the era of communication, personal development through sharing and community, but now not only has the "internet" as a medium become content itself but it has become dead content, i.e., generative AI. Thus we are in the heroic age in which the human person must again find development and peace in isolation again - so that again there is something to share. The highest form of human knowledge must be global and synthetic, partial analysis has as its ultimate aim furnishing the support for such a knowledge. Global and synthetic knowledge is the only knowledge that frees one from Plato's Cave. 

Can we reverse engineering generative AI using generative AI? Could we train an AI model M on the data D = (T,L) consisting of training data T and a LLM L trained on T, to solve the following problem:

Given a prompt P fed into L and response R, identify the portions of T which can be considered as having (in some mode or another) the largest weight in the generation of R from P.

Can we define the concept of essential dependence of response R to P via L on a portion S of T? Could we train M to return, for a given P, portions of T which have a certain structural similarity to R?  

Note on Large Language Models

LLMs have a certain analogy to compression. From the training data D we obtain a LLM T(D) which is supposed to contain (or "extract") the essential "information" or "statistical patterns" present in D. T(D) is much smaller than D. It is speculated that Claude models are trained on D of the size of a petabyte and that the models themselves range from 150 to 500 GB. The response to a given prompt is analogous to decompression. Supposedly T(D) can "generate" an approximation of all the information originally contained in D. Some questions:

1. Is it not true that the passage D -> T(D) is not lossless, that important information present in D is lost in T(D) and cannot be recovered by it?
2. Is there any way to study T(D) as a mathematical object, detect its structure and geometry? And to study likewise the correspondence between D and T(D)? If there are limitations to doing this are they practical or theoretical?
3. There is an analogy between passing from D to T(D) and passing from general to countable models of ZF set theory (which exist by the downward Löwenheim-Skolem theorems)?
4. Is there not some analogy between forcing using countable models and generic sets and the process of training to generate T(D)? In both cases there is pattern generalization from fragmentary data.
5. Is there any structural correspondence between the structure of T(D) and structures found in the world (not counting neurological analogues of MLPs)?
6. Can we construct toy universes, toy languages and toy training data and study how D -> T(D) works in this simplified idealized scenario to gain more insight regarding real world LLMs?
7. Do LLMs express an essentially emergent phenomenon in which hardware capabilities are a crucial factor? Can we formalize rigorously such a concept of emergent phenomenon or capability?
8. But most importantly LLMs are linear statistical predictors (next token predictors) and they are trained as such. We need to formalize clearly what LLMs are supposed to do in the first place. Suppose we have a (first-order) model M that represents the world. We want our LLM T to be able to deal with a good degree of approximation with the theory of M, Th(M). We are given a finite large set L of first-order formulae with probabilities of their belonging to Th(M). A transformation is applied to L to obtain the object T which is able to include the reliable part of L in Th(M) and to extrapolate to other elements of Th(M). Is this to be understood as both logical and statistical inference?
9. A LLM is just a finite state automaton. But recursively axiomatizable theories are in general not recursive. Can we can construct a theory T such that for any finite subset L of T all LLMs trained on L will err to an arbitrarily with regards to infinitely many sentences of T. We define metrics on expressions, that's the key.
10. And most importantly: are LLMs analogous to syntactic (and algebraic) models used in logic and category theory?  Or the training data is like the a poset P with the dense topology and the LLM is like the topos of sheaves over this site?

11. Do LLMs function essentially by analogy, metaphor, induction and extrapolation? This is an old idea in AI.