Note on Large Language Models

LLMs have a certain analogy to compression. From the training data D we obtain a LLM T(D) which is supposed to contain (or "extract") the essential "information" or "statistical patterns" present in D. T(D) is much smaller than D. It is speculated that Claude models are trained on D of the size of a petabyte and that the models themselves range from 150 to 500 GB. The response to a given prompt is analogous to decompression. Supposedly T(D) can "generate" an approximation of all the information originally contained in D. Some questions:

1. Is it not true that the passage D -> T(D) is not lossless, that important information present in D is lost in T(D) and cannot be recovered by it?
2. Is there any way to study T(D) as a mathematical object, detect its structure and geometry? And to study likewise the correspondence between D and T(D)? If there are limitations to doing this are they practical or theoretical?
3. There is an analogy between passing from D to T(D) and passing from general to countable models of ZF set theory (which exist by the downward Löwenheim-Skolem theorems)?
4. Is there not some analogy between forcing using countable models and generic sets and the process of training to generate T(D)? In both cases there is pattern generalization from fragmentary data.
5. Is there any structural correspondence between the structure of T(D) and structures found in the world (not counting neurological analogues of MLPs)?
6. Can we construct toy universes, toy languages and toy training data and study how D -> T(D) works in this simplified idealized scenario to gain more insight regarding real world LLMs?
7. Do LLMs express an essentially emergent phenomenon in which hardware capabilities are a crucial factor? Can we formalize rigorously such a concept of emergent phenomenon or capability?
8. But most importantly LLMs are linear statistical predictors (next token predictors) and they are trained as such. We need to formalize clearly what LLMs are supposed to do in the first place. Suppose we have a (first-order) model M that represents the world. We want our LLM T to be able to deal with a good degree of approximation with the theory of M, Th(M). We are given a finite large set L of first-order formulae with probabilities of their belonging to Th(M). A transformation is applied to L to obtain the object T which is able to include the reliable part of L in Th(M) and to extrapolate to other elements of Th(M). Is this to be understood as both logical and statistical inference?
9. A LLM is just a finite state automaton. But recursively axiomatizable theories are in general not recursive. Can we can construct a theory T such that for any finite subset L of T all LLMs trained on L will err to an arbitrarily with regards to infinitely many sentences of T. We define metrics on expressions, that's the key.
10. And most importantly: are LLMs analogous to syntactic (and algebraic) models used in logic and category theory?  Or the training data is like the a poset P with the dense topology and the LLM is like the topos of sheaves over this site?

New section in "Hegel and Modern Topology"

The most recent version of "Hegel and Modern Topology" includes a new 4 page section: "Formalization of Hegel's Logic of Concept".

The topological dynamics of plant physiology

We have mentioned Goethe's method in the context of the history of phenomenology as well as its connection to the modern topological and geometric theory of differential equations and dynamical systems. Anyone who has studied biology, or plant biology in particular, learnt about the systems of nutrition and transport in vascular plants, the two main types of vascular tissues being xylem and phloem. The first system is easy to understand enough (and its analogy with animal systems obvious) and not incompatible with a general Goethean conception of a plant with its vertically rising gradient. But phloem is a different matter.  Topologically phloem tubes - transporting sugars and amino acids synthesized in the leaves - are spread across the entire plant in a structure analogous to animal circulatory systems - a perplexing situation which begs for deep topological and dynamical questions and explanations. For if we consider the the system of phloem tubes to be topologically equivalent to a closed ball $B^3$, then the circulation of nutrients will define a vector field on $B^3$ which is tangent to its boundary $\partial B^3 = S^2$.  So there must be sources and sinks. We can ask: where are the sources and sinks in the plant? Is there an analogy with animal digestive systems - or even neural pathways? And even more importantly: does this vector field change in time (undergo a continuous bifurcation)? Modern plant biology of course allows us to answer these questions. The point here is the great philosophical and methodological importance of asking precisely these kinds of question and the role of topological-dynamical thinking.

A point about subjective idealism

Consider the opening sentence of Schopenhauer's famous book: "Die Welt ist meine Vorstellung/The world is my representation".  How can subjective idealism seem at once so utterly certain and so dubious? The answer is quite simple and lies in the definite article "Die/the" in Schopenhauer's dictum. The world in the sense of the world in which each one of us is acquainted with through first-hand experience - my world that I take immediately as the world - this cannot be in reality any more than a modification of the subject, of consciousness, a self-projection, self-occultation of consciousness inside itself. Even from neuro-reductionist physicalist assumptions there is no way of avoiding this conclusion without dogmatism (the ruse of invoking the objectivity or public nature of logic or language) or non-physicalist explanations and we can turn Benaceraff's overrated objection against Platonism against the physicalist who wishes to refute subjective idealism:  basic neuro-physiology and gestalt psychology show that a direct causal connection between nerve impulses and the "causality" of the phenomenal subjectively constituted world - is sheer nonsense. There are physical impacts on receptors which do not produce effects on consciousness and there are effects on consciousness which are not the direct result of impacts on receptors.

The world in the sense of that world with which I am acquainted with through first-person experience, is entirely phenomenal and exists in consciousness only, despite its objectifying externalizing intentions and pretenses.  However from this it does not follow in the least that a world, any world, is my representation. It does not follow that outside my world there is not another world. Nor can we rule out that there could be some sort of cognizable connection between our world and such a world, the world which is not our world.  In Kantian terms there does not follow that there is no thing-in-itself and it does not follow that we cannot know anything about this thing-in-itself (we already know that we know that it does not necessary not exist).  Despite Kant's own arguments, we may hold that such a world-in-itself is perhaps a grey, timeless, purely mathematically structured world but in which other world-creating consciousness subsist. We can even discern such a conception in Hume's Treatise. None of our causal scientific theories or explanation depend on the phenomenal intuition or concept of time, only on a vicarious geometric abstraction, a flattened timeless time.

Consider the stereographic projection of a sphere into a plane. The sphere is the subject,  the tangent bundle of the sphere is the modifications of consciousness,  the plane is the postulated world outside consciousness.  

But is this not logical dogmatism where we endow logic, a product (or condition?) of consciousness, with the right to judge that which is beyond consciousness? If logic corresponds to the ability to understand, carry out and check rules and to meta-interpret rule-systems,  where resource constraints can be taken into account,  then perhaps continuous processes - as infinitary rule systems - transcend logic. But logic functions through necessary principles of meta-logical agreement - thus it is reasonable to endow logic with transcendental epistemic scope. If a derivation in one formal system has to be considered sufficient for the cognition of a non-finitary property of non-derivability in another formal system, then surely the belief in the extensive epistemic range of formal systems - of logic - is reasonable.

If the above considerations do not seem convincing, we can point out that the validity and transcendental scope of logic (the highest a priori condition) is an essential assumption in Hume (as we discussed in detail before), Kant, Schopenhauer and Husserl.  We can always ask Kant: what gives reason the right to judge itself and its own limits? Does not transcendental logic have to follow the rules of general logic as well (even if it includes important differences)?  And another important approach involves the formal verification principle: challenge the meaning-as-use ordinary language philosophers and neoscholastic dogmatists  to set up their theory of knowledge and mind in a formal system and then proceed to critique those systems using their own rules. The weak conclusion remains the same: either physicalism is false or (qualified) subjective idealism is true. The stronger conclusion is that if consciousness is essentially non-physical and non-local (or even trans-temporal) then this fact tends to strengthen rather than weaken the case for qualified subjective idealism. And if the dogmatists invoke a direct intuitive "given" or direct intuitive acquaintance with allegedly real exterior things then they have no right to discard the direct intuitions of consciousness being the fundamental reality that can be experienced during spiritual development and altered states of consciousness.

If we are certain that there are universal moral laws known a priori then it would seem to follow that we should treat phenomena which appear to us as being sentient beings as if they really were sentient beings (treat them according to the moral law),  regardless of the non-reality of their first-person phenomenal presentations. This is because we do not have nor could ever have certain knowledge that they do not exist (either entirely or in part) and secondarily because regardless of the situation the practice of moral deeds can only strengthen character and morality itself.  If a child is nice to its toys it will probably be nice to people and animals as well. The similarity of this argument to Pascal's wager is noted.

But subjective idealism is a misleading term because it suggest than an individual ego-subject, finite personality or self-in-the-world is at the same time some kind of substrate of the world. But in reality cosmo-ontology and egology are illusions to be dissolved together. This is the task of transcendental dissolving and liberating insight,  true philosophical phenomenism/positivism/empiricism and skeptical dialectics.

Also considerations on temporality, finitism, mereology, the mysterious nature of concepts and meaning - and in general intrinsic intentionality, synthesis, constitution and constructivity - applied to both the self and experience  - are quite sufficient to establish the above liberating insight with regard to the egological and cosmo-ontological illusion.

How do we interpret quantification from a finitist perspective? How can we develop intentionalist finitism? What does alleged arithmetic knowledge mean from a finitist perspective? Was Hilbert really defeated? Was Leibniz actually a finitist?  Can finitism offer new perspectives on the theory of knowledge? Can we solve the problem of the stubborn non-finitism of category theory? Do we need a new interpretation of Carnap and Russell?

Can statements about numbers involve an infinite number of relations to actual numbers? What is the significance of Ockham here? Can psychologism be revindicated?

The new synthetic a priori principle: finite derivations must ratify and represent an indefinite but bounded set of truths. This is applied also to specification and verification of programs.

And yet, what are logical, semantic, phenomenal, physical atoms? What could they be? Can paraconsistent and infinitary logic be justified in the framework of our "Analyticity, Computability and the A Priori"?

Very short note on formal concepts in science

From whence do we get the impression that some mathematical models are closer to physical and spatio-temporal intuition,  more down to earth and intimately tied to concrete applications, while others hover close to the heights of allegedly less useful  'abstract-nonsense' ? The real numbers and differential equations, these are seen as tied to dynamic-geometric intuition and of vast applicability and interest in engineering and science.  But abstract algebra and category theory appear to have no direct relevance to applicable mathematical models or to their kind of concrete geometric-dynamic intuition. We present here a few short speculations.

Perhaps the abstract models do capture fundamental levels of reality (maybe even more fundamental than the so-called concrete spatio-temporal ones) and what is required is first of all the methodic development of a special kind of intuition or cognition to grasp the planes of reality which they model (maybe something like Goethe's method is called for?). And then further models are required which can effect a mediation or transfer between these two levels, or at least allow a continuity or gradual deformation between concrete spatio-temporal reality and higher levels of reality.

A beautiful illustration of such a meditation and construction is furnished by the double fibration in Penrose's twistor theory which allows a mediation and transfer between physically significant objects on Minkowski space and abstract cohomological objects on the complex algebraic variety $\mathbb{P}^3$.This operation allows solutions to conformally invariant differential equations on spacetime (e.g., Maxwell's equations, Yang-Mills, or linearized gravity) to be identified exactly with Cech or Dolbeault cohomology classes on specific regions of twistor space. This appears to have been Penrose's attempt to give a geometric (topological) interpretation of quantum non-locality. 

Another illustration is the theory of $\mathcal{D}$-modules which presents a mediation between derived categories (and monoidal categories) and concrete models of systems of partial differential equations.

The idea of Kant's schematism of the pure concepts of the understanding can be interpreted as finding the mediation between logic and geometry (including mathematical physics). This is of immense contemporary significance as we find versions of Kantian schematism in topos theory, homotopy type theory and in areas involving monoidal categories in algebra, geometry and physics and in linear logic and computer science. 

There is also apparently a mediation and transition between the theory of bifurcations of smooth vector fields, fundamental  and ubiquitous in concrete applied mathematics,  and monoidal category and operad theory. See also Physics, Topology, Logic and Computation: A Rosetta Stone by John C. Baez and Mike Stay.

An objection can be raised that our concrete allegedly intuitive mathematical models are themselves quite abstract but are only perceived as immediate and intuitive in our present cultural context by a deliberate forgetting of the complexity of their cognitive-historical past and genesis. For instance real numbers are constructed from the rationals by quite abstract, cardinality increasing, procedures involving equivalence classes and identification. And the same goes for the concepts of continuity, differentiability and so forth. Surely the ancient Greeks had other forms of intuitive perception and concreteness in their geometry, physics and engineering. We answer this objection by our arguments that the fundamental dynamic and geometric-topological intuitions of modern science were in fact identical to those of ancient Greece. It is the modern foundations in terms of Dedekind cuts and $\epsilon$s and $\delta$s that can be critiqued from a philosophical and logical point of view and alternative foundations (locales, realizability topoi, synthetic differential geometry, homotopy type theory) can be defended which are also more aligned to the theory of space and change found for instance in Aristotle's Physics.

Meditation, phenomenology and obfuscation

There is an important missing link in the history of philosophy which has significant consequences for the history of phenomenology. This is the work of the young Rudolf Steiner. It is not fair to judge this work by his later abhorrent and manifestly false views as an occultist and cult leader any more than it would be fair to judge Heidegger's Sein und Zeit by his later similar political and cultural-anthropological views. The most important work of the young Steiner (who attended lectures by Brentano) is a work which usually goes under the title of Philosophy of Freedom (1894) but was published and revised numerous times under different titles.  Along with this work there is also much value in Steiner's earlier work on Goethe's scientific method and worldview. 

There can be no doubt that The Philosophy of Freedom, like Sein und Zeit, is a very important work and one of the masterpieces of phenomenological philosophy. In this work (and in portions of his later books such as the one on "Initiation") we find something similar to an exposition of the approach and methodology involving the transcendent self-transparency and self-illumination of thought and consciousness which we have repeatedly attempted to expound, the correct combination of "epokhê" and "satipatthâna".  

Equally important is Steiner's Goethean method of variation and "imagination". This seems to us to be the original and correct form of what later was called the method of "eidetic variation" (and the affinity for Ruskin's approach in Modern Painters seems to have gone unnoticed).  But the consequences are enormous. The combination between the philosophical-spiritual insight offered by Steiner's exposition of the Goethean method on one hand and on the other the development of modern geometric and topological theory of differential equations and dynamical systems would have been a "marriage made in heaven", two complementary aspects of the same essence,  whose union would have furnished the key to a powerful new science.

But the harmful influence of 20th-century materialist cults and their pseudoscientific dogmas (Freudian psychoanalysis, evolutionary psychology) hindered the emergence of such a science. This is the tragedy of René Thom whose work flawed by Freudian and materialist-evolutionary dogma did indeed point to such a qualitative topological science with a method based on developing higher dimensional dynamic and geometric intuition. The fact that Thom never mentioned Theodor Schwenk's Sensitive Chaos: The creation of flowing forms in water and air (preface by Jacques Cousteau) which predates Structural Stability and Morphogenesis can only be seen as an example of intellectual dishonesty. As far as I know, Thom never referred to the wonderful book by A.T. Winfree, The Geometry of Biological Time (1980) which shows how advanced topological concepts in applied science can be presented in an accessible, rigorous and clear way (cf. the explanation of phase singularities on p.26).

A major fault of René Thom was also ironically to be found on the mathematical plane. Thom, like V.I. Arnold, engaged in obfuscation and sloppy presentation of mathematical ideas.

An expert in a field has all the right to develop a conceptual shorthand for their own use or that of fellow experts. In particular, a shorthand for writing down ideas to be developed and unfolded in lectures. Such texts can also employ beautiful drawings and diagrams which may impress even the non-expert. However all this has nothing to do with writing a book whose aims is to open the gates of a knowledge to those who, while not having such a knowledge, seek earnestly for it. A book that is a genuine introduction (i.e. initiation) in the full sense of the word, and which does not only "bring in" the neophyte but presents the beauty, clarity and order of the temple in all its details and adornments.  This is the kind of book, the kind of spirit, style and approach, which every student deserves, and certainly the kind of spirit. style and approach which should be employed when presenting important philosophical, scientific and mathematical ideas to the world. To present the young student with any other kind of book can only be considered a pernicious and immoral act of deliberate obfuscation and perversion. On another occasion we hope to develop the topic of cryptography, deliberate obfuscation and "secrets of the trade". Let it just be said that this has traditionally been a not infrequent vice of academia, the deliberately ponderous, opaque, pedantic jargon which not only hinders understanding of comparatively simple concepts and arguments but also masks the author's debt (or even plagiarism) to their contemporaries or predecessors. The aim is also to appear outwardly as outstandingly intelligent, original and sophisticated and to close the doors of science to anyone outside a select few.

Besides in phenomenological philosophy and Goethean science, a tragedy occurred regarding "meditation".  There is no such thing as "meditation" in the popular sense, there is no secret powerful technique of self-help where one sits down, closes their eyes, breathes in and out, focuses their mind or attempts to detach themselves from troubling thoughts, and thus finds a cure for mental anguish or a way to gain paranormal faculties or blissful states. This goes in particular for everything connected to the term "mindfulness".  And there is evidently no easy quick philosophical-phenomenological method for the self-observation and self-transparency of consciousness.

This is not what "bhavana" or "katharsis" as in neoplatonism and the ancient mysteries was about. Rather bhavana can be likened to the building of a beautiful high-towered palace or temple, including the preparation and purification of all the materials needed as well as the ground and foundations. The "mindfulness" at work here cannot be separated from action. The popular concept of "meditation" actually only corresponds to the final royal stage in which one enters the newly built palace and climbs up many flights of stairs to survey the beauty of all the sparkling rooms and luminous windows until reaching the most wonderful of all views at the top. Before that it is the long toil and active work of the artisan and soldier - which can be be interwoven with periods of peaceful surveillance and contemplation of yet unfinished work. We do not meditate as some kind of easy, all-sufficient, way to overcome our desires, rather we first diligently and actively uproot desire-images-complexes in order to attain the peace and illumination of "meditation".  Using a software engineering metaphor, we must start with cultivation at the abstract object-oriented level before proceeding to reverse engineering at the assembly level. Thus taught both the Pali suttas and Plotinus.

Unwanted popup on this blog

Unfortunately the theme of this blog employed an outdated and suspicious external script from polyfill.io. This script has been removed.