Common Lisp implementation of the Forth 2012 Standard

(github.com)

89 points | by wild_egg 11 hours ago ago

52 comments

  • packetlost 6 hours ago ago

    Very interesting! For those who do not pay too much attention to Forth, the forth standard [0] still sees fairly regular updates. GForth, the implementation that I've spent the most time with, gets several commits per week, so it's not dead, though I suspect that it's a small handful of individuals who use it frequently that are keeping it alive and not widespread use.

    Forth is absolutely worth learning, if for no other reason than understanding how stack-based interpreters work. The CPython VM, as an example, behaves similar in some respects to the average Forth program.

    There's definitely better examples out there, but here[1] is an implementation of Mage the Awakening 2e's (tabletop game) spellcasting system in Forth that I wrote to help with playing awile ago.

    [0]: https://forth-standard.org/

    [1]: https://git.sr.ht/~chiefnoah/spellbook/tree

  • vdupras 7 hours ago ago

    This is too much to bear and I had to create an account for this!

    Might I interest you in the opposite? Sort of a Lisp implemented in Forth that can compile Lisp functions to native code, all of this in less than 500 lines of Dusk code.

    https://git.sr.ht/~vdupras/duskos/tree/master/item/fs/doc/co... http://duskos.org/

    • kragen 44 minutes ago ago

      Condolences on finally opening an HN account. I greatly appreciate your work even though I disagree with much of it. DuskOS in particular seems excellent.

      I hadn't looked at comp/lisp before. It definitely looks like a real Lisp to me. In https://git.sr.ht/~vdupras/duskos/tree/master/item/fs/mem/co... I see that you have dynamic typing, implemented in assembly, and from https://git.sr.ht/~vdupras/duskos/tree/master/item/fs/mem/co... to line 70 you have a garbage collector. From https://git.sr.ht/~vdupras/duskos/tree/master/item/fs/comp/l... I see you have closures, but I'm not sure how memory allocation for their captured variables works; from https://git.sr.ht/~vdupras/duskos/tree/master/item/fs/comp/l... I infer that you are just using dynamic scoping, thus ruling out both upward funargs and tail-call elimination?

      What's the most interesting program you've written in comp/lisp?

      • vdupras 18 minutes ago ago

        > What's the most interesting program you've written in comp/lisp?

        None. I wrote comp/lisp because my lack of Lisp knowledge seemed to me like a liability as I invest more and more time in Forth. I had nothing I felt like writing in Lisp, so I thought it would be fun to implement one.

        While I can't call myself a Lisp programmer, I think I can say now I have an idea of what it's about. It didn't grow on me, so I leave it aside for now. Maybe I'll come back to it later.

        > I infer that you are just using dynamic scoping, thus ruling out both upward funargs and tail-call elimination?

        Tail-call elimination is possible with my implementation within the same function, but it's not "natural". I'm not sure of what it would take to have it like we see them in other lisps, or if it's possible at all. It's been a while since I swam in that pond, details aren't fresh in my mind.

        As for scoping, I'm unfamiliar with the theory. I don't think it's what we call "lexical scoping" because variable names have very little meaning in the compilation process. It's eliminated very early and only "compile time Parameter Stack offset" is kept in the AST.

        I don't know what you mean by "upward funargs", but if it's to access arguments from outer scope, yes it's possible. That's exactly the point of the last link you pointed to: lambda generation. A call to a closure generates a lambda which has its outer scope "injected" to PS.

    • wild_egg 3 hours ago ago

      This is excellent as well. I don't really jive with the Collapsnik aspect but I absolutely love powerful systems that can run in constrained environments.

      Thanks for sharing, I'll be diving into this for a while

  • codr7 2 hours ago ago
  • mepian 9 hours ago ago

    The author is also one of the developers of the Symbolics Virtual Lisp Machine used in Open (and now Portable) Genera: http://pt.withington.org/publications/VLM.html

    • amszmidt 8 hours ago ago

      Open in the "closed" sense. The legal status of anything from Symbolics is a minefield. Even this "portable Genera" might be of dubious legality...

      "Open" here was a popular term back in the day and bunch of companies used it .. sorta like AI, LLM, and blockchain.

      • fsckboy 4 hours ago ago

        >"Open" here was a popular term back in the day and bunch of companies used it .. sorta like AI, LLM, and blockchain.

        nah. what you are describing is Open in the present day, like OpenAI.

        back in the day it meant "open standards, allowing cooperation between secretive competitors". IBM and Digital could both implement to an open standard instead of their traditional proprietary so their devices would interoperate.

        This type of openness is what Microsoft liked to "embrace, extend, extinguish", meaning "we'll get kudos joining the interoperating consortium, we'll look cutting edge by adding features which would make us not actually interoperate but are useful so people will adopt them, then we will have killed off this threat to our monopoly"

        in that same period of time, open source started to be called that because what was open was the source.

        • anthk 4 hours ago ago

          Back in the day there was ITS, Maclisp and Emacs on top of TECO.

          • lispm 4 hours ago ago

            That was gone at a time when Lisp Machines were commercialized (81/82). Emacs there was called Zmacs and was written in Lisp. Maclisp was replaced by Lisp Machine Lisp / ZetaLisp / Common Lisp. UNIX was then a thing.

            Open Genera in the early 90s then ran on OSF/1 UNIX (https://en.wikipedia.org/wiki/OSF/1 , renamed to Digital UNIX) on DEC Alpha. OSF meant Open Software Foundation ( https://en.wikipedia.org/wiki/Open_Software_Foundation ), a group propagating "open standards" for UNIX.

            • anthk 3 hours ago ago

              Curiously, GNU was born in that era, to give the Unix users the freedom from ITS, Maclisp, Emacs and the rest of the projects from MIT.

              Instead of having slow-ish boot times in a Lisp machine, GNU's plan was to put several memory-protected concurrent interpreters under Unix, giving the user far more power by default with GNU/Hurd against a common Unix users. WIth namespaces, for instance, you could forget about suid and/or needing group perms to mount a device or access different media and soundcards.

      • Y_Y 7 hours ago ago

        I'm sure if they weren't incentivised by the certainty of creating a legal minefield in the future then this technology would never have been created and we'd be sitting at home banging sticks together.

        • amszmidt 6 hours ago ago

          Most of this ”technology” was created at the MIT AI Labs,

          Genera did a few cool things but in the grand scheme of things it is essentially not much (the Dynamic Listener probably being the most interesting thing, the rest of the system was essentially the same).

          The MIT version of the Lisp Machine is free software without any legal conundrums. So there you go …

          • lispm 5 hours ago ago

            > rest of the system was essentially the same

            not true

            • Y_Y 3 hours ago ago

              I usually decry this sort of response and as virtually useless and not letting to fruitful discussion. However I have to commend lispm and amszmidt on having a fascinating (if tense) discussion on the history of Genera and its variants.

            • amszmidt 5 hours ago ago

              Very true, try doing a M-x Source Compare. Genera is backwards compatible with System 99. And any differences are trivial.

              • lispm 5 hours ago ago

                I'm looking at a Symbolics file system. Where do I find Dynamic Windows in System 99? Common Lisp? CLOS? The Document Examiner? X11? NFS?

                • amszmidt 5 hours ago ago

                  Nobody was saying that Genera didn't do anything novel, Dynamic Windows being one, and the Dynamic Listner being another. Overall, the guts are essentially the same, ZMacs, Zmail (also not part of the base System), Flavors, and anything else in the base layer -- to the point that you can compare all of the files that get stuffed into the cold load image without much effort.

                  System 99 has support for Common Lisp. System 301 will hopefully have better support than Genera (seeing that it doesn't really support CL very well anyway) at that.

                  The Document Examiner is a separate system (and LMI did it differently, with Gateway), and not part of the base System. X11, and NFS likewise. So listing a bunch of orthogonal systems isn't really what was being discussed.

                  • lispm 4 hours ago ago

                    The core Symbolics SYSTEM system I see here is version 501.51. It consists of a bunch of other systems like SI, Embedding, L-Bin, Tables, Scheduler, Netboot, Common Lisp, SCT, Garbage Collector, Flavor, Error System, Language Tools, Network, Lisp Compiler, Bitblt, Fonts, FS, Bin, Time, TV, Presentation Substrate, CP, Dynamic Windows, Debugger, I Linker, Fep-FS, ...

                    I don't see much of that in System 99 from MIT.

                    Stuff like X11 and NFS is also not random stuff. Open Genera uses it for its console and its files.

                    > seeing that it doesn't really support CL very well anyway

                    Well enough that it is able to run a lot of its software written in Common Lisp. It's able to run stuff like PCL, ASDF, ITERATE, ...

                    System 99 is from 1984 (?), Symbolics started in 81/82. Open Genera appeared 1992/93. You are ignoring a full decade of development.

                    Earlier:

                    > Genera did a few cool things but in the grand scheme of things it is essentially not much (the Dynamic Listener probably being the most interesting thing, the rest of the system was essentially the same).

                    I would guess that less than 20% of the core SYSTEM is essentially the same (but often quite a bit enhanced). For the rest of the software, I would think it's even less. Which is not surprising, since System 99 did not support the 3600 systems, the G-Machines, the I-machines (XL400, XL1200, XL1201, UX400, UX1200, NXP1000, MacIvory 1/2/3, ...), Open Genera, ..., because all that was developed later by Symbolics.

                    • amszmidt 3 hours ago ago

                      > I would guess that less than 20% of the core SYSTEM is essentially the same (but often quite a bit enhanced). For the rest of the software, I would think it's even less.

                      ZWEI, TV and SI makes up the majority of the base system which is far more than "20%" percent, which is ignoring EH, WINDOW, and anything else that is essentially processor agnostic.

                      The target processor is not that important from the users point of view, to the point that Lambda and CADR shared the exact same source code, and microcode. The Explorer I was essentially a Lambda. Much was "feature" protected with #+/#- to the point where many of the systems worked on Genera as well.

                      So that seems at par with the CPU architectures that Symbolics made, which is "just" a change to the compiler .. and not the overall system .. which is what was being discussed. But yes, Symbolics did design hardware done by Symbolics and made sure it worked on their system .. not sure what your point is other than giving an incorrect view of history and what Symbolics actually did to the Lisp Machine system parts (Symbolics work was much more interesting when it came to other systems like the 3D stuff and what not).

                      • lispm 2 hours ago ago

                        ZWEI is no longer in the base system. EH does not exist, a Window system does not exist. Dynamic Windows is in the base system.

                        > The target processor is not that important from the users point of view

                        It's important for the operating system and the processor was used for very different machine hardware: standalone workstations, headless systems, embedded in a Mac, embedded in a SUN, embedded in a network switch, ...

                        > Lambda and CADR shared the exact same source code, and microcode.

                        But not the 3600, which was a 36 bit architecture and not the Ivory, which was a 40 bit microprocessor. A lot of the core routines were different on a 3600 and an Ivory machine. The VLM was even different, running on a 64bit microprocessor, implementing the emulator in C and assembler. There is low-level functionality which has two or three different implementations.

                        The Ivory machines could be very different from a LAMBDA and a CADR, for example as an embedded CPU in a very different host machine (Mac & SUN). That had a lot of consequences for the operating system.

                    • amszmidt 4 hours ago ago

                      SI, Scheduler, GC, Flavor, EH, Compiler, TV, etc all date back to before Symbolics entered the picture. And if you don't see that in System 99 .. then you must not be looking very hard since they are explicitly mentioned there. You mention a bunch of things that are also _not_ part of Genera, specifically bunch of Ivory stuff -- which is just the processor target.

                      > Well enough that it is able to run a lot of its software written in Common Lisp. It's able to run stuff like PCL, ASDF, ITERATE, ...

                      No it is not. You are thinking of Portable Genera.

                      > System 99 is from 1984 (?), Symbolics started in 81/82. Open Genera appeared 1992/93. You are ignoring a full decade of development.

                      System 99.32 is from 1987, which continued via LMI into System 130. Genera, or before when it got forked, is from 1978 when it got forked from System 78 or there about being re-branded into System 200. All three (four if you count TI -- which did quite more in renaming things making it hard to follow) with parallel development going on from there into the early 1990s.

                      Genera is a Lisp Machine system based on the work done at MIT, where most of the guts still the same. To the point that how you rebootstrap the system is the same, how the core areas look work, how the scheduler works, how the windowing system works, how flavours works, etc.

                      Open Genera did very little on top of Genera at that, mostly targeting and making it work on the Ivory. You are purposefully conflating Portable Genera, Open Genera and Genera.

                      • lispm 3 hours ago ago

                        I have literally started a VLM (Portable Genera 2.0.6) with a Genera 9.0.6 world. In the "herald" the software says that it is running Genera 9.0.6. You can claim "You are purposefully conflating Portable Genera, Open Genera and Genera.", but I'm actually looking at it now and you don't. Please don't tell me this BS, when I'm right now reading the screen in front of me.

                        > You mention a bunch of things that are also _not_ part of Genera

                        Of course they are. I have the thing right in front of me, running. I'm typing to it. I'm looking at the system definition of SYSTEM 501. On a live running Genera 9 on a VLM. If I would look at my MacIvory running Genera 8 it also would not look much too different.

                        Stuff like EH has been long superseded. The Scheduler has been redesigned & rewritten. TV has mostly been superseded by Dynamic Windows. The Garbage Collector has been extended, by new GC variants.

                        > Open Genera did very little on top of Genera at that, mostly targeting and making it work on the Ivory.

                        Open Genera does not work on the Ivory processor. It's a Virtual Lisp Machine.

                        > Genera is a Lisp Machine system based on the work done at MIT, where most of the guts still the same. To the point that how you rebootstrap the system is the same, how the core areas look work, how the scheduler works, how the windowing system works, how flavours works, etc.

                        The Scheduler works different (the old scheduler polled, the new own is event driven), the window system is now Dynamic Windows, Flavors has been updated to New Flavors & the new CLOS (the Common Lisp Object System), ...

                        • r40694 an hour ago ago

                          you know you don't need tell everyone how you have palter's vlm to look at rel-8-5's sysdcl which has been leaked and hosted on public sites since forever. for example https://archives.loomcom.com/genera/symbolics/sys.sct/sys/sy... rel9's sysdcl is not substantially different anyway, the list of module components is the same.

                          ams's hyperbolic perhaps point is that genera is significantly SYSTEM, that symbolics contribution is a kind of obvious extension of the grand vision that was already there in its totality and potential in the MIT's work. I think it's a valid argument, which I don't think can be resolved just by listing names of subsystems.

                          for example you can't just say "oh they replaced tv and window with dynamic windows", because dw uses both tv and legacy, for lack of better term, window. if you look at the flavor definition of basic dynamic-window it uses tv:stream-mixin tv:graphics-mixin and tv:minimum-window. and tv minimum-window is a venerable SYSTEM flavor. not to mention that other systems (like zwei) still use tv window directly. how thick a layer dynamic-window is on top of tv? answering that question require systems level knowledge and investigation.

                          other symbolics extensions are of similar nature.

                      • mepian 2 hours ago ago

                        >Open Genera did very little on top of Genera at that, mostly targeting and making it work on the Ivory.

                        Ivory support was implemented in Genera 7.3. Open Genera added the virtual machine to run on DEC Alpha hardware, it was Genera 8.3 + VLM, upgraded to Genera 8.5 with Open Genera 2.0.

                        • lispm 2 hours ago ago

                          and now it's a new portable VLM (DEC Alpha, x86-64, ARM64) with Genera 9.

      • hayley-patton 8 hours ago ago

        "We changed the name to OTP." "Three of the trendiest names we could think of. Open. Telecom. Platform."

        https://www.youtube.com/watch?v=rRbY3TMUcgQ

    • anthk 5 hours ago ago

        Not free. This is useless for preservation. Just compare it against Medley/Interlisp.
  • kunley 8 hours ago ago

    I hope there will come time for Forth to get more attention, or dare I say, to stop being so underrated.

    It is so great tool, even just for stretching your brain cells..

    • sph 3 hours ago ago

      I recently posted a paper on the Permacomputing Aesthetic (https://news.ycombinator.com/item?id=41818119) which argues there is a counter current of developers that want to go against the maximalist world of computing and rediscover programming as art on constrained systems. In this world, where we do not idolize the large teams of code monkeys, languages like Forth can see a resurgence. Forth is a bad language for a team of people of dubious and different skillset, but in the hands of a single person it is the most moldable environment, even moreso than Lisp.

      I have been designing some sort of "persistent" Forth OS, that boots into an interpreter and persists your defined words across sessions and reboots, so you can incrementally build your own personal computer from scratch.

      • packetlost 34 minutes ago ago

        I think I partially fit into the counterculture and have had similar ideas for a persistent Forth OS. I'd love to see your work!

      • mepian 2 hours ago ago

        >even moreso than Lisp

        How exactly?

        • kragen 27 minutes ago ago

          Let's take Laxen and Perry's F83 implementation as an example.

          F83 implements an assembler, a memory hex dumper, an interpreter, a sort of bytecode compiler, a decompiler, a single-stepping source-level debugger, a screen editor (which could jump to the definition of any function and jump back and forth between source and documentation, and which tagged each screen of source code with the last edit date and author's initials), cooperative multitasking with thread-local data, a print spooler, and virtual memory. It can also compile programs into standalone executables, and in fact it's written in itself, so it can cross-compile itself for other architectures and operating systems; it supported CP/M-86, MS-DOS, and CP/M for the 68000.

          The Forth programming language it implements is low-level but relatively powerful; among other things, it includes structured control flow, Turing-complete compile-time macros, closures, pointers, and multiple return values.

          This is nothing unusual for a Lisp system, of course. What is different is that F83 is only about 2100 lines of code by my count. That's what makes it "more moldable" than a Lisp system with the same features, which would require around an order of magnitude more code.

          There are some major compromises made in pursuit of this level of minimality, though: no type-safety (not even to the extent of calling functions with the right number of arguments), no built-in structs (though you can build them with its macro system and closures), no bounds-checking on arrays, error-prone manual access to the virtual-memory system, no syntax, no stack-allocated named variables, etc.

        • transfire an hour ago ago

          Probably because Lisp requires a GC. With Forth you have to write your code to fit a finite memory footprint.

          • mepian an hour ago ago

            >you have to write your code to fit a finite memory footprint

            This sounds like the opposite of having the more moldable environment.

    • ink_13 8 hours ago ago

      Too bad I no longer have my HONK FORTH IF LOVE THEN bumper sticker

    • codr7 2 hours ago ago

      I don't see standard Forth going anywhere; same for Common Lisp, Scheme & Smalltalk.

      They had a good run, and they're still fun to play around with and useful for teaching and stealing ideas from. But the world has moved on in too many ways.

      There's still a lot of ground to explore between Forth and Lisp though, and plenty of modern ideas to steal from scripting languages.

    • packetlost 6 hours ago ago

      I doubt it ever will. There's still pockets on the internet and a few tireless maintainers, but I doubt it'll ever see much of a resurgence. It's just too different. Maybe during "bring-up" of embedded systems, but even that I think is mostly on the way out sadly.

    • whobre 5 hours ago ago

      That time was the 1980s. After 1995 or so, Forth sadly started to disappear.

  • ofalkaed 4 hours ago ago

    The use of CODE/;CODE/;ENCODE for defining words in lisp is great. I can see myself using this a good amount especially if I can get it playing well with Tk either through the FFI or lTk/nodgui.

  • crest 4 hours ago ago

    Doing it the other way around would be a lot more impressive. ;-)

    • astrobe_ 4 hours ago ago

      There are more useful thing to do with Forth :-P

  • pmarreck 5 hours ago ago

    “CL-Forth compiles with LispWorks but crashes running the Forth test suite.”

    wouldn’t this normally preclude doing any actual work with it?

    • shawn_w 4 hours ago ago

      I wonder if they were testing with the paid, commercial version or the free evaluation version of LispWorks, which has significant restrictions.

      Most people playing around with this will be using sbcl though. It's overwhelmingly the most popular free common lisp system.

    • zulu-inuoe 5 hours ago ago

      You can use CCL

    • anthk 4 hours ago ago

      Use SBCL.