A Lisp REPL as my main shell

A Lisp REPL as my main shell(ambrevar.xyz)

149 points by qazwse_ 5 years ago | 30 comments

ssivark 5 years ago |

It’s interesting to note that while no single idea is revolutionary, the design choices in putting together this bunch make for a particularly slick interface where the whole is much better than the usual shell.

PS: I look forward to trying this out! Since I don’t know much Common Lisp, are there particularly useful libraries that would make it easy to cover functionality commonly used through the shell? I’d love to see a tutorial with a compilation of examples. EDIT: Found some useful stuff in the CL cookbook (Eg. section on files & directories) https://lispcookbook.github.io/cl-cookbook/

Ambrevar 5 years ago | |

The article presents a few libraries. There is https://github.com/ruricolist/cmd in particular which is very helpful.

I'm going to publish a few more libraries which should help with file manipulation (as I demoed it).

Stay tuned!

nerdponx 5 years ago | |

UIOP and Alexandria are very common. The former has a lot of functionality that you would normally expect out of a shell.

dTal 5 years ago |

>Terminals have no reason to continue to be used in my opinion. Note that this does not mean we shouldn’t use “textual” interfaces, quite the opposite: textual data is a bliss to manipulate. But we can very well manipulate text, along with other types of data, in something other than a terminal, that is faster, prettier, more powerful. (Graphical Emacs is one such example.)

I have recently switched to Jupyter's qtconsole for my REPL needs - for me, it hits the sweet spot of supporting rich media and interaction without the overhead of running a full web browser stack. I could see it becoming the basis for a next-gen terminal.

Ambrevar 5 years ago | |

Agreed, I think Jupyter got many things right, in particular when it comes to prompt handler and data visualization.

What I find limiting for now is interactions with the shell process, in my case the Common Lisp compiler: no interactive stacktrace, no debugger, etc. This is very limiting. I don't know if there is a way around it, as this could be a limitation of the Jupyter design with its kernels. Please let me know if there is a way out! :)

junke 5 years ago |

While not as extreme as what is demonstrated here, I integrate things in my environment with Lisp in a way that is similar to this. I run StumpWM too so I can grab the content of the current X selection and do actions from it (e.g. the content matches a JIRA ticket regex, open it, the text matches a filename, edit it (fun fact, anything can be a filename), etc.).

For processes, I wrote something that is not finished, not polished, but if anyone want to steal from it, go ahead:

https://github.com/christophejunke/pipeline

This allows to spawn processes or threads to pipeline input/output, like done usually with | in shells

    (with-pipeline ()
      (program "ls" (namestring (merge-pathnames "bin/" (user-homedir-pathname))))
      (program "sed" "s/a/aaaa/")
      (tee/error) ;; "tee" and output a copy to error
      (program "wc" "-c")
      #'read)

https://github.com/christophejunke/pipeline/blob/master/test...

The nmcli.lisp example starts an nmcli process and filters its output to build Lisp objects representing connections:

https://github.com/christophejunke/pipeline/blob/master/nmcl...

Ambrevar 5 years ago | |

Wow, looks like you've got a lot of cool stuff in there! I'll try it out later and give you some feedback!

vindarel 5 years ago | |

Hey please write a readme, your project is worth it!

max_streese 5 years ago |

I wonder what people think about Ammonite (https://ammonite.io/)?

It's not Lisp but Scala so may not be the authors language of choice however it can be used as a Shell: https://ammonite.io/#Ammonite-Shell

I am personally using it and compared to a classical shell like Bash it's really nice for more structured data related tasks (exploring some API, checking some data, creating a bunch of PRs at once, ...).

It also makes use of Scala's adjustable syntax and functional concepts so you basically get shell piping but in a strongly typed fashion (e.g. `ls.! |? (_.ext == "txt") | (os.read)` would produce a list of strings representing the contents of all files ending in .txt of the current directory).

Also I am curious what people think of PowerShell which as of PowerShell Core is usable on all platforms as well.

majkinetor 5 years ago | |

Looks almost exactly like PowerShell in syntax but I suppose its even more powerful given lisp heritage.

Its hard to beat PowerShell tho given its integrated in infinite number of tools on Windows platform and growing number on Linux.

pjmlp 5 years ago | |

PowerShell is the only mainstream shell that builds on the ideas of Lisp Machines, Xerox PARC and ETHZ workstations, naturally I am quite found of it, despite some of its quirkiness.

It does structured data, native support for extending it via .NET and native libraries and at least on Windows, can plug into OS Automation libraries.

What it misses is graphical display on the REPL.

I can easily see PowerShell Core replacing my uses of Python in what concerns Linux VMs.

swiley 5 years ago | | |

I think this talk actually came up in a thread on the guile mailing list about adding powershell as a guile language.

skissane 5 years ago |

> Implement automatic ncurses detection to automatically start a “visual” program in a terminal, even when it’s not known in advance. Is it even possible?

There are a couple of techniques:

1) Check for use of certain escape sequences in the output.

2) Check PTY terminal modes using tcgetattr, especially for raw mode

I think combining (1) and (2) you could come up with some heuristics that would work 99% of the time, without hardcoding names of fullscreen apps or anything else like that.

(Really this isn't "ncurses detection" per se, you can have a full-screen terminal app without using any curses library.)

regularfry 5 years ago | |

Can you switch mode once the executable's already been launched, though? It's already got its tty, so you'd presumably need to start up the ncurses-capable terminal on demand, switch it in, and play back any meaningful prior output so it's in sync. The page is dead so I can't see how they handle it, but each of those steps is potentially problematic, isn't it?

skissane 5 years ago | | |

> Can you switch mode once the executable's already been launched, though?

Yes, apps can change terminal mode at any time. The solution the author went with is to have a list of "visual program" names, and when a visual program name is detected, launch it in an external terminal (xterm or Emacs vterm). That is making no attempt to handle mode switches during program execution, it is making a permanent decision before the program starts based on its name.

I think the solution here is to have a visual terminal "widget" that can be embedded into the graphical REPL, and the REPL constantly monitors the output and terminal mode of the subprocess, and when the REPL detects initiation of visual mode, start an embedded visual terminal widget, and then stop the visual terminal widget when it detects visual mode is completed.

On Linux, it is possible for the pty master to be immediately notified of any mode changes in the slave [0], using packet mode and EXTPROC. I don't think that facility exists in most other operating systems.

[0] https://stackoverflow.com/questions/21641754/when-pty-pseudo...

marcodiego 5 years ago |

I don't get lisp. Actually I think I understand what people like about it. What I mean is that I don't know exactly how to "think in lisp". Is it like writing the AST instead of the text or describing what is needed instead of what needs to be done... I simply don't get it.

Not sure if I'm the only one though.

mapgrep 5 years ago | |

What you said is true and perceptive — it is at base like writing the AST - but most of the time you’re just doing function calls with the opening paren shifted to the other side of the function name.

In other words, in Algol like languages you’re typically calling like(this) (or the dotted.equivalent(form)) and in lisp it’s (like this).

You do eventually take advantage of the fact that you’re writing an AST, when you get into macros. But that feels pretty simple once you’re used to the function syntax, since macros just manipulate the resulting structures.

What used to throw me off was the special syntax for lists - ‘(structures looking like this) - but once you get used to using the lisp structure for function calls it becomes clear why and when this kind of quoting is necessary (any time you want a list as data instead of a function call).

jmkr 5 years ago | |

Take any idea of a data pipeline, something like with rabbit or kafka. Now describe that in functions by composition.

read-data -> push-to-rabbit -> read-from-rabbit -> send-ack -> push-to-http.

Turns out with functional composition + data structures you can do a lot.

With the idea of immutability or persistent data structure with say Clojure, a lot of idea start to make sense. Rich Hickey videos are pretty good at describing simpler programs. Even if you never use Clojure, or Lisp, or Scheme, these idea might make you a better programmer. At least they did for me. My Python code is now much simpler because I realized I didn't need everything I was doing.

1MachineElf 5 years ago |

You may be forced to use Scheme as as shell when encountering an error booting GuixSD. Back in 2016, support was apparently not good for the AMD bulldozer systems I was trying to install on. The systemd alternative they used, GNU shepherd, crashed on me, and left me in a Guile REPL.

jmercouris 5 years ago |

A very well done article! I wonder what an integration with a Python REPL might look like.

nethunters 5 years ago | |

Xonsh is a fully featured Python shell with shell syntax added. https://xon.sh/

Ambrevar 5 years ago | | |

(To reiterate what I answered for Ammonite, s/Ammonite/Xonsh:)

I've only scratched the surface of Xonsh, so please correct me if I'm wrong.

From what I understand, Xonsh was designed as a "readline shell" as I wrote in the article. It perpetuates this approach that everything is a command.

The thesis of my article suggests we do the opposite: I'm suggesting to rethink shells by starting from the interface (here the SLY REPL) and then implement the shell features. In particular, it seems that Xonsh does not support back-references and I'm not sure it has an interactive inspector (or does Emacs python mode provide one?).

While Xonsh seems to be a definite improvement over the syntax of Bash, etc., I'm not sure it brings much novelty in terms of user interface. But again, I know very little about it so I may have missed some features, in particular regarding the Emacs integration :)

markus_zhang 5 years ago | | |

Thanks! I was about to ask exactly the same question.

webgoat 5 years ago | |

You can use the magic command `! ` in IPython to write shell commands with support for f-strings i.e. ``` arg = "test" out = ! echo {arg} ```

konjin 5 years ago |

Back at uni I ran an experimental lisp shell. A few notes of what was painful and what worked.

Outer most parens should be optional, if you're running ls you don't want to type (ls).

Grouping flags, files and options is difficult: ls -lh file could become (ls (h l) file) (ls (flags h l) (files file)) (ls -l -h file) (ls -lh file) none of the options are particularly good or lispy.

You need a full editor on the command line. Shells are designed for teletypes, lisps are designed for glass terminals.

All in all not I'm sure that an s-expression based system is the right fit for a low level interface to the computer. That said I did enjoy my experiments a lot more than I did using shells. Were I to try it again I would have it is a higher level language that compiles down to a shell.

Ambrevar 5 years ago | |

I believe I answered most of your notes in the article ;)

ashkankiani 5 years ago |

"rather novel paradigm" is where you lost me. TempleOS's integration with HolyC is more featureful than anything you've proposed and it was completed a few decades ago. The idea was surely older than that, too. I'm just pointing out a very well known case.

A bit late on the uptake, but welcome to the party of realizing that the shell is not be all end all of computing.

(with-pipeline () (program "ls" (namestring (merge-pathnames "bin/" (user-homedir-pathname)))) (program "sed" "s/a/aaaa/") (tee/error) ;; "tee" and output a copy to error (program "wc" "-c") #'read)