Why Create a New Unix Shell?(oilshell.org) |
Why Create a New Unix Shell?(oilshell.org) |
https://www.gnu.org/software/bash/manual/html_node/The-Restr...
I believe Oil will be able to do this, because the architecture is very modular. See the last point in the post using the LLVM / GCC analogy.
(This type of feature isn't a priority now, but I'm interested in hearing use cases.)
https://www.reddit.com/r/ProgrammingLanguages/comments/7qn14...
Bizarrely (to me), more than one person thought the name was a play on the company "Shell Oil". Is that the connotation you got from it?
That's unfortunate, but I think as people use it more, the name will take on a different connotation. Guido was fighting "Python == snake" for a long time too (it comes from Monty Python). There were a lot of people that said the name Python was stupid and you couldn't convince your boss to use a language with a name like that.
I jest.
1) I prototyped it in Python; the dependency on the Python interpreter will be removed [1]
2) Oil went through many implementation languages, and one incarnation was 2000-3000 lines of C++. But I realized I would NEVER finish that way. The goal is to be compatible with bash, which is a tall order.
3) Oil is heavily metaprogrammed. It's only 16K lines of Python, compared to 160K lines of bash, and it can run some of the most complex bash programs out there. [2]
It's more accurate to say Oil is written in Python + ASDL [3], i.e. somewhat in the style of ML.
[1] https://news.ycombinator.com/item?id=16277358
If you expect the shell process to need to make use of true thread-based concurrency, then that might be a reason not to use python.
Do we have either of the above expectations? What other reasons are there for python to be inappropriate?
I said that you can't convince people not to use bash or PHP by writing posts on the Internet, which is true.
I also said that Facebook is replacing PHP, which is true. That's not a criticism of PHP. The fact that huge companies like Yahoo and Facebook can be started with PHP is amazing.
I think PHP is a good analogy for bash. It gets the core things right, and it gets a ton of work done. I like languages you can get work done in! That's why I use bash.
But both languages also evolved a lot of warts. That's inevitable when you have so many users. They have diverse needs, and you need to preserve backward compatibility, which leads to an awkward evolution.
The reasons for that are that shells must start very quickly (due to subshells, local ssh, etc.), be fast, have no complex dependencies since they are used to recover broken systems, be portable but also with full support for OS semantics and be written in a language that allows rapid development of robust software, none of which Python does well.
But I'm sure it is ease of prototyping and exploring the design space.
Best of luck to them. Another interesting shell to check out is elvish, lots of new ideas there (even if awkward to use).
(Disclosure: I’m one of the core fish devs/maintainers. Edit: The entire team is awesome and the others deserve virtually all the credit!)
However I don't see it being used AT ALL for the cloud/linux use case? Those are the cases where you tend to get 1000+ lines of shell scripts.
For example, I mention Kubernetes/Docker/Chef, and I've never seen fish in that space.
I also don't know of any Linux distro that uses fish as their foundation -- they all appear to use a POSIX shell like bash/dash/busybox ash. fish is "on top".
See Success with Aboriginal, Alpine, and Debian http://www.oilshell.org/blog/2018/01/15.html -- these distros are built with thousands of lines of shell scripts.
Either 1) I don't know about such usage, 2) people don't know that fish can be used this way, or 3) there is some problem with using fish this way.
I link to this post in the FAQ, which I think is a lot closer to Oil:
https://ilya-sher.org/2017/07/07/why-next-generation-shell/
It's basically the "devops" use case. (And as I mention the main difference between Oil and NGS is that Oil is compatible / has an upgrade path from bash.)
If you want to switch your existing distro scripts to fish, you need to rewrite a lot of stuff. If you want to start a new distro, "it's written in fish!" isn't a terribly compelling selling point, since I don't think anyone is actively picking distros based on their tooling language.
For the general devops case, there's a lot of existing example code out there for bash scripts, and far less for fish. "How do I do X in bash?" is probably going to get you a decent example on Stack Overflow. Devops (I feel) cares a bit more about the installed-everywhere thing, and fish isn't a standard part of most distros.
Oil having a goal of being completely sh/bash compatible gives it much greater odds of something like Debian switching to it, since it wouldn't carry huge technical debt along with it.
Now, I quite seriously believe that a 1000 line shell script only exists out of error. I still occasionally end up doing 2-300 line dense shell scripts, but not without feeling very dirty along the way. Either split into small, simple shell scripts (which is fine), or a different language.
In the cross platform build pipeline at work, I keep a strong discipline when it comes to scripts: They must be short (<=100 lines), and if their complexity exceeds a certain threshold (parsing more than a "grep | cut" here and there, total program states exceeding some low number), then a shell script is no longer acceptable regardless of length. And, well, it's not safe to assume the presence of anything more than a shell.
If you are writing and dealing with 1000+ lines of shell scripts, then experience tells me that you are shooting yourself in the foot. With a gatling gun.
(I used fish, btw. The interactive experience was nice, but the syntax just felt different without much gain, which was frustrating to someone who often writes inline oneliners. Unlearning bash-isms is not a liberty I can afford, as I need to be proficient when I SSH into a machine I do not own or control. I can't force the entire company to install fish on all our lab servers, nor is it okay to install a shell on another persons' dev machine just because we need to coorperate.)
> 1. People who use shell to type a few commands here and there.
> 2. People who write scripts, which may get into the hundreds or even thousands of lines.
> Oil is aimed at group 2. If you're in group 1, there's admittedly no reason to use it right now.
From my perspective, fish is basically the opposite of oil in that it mainly targets group 1 (admittedly I'm biased in that I rarely write shell scripts at all). IME fish scripts as a direct replacement for bash scripts are pretty rare, but that doesn't mean it's not a success if it's aimed at interactive use.
Second, I was recently chided by another fish user for writing fish scripts: "you're using it wrong" was the sentiment. The argument was it's a shell intended to be interactive, but extendable. I don't know how pervasive this line of thinking is, but who's going to learn all about their shell if they're not supposed to write scripts in its language?
Do you really need 1000+ lines of shell scripts? I almost have 20 years with shell scripting and I still have the rule of not writing longer than 1-2 page shell scripts and use something else for longer projects or break it down to much smaller chunks and invoke the parts with a main.sh.
Fish is a for making software engineering and system administration easier by having functionality like command completion (without tab) and it works like a charm. I would never ever would nother to rewrite my bash scripts in fish simply because bash is on all the servers and fish is on none by default.
I am not sure what you are trying to solve here but good luck.
The two shells target different use cases. Hopefully they are both successful in attracting a good audience.
But is this project really overlapping it? I see mostly fish as "UX" centered, while oilshell - as far as it's stated here - reminds me more of powershell (I found the idea of having a modern language behind it so cool when it was released, too bad it wasn't my ecosystem): looks like oilshell is targetting scripting more than UX.
Anyway, I love the idea of having several shells who try to have a new look on what a shell is.
EDIT: oh, btw, let's not consider no significant scripting can be done with bash, just look at the incredible work from dokku team ;) https://github.com/dokku/dokku
As for the scripting language part, however, I have always been wondering why do people use headache languages like bash/sh when we have Python. Might anybody have a clue - I'd appreciate if you could share.
That is, Fish intends to be a useful interactive shell and if it also scriptable, that because you need scripting for it to be useful. Fish doesn't make a serious attempt being the language in which system scripts are programmed in.
Oil on the other hand is a concerted effort to formalise and rigorously implement the language which existing system scripts are written in. The author believes that is a starting point for a good interactive shell -- but programming comes first.
The goals of mine are akin to Fish in terms of REPL use but with a greater emphasis on scripting.
Like Fish, murex also does man page parsing (in fact I wrote mine before realising Fish did the same), but unlike fish autocompletions can be defined by a flat JSON file (much like Terraform) as well as dynamically with code.
Currently I'm working on murexes event system so you can have the shell trigger code upon events like file system changes.
My ultimate aim is to make murex a goto tool systems administration tool - if just for myself - but the project is still young
I'll have to give yours a spin too. I'm glad there's a lot of shell innovation right now. I'm all for breaking posix shell standards and creating things that are way more usable. Fish's prompt customization, functions, highlighting, completion and searching are pretty amazing.
I realize a lot of these little projects will come and go. No matter what, they're great learning tools for the creators/developers, exploring what it takes to make an interactive shell.
Still, I hope we see more stuff like fish come out (and make no mistake, fish took a lot of years and a lot of devs. In the early days my instance would crash every once in a while in ways I couldn't easily reproduce). It's great that we're finally getting away from the traditional bash/zsh/ksh stuff and into newer shells that make coding and navigation easier.
Elvish is pretty nifty, but the biggest failing point to me is that the fancy rich pipelines really work just for in-process stuff, which for me kinda loses the point of being a shell. Of course I do realize that rich (polyglot) interprocess pipelines is a difficult problem; some might say a pipedream.
... | to-json | your-command | from-json | ...
fn f { to-json | your-command | from-json }It's a challenge for me to use well; not sure all that richness is composable. Better programmers than I am would know.
The feature I use the most is automatic history search by typing part of a command and hitting UP to search the history. I also find the scripting language more straight-forward.
Doesn't bash have that same feature? Or is there some subtle difference between what you're describing and what bash does?
This is available in bash/sh by setting this in your ~/.inputrc file:
"\e[A": history-search-backward
"\e[B": history-search-forward
"\e[C": forward-char
"\e[D": backward-charWhat's the process for requesting functions be added to core? I had to write my own to get bash's 'dirs -v' functionality. My solution depends on sed and is no doubt a hack.
The only time I run into issues is when a command expects manipulate environment variables via bash syntax.
I think the fish documentation WRT to scripting could be much better, but the language is more elegant than bash or PowerShell IMHO.
Not that the POSIX-like shell syntax is not all sorts of clunky and odd but I almost consider it a feature, it's a deterrent to force you to move to a "real" scripting language when the concepts become too complex to express in a shell script.
Excellently put. POSIX shell languages have fantastic capabilities you just can't get in most other languages. I would love to see a more safe, more sane shell language gain enough popularity to change the narrative that "shell scripts are dangerous and impossible to maintain."
The contrasts to Python and Ruby made me think of xonsh[1], a Python-based shell that can dynamically switch between Bash-like syntax and standard Python. It's not quite ready to become my daily driver, but I'm still excited about it.
[1]: https://xon.sh
For all sorts of interactive stuff I use fish, because it works the way you want for the most common tasks. I can use it to really quickly match and get back previous statements or do a completion.
Also much easier to configure and grok than bash, because it has saner syntax and is a simpler shell language.
However when writing shell scripts I use a real language like Julia. It integrates very well with the shell world so I don't find it problematic to do this. It is very easy to read output from processes and pipe stuff. Much nicer than say Python or Ruby.
You got built in syntax to deal with shell stuff, but then didn't make it crazy so you end up with a mess like perl. Julia is actually a very clean and nice language. Which also happens to blistering fast and have LISP style macros.
In an effort to understand the reasons for actively choosing against 3, does anyone know what problems those would be?
Im most excited about Elvish shell and the language that's being developed around it. The shell is built with Go and feels super fast compared to my plugin-heavy ZSH. The language design is quite nice too but still very alpha. Looking forward to see what it evolves into...
https://stackoverflow.com/questions/356100/how-to-wait-in-ba...
So whenever you want to do things in parallel there is probably a limit to the number of processes you would like to execute in parallel (e.g. the famous compiler limit formula: number of CPU cores +1). It would be great if Oil could support such a use-case out of the box, as easy parallelism without the ability to artificially limit the number of parallel executions is often useless.
I use xargs -P all over the Oil codebase, which does what you want. The trick is to end the file with "$@", and then invoke xargs -P 4 -- $0 my-func.
That way xargs can run arbitrary shell functions, not just something like sh -c "..." ! I'm going to write a blog post about this. I also do this with find -exec $0 myfunc ';'
https://github.com/oilshell/oil/blob/master/test/spec-runner...
However I think Oil will have something built-in to make this parallel process more friendly. I will probably implement xargs so it can run my own shell scripts without GNU xargs, but then add a nicer syntax. (Probably "each", since that's what xargs really does.)
This gets into your other question about standard utils, which I'll answer now. Short answer: yes I would like something like that, it's just a matter of development time and priorities. I agree with the problem you point out.
Learn from history. In the 1980s the world improved on the terminal paradigms, with TUIs that included things like directly addressable output buffers and unified and standardized keyboard/mouse input event streams. In parallel, GUIs took hold, and there are nowadays a lot of GUI programs in the world.
I have been keeping a wiki page:
https://github.com/oilshell/oil/wiki/Interactive-Shell
Although honestly I won't get to any of this in the near future.
Dead project according to the author, though.
I mean, they can't keep using Bash 3 forever, right? (Hope)
http://penguindreams.org/blog/the-philosophy-of-open-source-...
Maybe they just expect you to install your own shell? I think a lot of people do that with homebrew?
I think Apple has largely expunged shell scripts from the startup process with launchd too? That is like their systemd.
Screen tiling and visual 'tabs' would also be welcome additions. Not everyone needs a graphic environment, and I refuse to install X just for better keyboard shortcuts on my terminal.
Since dvtm also works as a terminal emulator, it seems to me that you could use loadkeys to setup various keycodes to send the proper vt100 escape codes. I've not tried it, but see no reason why it shouldn't work.
If you just want "No X install" you can use a frame buffer terminal (fbterm was one I used to use, but it doesn't appear to have been updated in a while, perhaps there is a spiritual successor, or maybe it already does what you want)
[edit] YAFT https://github.com/uobikiemukot/yaft looks like it's more up to date than fbterm.
Nevertheless, there is one piece in this puzzle I am missing. There does not seem to be a process which manages the 'core software set' across platforms. So after decades we finally have a shell which is available on most operating systems, but how long will it take before Microsoft, Apple, Oracle, etc. will adopt a new shell?
So why don't the large OS corporations form a consortium to define something like a 'cross platform run time environment' standard (maybe together with the Linux Foundation and some BSD guys?). I mean its not so much about which shell someone prefers, but more about a common set of interpreters and maybe tool kits. And even more than that it is not about the state but the process/progress.
What do you think, do we need such a process or is there another way to solve the cross platform dilemma?
The latter problem could probably be solved with a wrapper which would pipe and execute the shell (or a bytecode interpreter ala shuttle?) automatically - but I've seen no alternative shell project take this part seriously for the problem space.
erm, there's a big difference between a command scripting language and a programming language. These should be treated as different things.
I have years of experience using both, and I really don't want to be doing shell tasks in a programming language and I don't want to write programs in a shell language. Those sorts of hybrids are almost always mediocre. Horses for courses and all that.
There's a reason bash keeps being used - it's mature, it's simple, it's easy and people are productive with it.
Lets say there are two uses of a shell: 1. interactive and 2. non-interactive (scripting).
Lets imagine the commandline user is learning about her OS. She learns it is heavily reliant on shell scripts to build and (if desired) to automate starting services.
She realises that to understand the OS she will have to learn the shell that the OS developers used for scripting.
Then she realises that if she chooses another shell for interactive use, she will have to learn two shells.
Finally she realises that any script she writes in the "non-interactive/scripting" shell will also run under the interactive one. But not vice versa.
If she only has enough time in life to master one shell, which one should she choose?
Over time I found I really cared more about the scripting aspect of a shell than the interactive facet.
The scripting shell used by the OS authors might be an Almquist derived shell, for instance.
Occasionally the ash Im using gets a new "feature" but not too often. I like that it stays relatively small. The latest "feature" is LINENO.
But I also use a smaller version of this shell with no command line history, no tabcomplete, etc. IMO, there is no better way to learn how to reduce keystrokes. It has led to some creativity in this regard for which I am thankful.
After "mastering" ash, I started using execlineb, pipeline and fdmove. I am starting to use more components of execline and am continually replacing ash scripts with execline scripts for more and more daily work.
I guess we will never see execline on the front page, which I think would be interesting because I would like to hear whatever harsh critique HN can muster.
Seeking a better non-interactive/scripting experience, I have experimented with many other shells over the years, and written simple execve "program launchers", but in this vein, I have not found anything that compares to execline.
The speed gains and resource conservation are obvious, but with the ability to do "Bernstein-chaining" and the option to use djb low-level functions instead of libc, it is a rare type of project.
The speed and cleanliness of the compilation process is, compared to all the other crud one routinely encounters in open source projects, "a thing of beauty". Humble opinion only, but I think others might agree.
b = [ @a ]
As for new language, I feel like if you want to script things, you can use ruby or python, hell, perl will do and you could be fine. I don't want to be unfair to this effort, I just feel that it is not for me and I am tinkerer.
* http://perldoc.perl.org/functions/open.html
* http://perldoc.perl.org/IPC/Open2.html
* http://perldoc.perl.org/IPC/Open3.html
* http://search.cpan.org/~odc/IPC-Open2-Simple-0.01/lib/IPC/Op...
* http://search.cpan.org/~exodist/Child-0.013/lib/Child.pm
* http://search.cpan.org/~rkrimen/IPC-RunSession-Simple-0.002/...
* http://search.cpan.org/~trski/Proc-Forkmap-0.025/lib/Proc/Fo...
* http://search.cpan.org/~toddr/IPC-Run-0.96/lib/IPC/Run.pm
* http://search.cpan.org/~ayoung/IPC-Run3-Simple-0.011/lib/IPC...
* http://search.cpan.org/~rjbs/IPC-Run3-0.048/lib/IPC/Run3.pm
* http://search.cpan.org/~djerius/IPC-PrettyPipe-0.03/lib/IPC/...
* http://search.cpan.org/~xan/IPC-Pipeline-1.0/lib/IPC/Pipelin...
* http://search.cpan.org/~sscaffidi/IPC-OpenAny-0.005/lib/IPC/...
* http://search.cpan.org/~glai/IPC-Exe-2.002001/lib/IPC/Exe.pm
* http://search.cpan.org/~zefram/IPC-Filter-0.005/lib/IPC/Filt...
...people frequently ask this? Tip #21 of "The Pragmatic Programmer" states: "Use the Power of Command Shells."
In a general purpose programming language there's a lot of overhead for doing the same things.
For maintainability, there are now linters for shell languages that can help making the job easier.
Obligatory in case anyone hasn't seen it:
Works as a web app or local tool.
I did use fish a bit as a script language, but I decided for anything of any size I much prefer Julia. For typical file system navigation, fish is better, but Julia is actually pretty decent as a shell, despite being a real language. So writing shell scripts in it is pretty nice.
In the beginning I wrote separate programs executed from fish shell. But now I just fire up Julia as a shell and run functions directly there interactively.
It sort of is an "uncanny valley" for a text interface. It feels close enough to a traditional UNIX shell that I start to interact with it like one... but has enough differences that I found myself constantly tripping over them.
And in my experience 90% of those are in the form `FOO=bar command` which can be replaced with `env FOO=bar command` and works just fine in fish.
eval $(ssh-agent)
I use Fish as my shell, but I do scripting with Bash. There’s nothing that prevents you from doing so unless you’re sourcing a file.
https://github.com/oilshell/oil/wiki/ExternalResources
I guess what you mean by embedded is that it should be an embedded DSL in a full-fledged programming language? I don't quite agree, since there are at least 20 projects like that on the wiki page, none of which is popular.
Probably the most popular one is eshell, in Emacs Lisp?
But if there's something I don't know about I'd be interested in hearing it. This idea goes back at least 20 years, e.g. to scsh. And it hasn't taken off.
But certainly I don't begrudge anyone if their favorite environment is Racket and they want to stay in Racket. That's a perfectly reasonable thing. It's just not what I would expect anyone besides racket users to use.
One reason I'm interested in shell because it's the lowest common denominator between different groups of programmers. C/C++ programmers, use it heavily, Python, Ruby, JS programmers, Go, etc. Everybody uses it.
And of course I agree with that! That's the whole reason for Oil.
> concurrent processes
Job control is disabled in shell scripts, and your only other option is juggling process ids. Combined with nearly nonexistent exception handling using anything more than a single process at a time is like pulling teeth.
Things like ssh have super awkward workarounds using control pipes.
I'm not sure I've ever seen a shell script use concurrent processes in the wild.
Python's subprocess library is excellent and makes concurrent processes a breeze.
> file system
With some minor exceptions, I can't think of any FS ops I'd do in shell that isn't just a couple letters longer in Python and 10 times more flexible.
The only reason to use shell is that it has a simple syntax for pipelines.
I rewrite any script > 2 lines in Python and have no regrets.
tl;dr I used Python for prototyping; it will be removed.
Consider it an implementation detail -- building and running Oil does not require Python, as a portion of the interpreter is bundled in the tarball. Python 2 vs. 3 doesn't really matter. It was in Python 3 at one point.
More discussions here:
https://www.reddit.com/r/ProgrammingLanguages/comments/7tu30...
> I encountered a nice blog post, Replacing Shell Scripts with Python, which, in my opinion, inadvertently proves the opposite point. The Python version is more difficult to write and maintain.
Here's the link: https://medium.com/capital-one-developers/bashing-the-bash-r...
I think, roughly speaking, the fact that Python 3 is much closer to a sane language for engineering means that it's less suited for scripting.
What is A=$(cmd) in bash is an absolute pain in python.
def _call(cmd_str): return subprocess.check_output(shlex.split(cmd_str)).decode("utf-8")
Definitely a verbose monstrosity compared to doing it in bash, but more than worth avoiding the garbage fire that is Bash. And it only works for simple cases
They work well, if a bit magically, until you need to background a process.
Xonsh is an awesome fishy she'll that's a Python superset. $() is built right in.
_("ls -la") | _(lambda x: x.split()[-1]) | _(lambda x: os.path.splitext(x)[1])
A = !cmd
> /bin/bash your_script.sh
And if your script is written with bash in mind, use a shebang:
> #! /bin/bash
And it will work perfectly fine on fish. As long as I have a bash binary, why do I need COMPATIBILITY?
> CC=gcc make
And if your program is written with gcc in mind, put that in the Makefile:
> CC ?= gcc
And it will work perfectly fine on LLVM systems. As long as I have a gcc binary, why do I need COMPATIBILITY?
----
A big part of the long-term objectives of OSH is that it provides a way to move to a better language, without having to entirely rewrite your codebase. Think of it in a similar spot to C++ (originally); a big part of the design is that your old C (bash) code is already valid C++ (osh), and you can start using new C++ (osh) features wherever you see fit in the codebase, without having to rewrite anything first.
Having bash compatibility is a quality of life feature, no need to debug all the werid cases where it breaks for purely syntax reasons.
zsh and fish don't really belong in the same comparison IMO.
zsh is, like bash, a ksh-like shell with a Bourne-style grammar. Obviously it depends on the exact use case, but in practice, for basic scripting purposes, it is almost a super-set of bash, and it provides several emulation options (like sh_word_split) specifically designed to increase compatibility with POSIX and with bash in particular. It even provides shims to support bash completion functions. (It is fair to point out that, even with all the emulation stuff enabled, it's still not completely bash-compatible, nor POSIX-compliant. It's close enough that the changes required are usually extremely trivial, though.)
fish on the other hand has its own completely different grammar and makes no attempt to provide POSIX/ksh/bash compatibility at all.
Zsh isn't a new kid on the block. Both it and bash are actually about the same age: 28 years:
e: sorry some connection interruption and now this is redundant to other comments.
But of course your argument has the fundamental flaw that the Bourne Again, Korn, and even Bourne shells were not compatible with their various predecessors, but that turned out to be not as problematic in practice as you paint it to be. And we've had all sorts of things gaining "real adoption" over the years, from Norton Commander clones to Perl. The world is nowhere near as narrow as you think.
https://en.wikipedia.org/wiki/Shebang_(Unix)
I used fish for years, never had a problem with bash or zsh scripts.
So I ended up using the loop syntax:
for i in {0..9}; do
echo "$i" &
done
wait
Just an idea from the top of my head an idea for an optional named limit:
Lets call it 'flow':
flow [options] [command]':
-n number of max parallel processes
-c (optional) identifier of the counter.
Example:
for i in {0..9}; do
flow -c myCounter -n 4 echo "$i"
done
I think the modern "terminal" is the Browser. With URL's instead of file paths. But I think it's maybe time for something new. In the 70's we got the terminal. 20 years later we got the browser. Now another 20 years have passed. What's the next step ? Terminal -> Browser -> ?? -> AI ?
I thought more about a higher level standard like adding Python, Lua or Qt to every installation by default. As some of those things are pretty heavy I doubt that it would be a wise choice to include them in POSIX.
Just imagine a world were you could simply write a small python script which would start a complete GUI application on different platforms without any additional installation procedures. To my knowledge that is not possible today. AFAIK the only way today is to bundle the dependencies, but that has a lot of negative effects.
Probably the first cut will be an "app bundle" format for Oil + busybox + arbitrary user utilities.
I'm more interested in the subset of busybox that is PORTABLE. busybox doesn't run on BSDs, because a lot of it is tied to the Linux kernel. It's more for embedded Linux.
I actually worked on the toybox project around when starting Oil (toybox is the "busybox" on Android, started by the former busybox maintainer Rob Landley.)
So I don't want to necessarily create another package manager, which is sort of implied by your question (?). For shell, the package manager is traditionally the system one -- "apt-get" on Debian, maybe homebrew on Mac, etc.
But I definitely want to solve the dependency problem, and I think the best way to do that is through some kind of support for app bundles. Of course you can also create a container image if you like.
* https://news.ycombinator.com/item?id=12600807
Laurent Bercot no longer has xyr page about the compilation process. I have since picked up some of the slack there. Although I don't go into things like the way that M. Bernstein avoided autotools.
f() {
echo --
ls /
echo --
}
f > out.txt
f | wc -l #!/usr/bin/perl
use strict;
sub f
{
my $outputFH=shift;
print $outputFH "--\n";
open(my $lsFH,"ls /|") or die("pipe ls: $?");
print $outputFH (<$lsFH>);
close($lsFH);
print $outputFH "--\n";
}
open(my $outTxtFH,">","out.txt") or die("open: out.txt:$?");
f($outTxtFH);
close($outTxtFH);
open(my $wcFH,"|wc -l") or die("pipe wc: $?");
f($wcFH);
close($wcFH);There are a number of ways to do these same things. Some of them mirror your code more closely than others. Here's my first shot using a core module, since someone already did one with no modules that works much like your code.
use IPC::Run3;
my @lines;
sub f {
my @command = qw( ls / );
run3 \@command, \undef, \@lines;
}
f();
open my $out, '>','out.txt' or warn "can't write to out.txt : $!\n";
printf $out "--\n%s--\n", (join '', @lines);
print scalar @lines . "\n";
#!/usr/bin/perl -w
use strict;
use English;
use autodie;
$OFS=$ORS="\n";
sub f { my $h ; opendir($h,$_[0]) ; print "--",(readdir($h)),"--"; closedir($h);}
my $out;
open($out,">/tmp/out.txt") ; select $out ; f("/tmp");close($out);
open($out,"| wc -l ") ; select $out ; f("/tmp"); close($out);
select STDOUT;
@ary=split("\n",`ls`);
around if you feel so inclined.
#!/usr/bin/perl -w
use strict;
use English;
use autodie;
sub f { open(my $h,"/bin/ls $_[0]|") ; print "--\n",(<$h>),"--\n";}
open(my $o,">/tmp/out.txt") ; select $o ; f("/tmp") ;
open($o,"| wc -l ") ; select $o ; f("/tmp") ; sub output_of {
my(@commands) = @_;
my $pid = open my $fh, "-|" // die "$0: fork: $!";
return $fh if $pid;
for (@commands) {
my $grandchild = open my $gfh, "-|" // die "$0: fork: $!";
if ($grandchild) {
print while <$gfh>;
close $gfh or warn "$0: close: $!";
}
else {
exec @$_ or die "$0: exec @$_: $!";
}
}
exit 0; # child
}
my $fh = output_of [qw( echo -- )],
[qw( ls / )],
[qw( echo -- )];
while (<$fh>) {
print "got: $_";
}
close $fh or warn "$0: close: $!";
my $output = `echo -- ; ls / ; echo --` // die "$0: command failed";
chomp $output;
print "$0: lines = ", `echo '$output' | wc -l`;
If all you want to do is redirect anyway, run
system("{ echo -- ; ls / ; echo -- ; } > out.txt") == 0
or die "$0: command failed";
Maybe I am missing your point entirely. Do you have a more concrete example in mind?
1. Keep up the great work (both oil and the blog!) Both seem fun :)
2. Reading your blog really shows that despite the great ideas, shell languages come from an age where computing was still in its infancy. We’re now approaching our teen years, but we’re still ahead of infants. So many awkward design decisions and organic evolutions. A shame that replacing POSIX at this point is a gargantuan undertaking.
Where's "my shell is my desktop environment"? :/
Everyone's desktop environment is a shell.
But any shell you like is suitable for that use case since it doesn't have to be compatible with anything other than your fingers.
There's sh which is the most portable. Why write scripts in anything else but an actual programming language when things get more complex?
Like csh on FreeBSD, alt+(up/down) can be used to search history for arguments instead of lines, so if you do something like
touch some/long/directory/path/to/file
....
rm so<ALT+up>
As always, with great power comes great responsibility I guess.
I love using fish and enjoy scripting with it too, but I'm hampered by this fact as well, and mostly use bash at work when others might see it or have to use it.
$ time julia -e 'println("Hi")'
Hi
real 0m0.241s
user 0m0.216s
sys 0m0.196s
$ time python3 -c 'print("Hi")'
Hi
real 0m0.046s
user 0m0.020s
sys 0m0.000sBut a shell script is a script, not a codebase - they're written for one-off scenarios in very specific environments. I imagine that some folks have very large, complicated systems that depend on many bash scripts. But in such a case one can still use fish to just execute the bash scripts and pipe the output around as needed. If you have a single, large bash script that does some really complicated stuff, you can begin porting it by separating it into smaller scripts, calling those from a modern scripting language like Python and rewriting aspects of the process as you go.
A C or C++ codebase is in no way equivalent to a bash and scripting codebase since C and C++ are not nearly as easy to call from different versions - you would have to split up your build process across different versions of a compiler and build a library to accomplish the same thing.
Exactly? What you just said is true for CC and SHELL; I'm not sure what is confusing.
> But a shell script is a script, not a codebase - they're written for one-off scenarios in very specific environments.
Ahh, there it is. IMO, a big part of the reason why many people think that people shouldn't write shell scripts is that they've only had to deal with shell scripts written by people who didn't treat it as a real codebase.
It's hard to say what private codebases are doing but:
- Docker, which is expected to run in many different environments, has thousands of lines of shell. That isn't one-off, and isn't for a specific environment.
- Most GNU/Linux distros can generally be thought of of many separate upstream packages glued together by many thousands (if not millions) of lines of shell. It might be expected to run in a fairly specific environment, but it isn't one off, and is massive enough, I'd have a hard time not treating it as a codebase.
> ... you can begin porting it by separating it into smaller [programs], calling those ...
> A C or C++ codebase is in no way equivalent to a bash and scripting codebase since C and C++ are not nearly as easy to call from different versions
C is very easy to call from different versions, the ABI(s) has been very stable over the years (unfortunately, this isn't as true for C++). Just say `myobj.o: private CC=gcc-4.7` or whatever compiler is needed for myobj.c, to set the compiler for just that object.
You seem to be saying "I don't need C (bash) compatibility, I just need an FFI that makes it easy to call C (bash) code."
C/C++ and Bash/OSH are different, but not that different.
Rash also tries to be better than the competition by adding object pipelines (much like Powershell, it makes it much more reasonable to make system administration commands in the host language, and have rich interaction and inspection of command results), user-defineable pipeline operators, and generally tighter integration with the host language while still having a light syntax for basic commands.
I would like to be able to have my command language tightly integrated with my programming language, and be able to define my system administration commands (and interactive shell plugins and completion) as functions in the host language (while still being able to naturally use external programs). And I would like my shell scripts to be able to grow and potentially even turn into full-fledged programs, or modules of some larger program. I think there are a lot of benefits to the approach I'm using (which would be too long for a comment here).
That said, I'm not holding my breath for it to catch on widely any more than I'm holding my breath for Racket to take off as a popular programming language (although I frankly wouldn't mind either one). I think a better Posix shell is certainly a noble effort, because whatever better shell does become popular, we certainly need one. And an automatic upgrade path for existing scripts sounds great. So I salute you and wish you good luck with it. Also, as someone looking at shells and their features to copy the best ones, your wiki is a great resource. So thanks.
It might be time to start that again to see what ideas people have and what has changed. If you're interested e-mail me at andy@oilshell.org.
I'm also interested in the Shill shell, which as I understand was written in Racket, and then somehow they moved away from Racket? I'm not sure. I think it was because of the runtime. I also saw some efforts to move Racket to Chez Scheme.
I very much like Racket as an idea -- a meta language -- but I haven't gotten a chance to play with it too much.
And I did experiment with femtolisp as a basis for Oil -- the lisp used to bootstrap Julia -- but I decided against it.
Windows NT already is, and has been for many years. It has had two POSIX subsystems over the years, and then a Linux subsystem.
But everybody who has build an electron app and a Qt/GTK app, will agree that the tool kits which are available to electron apps are not as sophisticated as Qt/GTK (not talking about the ton of downsides of electron apps (huge app size, outdated versions, etc.)).
I am completely pro PWA, but as far as I can see it it will take a few more years (at least) before we will get to a state which will allow us to use them in the same way we develop and use normal desktop apps nowadays.
Sorry I got to this late -- I might do a blog post on it. I think your response, along with the 3 or 4 others i got essentially proves my point: "Perl is not an acceptable shell".
A good idea to double check using a text editor.
I talked with the authors of Ion on github almost a year ago. Ion was influenced by Oil, in particular this post:
Not sure whether I will actually get to it though. Parsing sounds like a lot of headache. Your work is very impressive, but I’m not sure I will want to go through that... :)
Maybe bind is easier than I think; maybe it's just a few readline calls. I haven't looked into it.
But also, I noticed almost no shell scripts use extended globs like @(foo|bar), whereas the bash-completions package uses them EVERYWHERE. Completion scripts are almost like a different dialect than other shell scripts.
If you want to source /etc/bashrc on most distros, you're going to end up with thousands and thousands of lines of plugins from the bash-completions project.
The parsing is a big headache, but I think it forced me to learn how people use shell, which will benefit Oil.
It's been a full year since our alternative shells thread... might be time to revive it and see what's changed! Congrats on the Elvish releases!
1) what you paste might not be exactly what you copied. i.e: http://thejh.net/misc/website-terminal-copy-paste
2) possible licensing issues
3) it enables cargo cult programming
Any new shell that hopes to compete in the systems space will have to be POSIX compliant. At least until there is a new widely adopted shell standard.
I would use the crap out of a Linux distribution built in python.
https://github.com/dcantrell/pycoreutils https://github.com/davidfischer/pycoreutils
http://u-root.tk/ does that for Go.
actually now you mentioned it, barring possible performance issues I'd also like a Python distro
I recommend Emacs. Maybe not as a daily driver (it's a matter of preference), but for the experience. I recommend at least a month with it, make sure to write some original Lisp code for your customisations. (You WILL end up customizing it, the defaults are crap.)
> or a window manager
I recommend Awesome. The core is in C, but that's basically the low-level stuff, the actual WM is written in Lua. If you'd start with an empty init file, you'd have a moderately sized side-project on your hands to put something usable together. But if you'd start from the stock rc, there's a world of endless tweaking and customisation waiting for you...
Watch out, both Emacs and Awesome are rabbit holes.
I'm not sure how much work one needs to do to implement things like extended globbing, but my gut feeling is that once you can parse them it's more than half done. The functionalities are quite simple, as my impression goes.
I was mainly thinking about environment variables, aliases and wrapper functions when I said sourcing bashrc; that's probably what most people's bashrc files are mostly made of.
We totally should revive our alternative shells thread! I'll try to summarize my progress in the last year. Congrats on your progress with Oil too! ^_^
So basically you should never make /bin/sh fish, because then you will no longer have a Unix system (according to POSIX).
On ubuntu, /bin/sh is always dash and it's not possible to change without changing the symlink:
https://wiki.ubuntu.com/DashAsBinSh
So, if you use chsh to make your shell fish, it would have no effect on os.system or the system function in C.
Actually, my zsh config mostly behaves like fish with great completion, syntax coloring and sensible history handling, with zsh syntax which is close to bash. So I can copy paste stuff from the internet and reuse my knowledge from the time I used bash because it was the default, and still leverage the improvements brought by zsh.
I don't understand 2.
For 3, of course you should understand what you run, but I don't want my tools to get in the way. My tools should allow me to do what I want, not prevent me from doing something for technical reasons. This is a question of education. If people want to run something without understanding it, I bet they will type it blindly, if they can't copy paste it. They will just be slower at running things blindly, leading them to a disaster just a bit slower. Please let me not lose my time even more when I cause disasters.
(Yes, true, retyping stuff forces your more to think about it, but I already review the things I paste into my shell)
The thing that is working for me is to explore VB.net . It is really an interesting language with many historical features. I've noticed that all the languages I use (Python, C#, Java, Go, etc.) distracted me from the essence of programming. When I started programming I immidiately thought about patterns, architecture, unit-tests, integration-test etc.
In VB.net I feel like I'm slowly reexperiencing the joy I've felt in my earlier days of programming.
Ruby was the first language that I experienced that got out of the way, so I felt like I could think about what I was trying to achieve, rather being distracted by house-keeping. Of the other languages that I've tried, I've only gotten close to the same feeling with JavaScript, Go and Python.
I suspect that a lot of it is to do with your own mind-set at the time: I first learned Python as an applications language, with all of the add-on tools and baggage, and didn't like it much, but recently I wrote a one-file, no-dependency thing to do exactly one job in Python, using just the standard library, and it was a joy.
The difference was, I think, that I did not feel the internal pressure to meet best practices, and could just write the code. For some languages, the stuff that you are supposed to do and the tools that you are supposed to use are a heavy burden.
the illusion is almost real, because sometimes i also run linux in vmware/docker/... and when i am not paying attention i forget easily which os i am on.
the unix subsystem for win is quite boring as you can't mix windows applications with linux ones, for example running visual studio build from bash.
powershell would be probably worth learning, if i'd have to, but i do not. it may be godsend to win sysadmins though, as classic cmd is a 40 year old joke.
that was not possible when i tried it, thanks for pointing it out
As someone who has recently joined the 1k LOC Emacs configuration file club, I have to disagree. The default settings are a well thought out starting point that needs minimal tweaking to get to exactly where you want. I try out a lot of packages on ELPA, and many that purport to provide alternatives for defaults end up being inferior to using the defaults with some small tweaks. The convenient thing is that you can customize the third-party packages in the same way and just use the parts you like from them.
The other good thing about Emacs is that you don't need a separate window manager[1], and you don't need to use a separate command shell (Emacs comes with eshell), or to use a terminal emulator for SSHing into remote machines (eshell over TRAMP is absolutely amazing), or a file manager (Emacs comes with dired, which works over TRAMP which is amazing), or a...
org-mode is a good one, too, but I feel like I've barely scratched the surface there.
When it comes to magit and TRAMP, definitely the best part is with-editor[1]. This is one of those ingenious "why didn't I think of this?" hacks to let you use your local Emacs as EDITOR on remote machines without having to do SSH port forwarding for emacsclient.
I even started to suspect that an easy customization of a tool can be a bad sign. It is almost like developers delegates all usability issues to the end user making the default case really bad.
I've used -- for longer periods of time, of course -- Awesome, Xmonad, i3, and dwm; while some of them are better at some things than StumpWM, StumpWM is the only one that provides an Emacs, live-hackable experience. So these days, I strongly prefer that.
My rough heuristic is, "no more than ten lines, nor more than two variables." Yes, that's short almost to the point of absurdity. The only good thing that one can say about Bash as a scripting language is that it's better than csh. Bash is taken seriously because of its longevity and ubiquity, but it's fundamentally limited in what it can express, and it is quite trivial to exceed those limitations.
It's definitely easier/safer to write longer programs in other languages, but with proper discipline it's also possible to write really big, robust programs in Bash/shells--people have done this, and some of that code is probably still running today.
A lot of that code is still being written today, which is one of the things that OP is trying to change.
It's funny though, after decades of shying away from them I've now gone full circle and embraced Makefiles once more. Perhaps it's the cleanness of the bmake/pmake extensions as compared to those of GNU Make, but the determinism, zero dependencies (no CMake or meson and its python baggage), automatic parallelization, and strict error handling call to me each time I have to write mission critical "glue code" that must. just. work.
That's a fair point. Nothing beats shells for ubiquity and supportedness everywhere . . . in theory.
In practice, even people who write "portable shell scripts" (almost) never really write portable shell scripts.
If you learn the POSIX sh standard like the back of your hand, and stick only to the features in it, never using bashisms or equivalent, most shellscripts still rely on external programs (even if only grep/sed/etc) to do their heavy lifting.
And that's where you get into trouble. Because compared to the variability in behavior of even "standard" ultra-common programs, the variability in behaviors between shell syntaxes/POSIX-vs-non-POSIX shells is tiny. The instant your code invokes an external program, no matter how ubiquitous that program is, you have to worry about:
- People screwing with PATH and changing the program you get.
- People screwing with variables that affect the external program's behavior: LD_LIBRARY_PATH for dynamic executables, or language-specific globals for programs' runtime behavior (CLASSPATH, PERL5LIB, PYTHONPATH, etc.).
- External program "editions" (e.g. non-GNU sed vs GNU sed). Good luck using PCRE with a non-GNU "grep"! Oh, and if you're sticking to POSIX-only shell semantics (no bashisms), you don't even get full BREs in most places you need them in the shell; you're stuck with limited BRE or globbing, which makes editing 100-line PCRE regexes feel like a dream.
- External program per-version differences.
- External program configuration.
- Etc.
Dealing with those issues is where "self-test"/"commandline program feature probe" things like autotools really shine. Raw shellscripts, though, very seldom live up to the "ubiquity" promise.
After learning and using many other "modern" Makefile replacements of various complexities (ninja, cmake, scons, tup, meson, bazel, and others) I realized if you're not using exactly what the tool was designed for, you end up recreating a Makefile (e.g.: meson is awesome for cross-platform C++ builds, but if you try to use it to build an environment composed of output from various processes that aren't C++ compilers, writing a Makefile is easier). CMake, apart from also being too C/C++-specific, would be nice if it didn't require a million different files and didn't have such a god awful language.
The only one I liked as a general purpose build system was ninja but it is too restrictive to code in (requires duplication of code by design, not meant to be written by hand though I still do from time to time) and tup, but tup is built around fuse (instead of kqueue/inotify/FindFirstChangeNotification) and so is a no-go for anything serious.
Once I embraced Makefiles, it turned out that most things traditionally built with shell scripts should actually be Makefiles for determinism. For example, I just used bmake to take a clean FreeBSD AMI and turn it into the environment I need to perform some task every n intervals (the task itself was turned to a Makefile rule), in place of where Puppet and other tools would normally have been needed, but would have been overkill for my needs.
The only drawback to Makefiles that I haven't found a clean solution to is when you need to maintain state within a rule (without poisoning the global environment). The only solutions I can see are either a) using a file to store state instead of a variable, which is just stupid, b) calling a separate shell script (with `set -e; set -x` to try and mimic Make behavior) which sort of defeats the point of Make, and c) multiline rules with ;\ everywhere, which is hideous and error-prone but works (though it makes debugging a nightmare as the rule executes as one command, again defeating some of the benefits of Make).
Perhaps the best use case for Oil is to provide a debugging environment where you can figure out what your legacy shell scripts are doing, and rewrite them in another language.
> Unlearning bash-isms is not a liberty I can afford, as I need to be proficient when I SSH into a machine I do not own or control.
This is a slippery slope. I have heard things like, "don't make your own custom aliases / shell functions, because they won't be available when you SSH to another machine." Forcing yourself to always use the lowest common denominator of software is not a fun path.
Well, the question is: what do you use as a replacement? For my scripts I tend to pull in some version of PHP as soon as needed. Perl is something that's universally available in anything based on Debian or Ubuntu, but it's a maintenance nightmare, and PHP doesn't care whether I use spaces, tabs or a mixture, or if there are due to any circumstances mixed line endings in a file whereas Python may or may not barf on encountering any of said things...
https://github.com/roswell/roswell
(As a side-note: Perl 5 was really the ideal language for this, by design; we might be in a different world today if it hadn't been derailed by Perl Forever^W 6).
Anyway, I usually avoid perl. While definitely a true programming language, I really don't like it. I find that it lacks clarity, and quickly collapses into bash-like hackery.
I personally use Python when it's a notch above shell, but not enough to pull out the big guns.
(I use to write a lot of perl when I was a kid, even made an accounting system in it, so my dislike of perl is from experience. We also had a 6000 line perl build script at my current job which was the last drop. Rewrote it to 40 lines of batch and 150 lines of Python. Likewise, I also wrote a lot of PHP, and can no longer consider it a proper programming language.)
Are you claiming that its size was because Perl forced it to be long winded?
http://www.oilshell.org/blog/2018/01/28.html#toc_7
I didn't write those scripts, and you rely on them, whether you know it or not.
Do you use Unix? Do you use the cloud? You rely on them. See:
https://www.reddit.com/r/linux/comments/7lsajn/oil_shell_03_...
Also, shell has functions.
However I do think it is a disaster that people can't break up their shell code into more modular components, and there is no excuse for it.
And I especially do not wish to discourage you from trying to reinvent shell, because someone has to do it, but at minimum I don't see myself moving away from my shell for the better part of a decade. Hopefully at that point Oil's community has matured to the point where I feel comfortable trusting it to be stable and free of any critical privilege escalation bugs.
Good luck!
People also implement new SQL injection bugs every day—just because a lot of people do it doesn't mean we should just let it slide.
(Although note that bash actually has a debugger called bashdb, which I didn't know about until recently, and I've never heard of anyone using it.)
One intermediate step I would like to take is to provide a hook to dump the interpreter state on an error (set -e). Sort of like a "stack trace on steroids".
If anyone is running shell cron jobs in the cloud and would like this, please contact me at andy@oilshell.org.
I want to give people a reason to use OSH -- right now there is really no reason to use it, since it is doing things bash already does. But I think a stack trace + interpreter dump in the cloud (like Sentry and all those services) would be a compelling feature. I'm looking for feedback on that.
Shell scripts have a very small amount of in-process state, so it should be easy to dump every variable in the program.
Also some kind of logging hook might be interesting for cloud use cases.
Second this. Been there, and back, and there again, and recently back again.
Customize the hell out of your shell, make it your place, make it nice. You're spending your day there, every day. Treat it like you'd treat your work desk.
If it's a stranger's machine, well, OK, suffer through the 10 minutes of troubleshooting, an occasional session with busybox also helps keep "pure POSIX" skillset fresh. If you're becoming a regular there, it's time to think how to make your dotfiles portable. google.com/search?q=dotfiles
OTOH, I’m also against, for example, installing zsh fleetwide in production fleets to accommodate choosy folks. So I’m on both ends of the problem, and know it.
My setup is customized, but the core experience is kept relatively untouched. If someone experiences a problem, I need to be able to debug it in place efficiently. I write drivers for the company's hardware for a living—troubleshooting takes more than 10 minutes.
My approach to that problem is to just instrument scripts to write out the command line every time a tool is called, and infer the logic behind it. When I started here, build/package/test on Windows were controlled by 6000 lines of Perl... A printing version of system() and a few days later, it's 40 lines of batch and 150 lines of Python.
> Forcing yourself to always use the lowest common denominator of software is not a fun path.
Never said it was fun. Also, I need to be able to use the lowest common denominator, I don't necessarily have to be the lowest common denominator. I use a configured zsh and nvim instance myself, but I take care to ensure that if I'm stuck with vi and (ba|da|k|c)sh, I'm still productive. The core behavior of my VIM instance stays close to stock, but it has a bunch of extras like fuzzy search, linting, a nice theme, etc.
And, if I can make a choice without caring, my needs for a shell and a programming language are in direct opposition. I want my shell to put all of its energy into a useful interactive experience.
If you're logging into a machine to the point you're editing files or require your customizations in this day and age you're doing it wrong IMHO.
I feel like some programming languages, no matter the structure of the application, feels unstructured and write-only in nature. To me, Perl is one of such languages. Python has a similar effect, but I feel like it is more controllable as long as the project is kept within a sensible length.
So while that abomination was primarily the fault of the designer, I do think that Perl invites this behavior one way or another.
(I gave up entirely on reading that disaster of a build script, and instead instrumented system() so that I could trace the calls to various tools and infer the logic myself.)
echo "${myArray[2]}"
Bash is weird.
- So you've found yourself SSH'ing into a particular box, because that's just the best way to work on a problem. Either it's a pet box (in which case it makes total sense to drop your dotfiles there), or it's a cattle box and you don't care about the trash you left behind, because autoscaling will clean it up for you.
Subprocess is a bit verbose to use, though... A wrapper would be nice.
... unfortunately, I'm stuck with python2.4 or 2.6 on most machines my Python code has to run on. :/
Not even 2.7?! 2.6 hasn't had a security update since 2013[0], I dread to think how old 2.4 is.
You can install a newer version, but our CentOS 5 machines are meant for compatibility tests, so modifying their setup is unacceptable.
I believe they are available for CentOS 5.x?