The Man Behind Windows PowerShell(heavybit.com) |
The Man Behind Windows PowerShell(heavybit.com) |
It's a miserable language, full of unexpected behaviors and badly designed features. Oh, it's got some interesting stuff going on, but things like:
- including the text of 'echo' (and stdout generated by invoked tools) in a function's result was a big surprise. You wind up piping stuff to "Out-Null" a lot in defense of this, since a lot of Windows tools are stupidly chatty ("The Blffgh command succeeded with status 0!") and that's awkward.
- so was the "unwrapping" that happens when a function returns a list (return [a,b] you get the list you want, try to return [a] and it rips off the list and just returns the inner 'a', and I think an empty list results in null . . . just wow).
The sole reason I use PowerShell is because it's good for mucking with OS-level objects in Windows, stuff that I would otherwise have to write native code to frob. And sometimes that is actually kind of attractive.
Yes, the output from calling native commands within functions can be a little tricky, and something that annoyed me initially, but once you understand the concept of passing objects around, not strings, you quickly get used to either capturing the output into a variable ($BlahResult = .\blah.exe /foo bar /baz 2>&1), where you can then access the string output (stdout) and any ErrorRecord (stderr) lines, or as you suggested, piping to Out-Null if you don't actually need the output. If you're writing your own 'echo' type statements for debugging or information purposes, you learn to use Write-Verbose or Write-Debug instead of Write-Output (which is what happens by default).
The unwrapping example was something that got me initially, because I was used to defining a collection (array, List, whatever), populating it, and then returning that collection... and then having the calling code iterate over that collection. "The PowerShell way" is to not focus on the collection/array at all, but instead just let it take care of that and deal with the items themselves. If you pipe the output of a function to Foreach-Object, it will work correctly - if it's a single object returned, that's what will be $_ in the loop, if it's multiple, they will all be passed in one-by-one as $_... if you actually want to access the array itself (for example to count the number of items), then you can enforce the array type by wrapping it in an additional array (i.e. @($Users).Count).
But "... quickly get used to" is more readily translated as "Spent an afternoon debugging, found the reason, did a forehead plant on the keyboard, then tried to remember on every invocation, so as not to get burned again."
Python, LISP, Smalltalk and a host of other languages made much better decisions in similar areas.
In about one-and-a-half months, earlier this year, I gave my Lisp dialect an FFI, using which I was able to completely translate the MSDN "Your first Windows Program" C example, almost line by line:
http://nongnu.org/txr/rosetta-solutions-main.html#Window%20c...
After a bunch of FFI definitions of constants, structures and functions, the core of code is just:
(defun WindowProc (hwnd uMsg wParam lParam)
(caseql* uMsg
(WM_DESTROY
(PostQuitMessage 0)
0)
(WM_PAINT
(let* ((ps (new PAINTSTRUCT))
(hdc (BeginPaint hwnd ps)))
(FillRect hdc ps.rcPaint (cptr-int (succ COLOR_WINDOW) 'HBRUSH))
(EndPaint hwnd ps)
0))
(t (DefWindowProc hwnd uMsg wParam lParam))))
(let* ((hInstance (GetModuleHandle nil))
(wc (new WNDCLASS
lpfnWndProc [wndproc-fn WindowProc]
hInstance hInstance
lpszClassName "Sample Window Class")))
(RegisterClass wc)
(let ((hwnd (CreateWindowEx 0 wc.lpszClassName "Learn to Program Windows"
WS_OVERLAPPEDWINDOW
CW_USEDEFAULT CW_USEDEFAULT
CW_USEDEFAULT CW_USEDEFAULT
NULL NULL hInstance NULL)))
(unless (equal hwnd NULL)
(ShowWindow hwnd SW_SHOWDEFAULT)
(let ((msg (new MSG)))
(while (GetMessage msg NULL 0 0)
(TranslateMessage msg)
(DispatchMessage msg))))))I'm kidding, I'm kidding. But on the other hand, related to your example: the Windows APIs are in my opinion awful. Look at that CreateWindowEx call...
regarding your second point, I don't like this unrolling either.
The trick seems to be instead of :
return $mylist
return ,$mylist
This will make sure [a] never gets changed to a, and [] never gets changed to $null
For example, instead of
return $mylist
return @($mylist)
@(mylist)Yes, powershell is an insufferable language. It is in good company with every other language ever invented by humans.
"Unexpected behaviors and badly designed features." Are you talking about ... C? PHP? Javascript? Java? Ruby?
The design/implementation faults in C, for example, come at a cost of billions of dollars a year in damages (and that doesn't include the cost of mitigation).
Sure, Powershell has plenty of warts, point me to a perfect language that does what Powershell does as well as it does but without the warts and then people can stop using Powershell.
However, he's really making it worse because now all return values from that function have to be type-checked to see whether they are scalars or lists.
It was aweful when PowerShell team tried to shove their ugly creation down people's throats by removing Shift+right click menu for "Command prompt here" by "PowerShell here". Not a sign of good product when you have to force it upon people. The designers of this thing should have been demoted, let alone making them "Distinguished Engineer".
It's slow (I have to wait 5-10 second and before it responds on an i7?!?), inconsistent (some things return local time and some UTC), unreliable (it will just fail randomly after working for two months), has terrible memory usage problems (try copying a large file with it via WinRM), has a terrible scheduling and security model (scheduling something that actually doesn't barf with errors is difficult), spooges out UTF16 randomly, and is full of nasty surprises which are constantly lurking waiting to bite your face off.
It is entirely the opposite of what I want from a language of any kind.
That man owes me at least 3 months of my life back.
The lure is a few one liners work pretty well to start with but the moment they turn into two liners then problems start but you're invested in it then and it's too late.
I use python for automation now. No such problems.
The one thing I do like is I can make assumptions about how the scripting language works - like writing a if-statement for instance. This lets me write simple scripts pretty fast. Especially compared to bash which tends to be quite quirky and requires a lot of checking docs.
Had I known how long it was before I started listening I probably would not have listened to it.
I am glad that I didn't know, so that I began listening.
Once I started listening it didn't take long before I was convinced that I ought to listen to the whole thing.
Very interesting interview.
Very glad they provided a transcript also. Having both the text to read and their voices to listen to was much more enjoyable together than would either of those have been alone.
There were some typos here and there in the transcript but since I am on mobile I didn't dare leave the page to point them out. Someone else will have to take care of that to have them fixed. I do remember one error that occurred in a couple of places was that the transcript said SANS instead of just SAN (singular) or SANs (with a lower-case "s").
> On the other hand, if I'm at a lower level and in fact I'm at a really higher level, it takes time, but it'll eventually rectify itself.
A lot of people get wrapped up in a desired title and recognition, but the confidence Jeffrey had to know that his work would speak for itself is instructive. Additionally, it's not like it was instantaneous; it took 5 years to get that rectification.
If you are at a low level, you move to Facebook/Google and you come back to Microsoft the next year. The world is filled with people who are waiting for a raise and will not get it in the next decade.
I'm not trying to discount your perspective, because it's one that I have lived out and fully endorse. However, I took this quote as a nice check, or even corrective, to my default mode of operation. I think that these checks to our own personal status quos are important to our growth as human beings.
Learning it is one of the most important things a Windows sysadmim should do, today. Hell, if you consider yourself a Windows power user, you should learn it.
A single sub-1000-line script I wrote in a couple days of downtime has saved over $10,000,000 in taxpayer dollars and has never stopped working since the day it added itself as a scheduled task. I am not a programmer.
It's very powerful and generally limited only by your knowledge of how to do it with PowerShell. And all you need to start and learn everything are two commands. Three if you count update-help.
Get-help Get-command
You can do stuff like putting callbacks on a handle to do async-io when you have a pointer to work with.
I have a UNIX background and I really, really, dislike doing any sort of programming on Windows. Powershell is a giant bucket of WTF.
I apologise, but your comment doesn't list any specifics to back up your broad assertion. Do you mind expanding on, and backing up, that assertion? Your view could apply to any technology or shell.
I could go on and on
Reminds me of Dave Korn's story.
https://news.slashdot.org/story/01/02/06/2030205/david-korn-...
See Question 5
"I think this is symbolic of the way the company works."
The player itself is built with video.js which, when combined with the ttml format for the final transcript, allows us to provide that click-to-time interactivity.
EDIT: I just incorporated this quote into the PowerShell article. The transcript differs a bit, so you might want to update it. Notable differences are the transcript's "would turn structured data" and the omission of mention of awk, grep, and sed. (Presumably, the latter sounded like unintelligible false starts to the stenographer, who just cut it entirely.)
Here's the quote:
> I'd been driving a bunch of managing changes, and then I originally took the UNIX tools and made them available on Windows, and then it just didn't work. Because there's a core architectural difference between Windows and Linux. On Linux, everything's an ASCII text file, so anything that can manipulate that is a managing tool. AWK, grep, sed? Happy days. I brought those tools available on Windows, and then they didn't help manage Windows because in Windows, everything's an API that returns structured data. So, that didn't help. [...] I came up with this idea of PowerShell, and I said, "Hey, we can do this better."
EDIT 2: The references to "PEARL" are all a little weird, too.
Perhaps it would be possible to only seek on click, not selection.
It's like any other language... until you bite the bullet and really learn it, you don't know. And the idea that Windows isn't built for scripting is from the 1990's. For the last 15 years, there's (almost) nothing that Microsoft ships that can't be automated/scripted/controlled from a command-line.
In what way? I use PowerShell daily and hardly ever have to Google [things that I've used more than once]. I imagine if I picked up another language, I'd be Googling furiously, but that shouldn't disqualify the language.
Powershell is like a gas station men's room in the middle of nowhere. It's better than nothing but nothing to celebrate.
When it launched there also wasn't a whole lot you could do with it, windows wasn't built to be automated by scripts like this. This hurt the uptake right of the bat and it never really reached that critical mass to become adopted.
A decade after powershells creation I still write batch files for windows specific things and stick to cygwin/bash for everything else. It's not even a familiarity thing, I learned (or at least tried to) powershell before I (properly) learned to use bash.
MS would have been better off creating language/environment independent binaries to manipulate windows.
> Not a sign of good product when you have to force it upon people.
This sums it up nicely, the only reason PS has any use whatsoever is because it has the MS name attached.
I feel like I should stick up for C# as it's probably one of the best languages available today. I'm biased as I prefer statically typed languages so none of my brain is taken up by utterly useless information like having to remember the names of properties, methods, method signatures, etc. which the computer should bloody well tell me.
I am doing a lot scripting now with CSScript. It's much nicer than PowerShell. The main advantage of PowerShell is interop with .NET so why not use C#?
Because you need to compile your C# source first. With PowerShell you just write and run. Also in a PowerShell shell you can fart about and test stuff without actually having to write a fully working script in a file, i.e. it's got its own REPL.
It's an extremely flexible language that's fairly readable.
And I _REALLY_ like the idea of an object-oriented piping system. I would love to have that built into Linux/bash/etc instead of every program having to have some sort of "human readable" mode and (if you're lucky) a "machine readable" mode. Except every machine readable mode is different and still requires a parsing pass to get it filtered into your program correctly. I find it fun and even relaxing to design parsing passes to bring say, smem, into a Python script. But that's because (Thank God) I don't have to do it for a living. [And the newest era of "some" programs supporting JSON or XML output but there's still no easy tools for filtering through it into the next program--requiring yet another translation layer.]
Now back to Powershell. It's fun to learn the first day. And then after that, it's just an 80 degree walk uphill after you learn one stupid idea after another. You want to do something insanely simple in concept and you're pouring through ours of documentation and examples for the exact use-case.
Personally, I've started using Python2/3 for scripts lately. Most systems I use already have Python, and Python is easily 10x-100x faster in many situations and that's not even using a compiled variant. And while I enjoy C variant programming, Python is still a huge step up from all those stupid Bash idiosyncrasies like string -eq non-string but wait, you added a space before the equals, so it explodes anyway. (I can't remember a great example off-the-top-of-my-head, but anyone who has used Bash has had those days of an obscure error being related to a single incorrect whitespace.)
A much more lightweight approach is to exchange structured data. This can even be easily done on top of existing byte streams, just standardizing the format.
FreeBSD started adopting this approach via libxo for its base utilities: https://github.com/Juniper/libxo. I hope it spreads into Linux.
As someone who's been working with aws command line tools quite a bit recently that accept and emit lots of json, I've found jq to be a life saver! https://stedolan.github.io/jq/
That's sort of what ansible/salt do. Provide the most useful system commands with structured arguments as input and output, instead of crazy text everywhere.
The only thing I like about it is it's object oriented nature. That everything is an object ( PSObject ) that can be piped is very nice.
However, the shell itself is slow, ugly and cumbersome. I just don't understand why windows can't create a shell like bash. The old cmd shell was horrible. PowerShell is a step up but it's still lightyears behind all the terminals we have in linux/bsd/etc.
> mind bogglingly verbose.
It's verbose if you don't use the aliases. But I don't mind the verbosity as you can understand the command just from the name.
I like the idea of powershell ( OO, Verb-Noun commands, etc ). But the execution has been disappointing.
One small detail is that HANDLE covers more things with the same table. So for example, a process, a thread, a cross-process semaphore or mutex - those are handles, whereas Unix-like systems have different types for each of those (pid_t, pthread_t, sem_t, etc.).
But then Linux started introducing fds as synchronization primitives too - like signalfd(2) or eventfd(2).
One of the more annoying things about Windows handles is there is no equivalent of dup2(2) - you can't swap the kernel object of an existing HANDLE with another existing HANDLE. This makes imitating certain unix idioms involving redirecting I/O after the process has already started kind of clumsy.
[Worth noting in a discussion of the HANDLE typedef that the type is literally a void pointer, and some user-mode code abuses that to make them pointers to heap objects instead of a proper kernel handle - for example FindFirstFile()'s handle is not a true kernel handle, which is for example why you need to close it with FindClose and not the normal way to close handles.]
The main difference I know of is that file descriptors are sequential, while handles are random. That is, a function like open() will always return the lowest-numbered available file descriptor, while CreateFile() can return any available handle.
Other than that, both are basically identical as far as I know: both are just keys to an in-kernel table which points to the real object.
Was this difference so good that it was incorporated on other platforms?
I really liked the idea of the PowerShell, but somehow it felt to me that it never really became "a thing" and the Windows Subsystem for Linux move from MS felt like they gave up on the whole shell thing in the end. :\
We're generally not as publicly chatty as developers, and don't tend to do Windows platform admin as a hobby. We have varying levels of programming experience.
For us, PowerShell has been nothing short of miraculous, though some people are put off by anything that's not a GUI. It's a shallow initial learning curve and easy enough for semi-automating some very repetitive user management tasks. The more skilled scripters can switch pretty easily into C# once they need the performance, because they already understand the .Net framework from mucking around with it in PowerShell.
And if you're ever at a small conference where Jeffrey Snover is speaking, say hi - he's generally eager to talk with PowerShell users.
Yes, it's very popular with Windows sysadmins, which is more or less who it was designed for.
It hasn't really taken off outside of Windows. Unless you're in a mostly Windows environment, PowerShell on the Mac or on Linux doesn't make a lot of sense.
Please feel free to take a few days and learn some PowerShell. After you feel a little more familiar with the documentation, and have used the shell enough to discuss, please feel free to reach out.
[1] https://docs.microsoft.com/en-us/powershell/wmf/5.0/feedback...
> The main advantage of PowerShell is interop with .NET so why not use C#?
Anyhow, I've written tens of thousands of lines of CScript and PowerShell over the past 15-20 years, give me PowerShell any day of the week for automation and admin tasks. Why anyone would persist with CScript/WSH for sysadmin tasks and automation (unless you're still managing an ancient fleet of Windows 2003/2000/NT) makes my mind boggle.
Also you there's a reasonably useful IDE with ships with PowerShell so you can actually set breakpoints, step through your code and inspect variables (though I prefer to use PowerGUI). That alone is a massive productivity booster.
Is it an API guarantee, or just an irrelevant implementation detail?
"[...] atomically allocate the lowest numbered available (that is, not already open in the calling process) file descriptor [...]"
Since .NET objects can't be copied from one session and pasted through RDP into another, I'll often take an array or other object, and pipe it through ConvertTo-Json into clip.exe, which throws the JSON onto my clipboard. In the RDP session, I'll pipe Get-Clipboard into ConvertFrom-Json, all written into a variable.
Piping ( | ) sends objects from one cmdlet to another. Many cmdlets and functions have a preset parameter for pipeline input, and it's easy to specify this when making your own functions. Foreach-Object ( % ) can have cmdlets act against each item in the pipeline, with $_ being the "this" character. Just remember to wrap them in curly brackets, as this marks them as as the script block being invoked.
Additionally, Powershell can interoperate with CSV just as easily, with ConvertFrom-CSV and ConvertTo-CSV.
Want a simple API? Here's a one-liner - just host the output with IIS:
While ($true) {Get-Process | ConvertTo-Json > C:\www\ApiFile.html; sleep 60}
For a more interactive API, the .NET HTTPListener class can easily be called and built upon.
My favorite part of the language is the type system, where you just put the .NET type in square brackets before the variable. [string]$myString = "Hello World"
PS> $x = @('a', 'b', 'c', 'd,xx', 'e')
PS> $x
a
b
c
d,xx
e
PS> $x | ConvertTo-json
[
"a",
"b",
"c",
"d,xx",
"e"
]
PS > $x | ConvertTo-Csv
#TYPE System.String
"Length"
"1"
"1"
"1"
"4"
"1"
The official documentation is not much help. I could go do some research on StackOverflow and so forth, but . . . I've lost the will to live. For this little exercise I've reached the point of uncaring.
This is basically my everyday interaction with PowerShell; try out something new, have it fail in a way that this LISP / Python / C++ / etc. veteran would never expect, and spend 20 minutes or maybe a whole bloody day researching why, or working around the behavior with other tools. I could become an expert in PowerShell, but I'm not interested, I just want things to work in reasonable ways. I've got shit to do. [I'm writing this HN post because in ten minutes I get to dive into some more PowerShell]
PowerShell has repeatedly failed the "reasonable expectation" test. I can't depend on it to be a lightweight and useful tool. Instead, every time I use it I can expect to spend 60 percent of my time working around capricious nonsense, dredging through postings by other people who have thankfully preceded me.
What you got seems unexpected initially, but as you read the doc on this function (which took me a good 20 secs) you may think it is indeed well conceived, and realize that it was weird to expect passing an array of strings to it and expect it to work.. I don't even know what it should've done.
Should it have given you a single line of comma separated values? But then what should you give to the function to have multiple lines of csv? An array of array of strings?
Do you have an alternative? Maybe for the edge case where the contents of the CSV is a list of primitives, it should give you a "Value" column? Or did you expect that, given an array of stuff, the best approach is to encode the whole array in a single CSV row?
Please guys, demonstrate a better way. Make that better language.
There is a reason most kernels are developed in C, most databases are developed in C, most (AAA) games are developed in C. And it's not "C has the necessary mind share". Man, these people make tons of scripting and DDL languages to get their game done. If they knew a better (more practical) approach than using C in the critical places, they would do it.
Yes, you get buffer overflows and memory management bugs. Much more so if you are a beginner, but also if you're very experienced. But people just haven't found a practical alternative.
How would you even be able to define what a valid buffer region is, if so many functions are basically custom allocators, declaring a certain sub-slice of the buffer they were given (taking pointers / indices), and handing it to the next function? It's just not practically possible to make a formalism that guards against buffer overflows here. You could make a "slice"/"buffer" datatype, but that would just just increase the line noise and keep the bugs at the countless locations of wrapping/unwrapping.
Rust, Go, Swift, etc. Even C++.
> There is a reason most kernels are developed in C, most databases are developed in C, most (AAA) games are developed in C. And it's not "C has the necessary mind share".
The only thing that is still pretty much all C is kernels. Game engines are usually C++ (a much safer language). Databases are written in all sorts of languages. Sure the big old ones are C or C++, but that's because they are old!
> But people just haven't found a practical alternative.
They have. Garbage collection works in a lot of cases. Rust has its lifetime & borrowing system. C++ has proper smart pointers (finally).
> It's just not practically possible to make a formalism that guards against buffer overflows here. You could make a "slice"/"buffer" datatype, but that would just just increase the line noise and keep the bugs at the countless locations of wrapping/unwrapping.
I take it you haven't used Go. It has slices. They work fine. No buffer overflows.
Sorry to poop your meme frolicking here, but your statement is flatly untrue at face value; other non-native languages exist that provide access to Windows API's.
Obviously, since I made this for myself at considerable effort in my free time over more than eight years, developing in Lisp must obviously be something I really want, not something I have to do.
Regarding the second part, don't take it so personally, I was just joking. You're also on the internet: if a lighthearted comment such as mine annoys you, a real internet troll will ruin your day...
This included the object-oriented nature of PowerShell. The ones who have fully accepted that it is a command line wrapper for .Net like it way better than they did when they were trying to use it as a crappy version of bash.
I, on the other hand, routinely astound them with my ignorance about the brave new world of build tools that assume that you have constant access to the internet (among many other points of my ignorance).
I'm now trying to get re-accustomed to Unix-style command line tools, and it's irritating as anything not to be able to ().somepropertyname to get exactly what I want :)
as opposed to bash/posix with its confusingly named utilities?
want to rename something? it's mv
want to tell the time? it's date
why is the command to download something curl/wget?
what's the difference between if [[ stuff ]] and if [ stuff ] anyways?
There is a rename command as well. But apart from the first time I tried to rename a file I've never had trouble remembering it. There is a good reason for it to, internally it is a move operation.
> want to tell the time? it's date
Is this different in powershell? Some quick googling indicates the Get-Date is the method you want and being .net based everything deals with DateTime objects.
> why is the command to download something curl/wget?
I find this easier to remember than Invoke-WebRequest, or is it Invoke-RestMethod?
If they were so hard to remember that could be aliased to something similar, but I've never seen anyone bother.
Edit - I mean I've never heard anyone going the other way and suggest doing "alias Invoke-WebRequest=\"curl\"" in bash.
I find this easier to remember than Invoke-WebRequest, or is it Invoke-RestMethod?
easier to remember: maybe, but much less discoverable than Invoke-WebRequest/Invoke-RestMethod. this seems to be the overall trend for linux shell commands. a few characters long for easy typing, but otherwise arbitrary strings that you have to remember and have 0 discoverability. GGP's comment was about how you had to search for how to do everything in powershell, so in that respect powershell is at least not worse than bash.
want to tell the time? it's Get-Date ("time" won't work, neither will "get-time")
why is the command to download something Invoke-WebRequest?
what does '&"C:\Program Files\TortoiseSVN\bin\TortoiseProc.exe" /command:about' do?
and if you want to complain about ridiculous syntax, why the hell is ` the escape character instead of \ like every other language?
so... working as intended? (unless you subscribe to the belief that rename is the same operation as moving)
>want to tell the time? it's Get-Date ("time" won't work, neither will "get-time")
fair point, although at least it doesn't alias any existing commands (typing time will get you a confusing output of 0 user, 0 kernel, 0 real)
>what does '&"C:\Program Files\TortoiseSVN\bin\TortoiseProc.exe" /command:about' do?
& = run program in specified path. i agree this isn't used in any other shell language, but then again all the linux shells are fairly similar to start with. it's probably there because otherwise it won't be able to differentiate between a command that you want to execute, and a string literal.
also for windows command line programs, it's fairly stander to use /arg:value for command line arguments instead of --arg value
>and if you want to complain about ridiculous syntax, why the hell is ` the escape character instead of \ like every other language?
probably because \ is used in windows paths, so to avoid unnecessarily quoting every path, they chose an odd line continuation character.
also, about what you were responding to for the last two points: that was more a poke at how '[' isn't actually part of the syntax, but in fact an alias for test.
What tool(s) have you tried using to write PowerShell scripts?
You find it hard to believe that someone who uses a tool daily gets to know it?
The PS and WSL story seemed like they tried to do better but gave up.
But what can I say, I'm no Windows guy, just talking from an outside view :)
The PowerShell story is pretty simple: a cross-platform, object oriented shell. Having used it, I attempted to locate a similar Linux solution, to no avail. I'm tired of cutting text and object translation makes my life a great deal more straightforward.
As for WSFL, it seems "Windows can run everything. Docket support is stellar" is a much more likely story than "we gave up and are slowly becoming Linux with a GUI".
Or.. you could actually encourage people to use sane path separators. The rest of the world uses a normal slash which works most of the time in Windows too.
CP/M lost. URLs won.
It's like UTF-16. They somehow had the means to replace Windows-1252 with UTF-16, but they can't bring themselves to move to UTF-8 which everyone else and the web uses instead.
I like Bash but writing anything remotely complex in it is... not recommended
If you carefully read my post, you noticed C here means actually C(++). And a great majority of developers from these domains actually write C++ like this: C(++).
Rust, Go, Swift. Show me the serious kernels, databases, AAA games. Rust hasn't even descended the hype train.
Garbage collection is a bit like exceptions. It works for some cases, but not for serious engineering of long-lived applications. If you have ever written a Java application with millions of little objects, you know what I mean. For starters, we can not even afford to pay for the extra memory overhead that comes with each object. For the advanced, at some point we want the application to do exactly that: advance, instead of doing GC only.
> I take it you haven't used Go. It has slices. They work fine. No buffer overflows.
These slices all come with the (huge) overhead of adding a reference to the original runtime object, and on top of that each array access is checked. Correct?
Oh come on. Does Lucene (or Solr or Elasticsearch built on top of it) not qualify as serious engineering? Elasticsearch is quite successful, and is indeed intended to be used a long-lived application!
Does this mean that the likes of Lucene don't run into GC issues? Of course not. I've certainly diagnosed problems in Elasticsearch related to GC (which, more often then not, is a symptom of something else going wrong), but saying it's not qualified for "serious engineering" is just patently ridiculous.
And that's only one example. There are loads more!
> These slices all come with the (huge) overhead of adding a reference to the original runtime object
Huh? This is the representation of a slice: https://golang.org/pkg/reflect/#SliceHeader --- It's pretty standard for a dynamically growable region of memory.
I don't know elasticsearch, but if this is something like a database where millions of objects are tracked (like in an RDBMS, or in a 3D Game if coded by an unexperienced coder who likes to isolate everything down to the Vertex or scalar level into "objects"), then I would assume at least one of the following applies
- The objects in the datastore are not represented as individual runtime object after all
- The GC for objects in the datastore is highly tuned (GC only done manually, at certain points),
and the memory space overhead of having individual DB objects represented by runtime objects
is just accepted.
> Huh? This is the representation of a slice: https://golang.org/pkg/reflect/#SliceHeader --- It's pretty standard for a dynamically growable region of memory.
and object/GC overhead? It's GC tracked objects after all, right? (again, I admit to knowing next to nothing about Go's runtime)
Go has value semantics. So when you have a `[]T` ("slice of T"), then what you have is 24 bytes on the stack consisting of the aforementioned `SliceHeader` type. So there's no extra indirection there, but there might be a write barrier lurking somewhere. :-)
> I don't know elasticsearch, but if this is something like a database where millions of objects are tracked
Elasticsearch is built on top of Lucene, which is a search engine library, which is itself a form of database. I don't think there's any inherent requirement that a database needs to have millions of objects in memory at any given point in time. There are various things you can ask Elasticsearch to do that will invariably cause many objects to be loaded into memory; and usually this ends up being a problem that you need to fix. It exposes itself as "OMG Elasticsearch is spending all its time in GC," but the GC isn't the problem. The allocation is.
In normal operation, Elasticsearch isn't going to load a millions of objects into memory. Instead, its going to read objects that it needs from disk, and of course, the on disk data structures are cleverly designed (like any database). This in turn relies heavily on the operating system's page cache!