The weirdest bug I've seen yet(engineering.gusto.com) |
The weirdest bug I've seen yet(engineering.gusto.com) |
I mean, you basically do! You can just go check out the chromium source.
It's mentioned in the bullet points in the "trouble reproducing the bug" section that chromium wasn't affected.
> Using open-source Chromium instead of Chrome did not cause crashes, so we couldn’t see what Chrome code was failing either.
I guess some things never change.
Python tests were taking ages on VSCode due to an SVG spinner:
https://bugs.chromium.org/p/chromium/issues/detail?id=103626...
lolol I can just imagine the engineer working on this: "huh, it keeps crashing, that's weird. Oh well, it was like that when I got here. ship it"
Grammarly is an application that I don't get. The fact that people are installing, basically spyware, on their computers just to get grammar suggestions to make their writing more boring and add a spellchecker (which is already inside web browsers) is pretty astounding to me. The fact companies allow employees to have it, despite obvious security issues of sending everything one types to a saas, is even more wild.
[1]: "How a flawed idea is teaching millions of kids to be poor readers" https://www.apmreports.org/episode/2019/08/22/whats-wrong-ho...
[2]: "Sold a Story: How Teaching Kids to Read Went So Wrong" https://features.apmreports.org/sold-a-story/
[3]: "According to the U.S. Department of Education, 54% of U.S. adults lack proficiency in literacy, reading below the equivalent of a sixth-grade level" https://old.reddit.com/r/todayilearned/comments/rqulik/til_t...
The lack of "continuing education" in the era of the internet is baffling to me.
Last I checked US students rank well and are near the top in most education global rankings, so I think bad education is more of a global problem than Americans think it is. Maybe that's outdated though, I'll do my research.
I know a guy who used to get inexplicable feedback about his communication that boiled down to “write better.” This limited his ability to get promoted. He runs all his comms through ChatGPT and asks it to “make this more professional” and doesn’t get that feedback anymore.
You write like a native speaker, so I'm not surprised. But imagine having a few years of school-German, and then taking a German language job. I'd bet there would be times you'd want a writing assistant, too.
There are also plenty of native English speakers who for whatever reason got a crappy education, and didn't get a lot of writing feedback.
As far as corporate security goes, you are correct, and we ban it. But I get why people want it.
That is what stuck out to me: Installing rando applications on your corporate computer that has access to internal stuff... Whoooaaaa Baby! That's just a security disaster waiting to happen. It's stuff like this that eventually leads to draconian and crappy "Nobody gets admin access to their machines" corporate policies coming down.
Most TechCorp places I worked, if someone installed something like that on their corporate device, they'd get at least a stern talking-to and probably sent back to security training.
especially what is puzzling me is:
> For security reasons, we do not have Chrome crash reporting enabled.
so we do not want to have stack traces or whatever else this includes for security reasons, but installing basically keylogger that does spell check is ok
there are companies that forbid using chatgpt for even html development because this could leak company secrets, but grammarly on confluence/jira is just fine
If you're going to generalize "everyone" you need to understand your business type is a tiny tiny minority of what most people do.
I understand why Grammarly sells their product as a service, but it's irresponsible and they are just waiting for their Okta moment.
It is an abnormal developer and an even more abnormal business that actually spends enough time to find the root cause of outré glitches. Especially when you start having to debug complex third party systems to debug them properly - requires skills and motivation plus a company that will encourage a developer to do that.
The story is not specific to Gusto - it is the story of every developers life. I have chased down bugs in my OS and my browser - it is rarely well rewarded! Fixing a compiler bug should be on my bucket list! A long time ago I worked around a compiler bug by inserting a label: (I think the label prevented certain optimisations where the label was put).
The customer response team was extremely quick and responsive telling me their hands were tied.
Fuck Gusto.
Then we get to the punchline: "Uh, we fixed the bug, but sorry folks; we didn't solve the puzzle". So I guess we'll never know why that particular anigif crashed Crome but only Chrome, and only if Grammarly was installed (or had been installed during the same session).
I hope Amy Lai lets us know if the story ever gets an ending!
> We also confirmed with many of our affected users that they had Grammarly installed on their computers.
Ironic.
This smells like a potential security vulnerability.
I have absolutely no idea how a combination of grammarly and a specific gif would cause a browser crash though…
Anyone here use the grammarly desktop app? Any additional clues?
The main reason –other than curiousity– is to ensure that a future regression (in Chrome/Grammarly) wouldn't lead to it again.
The mainstream approach that really made a hash of things though was "Whole Language" learning, largely commercialized by Fountas and Pinnell, which eventually provided the program and associated materials to classrooms around the world. This program relied, in part, on guessing: if a child got stuck on a word, they would be asked to guess what word might fit in the sentence. Sometimes they were given context clues, like a picture on the page. Any efforts to associate individual letters or letter combinations with sounds was abandoned.
Wikipedia has something of a watered-down overview of this at https://en.wikipedia.org/wiki/Whole_language, but I really recommend listening to the podcast in the second link in my parent comment (https://features.apmreports.org/sold-a-story/), APM Reports put together a compelling examination of what happened, and it's explained well. (I do wish it was also available in a more typical article format.)
“Sorry, the system says no”
Had similar situations with PayPal and Uber recently where their support have absolutely no information or ability to take a decision.
Support essentially becomes a glorified text-to-speech system.
GIF stands out as a widely understood file format [1][2].
To kick things off, delve into the GIF file using a hexadecimal editor. HexFiend [3], for instance, offers a template for visualizing GIF file structures [4]. Another excellent option is Synalyze It! [5], which comes pre-loaded with an extensive list of file formats, encompassing GIF among others.
These visualizations serve as a guide to pinpoint any irregular byte clusters that might pose issues when loading the file into an application with an image reader lacking support for that specific byte group or its arrangement. Once you've identified such a cluster, consider it the bug.
[1] https://en.wikipedia.org/wiki/GIF#Example_GIF_file
[2] https://www.w3.org/Graphics/GIF/spec-gif89a.txt
[4] https://github.com/HexFiend/HexFiend/blob/master/templates/I...
If you don't know why the fix worked, you may not have actually fixed it.
Discord client uses Electron, which is in turn Chromium.
This company would sure make a great asset for the FBI, NSA and CIA given that they're so interested in snooping in on foreign language speakers; which happen to be grammarly's main demographic. The thing is malware.
A friend of mine had an aunt who passed away, and so he ended up inheriting her car. The car came with a petrified apple pie in the back. He was insistent that the car would not start without the pie in the back window.
Several of his friends who he played in a punk band with confirmed this, that they had tested it. Take the pie out, car won't start. Put the pie back in, the car starts.
I don't think anyone ever figured out what was going on, I graduated a couple of months after hearing the story, and fell out of touch. But - timing and vapor lock makes sense, if they were always testing it by first starting the car, removing the pie, and then putting the pie back in.
As an aside, the aunt who had passed away was one Aunt Martha (after which the car was also named), which in honor of the strange car and its strange pie was what their garage punk band was named after. There's some totally unrelated band now called Aunt Martha - any evidence of their band is not on the internet.
One more point for Don't Be Clever. As Brian Kernighan put it: “Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.”
It was really hard to generate thoughts of the form "subsystem x causes the crash" and not of the form "try to do experiment x next", but here's what I came up with:
- a headache trying to think about it - if it's not an extension, it's likely another piece of software that has code running in the browser, like PDF plugins used to before Chrome started handling PDFs itself - the crash is in one of the browser subsystems, so either in the JS engine, the GPU code, the layout engine, the parser, some format plugin (video? pdf?) or the network code. Some format plugin is most likely of these to only crash for tech support and remote developers but not devs
Seems I was pretty close. After doing this exercise I would probaply have compared lists of installed software between staff who see the bug and those who don't, which would have solved it. My takeaway is that Zen debugging (just thinking what possible causes the seen behavior could have, without atking any new actions or measurements) is useful even when it seems completely useless and causes a headache.
One thing that comes to my mind happened in a company where they used a simple software to create medical reports. One clerk suddenly started creating "broken" PDF files. I debugged this for hours and could not find anything. At some point, I just copied the full raw text content that the clerk had written into one of those files out of the database into Sublime Text and I suddenly saw a small box appear between two characters of one word. It turned out that this clerk had created a Word document with text templates. Some of those texts were copied (at least partially) from other documents or websites. Apparently, either something went wrong while copying some text, or there was some control-character embedded for whatever reason in that text. I don't even remember what we did to prevent this from happening anymore, but one single "invisible" character can really cause a lot of trouble.
Another point is of course if you have strange side-effects just by combining specific software. Back in the day when Crysis 1 was new and popular, I was a little involved in the modding-scene. At some point, my CryEngine editor would refuse to start up. I reinstalled it and tried to debug it, but I could not find anything myself. So I turned to the forums and found a thread where a small group of people had the exact same problem. We all shared our setups and - basically by accident - found out that we all had Wacom tablets plugged-in to our computers. I don't remember if it was enough to unplug the devices or if we also had to uninstall the Wacom drivers, but one of those things fixed the issue and we could all continue to build beautiful maps in the CryEngine.
Btw, I am now suddenly constantly aware of my Grammarly plugin ...
But stranger things have happened, and given the enormous surface area of a modern computer (hardware, software, drivers, state, etc.) can anything truly be deterministic?
alert(banana++)
If it is just you and your application you can just spray paint from the hip the alerts from top to bottom. After each line of html you can have one more <script>alert(banana++)</script>, in the middle of your css </style><script>alert(banana++)</script><style> etc
If there are uhh thousands(?) of people actively using the page you put just one alert some place in the middle.
Everyone will have to click on OK, the page crashes. You ask the crashee: Did it crash before or after the "make America great again" alert message. (call it something they would remember)
Now you know the issue is in the top or bottom half. You move the alert to half way the half with the issue in it. If you can get some sort of reasonably consistent crashing you will find it pretty quick even in production.
Hope this helps, or maybe it is a stupid idea and you could explain why.
Improper animated GIF decoding is potentially a bug and potentially a security vulnerability. The reality is this leads to duck-tape workarounds and greater tech debt on the production side of the web. Sigh. There are standards, there are expectations, there quirks across N implementations, and MxN layers of duck tape in M consuming implementations. Just one implementation intentionally or unintentionally being different causes M headaches.
Also fun Chromium-derived browserisms:
- Updating while open is allowed and leads to silent and not-so-silent breakage.
- MSFT MDE causes unexpected breakage in fun ways every now and then, including the cause of a crash while dragging a tab on Windows.
- Some flavors of Chromium browsers are broken with IPv6 enabled, leading to an ERR_NETWORK_CHANGED on every nth page visit.
We finally figured it out when a coworker couldn't replicate and we noticed they were a version of chrome behind. We were able to track down the specific commits in chrome that broke & fixed it in our case which was pretty cool to see at the time.
If you have any more time I recommend reading through the commit log and see if you can find the changes that broke/fixed this for you. I'd bet on another rendering bug.
They don't address why they didn't just run Chromium. Or Firefox.
(This is potentially better than the 'solution' they much later ended up with, in which they probably only relieved a symptom of an underlying problem that can exhibit again, and in the meantime is a zero-day exploit waiting to happen. At least, with a different browser, there's a chance that the vulnerability doesn't actually exist, when it's known to exist in their Chrome configuration.)
That's insane!
Can anyone think by what possible mechanism the installation of Grammarly could affect whether a .gif file would crash Chrome?
The company seems to be on Macs since they report that the problem doesn't surface in Safari.
Is there some kind of dynamically-linked GIF decoding library used by macOS that Chrome relies upon, and Grammarly somehow installs one that takes precedence for all applications? I didn't think this would be possible -- I thought image decoding was done natively in the browser and not outsourced to the OS, for security reasons.
Chrome does in fact attempt to block some apps from doing this because they're known to cause crashes. You can see the blacklist here:
https://source.chromium.org/chromium/chromium/src/+/main:chr...
They don't like apps doing it but you can apply for an exception:
https://sites.google.com/a/chromium.org/dev/Home/third-party...
So my guess is that Grammarly is injecting code into other processes, probably to try and gain access to their text controls so it can Grammarlyze stuff, and somewhere along the line corrupts the heap or stack in some way. The GIF aspect might be a red herring, as the crash was IIUC never reliably reproducible. Random crashes that happen sometimes but not others is strongly indicative of something being randomly corrupted.
Or it could be something else. Chrome moved to blocking third party injects by default some time ago, so maybe that's also a red herring.
I see some comments highlighting RAM, which could totally have been the issue. Totally looking forward to a follow up to this later down the road, I am sure this isn't going to be the last time we hear of this.
> Unfortunately, with access to neither the Chrome source code [...]
I mean, it's still very possible to debug (especially with the fact in mind that Chromium is open-source, and for me has been a very useful source when reverse-engineering and debugging Chrome), but I understand why web developers would not be trained in reverse-engineering techniques.
Never do you see the guy investigating memory usage, which is weird.
> This was fairly far outside the usual scope of our on-call issues. Our team is generally well-insulated by other teams from issues like browser compatibility, so I didn’t know the first thing about browser debugging.
I do reasonably okay at self-guided education when I want to, but there's definitely a difference vs. a structured secondary education environment, where there is accountability and other people to guide each other through the process. And, that's coming in to those subjects with already a better-than-average literacy and numeracy; I have to expect that for people who struggle with grade school reading comprehension or math, trying to bootstrap those abilities alone would be daunting.
Also, there's just less room for pursuing those now. Lots of people are getting squeezed by concerns that aren't part of most childrens' awareness -- housing costs, bureaucracy, the treadmill of maintaining all the machines that get us through daily life. Those add enormous pressure to dedicate more time towards professional development and "getting ahead", or at least not falling further behind, and that has been eroding all of the unstructured time that I would spend working my way through a textbook (or online class). People with poor literacy are probably more likely to have lower-paying positions, so all of those demands are even more severe.
Not that it's impossible. Lots of people do manage to self-educate their way out of poorer circumstances, and certainly the internet has made that far more accessible than it was before the turn of the century. But, let's not underestimate how challenging it is, either.
My suspected culprits:
1) The massively increased complexity of ordinary living is overconsumining our personal resources. and
2) For post-GenX and later, the erasure of childhood (free-roaming & peer-only hours) sabotages[1] the reward systems (joy) that supercharge early learning.
The less joy there is to facilitate learning, the more effort is required (from otherwise overly depleted resources).
[1] similar to what abuse and neglect do
Modern living is so monotonously boring and devoid of any challenge that people are find more and more creative ways to try and get an ounce of that physical and mental stimulation back in their lives (hobbies, exercise, gaming, etc).
One friend literally couldn't read. He took himself through adult reading courses. He ain't no Shakespeare, but I can now txt him and get a written reply.
Tech is helping, but the underlying reason for the change is their own initiative.
I would judge that none of the friends or family illiteracy was actually caused by our schooling system in New Zealand. Some people just struggle and our pedagogy will always be imperfect. Certainly I can see some failures in my own schooling that still exist and I would like to see fixed (mostly get rid of 99% of the deep crap).
It's all about incentives. That is companies are incentivised to give continuing entertainment for ad clicks, rather than building a world of the educated that may have a better all outcome for society (but probably not the ad companies at all).
My favorite thing about the Kaitai Web IDE is that it instantly updates the parsed tree as you make changes to the format specification, which makes it viable for reversing unknown formats instead of just specifying known ones.
The best they could do was work around it.
Sometimes workarounds are the best you can do until your vendor provides a real fix.
If your system doesn't work, and you just plonk around at values, until, very surprisingly, the system starts behaving well and you the call it working... well it might be working now. But it's just accidental correctness. As soon as something causes the system to bank left, something's gonna break and no one knows how to fix it - and you're back to square one.
On the other hand, as hard as it is, if you can clearly tell why your fix will restore function to the system without even applying it, you have deliberate correctness and function. If done right, it is very boring, because exactly and only the expected thing will happen. You should know about the unknowns and plan around those as well, so even if an unknown bites you, it's a known and handled unknown. This can be exhausting to make happen, because it is much harder, but those systems will just work.
But this is a fight I have with some development teams probably forever. "But we poked at the values, and that stopped the flames. It is fixed!" "but why?" "Dunny. But no fire anymore. All good." And then 2 weeks pass, and there is more fire and everyone is like "Oh but why would this happen? How should we have known for this to happen again"
I hypothesize that Chrome simply has a global (i.e. cross-tab) per-toplevel-origin limit to the number of allocated accelerated drawing canvases it's willing to allow; and that when you go over it, Chrome forcibly de-allocates all the existing drawing canvases used by other tabs that have that toplevel origin loaded, thereby causing them to crash. It's probably a measure designed to prevent a site from from DoSing your computer by just allocating an infinite number of canvases.
I suppose since they dont know the root cause it’s impossible to say. But I think the saying would fit better if they kept the gif but made some change that seemed to fix it without knowing why.
The article implies that Gusto's employees can whatever browser they want.
And, honestly, telling your employees to run a browser that only techies have heard of sounds like a really dumb idea.
For users of their security-sensitive internal software?
> And, honestly, telling your employees to run a browser that only techies have heard of sounds like a really dumb idea.
Sounds like they're using this for internal tools, as a kind of thin-client layer. They could recommend or mandate a particular browser, and people would just use it. ("Click this icon, and a window opens with our internal tool. It's pretty much the same as any other browser, as far as you care.")
Probably:
1. Because it is reasonable to expect the application to work in Chrome.
2. Chromium isn't intended for production use cases.
Back when IE and Chrome had about equal market share, I worked somewhere that had one team insisting that all employees must use IE for one of their applications, and another team insisting that all employees use Chrome for their application. 50%+ of support calls were employees confusing the two browsers.
It makes much more sense to try a different browser first and see if the problem persists. Instead of test versions and extensions.
It's things like:
- total stream-of-consciousness gibberish that could probably be assembled into a coherent statement if the writer would re-read what they wrote and edit it
- A complete lack of punctuation, or even understanding of sentence and paragraph structure; at a glance, it looks like what I described above, but it's different because there's definitely a topic and a point they're looking to make, but they can't put the words together correctly.
- spelling so bad, that even with context, it's unclear what word they're intending to use.
- A wild misunderstanding of how to start and stop conversations online. (One recent example is me asking someone on Facebook if I could stop by to check out a garage sale, and a clarifying question about a term they used, only to get the response "ok." Note that in their post, they didn't specify an address beyond the name of the town they live in.)
You can definitely point out flaws in the way I grew up - somewhat solitary, spending a lot of time alone in my room on a computer connected to the internet - but I think that it at least taught me how to make myself understood in written form.
This drives me nuts. Did anyone see this [1] on HN the other day? People in comments were springing up to defend this atrocious writing style.
Make a paragraph. Make a point.
That's a whole different beast from an email I'll get from a coworker/neighbor where I cannot parse what's even being asked of me, and where the writing is so confusing I don't even know how to ask them to clarify their statement other than to tell them to start over, possibly all the way from kindergarten.
US is at the bottom of the OECD PISA rankings (as it is with life expectency too), though on a global basis you're right (better than Morocco or Indonesia on both criteria).
Shockingly Australia has fallen quite a bit from the initial PISA study where it was ranked #4, now almost as bad as USA.
(Though I dislike the way french and European higher education in general works. You're basically boxed in to your specific domain or degree that you often don't even really choose and changing or switching careers is almost impossible. The choices you make in high school basically define what you can even study in, and thus what you can do for the rest of your life. I think that's one thing the US does super well, even more so considering that degrees are less important there in the first place.)
This was true 50 years ago but hasn't been true since the 90's or so. France may well be the exception in this, but then again, France is an exception in many ways.
Thus the US’s “Level 1” literacy rate, which represents being able to follow basic written instructions, was 92% in 2014. But in 2020 the standard changed yet again to: “54% of adults in the United States have English prose literacy below the 6th-grade level.” Noticeably being literate in a non English language suddenly doesn’t count, the prose at 6th grade level is also higher than it’s been in the past.
Or as Wikipedia puts it: In many nations, the ability to read a simple sentence suffices as literacy, and was the previous standard for the U.S. The definition of literacy has changed greatly; the term is presently defined as the ability to use printed and written information to function in society, to achieve one's goals, and to develop one's knowledge and potential.[3] https://en.wikipedia.org/wiki/Literacy_in_the_United_States
I can also extend my speculation about all the times I've personally seen "Error 5", although this is probably dipping into superstition territory: I think the per-toplevel-origin draw-context limit isn't static, but is influenced by the amount of VRAM available on the computer as a whole.
So on an extremely VRAM-constrained system like a Raspberry Pi (or on a system that's running Cyberpunk 2077 on one display and Chrome on another, sharing a GPU) the per-toplevel-origin limit on accel draw contexts might be dropped as low as, say, 5: an amount low enough that just having one extra .gif on the single tab of the site you have open might be enough to make the tab fall over.
For example, the article doesn't get to a root cause in an absolute way. There is no absolute SEGFAULT of the OS causing the misbehavior. However, they nail down the crash to a gif, and if the gif is in, it crashes, and if the gif is out it doesn't. If the gif is loaded otherwise it crashes, too. At that level, to me, that would be enough, because we're users of the browser's rendering there.
Finding a solid cause that can demonstrate and reproduce a problem, and basing a workaround around that at a boundary you're unwilling to cross can be fine. If it's within the company, it absolutely is fine as long as you escalate beyond that boundary.
However, I have enough teams who are like "Oh, we set all values to 25 one by one and when we arrived at flum-value at 25 it stopped crashing. Fixed." Why 25? Who knows. Why flum? Who knows. Maybe the other value changed at the same time fixed it? Who knows. Do we use 26 once it starts crashing again? Fuck knows. Maybe 24 is better?
We have no explanation for 25, so why would 25 be a good fix?
Not to be blunt, but you might get a closer shave with Occam's Razor.
If my wife said "I'm going to the shops" I wouldn't assume she was lying until proven otherwise.
The conclusion does not follow from the premises.
My last job was Director of IT and I'm absolutely sick and tired of security fear mongering. Turn on MFA for everything. Boom. You are now secured from 99.9999% of attacks. Chat GPT is absolutely the least of my concerns.
Where it will go through teams, outlook/exchange, or O365.
Not leaking data is no longer as easy as it used to be. Just some forms are more accepted than others.
Compared to the alternative of having your colleagues think you’re a bit stupid just because you were raised speaking a language other than English, or your parents weren’t middle class… using Grammarly or ChatGPT is a no brainer. I’d support anyone using whatever tools they can to overcome discrimination and thrive.
The alternatives are:
1. Educate everyone in the company to stop discriminating against people based on language ability (impossible??)
2. Provide a local self-hosted version of the tools (although as a worker at RandomCorp, I would probably prefer to forward all my comms to Microsoft than to management!)
3. Tell people facing discrimination to just shut up and deal with it.
I wish they tried to simply rename the file instead of remove it.
I would guess grammarly is hooking chrome and potentially trying to read metadata about images, and the particular gif had metadata in a format they hadn't expected.
> If my wife said "I'm going to the shops" I wouldn't assume she was lying until proven otherwise.
I certainly hope you wouldn't! But the null hypothesis would indeed be that the words she speaks have no bearing on whether she is going shopping. I don't know your wife, but I find it easy to believe that your prior probability of that hypothesis is quite low.
(At least, until it becomes a pattern where she's regularly staying out all night with more and more flimsy excuses....)
In France, from what I know there is a fairly strong culture of secondary education that creates an 'in-group', not unlike what you see in the UK or the USA with their top-tier universities, and you are either 'on the plan' or you won't be able to get in unless you are of exceptional abilities and that rarely happens later in life, so I think that alone is sufficient to explain the discrepancy.
In NL you can enroll in higher education basically whenever you want, quota permitting and with the intense competition for such spots from abroad by very qualified young people this too can be tricky, depending on the field. But in NL a university degree isn't a pre-requisite for many jobs outside of academia (and teaching) itself.
Anything to do with technology tends to be more merit based, and achievement there tends to trump formal education, and by the time you are 40+ that formal education tends to be weighted far less than when you are say 25 and just out of school.
Other countries would add more to the pattern of variability, there is a huge difference between say Poland or Romania or the Nordics or the Baltics, further reflected in the weight that which a diploma or degree from such institution would carry, especially abroad. For instance, right now in the Baltics there is something of a brain drain happening with the younger generation moving West in droves and so as an older person it is stupidly easy to enroll in a university program. But that degree isn't going to help you much unless you remain in the local economy and the degrees from a decade or more ago are given more weight than the ones that you get there right now because they are fairly desperate for students just to keep the departments up and running.
The first tiers are more targetted towards craftsmanship (e.g. arithmetic and trigonometry you can quickly do in your head, you start working earlier in life, as early as 16) while the latter tiers are more universal and abstract (e.g. math concepts that have better use for computer science, you start working much later, around 25 years old).
You can switch between tiers or fetch later.
I know that here in Quebec, you can enroll in university no matter what as long as you are 21 years old and finished high school. Does that happen in Germany? Or is it rare to actually be able to switch between "paths"?
The other key thing was that the bug only appeared on our staging preview urls, not on the live website. It turned out it was because of a bad regex in the grammarly extension that caused the page to hang if the domain name was more than about 100 characters. Our staging domains were pretty long, I think they contained a few subdomains and had a job number or something in there.
This one is more crazy though if it is really caused by the desktop app - that's pretty scary!
Just today I debugged a regex that would DoS our backend whenever the user enters the wrong thing in a form.
Now I'm reading up on regex engines: https://swtch.com/%7Ersc/regexp/regexp1.html
https://ota-meshi.github.io/eslint-plugin-regexp/rules/no-super-linear-move.html
https://www.npmjs.com/package/eslint-plugin-redos/v/1.2.0A manager of some sort had his aging laptop replaced due to a company wide Windows 10 upgrade project. Super friendly older guy, probably in sales. IT went through all the procedures of inventorying software and network needs, backing up user profile and docs, etc. Great processes in place. I remember this because I saw the device assessment and it was like a 10 year old Thinkpad with 4G of RAM and a note saying he had to keep it plugged in at all times or it would shut down. Who puts up with that? Patience of a saint. Anyway.
Laptop was deployed by onsite IT to verify everything was gravy. All checked out except for Grammarly. License didn't get transferred properly or something so they had to put in a request to get his licensing working.
Fast forward a week and he gets his license key and Grammarly is tested good to go. He's checked off the list.
Later that day we get a call about not being able to see security cameras because the web page is crashing. Helpdesk tries the basics, reboot, clear cache, reinstall browser, rebuild profile, etc., nothing works and it gets escalated to me. I check the network, firewall logs, log into another PC, onsite, off-site, etc. All working for me, no one else having issues.
I tell him "I'm completely baffled here, have there been any changes lately? In your office? With your PC?" He jokingly says "Well yeah they installed Grammarly today maybe that's it?" We both laugh and I say well, I'm literally out of ideas, fuck it let's try it.
I remote in and uninstall Grammarly. "Ok go ahead and try the cameras lol". I then watch him open up Outlook, go to a folder named "Cameras", and open an email with a link to his cameras "home page". It fuckin worked. I turned Grammarly back on and clicked the link and sure enough it failed.
I made him a browser shortcut, moved his "email shortcuts" into his browser, blew his mind, and closed the ticket, but it definitely bugged me.
This tracks because it was some very dated camera software (you'll know what I mean if you've seen it) and the link was to his customized homepage with a super long php (or something) generated url. He was the only one at the site with Grammarly as well so it was the only time we saw the problem.
Thank you, I can finally close this cold case out in my brain.
Over three hours later, we discovered that the combo of her specific video card driver version along with her specific printer driver version would keep text from printing out underlined.
https://www.zdnet.com/article/xerox-scanners-alter-numbers-i...
Ouch
Basically using the OS's rendering to make a raster that is then sent to the printer. The main thing the printer's driver does in this case is know how to take the bitmap and tell the printer to print the bitmap (i.e. chunking data and/or sliding the right commands into the bitmap stream)
Contrast to, say, PostScript which allow for more compact and better scaling definition of what to print. This obviously works better for quality, however for a long time the issue was you then had to have sufficient processing power on the printer itself to handle it.
[0] - Search for 'GDI Printer' for a little more info.
If you can't share the dump for similar reasons to why you have crash reporting disabled, you can build minidump_stackwalk from Chromium and use it to generate an unsymbolized stack trace that you can post to the bug. A Chrome developer can then symbolize it.
https://www.chromium.org/developers/decoding-crash-dumps/ has some more details.
Also, I am fully here for Gusto posting this to say "wasn't our fault" and to throw some shade at Grammarly in the process
When restarting from Windows, Windows doesn't shut down my realtek audio device, but only puts it to sleep and Linux fails to start it. Only solution is to always do a shutdown from Windows and then hitting the power button. The issue is still there: https://askubuntu.com/questions/1032543/no-sound-in-ubuntu-1...
Unfortunately, with access to neither the Chrome source code nor the Grammarly source code, we can only guess.
Is this what the "open source" movement has created --- developers who are totally lost without source code and refuse to dig deeper? Of course the corporate interests who don't want us to know the truth, because their profit depends on it, would absolutely adore such propaganda...
I still remember a time when a lot of people would disassemble, understand, and patch programs without source --- and many of them weren't even career developers; it was just a way to get software to do what one wanted, and driven by that motivation, one would naturally learn enough to do so.
The article also touches on another point worth mentioning: the amount of complexity in the whole stack is insane. Seeing all the frameworks upon frameworks being name-dropped, I can't help but feel like a lot of this is self-inflicted.
When I removed loader-spinner.gif, the placeholder we display while the menu options load, the page stopped crashing.
Do menu options take long enough to load that they need a loading animation?
The world of software engineering is enormous, and knowing all parts of the software stack hasn't been remotely feasible for a very long time.
Also consider the fact that software engineering is a journey of learning, and every one of us is on a different part of that journey.
Then I noticed in their screenshots that some of the menus had weird wording too. Turns out they had Chrome's "Translate this page" option on. Problem went away when we showed them how to properly switch languages in the app.
<meta name="google" content="notranslate">
Apparently the new incantation to fix an app (can be applied to an element) is (ugly: I presume it isn't CSS to avoid supporting dynamically changing it):
<html translate="no">
Every now and then I would look at the meta tags for a major single page app and discover some new horror when searching for the reason for the meta tag!
I have seen accessibility tools in Chrome lead to this kind of issue in the past with a dropdown menu -- to the point where it could be replicated with a miniscule amount of HTML. The particular bug I hit 2 years ago was in Chromium-Edge, but the symptoms and cause were very similar.
Grammarly almost certainly leans on some of the accessibility tools in Chrome. These tools are somewhat different in the various Chromium flavors (Edge, Brave, Chrome, etc.).
It was a Sign from the universe that it was time to make the switch. Who are we to reject Signs?
(Full disclosure: I'm an engineer on Firefox. But that has nothing whatsoever to do with my advice here, no siree Bob, not in the least.)
As a PM, we spent 4 months making the onboard easier, and now you want people to install a new browser?
And business continuity investment is like insurance: A waste until you need it.
Too much monoculture is short sighted and long term expensive.
To the author, did you ever consider contacting the Chrome dev team about this? They're pretty responsive to bug reports.
(Grammarly has a bug bounty btw... and their chrome extension has quite a large surface area...)
If OP is here: can you provide the raw .gif file? (And if you're feeling generous, maybe even a minimal ruby example that replicates that templating setup, although it sounds like that wasn't required to reproduce it in the end.)
P.S. "For security reasons, we do not have Chrome crash reporting enabled" - maybe consider disabling Grammarly extension for the same reasons ;)
This is a great post anyway. Well written and still quite intriguing right up until the end. And it seems lots of comments on here seem to also know about this problem, so I think I can still satisfy my curiosity.
I had a funny one a couple of weeks ago: Text on a button would sometimes not display. After much experimentation, it turned out that the text would display unless the character 'D' was present. It turned out that the sizing of the buttons was just barely large enough for the font-size selected, and apparently the 'D' in the selected font was microscopically taller than other characters.
FWIW I have to thank ChatGPT for helping solve that one. I explained the problem, and it gave me a list of things to try...
https://support.grammarly.com/hc/en-us/articles/360003816032...
But alas, I hadn't discovered some new quantum effect... It turns out to have been because the race condition was close within maybe 1 millisecond and adding the console.log() statement there meant that one part of the code would take a bit longer to execute and so the race condition would not occur.
Admins had a fun day when that was found!
Can’t remember the exact event name now, but the browser fires a play event when gifs loop.
This particular gif was issuing too many of these play events that made the app super slow and freeze since it was doing some work in those handlers.
We had a bunch of crash reports and bug reports. None of those showed the actual gifs our customers were using. When we asked for the actual gif, we immediately spotted the problem.
The Larvae of leafhoppers are commonly known as spittlebugs, which create protective bubble nests while feeding on plant stems
For another fun debugging tale, google: Mazda radio Seattle NPR bug
The auto conversion to webp on the backend, signaled by chrome, resulted in a bad image that crashes the browser due to grammarly parsing of said bad webp.
Safari doesn't tell the server it does webp and so it downloads the actual gif, and doesn't crash.
<html translate="no">
> all pages in a single-page-web-app
[1] https://bugs.chromium.org/p/chromium/issues/detail?id=872770
it's hilarious
I’ve alternated between frontend, “full stack”, and backend roles for over 20 years. It’s my experience that what you’re describing is “dev in a nutshell”—neither “modern” nor “web” in the sense you seem to mean. And in any case it’s highly situational and variable depending on the dev and their team.
I would love to have the original and the un-crashy gifs mentioned. It's super easy, generally, and even without an extensive knowledge of image formats to get a grasp of what might be going on and then going down some really exciting rabbit-hole of image encoding/decoding issues.
Just take the two gifs and run them (one by one) through ImageMagick or GraphicsMagick to print out the details of what's in them, and look for differences.
Assuming ImageMagick is installed (or GraphicsMagick) installed, something like:
#imagemagick
$ diff <(identify -verbose loader-spinner-CRASHY.gif) <(identify -verbose loader-spinner.gif)
#graphicsmagick
$ diff <(gm identify -verbose loader-spinner-CRASHY.gif) <(gm identify -verbose loader-spinner.gif)
...and rabbit-hole away
EDIT: formatting
Spending time to create a test case and sending it to the browser bug team gets the bug fixed? Riiiiiiiiight.
Actually I did that for a while and the Chromium team would occasionally fix some fairly subtle issues: assuming I could make a demo of the problem and took the time to write up a good bug report. Maybe they just liked me! The Chromium team also wrote fantastic public followup to bug reports (whether fixed or not).
Certainly I never had any luck getting even extremely serious browser bugs fixed by anyone else (Apple, Mozilla, Microsoft) regardless of how much time I wasted trying to give good informative bug reports. And you never found out anything further - talking to walls is more productive.
It feels good trying to help the world be a better place, but it wasn't worth it.
Don't waste your time fighting windmills. Find a workaround, document it with a comment, forget about it. Do something that makes your business successful instead.
What is effectively guessing which is what this article entails - without knowing the actual cause - hardly qualifies one as "relentlessly curious".
Personally I deeply dislike the random walk towards insanity that modern dev takes with the constant churn and layers of fixes upon fixes - react state is no good! use redux! use this! use that! And before you know it knowing what is actually going on becomes nearly impossible!
I'd rather call it scientific method: observe, form a hypothesis, experiment and analyze results.
I agree it is anticlimatic to not know the root cause, but the rant about the current state of web dev seems out of place. We dont even know if it was because a web technology.
Some folks were mentioning issues from printers being caused by graphic cards drivers. One would love to blame printers, but it turned out they were not the culprit.
> Unfortunately, with access to neither the Chrome source code nor the Grammarly source code, we can only guess.
Chromium source code is open. They could also certainly try different versions of Chrome to bisect when the issue has started to happen. Isn't there a chance perhaps this crash might actually be disguising a buffer overflow vulnerability as well? Typically user inputs aren't supposed to crash browser/renderer processes.
Sorry but there is no rule that every comment made about an article must be specific to the outcome of an article about troubleshooting. I can comment about other things covered in the article too.
For engineers, absolutely, two browsers because some are liable to hit (self or team induced) bugs. But for sales? Support? Nah.
I had thought printers could be trusted with their own rendering, but of course that is another can of worms...
Grammarly is honestly insane, I can't believe corporations allow it to run on employee machines.
It seems like that must be the case. If we have the details right about desktop app only (which seemed pretty clear).
..+ or ...* are ways of writing "at least two characters".
Your comment had me wonder if a kickstarter for an rpg game called HP printers would take off. The potential fan base is HUGE, I imagine even people who would never play a game would want to see it.
Let the GUI be like little computer people (only an office)
https://youtu.be/SkTgX1mGmDg?t=17
Then the player has to go though rituals depending on the HP printer type, swap old for new then new for old then old for new ink cartridges after finding the instant ink package in storage, if any spawned of course.
If you try to go to fast the hero starts destroying the office, starting with the printer. Then a new different model is delivered. Ready for setup.... or maybe not?
In some laps of consciousness I purchase one more some years ago. The hours of configuring it made me feel dumber than the wrath of the HP dungeon master.
The ISP has a wifi router fixed on channel 11, half the neighborhood sits on channel 11, range is about 4 meters, the printer doesn't reconnect unless unplugged first. But there is wifi direct, this doesn't have 50 cm of range, it wants to use channel 11.
in the game our hero at least at first doesn't know about channels. The next printer arrives: Something seems off, there is no usb cable in the box? Do you want to A) order a cable B) we will use wifi! Of course if you order the cable the boss will complaint that it takes to long to set up the printer but when you eventually get the cable the boss wants to print from his phone...
The thing is, a lot form people are already playing this game, which limits the appeal of the simulation version.
We were bleeders, but there still existed a vestigial login page, and some other evil cthulic pages (I know whence they were begat for I was their father).
https://www.eia.gov/energyexplained/electricity/electricity-...