One of the main contributors of this project[0], was the core contributor (creator?) of Puppeteer[1], but then I guess left Google to join Microsoft and work on this[2][3].
[0] - https://github.com/aslushnikov
[1] - https://github.com/puppeteer/puppeteer/
[2] - https://github.com/microsoft/playwright/graphs/contributors
[3] - https://github.com/microsoft/playwright/graphs/contributors
- Cypress: https://www.cypress.io
- Playwright aws lambda: https://github.com/PauloGoncalvesBH/running-playwright-on-aw...
- Checkly: https://www.checklyhq.com
Meanwhile Playwright's API is just promises. Everybody knows how to compose promises.
The Cypress approach does in theory completely eliminate latency-induced flakiness, but in my experience, Playwright/Puppeteer's usage of bi-directional real-time protocols to communicate with the browser makes latency low enough that latency-induced flakiness is no longer a practical concern, especially when paired with the ability to selectively run test logic in client-side JS in the page with 0 latency.
Selenium did suffer from latency-induced flakes all the time due to its slow uni-directional request/response model. I personally believe the Cypress model is an over-correction for Selenium PTSD, and isn't making as good a set of tradeoffs compared to Playwright/Puppeteer.
Having said that, I would love to support Cypress if I had infinite time and focus.
Side note: Selenium is not on the menu, even with its large install base.
We are aiming for where the puck is going and it’s going to Playwright.
Full disclaimer: I’m CTO at Checkly.
https://blog.checklyhq.com/cypress-vs-selenium-vs-playwright...
Cypress vs Selenium vs Playwright vs Puppeteer speed comparison
doesn't playwright use puppeteer for chromium-based browsers(even edge-based), I thought it's just a wrapper for puppeteer with extra support for firefox.
I switched to Playwright 6 months ago - I'm much happier now and the switch also allowed me to delete 3/4 of the helper code that I needed before.
https://docs.cypress.io/guides/references/trade-offs#Permane...
- Playwright is more lightweight. Can be good or bad on what you expect. But I definitely prefer Playwright.
I don't know if it's Selenium specifically or some problem with the .NET binding, but I figure Microsoft must have better .NET integration so it will at least eliminate that possible source of problems.
I just want to plug Playwright by Microsoft as I've been using it over the past month and have had a really great experience with it: https://playwright.dev It's built by the founders of Puppeteer which came out of the Chrome team. Some things I like about it:
1. It's reliable and implements auto-waiting as described in the article. You can use modern async/await syntax and it ensures elements are a) attached to the DOM, visible, stable (not animating), can receive events, and are enabled: https://playwright.dev/docs/actionability
2. It's fast — It creates multiple processes and runs tests in parallel, unlike e.g. Cypress.
3. It's cross-browser — supports Chrome, Safari, and Firefox out-of-the-box. 4. The tracing tools are incredible, you can step through the entire test execution and get a live DOM that you can inspect with your browser's existing developer tools, see all console.logs, etc...
5. The developers and community are incredibly responsive. This is one of the biggest ones — issues are quickly responded to and addressed often by the founders, pull requests are welcomed and Slack is highly active and respectful.
My prior experience with end-to-end tests was that they were highly buggy and unreliable and so Playwright was a welcome surprise and inspired me to fully test all the variations of our checkout flow.
I've seen some sites that behave differently when the browser is being automated. E.g., if I access fanfiction.net from a browser being automated with Selenium it gets stuck in an endless Cloudflare CAPTCHA loop. Accordingly I've come to prefer automation methods that are less revealing to the site.
We’ve seen a consistent growth of interest in people wanting to use Playwright for browser automation (and testing).
Pros of Playwright/Puppeteer:
Reuse existing HTML/CSS knowledge
Cons: Requires an external service or shelling out to an external process
Pros of using a pdf lib:
Probably better performance, simpler architecture by being in-process.
Cons: Ad-hoc language for designing the PDF.
I used one based on docker, and the bottleneck was actually sending the html, css you want to print (if it's not already served over http). I used a shared docker volume to write to from one process (python) and read from another (the node pupetter).
It all comes down to, load html, wait to load, save to pdf. Very simple, fast, and reliable. More so than weasyprint for example.
There's so much potential to use playwright in CI/CD with GitHub Actions cron jobs. Really enjoying it so far.
We use TestCafe at work for this purpose. I personally hate TestCafe as it's is an absurd unfocused mess of a browser remote, but it lets me control my browser by navigating to a URL which no other browser remote system does.
test.use({
...devices['iPhone 13 Pro'],
locale: 'en-US',
geolocation: { longitude: 12.492507, latitude: 41.889938 },
permissions: ['geolocation'],
})https://github.com/microsoft/playwright/blob/0d277fa589e9508...
edit: ah puppeteer was a Google project. I forgot
Would love to help any of your testing needs
All of the above are uploaded automatically as Github artefact as part of auto-generated Github Action!
Br
The freemium service provides access to compute-heavy machine learning models running on GPUs.
Hackers blast 50-100 requests in the same second, which clog the servers and block legitimate users.
We reported IPs to AWS and use Cloudflare "Super Bot Fight Mode" to thwart attacks, but the hackers still break through.
We don't require accounts, but could impose account requirements if this helps.
Any suggestions?
- Hard-coded waits in your code, like "Thread.sleep(1000)". A better alternative is to replace hard-coded waits with something that waits for an element or value to appear on the page. i.e. click on a button and wait for a 'Success' message to appear. Puppeteer and Playwright both have good constructs for doing this.
- Needless complexity in the tests. Conditionals in particular are a code-smell and indicate there's something needlessly complex about the test.
- No test data management strategy. The more assumptions you can make about the state of your application, the simpler your tests become. Ideally tests are running in an environment that nothing else is touching, and you're seeding data into that environment before tests run. I personally don't believe in mocking data in regression tests since that quickly becomes hard to manage.
We spend a lot of time thinking about these issues at my company and wrote a guide that covers other common regression testing issues in more detail here: https://reflect.run/regression-testing-guide/
We don't do any timed waits, all of our waits are for an element or value to appear, but these waits never complete sometimes, non-deterministically. We then added a long 5 min timeout on these waits because we know the test will never complete at that point. It's always fine in manual testing though, and if we don't run the browser in headless mode and watch it work. Very frustrating.
Sometimes the HTTP requests themselves timeout after a few minutes, but this never happens in manual testing either. That's actually the most common issue these days, and this happens non-deterministically too. This is what I meant by "flaky".
// Expect an element "to be visible".
await expect(page.locator('text=Learn more').first()).toBeVisible();
> I don't know if it's Selenium specifically or some problem with the .NET binding
If the execution in .NET is slow then I suppose it could be .NET. But it could be (and often is) the suite design. You must wait for /everything/ before interacting with it because the code execution is quicker than the page.
Large Se/Webdriver suites are often a PIA. I find it's nice to write them with Python or Ruby so they can be debugged interactively with the an interactive shell.
That's what I do, but the wait for an element in certain tests times out after a few minutes, even though the elements are clearly visible, and manual use never has an issue.
From other comments it sounds like Puppeteer and Playwright are better on this, so will look into switching.
With great care and effort you can make your tests reliable (especially if you are happy to allow a "best of 3" type test strategy to allow for 1 flake and 2 passes) though. Prodigious use of the wait (i.e. stdlib polling) primitives seems to give you the most bang for your buck.
I am note sure if this is just the nature of web automation, or if selenium is just crap? My gut is to say it is selenium's fault since we never get the same issues when using javascript in the DOM or in an extension)...maybe it is the browser APIs I guess? U have no idea but if this playwright is any better than that would be superb.
We have an AI model that understands the page structure as humans do. So we can do this "Click on 'Sign in' on the 'Login' page".
We have a no-code tool as well which adapts to the changes. But we want to generate the code for people who want to keep things internally.
Would love to discuss it more: m@preflight.com Our website: https://preflight.com
In Playwright that manifests itself as the "auto-wait" feature: https://playwright.dev/docs/actionability
You can do this kind of thing with Selenium too but it wasn't designed in from the very start of that project.
You can also give the browser a virtual clock so that you can use time based timeouts and give every test a timeout of 3 hours, but those 3 hours only take milliseconds in real time. That approach gets CPU expensive if your site has any background polling scripts or animation, because obviously the animation will end up animating a lot during the test!
Yes, this is what I've done but the elements non-deterministically do or do not appear according to selenium, and then the wait times out. This happens for dozens of tests across dozens of pages with no issue with manual use, so something fishy is going on.
No-code is the best in my opinion :D https://preflight.com
We have done all the ground work. Like: - Concurrency - Adapt to the changes. Our selectors are like this: "Click on 'Login' button in the 'Sign in' form" - Update the tests with an HTML/Video player etc
The one event that cannot be automated is cursor movement/position. Put a check into your event handlers that check that the cursor is actually over the event target.
Are you saying block form submission unless the cursor is over the event target?
If so:
* How to handle legitimate requests from mobile users?
* How to handle form submissions with the "return" key?
Are there proxy examples you could point us to?
Thanks for your help.
The freemium service provides access to machine learning models on GPU instances, served with FastAPI.
Each request invokes a compute-intensive ML model, but perhaps there is something wrong with the FastAPI configuration as well?
It is possible, but this degrades the experience for legitimate users.
We prefer solving this without impacting/taxing normal users if possible.
It is possible, but this degrades the experience for legitimate users.
We prefer solving this without impacting/taxing normal users if possible.
Return key on a control in a form will fire a submit event. Check for cursor position in your submit handler.
I'm not familiar with it specifically, but that's my go-to starting place in weird automation issues like that. Normally it gives some kind of hint as to why that's happening (or why Selenium thinks it's happening).
edit: Actually, since you use CF, just make a firewall rule that forces the captcha for those ASNs before it even gets to your app. They have a field named "ip.geoip.asnum" for that, and an action called "challenge" which will force a captcha.
Thanks, I'll double check, but I think we do this now. In looking at the history of test failures, those failures are indeed less common, but still plenty of false positives of other types. Most persistent recent failures are the WebDriver timing out when loading a URL, which has never happened while manual testing or when being used by end users, so not sure what's going on there.
In any case, if the Playwright API encourages better idioms for writing tests that avoids these pitfalls, that would be cool because I deal with a lot of work term students that aren't adept at this kind of stuff so that would save a lot of headaches.
This is more likely when the system is loaded / shared e.g. a CI Vm.
It’s instructive to watch a screencast / recording of UI tests, because you don’t necessary intuit how spazzy and fast the harness will perform its interactions.
I've only been using/playing with it for a few weeks, but I find it more contemporary and less clunky than selenium. And it's lighter and runs faster. I dislike passing around a driver to tests and the playwright built-in testing (which I think is wrapping the expect library) is pretty nice.
I haven't fully made peace with the selector API in playwright, which uses a lot of experimental css selectors and a custom DSL, but these are minor learning curve issues.
It’s faster than cypress and you have access to a massive ecosystem as well of things that speak the WebDriver protocol, as well as the ability to do hybrid and native mobile testing (through Appium).
I’ve been really happy with it and we’ve decided to go all-in on it for apps in the Ionic ecosystem.
I watch the FastAPI repos a lot and tones of people do not understand how async python works and put their models with sync code in an async context.
We tried removing "async" -- thinking it would force sequential processing -- but it unexpectedly seemed to cause parallel processing of requests, which caused CUDA memory errors.
Before removing "async", this is the weird behavior we observed:
* Hacker blasts 50-100 requests.
* Our ML model processes each request in normal time and sequentially.
* But instead of returning individual responses immediately, the server holds onto all responses -- sending responses only when the last request finishes (or a bunch of requests finish).
* Normally, request 1 should return in N seconds, request 2 in 2N seconds, but with this, all requests returned in about N50 seconds (assuming batch size of 50).
1. Any suggestions on this?
2. Mind clarifying how sync vs aync works? The FastAPI docs are unclear.
Any help would be much appreciated.
This has been extremely frustrating.
The memory errors you're seeing could suggest that you may not actually be able to run multiple instances of the model, and even if you could it may not actually give you more performance than processing sequentially.
Seems like ultimately your current design can't gracefully handle too many concurrent requests, legitimate or malicious - this is a problem I recommend you address regardless of whether you manage to ban the malicious users.
There is an event loop that goes through all the tasks and runs them.
The issue is the event loop can only move on to the next task when you reach an await. So if you run a lot of code (say an ML model) between awaits no other task can advance during this time.
This is why it is co-operative, it is up to a task to release the event loop, by hitting an await, so other tasks can get work done.
This is fine when you have async libs that often hit awaits at things that are IO related like say db, or http calls.
FastAPI will spawn controllers that are not defined as async functions on a thread pool but it is still a python so GIL and all that.
You should do as the sibling comment says and decouple your http from your ML and feed the ML with something like Celery. This way your server is always there to respond to things (even if just a 429) to hit a cache or whatever else.
The previous team that developed Puppeteer indeed moved to Microsoft and have since started Playwright.
While it is true that staffing is tight (isn't it always), the number of open issues does not tell the full story. The team has been busy with addressing technical debt that we inherited (testing, architecture, migrating to Typescript, etc) as well as investing in a standardized foundation to allow Puppeteer to work cross-browser in the future. This differs from the Playwright team's approach of shipping patched browser binaries.
Wow, not writing stuff in TypeScript is now considered technical debt? I knew people were already rushing to rewrite everything in TypeScript if they could, but didn't knew we'd come this far along the hype-cycle already.
Can you expand on that?
They patch and compile browser binaries so they have the functionality Playwright needs.
Their build of Chromium is one release ahead of what's out but it looks like one could maintain a library of older Playwright browser binaries to test with. They probably have an older Firefox 91 binary that's feature-equivalent to the current Firefox ESR. Their WebKit builds won't ever be exactly the same as Apple Safari.
https://pptr.dev/#faq:~:text=What%20is%20the%20status%20of%2...
Right now, we use ELB (Elastic Load Balancer) to sit in front of multiple GPU instances.
Is this sufficient or do you suggest adding Celery to this architecture?
Edit: It looks like it's using Chrome's DevTools protocol. I'll have to read up more on that. I thought Google deliberately didn't want that to be used for automation, but I'm probably thinking of something else.
But it's not itself a test framework, nor is puppeteer, so it's nice having a more modern toolset on top.
"WebDriver Bidi is the next generation of the W3C WebDriver protocol and aims to provide a stable API implemented by all browsers, but it’s not yet complete. Until it is, Selenium provides access to the CDP for those browsers that implement it (such as Google Chrome, or Microsoft Edge, and Firefox), allowing you to enhance your tests in interesting ways."
https://www.selenium.dev/documentation/webdriver/bidirection...
P.S. OpeH264 doesn't help mainly because it's decoding bugs, not encoding bugs.
That's only a concern for video chats. Teams does a lot of other stuff too, and there's no reason you shouldn't be able to use that on Firefox.
It appears less like malicious DDoS and more like pragmatic theft.
@headlessvictim2 search for "Asynchronous Request-Reply pattern" if you want more information about this kind of architecture. You will remove any bottleneck from the API server and can easily scale out from the task queue.
How would this work with GPU-bound machine learning models?
The model processing takes > 30 seconds and would still represent the bottleneck?
Here is an example of what it could look like: https://docs.microsoft.com/en-us/azure/architecture/patterns...
Right now, we use ELB (Elastic Load Balancer) to sit in front of multiple GPU instances.
Is this sufficient or do you suggest adding Celery into this architecture?