SafeTest: A novel approach to front end testing(netflixtechblog.com) |
SafeTest: A novel approach to front end testing(netflixtechblog.com) |
The one thing that at least initially rubs me the wrong way is how the overrides work. Like I get why that's a solution to how we can inject some data, but I don't like the idea of writing test-specific code in a component just to enable tests with this tool. That being said, I've done similar things in the past when I've run out of options and this looks pretty clean, I just wonder if there's another way.
I don't think the author has used Cypress or Playwright. Their real value is that they do not drive browsers from the outside like slow, flakey WebDriver. They also allow injecting test doubles. And to override app functionality (not having to wait for a "normal" 60 second timeout like the author hypothetically suggests).
This test framework seems to solve non-problems.
Is Netflix not quite so FAANG these days?
export const Countdown = () => {
const [time, setTime] = React.useState(60);
React.useEffect(() => {
const interval = setInterval(() => {
setTime(t => t - 1);
if (t === 1) clearInterval(interval);
}, 1000);
return () => clearInterval(interval)
}, []);
return time > 0 ? <>Time left {time}</> : <>Done</>
};
I also found an issue where the only solution was to inject sinon into the page which I wouldn't consider a great option: https://github.com/microsoft/playwright/issues/10230
So, imagine a page that rendered a version of this component that a human could navigate to (this is what was historically called a test fixture before Rails rewrote the meaning of this word), then imagine that that human could have complete control over this interval by setting a query string argument. A human can do all of the interactive testing they need. Then, when it comes time to automate, all we need to do is automate what the human can already do.
This is another principle of automation that has been lost in history. We should first be able to do things manually before automating them. When we (as automaters) jump right to automation we often simultaneously necessitate and embrace (again, because we identify as automaters) additional complexity in the form of tooling.
I'd venture a guess that SafeTest is not likely to be necessary for the things that it was built for. Software design could have solved the problems with significantly less complexity and tooling while simultaneously providing useful test fixtures for humans to explore.
Storybook kind of enables, but it's also tooling fixation in my opinion. That's another post, however.
Oh, and I saw your other post about rewriting components to allow testability. You may be attempted to accuse me of suggesting that here. I'm not. I'm suggesting that components are written with fundamental design principles in mind, especially the necessity to exert control.
There's more to say about this that touches on the example of the sign in, and I can expand if interested.
I'd also recommend refactoring to a more mock-friendly way to do that countdown if you don't want to cover up all the internal logic.
If the timeout interval is loaded remotely from some API (and it probably is if you have reasonably configurable nag popups), then you can always mock that API call.
Yes that does mean wading through each interaction pathway every time you make a release, there really is no substitute.
I hugely support tests and I write a lot more of them than most people. I just don't think it usually works on the frontend.
From what I can tell, there are two types of tests that are worthwhile in UI: unit tests on functions (not UI elements), and basic integration tests. I believe it's possible to write valuable tests that don't fit into this framework, but from what I can tell every other UI engineer feels compelled to write tests that just regurgitate component implementation details.
const FetchPerson = createOverride(fetchPerson);
must be less performant by some amount than just using fetchPerson? How many over-rides do you find a reasonably complex component needs?
Overrides are opt-in so you can just expose any overridable value as a prop and run a isolated component test on it.
any common mistakes you see people make in writing overrides or is it too early to make conclusions?
https://scribe.rip/introducing-safetest-a-novel-approach-to-...
Pinterest: https://medium.com/pinterest-engineering Workable: https://engineering.workable.com/ Skyscanner: https://medium.com/@SkyscannerEng Agoda: https://medium.com/@agoda.eng
Not being toggle parts of the app is the root of the issue when creating e2e tests. For example overriding a feature flag, you could find the API call for the feature but what if it's a grpc call and part of your dev build pulls in any update, you can't easily change the binary response with confidence anymore.
The current solutions are good enough to do smoke tests, but nitty-gritty e2e tests don't scale well.
In the example above it's simple
export const CountdownTime = createOverride(60);
export const Countdown = () => {
const initialTime = CountdownTime.useValue();
const [time, setTime] = React.useState(initialTime);
React.useEffect(() => {
const interval = setInterval(() => {
setTime(t => t - 1);
if (t === 1) clearInterval(interval);
}, 1000);
return () => clearInterval(interval)
}, []);
return time > 0 ? <>Time left {time}</> : <>Done</>
};
it('tests faster', async () => {
const { page } = await render(
app => <CountdownTime.Override with={() => 5}>{app}</CountdownTime.Override>
);
await expect(page.getByText('Done')).not.toBeVisible();
page.waitForTimeout(5000);
await expect(page.getByText('Done')).toBeVisible();
});
const useCountdownValue = initialTime => {
const [time, setTime] = React.useState(initialTime);
React.useEffect(() => {
const interval = setInterval(() => {
setTime(t => t - 1);
if (t === 1) clearInterval(interval);
}, 1000);
return () => clearInterval(interval)
}, []);
return time;
}
const Countdown = ({time}: {time: number}) => {
return time > 0 ? <>Time left {time}</> : <>Done</>
}
const ActualCountdownInContextSomewhere = () => {
const remainingSeconds = useCountdownValue(60)
return <>
<Countdown time={remainingSeconds} />
</>
}
<Countdown time={-1} />
<Countdown time={0} />
<Countdown time={1} />
but like, whatever works for your team! i think it's fair to argue for either direction, but neither is zero-cost unless you have buy-in for one direction vs another from your coworkers, which honestly is all that matters especially if you have to eat lunch with them occasionally.
Note that if you needed a specific reference in an override there isn't a good way to get that via Playwright, consider this silly example:
it('can test spies', async () => {
const spy = browserMock.fn();
const { page } = await render(
app =>
<LogoutButtonAction.Override with={() => spy}>
{app}
<LogoutButtonAction.Override>
);
await page.getByText('Log out').click();
expect(await spy).toHaveBeenCalled();
});