This doesn't seem right to me:

> 3) How do you determine which devices and OS versions we should test on?

> This should generally be an easy question for the candidate. Good candidates will point to app analytics as the best measure, looking for the most used devices for their particular app. Another good answer would be to check the app reviews where people might have complained about specific issues happening on their devices. Less creative candidates might simply suggest “looking at the top devices” on the market

Looking at top devices is also incredibly important, and it seems foolish to dismiss that strategy.

What if your website doesn't work on iPhones, and so you have no iPhone users? What if, because people can't even sign up on an iPhone, they don't bother complaining about specific features not working?

Surely it makes more sense to test based on your target market, rather than the subset of people who are self-selecting to use your product?

Ideally you should be looking at both usage and market data. If your target market contains lots of iPhone users, but everyone's on Android, this points to a potential problem.

This is actually awful. A QA manager should be asked how testing is integrated into the development process, how to infuse quality into everything, how to hire/train/retain testers, how to estimate for QA. Writing test plans is an antiquated approach so I'd rather hear what they've done that works and why.

Hmm. I find proposed answers to these questions somewhat... unexpected. Lacking, in fact. They focus so much on technical ability, and seem to ignore the human side of being a manager. Starting with the first one... "1) Let’s say you are the first QA manager joining our startup. What are the first three things you would do?" The kind of things I'd expect to hear here would never be of the kind 'write a test plan'. And I hardly believe anyone would be able to contribute a test plan right away, would they? I expect ability to write a test plan, or setup a process where it is created and managed, to be rather easy to check by just asking someone to do a test plan on an imagined scenario. The answer I would like much more would be: '1. meet the team', '2. understand the product, its vision, goals', '3. understand current dev/QA setup and code before I proceed'.

In other words, it seems to me that the first things to do should be to see where we are, connect with people rather than making things up. How can you propose first actions without at least basic understanding of where you are?

I like questions 5+ much better, but still these seem more suited for interviewing an experienced individual contributor, not a team manager.

How about setting up a team? How about hiring? How would you define and split tasks, assuming we agree on tasks? What kind of testing would you personally do, which would you delegate to QAs and which would you require to stay on the dev team? What would you do in situation X? (any difficult multi-optimization problem where people, technical and philosophical issues are to be considered)

I can imagine a great individual contributor to ace every single question on ths list, and then fail utterly when he has to manage even 2 people effectively.

The issue of where to draw the line between developer testing and QA testing is an interesting one.

I think it's best to answer it in reverse and start with a bug.

Did the bug make it to production? Is it a critical path? Was it covered by the test plan? Was it covered by a functional/unit/integration test? Was it clearly defined in the product requirements? Was the behavior clearly defined as part of a business objective?

Most bugs could have been discovered earlier in the chain. The earlier the better. Many times bugs are a result of systems failure, bad communication, assumptions, etc.

I think the two most important concepts in QA are 1) answering the question of "can we ship this now?" and 2) encouraging systems thinking so that all of the processes related to executing business goals in a software company can be continuously improved upon.

Pretty much stopped reading when the answer to the first question involved writing test cases. Testing does not have to mean churning out hundreds of test cases, they are inefficient and not that effective at finding bugs. If a QA Manager ( or tester ) had such a strong attachment to them then the interview would be over. Definitely stopped reading when answer2 was 'detailed test cases'

This is a pretty good list. It's a bit self-serving on the technology questions as well as the general line of questioning for TestMunk (understandably).

If you are really hiring a QA Manager, also consider asking them about their team structure/building philosophy, elaborate on the technical requirements for those roles, if they have experience developing a testing process that works with your company/group's development process. What experience they have with problem hires and how they've dealt with them.

This is really a list of questions of a principal test engineer. A good manager should be able to answer them, too, but it's far more important that they can make a group of QA engineers perform well (otherwise you should be hiring a QA lead).

11. There are two number 3s. Or this is an attention to detail trick for QA managers.

Most of this is pretty good. Some of this is a little old-school.

There are a couple of things I disagree with, speaking from my own experience as a test lead. The biggest is the definition of "a good testcase."

If you define your test cases at the specific UI widget level, they have to be updated whenever any UI change happens, period. Since you usually end up with a number of test scenarios for a given area of the UI, that usually means that a UI update actually requires a number of test updates.

That has the same maintenance as UI automation tests, but without the benefit of a widget map, reusable functions, or an automated callout when the test is wrong. And as anyone familiar with the test automation pyramid knows, UI automation tests provide a huge scaling issue with maintenance even with those. Echoing that scaling issue into your test documentation is a huge mistake.

Tests should be defined at the level that makes the intent of the test clear to an experienced tester or user--no more, no less--unless for some reason you're shipping them to an inexperienced audience like an outsourcing firm. That usually means being very specific about input data, and -if- it's microtesting UI behavior (tab from here, end up here) being specific about that. Everything else, generalize. You do not need to spell out stuff 123abc for the person who's been running tests for you on a daily basis.

Maybe you don't write it at "check that signin works," but maybe more like "check that sign-in works with a valid username/pw, at no more than 2 seconds latency."

But you know, if you don't care about latency, just say "Check that valid sign in works." That's fine. You only have to get specific about what kind of invalid data should trip it up, and even then "without a digit," "without a symbol," "with less than 8 characters," that's all fine too. The test will get executed correctly if your tester isn't a total loss.

Also, people suck at manual regression testing but are great at shortcutting and will wander off a script that specific anyway. So you may as well accept that and write them at the level they'd execute. What you lose in repeatability, you 100% gain in maintenance and productivity.

Repeatability is overrated when humans do it: take advantage of the fuzziness. It's more coverage. Just make sure they can write a good enough bug report to tell you what they actually did that caused the issue. If you want that level of repeatability and definition on the tests, do it with automation. At least there you can define a widget map and some flow-encapsulation methods.

Honestly, if I didn't get the job with at least the nice and less-ranty version of that answer, probably not the right company for me anyway. If you're not legally IEEE-compliant or an aerospace/medical company, wasting time benefits nobody.

One of the most exciting movements in QA is context-driven testing. It has 7 tenets here:

http://context-driven-testing.com/

...but they all boil down to "do the right thing for what you want done, and quit being so damned dogmatic about it." That includes not generating artifacts if nobody else will read them, and not defining tests past the level of detail needed to make them effective.

QA has a crappy reputation because it's slow, ponderous, and often not very effective. Maybe part of that is because people are updating overly-detailed docs all the time because other people have told them they should, rather than because they really need to.

If you need that level of product documentation, write the product documentation and refer the tests to it. At least that way you have "single point of truth" for flow. Fragmenting and repeating the docs across a bunch of disparate tests is just like copy-paste coding: a maintenance nightmare.

I'm personally in the middle of helping my org move to checklist-based tests. They can be used as loose regression tests, or as missions for exploratory testing, and only have to be updated when the basic layout of the app changes instead of every single flow detail. It'll get us out from the currently crushing maintenance of reviewing and potentially updating 6000+ micro-documented UI tests on every release, and will be a huge win.

People really should compare this sort of thing to coding maintenance. We've learned so much there that can be applied here as well.

I haven't worked in a continuous integration environment but why would that fall on QA?

May be any personal question which may express your sense on "compromise". If you say i am deep rooted with perfection, have a added value.

I manage a QA team where there was none until very recently. If you have any reading on point #2 that you'd recommend, it'd be hugely helpful to me.

Couldn't agree with this more; especially:

>So echoing that scaling issue into your test documentation is a huge mistake.

and

> QA has a crappy reputation because it's slow, ponderous, and often not very effective. Maybe part of that is because people are updating overly-detailed docs all the time because other people have told them they should, rather than because they really need to.

I started my career in QA and moved over to Dev, then back to QA, and now back to Dev. One thing that frustrates me most as a Dev about QA folks that I've worked with is their insistence that the script is God, not the right functioning of the application to serve the customer/user. Often that comes down to these two issues.

Which organization do you work for? How long have you worked there? Take a few minutes to read over what you just wrote and see if it makes sense to anyone else besides you?

Typical process might be - and this is at a very hight/crude level - 1) job that checks out & runs unit tests triggering 2) a job that creates a build triggering 3) a job deploying to an integration/staging environment and possibly even triggering (or with an air gap, requiring manually running) 4) a job to promote to a production environment.

Testing can/should be added between various steps there depending on the type of product you are working with. Automated functional tests be integrated into a process like that at several points (qa ownership of portions of unit/integration testing framework and automated tests that smoke test new deployments to a staging environment) and knowing those points/tools that integrate well with your CI system/ability to implement those integrations is probably something you want a qa manager to be familiar with.

Because depending on your level of CI integration, many of the automated tests that QA writes can be run as part of the CI process. Developers are really bad at maintaining CI systems, so it usually falls on the QA team to ensure new tests are included in the CI workflow.

Ultimately, it's QA's job to certify a release. With CI, that certification is often done in an automated fashion. Thus, QA should have ultimate responsibility for the configuration and execution of the tests as part of CI.

In my experience QA wants it a lot more (test from known artifacts) than dev.