Testing Sync at Dropbox(dropbox.tech) |
Testing Sync at Dropbox(dropbox.tech) |
1) when we upload a file, we first ask the server which blocks we need to upload. this allows us to deduplicate blocks with other files or previous revisions of files you have. I had written some code like...
let blocks_needed: HashSet<BlockHash> = ...;
let file_blocks: Vec<BlockHash> = ...;
let to_upload: Vec<_> = file_blocks
.into_iter()
.filter(|b| blocks_needed.contains(b))
.enumerate();
for (offset, block_hash) in to_upload {
...
}
2) nontrivial interactions between different components in the system are hard to test manually and come for free with trinity. trinity will simulate cancelling file transfers, crashing the system, and even dropping the local databases (to simulate disk corruption). these are all hard enough to test in isolation, but knowing how to combine them is almost impossible. so, having "automated test generation" for these cases is really useful.
3) the "theory of trees" seems like it'd be pretty simple, but canopycheck has found some really interesting cases, especially around moves. when we started, we hadn't really thought about how to handle move cycles, concurrent moves that cross ancestor chains, ...
Why is that? Google Drive doesn't seem to need to struggle that much.
1. I'm a bit surprised that you don't persist the scenarios of the tests after a failure, only its seed. Does it mean that when you want to replay it, you have to redo the minimization phase? Or, do you have a way to find a seed for the minimized scenario?
2. Do you have some tests where you generate a set of operations and play them twice: one time with the mocks and one time on the real servers to check that they have the same results?
3. The article says "Note also the importance of the commit hash, as another type of “test input” alongside the seed: if the code changes, the course of execution may change too!". How are you ensuring that a commit really fixes a bug, and not just change the execution path to a happy path where the conditions of the bug are not met? By playing again a lot of tests, or do you write a new unit test that exhibit the bug to ensure the reproductability?
4. Do you think we can say that CanopyCheck is applying randomized testing at the unit tests level and Trinity is applying it at the integration tests level?
1. It does redo the minimization phase, but the actual execution is extremely fast, so this cost is minimal. Storing test outputs gets pretty expensive when you are running millions of tests, and since there are very few failures, recomputing this is worthwhile
2. Yes and no! the article talks about this a little, but the "heirloom" system does essentially this, and the "native" filesystem variant of Trinity runs the same Trinity tester code against a real filesystem. The "no" is due to the issues with randomized testing -- since any operation that you do can affect the RNG, the exact operation that is run for a particular seed can change if you swap any part of the system. For regression tests, the operation sequence can be put into a separate, non-randomized test.
3. both
4. Testing in Nucleus is a sliding scale from "unit-test-like" to "integration-test-like" -- Trinity is mocking plenty of functionality; CanopyCheck is simultaneously testing many different components. It would probably be more accurate to say that CanopyCheck is testing a smaller subset of components with much greater fidelity, and Trinity is testing as much of the sync engine is practical.
Like, if both Alice and Bob has changed somefile.txt, but Alice has added a few lines in the beginning, and Bob a few lines at the end (i.e. something that would "git merge" cleanly), will the sync engine merge that into one file?
I'm very curious about what thoughts you guys had on this issue, because I can see arguments for both sides.
I can see us perhaps doing this in the future for more targeted file types where the "rebase" behavior is well-defined, though.
The file object model wouldn't work for merging Word documents and such
What platforms do you test in “native” mode? What hardware backs it?
here's a talk from one of our engineers on our macOS CI infrastructure: https://blog.macstadium.com/blog/virtualizing-mac-infrastruc...
I thought Dropbox was actually testing Sync (https://www.sync.com/) in-house and doing a public comparison.
If you have time, I have another round of questions:
1. Did you try formal methods like TLA+ on the client? I think that the logic covered by CanopyCheck may be a nice target.
2. Do you have some tests with several clients running at the same time on a shared directory? In particular, I think of the termination invariant where the clients are fighting because several users have reorganized the directory by moving a lot of stuff, and each client is trying to converge in a different direction (ie they are making operations that cancelled the ones made by other clients).
3. The article says "In the Nucleus data model, nodes are represented by a unique identifier". Does it happen that a node has to change its identifier? For example, in a scenario like this one:
(ada) $ offline
(grace) $ offline
(grace) $ mv ~/Dropbox/shared/TODO.txt ~/Dropbox/private-grace/
(ada) $ mv ~/Dropbox/shared/TODO.txt ~/Dropbox/private-ada/
(ada) $ echo 'foo' >> ~/Dropbox/private-ada/TODO.txt
(grace) $ echo 'bar' >> ~/Dropbox/private-grace/TODO.txt
(ada) $ online
(grace) $ online2. Kind of. There are tests for interactions between instances (in particular, between different Dropbox folders on the same machine), but running Dropbox twice on the same folder is explicitly not supported.
3. Yes, this happens. There's logic to handle these changes, since some applications expect to use in-place edits and others swap in a new file -- you don't actually need grace to have an interesting situation. You can experiment with this yourself in a Dropbox folder :)