Global Kernel Locks in APFS

Global Kernel Locks in APFS(gregoryszorc.com)

141 points by indygreg2 7 years ago | 46 comments

kiwijamo 7 years ago |

I wonder if this is why recent versions of macOS performs so poorly on old HDD-based MacBooks. The difference between my work's 2012 MacBook (HDD) and my personal 2015 (SSD) MacBook is like night and day. Most apps open in a second on my SSD MacBook yet take up to a minute to open on my employer's HDD MacBook. It's not just this MacBook, I also see the same on a work iMac as well. Such a shame as these older machines previously didn't run so slowly.

bradknowles 7 years ago | |

Replacing an HDD with an SSD really is a night-and-day kind of difference for any laptop I've ever seen, regardless of the OS or filesystem.

IMO, that's likely to be a much bigger factor than the filesystem.

ce4 7 years ago | | |

Maybe it's analoguous to the old adage "what intel giveth microsoft taketh away". As soon as a critical mass has SSDs, no one cares to test regressions with spinning disk systems...

awiesenhofer 7 years ago | | |

Unfortunately not the biggest one though. I too have a old 2012 macbook pro with hdd. While it was flying under snow leopard it now grinds to a halt every time i open any app or document. Already ordered an ssd but still dreaming of downgrading it back to snow leopard.

GeekyBear 7 years ago | |

2015 was the year that Apple moved to NVMe SSDs, so they are bound to feel much more snappy than spinning rust disks, regardless of the file system involved.

https://www.anandtech.com/show/9136/the-2015-macbook-review/...

DavidNielsen 7 years ago | | |

Except for their desktops models, the base models of even the 5K iMac still ship with a Fusion Drive, and we only just got support for APFS on those configurations with macOS 10.14. As the base models are typically all authorized Apple Resellers and pop up stores get they must sell a bunch of those especially in regions where there aren’t a lot of Apple Stores (and where people don’t want to pay full price for a year old system).

They are going to need to support and perform well on non-SSD systems for quite a while.

js2 7 years ago | |

APFS is still (afaik) only used on SSDs. So your HD boot volume is probably HFS even on Mojave.

chmaynard 7 years ago |

The author worked very hard to determine the root cause of the problem but he was stymied because the APFS source code is not available. Why not? Apple has open-sourced the Swift compiler with great success, but there seems to be no movement within Apple to open up other system software components.

mikhailt 7 years ago | |

They do plan to document it fully later:

> Is APFS open source?

> An open source implementation is not available at this time. Apple plans to document and publish the APFS volume format specification.

Source: https://developer.apple.com/library/archive/documentation/Fi...

My theory is that Apple rushed APFS and its implementation isn't in a great shape for the public review. APFS doesn't feel any faster on my SSDs on macOS. I suspect they'll add it in a few years, after a few iterations.

mbell 7 years ago | | |

> My theory is that Apple rushed APFS and its implementation isn't in a great shape for the public review.

The rollout of APFS was _incredibly_ well done; I don't think there is even a remote comparison to how successful it was. A new file system was rolled out to millions of devices in an automated fashion with very few issues. I don't think anyone else has even considered such a thing. I highly doubt the code is that rough, I can't imagine pulling this off without fairly solid core code in place.

twotwotwo 7 years ago | | |

They released the format spec here: https://developer.apple.com/support/apple-file-system/Apple-...

MBCook 7 years ago | | |

What makes you think it’s a rush job?

Perhaps it’s just the lawyer reviews holding it up.

JadeNB 7 years ago | | |

> They do plan to document it fully later:

> > Is APFS open source?

> > An open source implementation is not available at this time. Apple plans to document and publish the APFS volume format specification.

They claim to plan to document it fully later.

saagarjha 7 years ago | |

The APFS source code isn't available, but you're free to disassemble apfs_vnop_readdir in apfs.kext. I'm seeing a couple of calls to lck_rw_lock_shared, so it's entirely possible that there's a lock here.

userbinator 7 years ago | | |

This. It's what everyone I know who works in the Windows space does when they have a deep problem to investigate. Lack of source doesn't mean lack of ability to investigate, and sometimes the source doesn't tell the whole story either.

(Yes, I know about the anti-RE clauses in EULAs. If they were actually enforced as strictly as they claim, people like Mark Russinovich and Matt Pietrek would've been sued out of existence long ago, along with just about every Windows security researcher.)

wmf 7 years ago | |

Apple releases most of the macOS kernel but I guess it doesn't include APFS: https://opensource.apple.com/source/xnu/

NegativeLatency 7 years ago | | |

That's missing a good chunk of the interesting stuff related to how users interact with the os though. (app and framework source code)

jarym 7 years ago | |

Pretty sure the code would expose just how much of a rush-job apple did with APFS. It may be released one day..

twotwotwo 7 years ago |

I'm too late to the thread, but wonder if locking might have to do with updating access times on the dirs on read--that would help justify a read-only workload taking some sort of exclusive lock. The "fix" there would be something like relatime to get atimes set infrequently.

There're def a ton of other possibilities. Some locking could be overbroad, like an exclusive lock where a reader lock would be enough. I don't see super-strong evidence that any locks involved are actually global, just that they're under contention here. And I can understand shipping a product with somewhat excessive locking because perf issues with specific workloads are a better problem to have with your shiny new FS rollout than data loss.

Anyway it's mostly shooting in the dark for us here (though cool some folks disassembled the functions), but there are some shots in the dark that look different from what I could find in the comments already here :)

lelf 7 years ago |

Absolutely nothing there indicates the presence of global kernel locks. readdir (obviously!!!) needs to hold a lock, it’s not global, it’s per-DIR.

georgn 7 years ago |

As someone who used to work on BitKeeper, I'm tickled to see a more popular SCM's test suite as a kind of de facto filesystem benchmark -- confirming what we had been observing ourselves forever.