A more general formula is known as the Woodbury matrix identity (https://en.wikipedia.org/wiki/Woodbury_matrix_identity), which comes up quite a lot in numerical algorithms.
Also appreciated the derivation of sorts of the formula, starting with the case where the main matrix is the identity.
One suggestion: when experimenting, monitor numerical precision. This is important if matrices are poorly conditioned or if the inverse is updated over many iterations.
Also possibly of interest are Suitesparse and a Python-based guide that uses it [1,2].