Currently, ExtendedKey.Derivation.toPrivate(at:) will throw an exception if called from a parent public extended key. This is obviously not good...and after several days of attempting to fix it, I've given up for the time being.
For more info: BIP32 Spec