I see the pyo3 crate constantly rebuilding

The pyo3 crate uses checks $PYO3_PYTHON for a path to the python binary it should use during the build. If this environment variable is not set, pyo3 will look for whatever version of python is on the $PATH, which unfortunately results in the crate being rebuilt when $PATH changes (i.e. whenever the cargo artifacts are used in a derivation which may have different build inputs).

The way to remedy this is to explicitly set PYO3_PYTHON to point to the version of python that will be used by the derivation:

let
  chosenPython = pkgs.python3;
in
craneLib.buildPackage {
  env.PYO3_PYTHON = "${chosenPython}/bin/python";
  nativeBuildInputs = [
    chosenPython
  ];

  # etc...
}