Refresh Comparison should have an option: "Selected Files only"
A common use case is:
1. Compare default(66) source and target
2. Select some files
4. Get some error indicative of either missing file or out-of-date file(s)
5. Fix file(s) in underlying org/source branch.
6. "Back to results" Optionally mark additional files as selected.
7. "Refresh Comparison"
This takes too long, you just want the selected files to compare (and really, just get on with deploying). You don't want to wait for both source and target metadata to load and compare. You don't want to uncheck/check specific metadata components that represent your selected files. You just want a refreshed set of selected files from the last compare