Refresh Comparison should have an option: "Selected Files only"
A common use case is:
1. Compare default(66) source and target
2. Select some files
4. Get some error indicative of either missing file or out-of-date file(s)
5. Fix file(s) in underlying org/source branch.
6. "Back to results" Optionally mark additional files as selected.
7. "Refresh Comparison"
This takes too long, you just want the selected files to compare (and really, just get on with deploying). You don't want to wait for both source and target metadata to load and compare. You don't want to uncheck/check specific metadata components that represent your selected files. You just want a refreshed set of selected files from the last compare
Often times I find myself updating 1 or two files since a prior comparison. Rather than refreshing the entire thing which can take a while, it would be nice if I could specify which ones to refresh.
I'm aware that I can set which metadata types to compare, but even in large orgs, the 4 or 5 common ones can still take quite awhile from a full refresh.