As per the official reply from Gearset support

".....The issue is, if you manually delete the app, all the app visibilities will be deleted as well. But when using Gearset to delete the app visibilities, they are not completely deleted but set to false. The best workaround for this is to manually delete the xml from your git branch. ..."

If Gearset can incorporate this feature, it would be nice.

The workaround is not that hard as it may sound. 4 apps in 60 profiles takes about 10 mins to mass-find and mass-replace in VS Code.