Я понял, что это как-то связано с самим проектом android.
1. Created Xamarin Forms Project with the same name
2. Backed up my version old android project and moved it to another folder (the one that was not working correctly)
3. copied over the new android project I created into the solution
4. did a diff between the two projects and figured out what was different
5. Copied over what changes I had made to the new android project
После этого все работает гладко до горячего refre sh сейчас.