Я решаю эту проблему так:
<FrameLayout
android:layout_width="match_parent"
android:layout_height="match_parent"
android:orientation="vertical"
android:visibility="@{!vm.showMap}" // move here
>
<fragment
android:id="@+id/map_fragment"
class="com.google.android.gms.maps.SupportMapFragment"
android:layout_width="match_parent"
android:layout_height="match_parent"
/>
</FrameLayout>