попробуйте это:
DisplayMetrics displayMetrics = context.getResources().getDisplayMetrics();
int height = displayMetrics.heightPixels;
int maxHeight = (int) (height*0.80);
View bottomSheet = findViewById(R.id.bottom_sheet);
BottomSheetBehavior behavior = BottomSheetBehavior.from(bottomSheet);
behavior.setPeekHeight(maxHeight);
и в вашем XML:
<LinearLayout
android:id="@+id/bottom_sheet"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:orientation="vertical"
app:behavior_hideable="true"
app:layout_behavior="android.support.design.widget.BottomSheetBehavior"
.
.
.
</LinearLayout>