There is already an option to limit visibility of custom fields: toggle "Visible to guests and limted members".

Please expand to other roles. For example, there are custom fields that only admins care about. I don't want those custom fields to clutter everyone else's views.

Thanks