@Sebastian Hans, Thanks for posting in Q&A. According to the following article, I find when we configure app under "Show or Hide apps", others such as the built-in apps Contacts/Camera/Appstore will be hidden from the user, giving you an experience similar to that of multi-app kiosk mode.
We can configure the required apps under "Show or Hide apps" under device restriction and see if the new Passwords app will be hidden.
If there's any update, feel free to let us know.
If the answer is helpful, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.