You should be able to delete the email address book, or manage them because they are (or can be) personal.

Your (Outlook/Gmail) address book is imported into Clickup at the time you connect your email account.

When you share this mail-connection within Clickup, you also share your imported address book to everyone. This might not be what you want. I certainly was surprised to see all personal contacts everywhere. And that I couldn't manage them.