It depends on what you are using the email address for. Some things you might want to use it for:
* knowing this is a real person
* marketing communication
* security/data breach communication
* forgot password flows
* step up authentication (send a code to an email)
* user lookup if one user wants to share something with another
* putting users on teams/allowing special access based on domains
In most of these cases, you want to verify not just that someone knows an email address, but that they have access to and/or control the inbox. This is what verification proves.
If you are just looking for a globally unique identifier and aren't doing any of the other stuff, you could skip verification. But in most cases you'll want to do this.
Incidentally I hope you are using a drop in auth service or a library--there's a lot out there and no need to roll your own.