I got following review from Apple while posting to Apple Store
We noticed that your app requires push notifications in order to function.
Specifically, we found that when selecting “Don’t Allow” no access to the app is granted.
Next Steps
Push notifications must be optional and must obtain the user’s consent to be used within the app.
the question is how to make it optional, can’t find documentation about this.