It depends on which piece of the application you are using email notifications with. If you are using it for Syslog Automated Actions you'll want to turn on debugs for SyslogAnalyzer.
For something like DFM you would turn on NOS debugs:
DFM > Configuration > Other Configurations > Logging
Put a Checkmark on Debugs for Notification Services
you can also look in the smtp.log in your logs directory and a sniffer/packet capture is a very useful tool to troubleshoot email issues.