Less an issue of privacy, more an issue of trust. I don't care if he accesses my email and he doesn't care if I access his, but we both trust each other not to unless there's some sort of emergency or something.
I wouldn't have gotten married in the first place if that basic level of trust wasn't there.
I did once by accident a long, long time ago. Picked up the wrong iPhone. Spent about 3 seconds thumbing through the subject lines before it clicked in my head what I was doing and how weird it was. Seems like if you're intentionally looking through your significant other's e-mail, the trust is long gone before you find anything actually worth being distrustful of.