I think we have translation problems.  If you are creating a mail
server, why do you need to know how to run mail servers?  If you mean,
what do I need to know in order to comply with smtp and pop standards,
the documents are refered to by their RFC numbers.  search "rfc smtp"
and you get technical documents relating to the smtp protocols.

at no time would you use tomcat or jsp to handle smtp or pop requests.
These would be handled by deamon processes running on your linux
server.  Writing a java program to handle incoming smtp requests would
be possible, but silly.   I use sendmail and qpopper.  Does anybody not
use sendmail?

Look through your linux docs for setting up the smtp and pop daemons
that come with the distribution.  Use tomcat and JSP for web services.

Rarely would you write one from scratch, unless, possibly,  it was
just to send an canned alert email to the site administrator. You use
the JavaMail classes.

See http://mindprod.com/jgloss/javamail.html