You can, of course, encrypt things.  In general, there are two
possibilities:

1. You eventually need to recover the original.  In this case, you need
to have some software installed on the system somewhere that contains
the encryption key.  This is the weakness in your security.

2. You don't need to recover the original.  For example, UNIX password
files work this way; instead of decrypting passwords, the login software
encrypts the candidate, and compares the encrypted forms.  If you can
pull a trick like this, then you're in better shape, but it's worth
noting that you're still not entirely safe, especially if the data is
predictable or short.  If someone can obtain a copy of the encrypted
data, then breaking the system becomes possible by simply testing
guesses.  For this reason, modern UNIX operating systems do not make
their password files world-readable as they once did.

If you're just trying to make this HARD, then it's possible.  If you're
trying to make it impossible, then you've got problems.

You originally mentioned a private key store.  Here's how that's
generally done.  First, you create the key store itself in some
unencrypted binary format.  Then you ask the user for a password.  You
munge the password with some non-reversible encryption scheme such as
that used for UNIX passwords, which doesn't require a key.  Then you use
THAT as the key for a reversible encryption scheme, and encrypt the
keystore.  To recover the keystore, you ask the user for their password
again, munge it the same way you did before, and then use that key to
decrypt the keystore.

That has the advantage of scheme 2 above, in that there's a non-
reversible (by which is meant that it's computationally hard to reverse
it) step, so you don't need to store the password anywhere on the
system.  Additional gains can be made by designing a dense enough binary
format that something reasonable -- as opposed to an obvious failure --
comes back from trying to decrypt the keystore using the wrong password.  
Incidentally, the java.security package contains APIs that implement all
of this and use it to manage a keystore.

Then, after you've gone to all this effort to protect the keystore,
someone will enter the keystore password into a plain text file such as
Tomcat's server.xml and thus break the whole thing.