The simplest way is definitely to use Swing and Matisse.

http://java.sun.com/docs/books/tutorial/uiswing/learn/index.html

and

http://java.sun.com/docs/books/tutorial/uiswing/

Now if this is an assignment where you cannot use Matisse or Swing,
please post up a short compilable example of what you have and I'm sure
someone will give you some advice on it.

You need to be going through the layout tutorial.
<http://www.google.com/search?q=java+tutorial+layout>

Here is the simplest way to layout a calculator
keypad..

<sscce>
import java.awt.*;

class CalculatorKeypad extends Panel {

 String[] num = {
   "7","8","9","/",
   "4","5","6","*",
   "1","2","3","-",
   "0",".","=","+",
 };

 public CalculatorKeypad() {
   
   // adjust spacing (last two numbers) to suit need..
   super(new GridLayout(4,0,5,5));

   Button b;
   for (int ii=0; ii<num.length; ii++) {
     b = new Button(num[ii]);
     add(b);
   }
 }

 public static void main(String[] args) {
   Frame f = new Frame("Calculator");
   f.setLayout( new BorderLayout(5,5) );

   CalculatorKeypad ck = new CalculatorKeypad();
   f.add( ck, BorderLayout.CENTER );

   Label output = new Label("Result goes here..");
   output.setBackground(Color.black);
   output.setForeground(Color.white);
   f.add( output, BorderLayout.NORTH );

   f.pack();

   f.setVisible(true);
 }
}
</sscce>

Note the layout is exactly the same for Swing
based components (JButton, JFrame, JPanel
and JLabel).  Most modern day projects would
be using Swing for these components.