Oops, whilst boggling at the JComboBox I failed to notice I hadn't
really addressed the OP's problem

The Op might enjoy reading
http://java.sun.com/docs/books/tutorial/uiswing/layout/box.html#features

I suspect the OP wants something more like
----------------------------8<-------------------------------------
import javax.swing.BoxLayout;
import javax.swing.JButton;
import javax.swing.JComboBox;
import javax.swing.JComponent;
import javax.swing.JFrame;
import javax.swing.JLabel;
import javax.swing.JPanel;
import javax.swing.SwingUtilities;

public class VerticalOddity {

    public static void main(String[] args) {
        SwingUtilities.invokeLater(new Runnable() {
            public void run() {
                new VerticalOddity();
            }
        });
    }

    VerticalOddity() {

        // smallest components
        JComboBox box = new JComboBox(new String[]
                {"apples", "pears", "oranges"});
        JButton button = new JButton("Button");
        JLabel label1 = new JLabel("label 1");
        JLabel label2 = new JLabel("label 2");

    // collect all but button into a left aligned container
        JPanel p1 = new JPanel();
        p1.setLayout(new BoxLayout(p1, BoxLayout.PAGE_AXIS));
        for (JComponent c: new JComponent[] {box, label1, label2}) {
            c.setAlignmentX(JComponent.LEFT_ALIGNMENT);
            p1.add(c);
        }

        // assemble container and button into a centered container
        JPanel p = new JPanel();
        p.setLayout(new BoxLayout(p, BoxLayout.PAGE_AXIS));
        for (JComponent c: new JComponent[] {p1, button}) {
            c.setAlignmentX(JComponent.CENTER_ALIGNMENT);
            p.add(c);
        }

    // add outer container to a frame
        JFrame f = new JFrame("Vertical Layout Oddity");
        f.add(p);
        f.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        f.pack();
        f.setVisible(true);
    }
}
-------------------------------8<---------------------------------------

I still haven't worked out why the JComboBox has an inordinate
propensity for growth.