Well, man ulimit -v, since you seem to be on UNIX.  That would need to
be run within the context of some parent shell whose sole purpose is to
run the application.

This is very posssibly a bad idea, though.  The virtual memory size of a
process hardly matters with the current price levels for disk storage;
and the physical memory size may just mean that you don't have anything
else running that needs memory.  Processes using memory isn't a bad
thing until you need that memory for something else.  Operating systems
do a good job of allocating working sets to processes based on their
actual observed locality at runtime compared to other processes.  Are
you sure you know better than your OS kernel?  You might just be getting
in the way.