You have to edit sencha.cfg file which is located in your Cmd folder.
If you are on Windows - most probably here:
C:\Users\<user>\bin\Sencha\Cmd\6.1.1.76
In it, you will find the part with JVM startup arguments:
#------------------------------------------------------------------------------
# These are the JVM startup arguments. The primary things to tweak are the JVM
# heap sizes.
# java.awt.headless=true - required to make phantomjs (used by theme slicer)
# work in headless environments
# cmd.jvm.args=-Xrunjdwp:transport=dt_socket,server=y,address=8888,suspend=n -Xms128m -Xmx2048m -Djava.awt.headless=true
cmd.jvm.args=-Xms128m -Xmx1024m -Dapple.awt.UIElement=true
You can find more info about configuration files in the Sencha Cmd Docs.
EDIT: From Cmd 6.5.0 this no longer works. You will have to use _JAVA_OPTIONS
# On previous versions, this file provided a way to specify the cmd.jvm.* properties
# to control the execution of the JVM. To accommodate all possible execution scenarios
# support for these properties has been removed in favor of using the _JAVA_OPTIONS
# environment variable.