Launching MATLAB on a server without the GUI is covered thoroughly at
http://blogs.mathworks.com/desktop/2010/02/22/launching-matlab-without-the-desktop/
you should also read the user comments/discussion there
e.g. you can use
start matlab -nosplash -nodesktop -nojvm -minimize -r
"testcommand,quit"
if not using Java is too restrictive for your needs:
Regarding ‘-nodesktop’ vs. ‘-nojvm’, there is a third
(undocumented/unsupported) option: ‘-noawt’. -noawt loads Java (thus
enabling Java I/O, data structures etc.) and just prevents Java GUI
The discussion there also speaks of using the undocumented com.mathworks.mde.desk.MLDesktop.getInstance.getMainFrame.hide to hide the command-line window, but that command doesn't seem to work
BTW, instead of launching MATLAB via Windows shell command you could launch as a COM automation server:
http://www.mathworks.com/help/techdoc/matlab_external/brd0v3w.html
or via existing C API for launching MATLAB:
http://www.mathworks.com/help/techdoc/matlab_external/f29148.html
If you use this often (e.g. from a web service), it is best that you keep an instance of MATLAB in memory all the time (since Windows apps for example share code and have separate data this can spare much time by avoiding the reloading of MATLAB code into memory at every script run). That instance could be headless too (with no UI) using this command at server boot (e.g. by adding an entry at HKLM/Software/Microsoft/Windows/CurrentVersion/Run in the Windows registry using "regedit.exe"):
start matlab -nosplash -nodesktop -nojvm -minimize