The new "Prefer Tabs" system setting in Mac OS X 11 (Big Sur) causes issues with Java applications. If the setting is set to "Always", or to "In full screen" and the application is running in full screen, JDialogs open as tabs and become unresponsive.
There is a way to set the "Prefer Tabs" setting on a per-application basis. For example, running
defaults write net.java.openjdk.cmd AppleWindowTabbingMode manual
in Terminal makes sure that NetBeans (and, seemingly, any application running on openjdk) works correctly.
Our application still supports Webstart on Mac. The above setting does not affect Webstart, so the problem still occurs there. I am not sure where the "net.java.openjdk.cmd" in the above command comes from, and where I can find the equivalent value for Webstart. It doesn't appear to be in Info.plist. Does anyone know what the value should be for Webstart (build 1.8.0_271-b09)? In general, how can I find the appropriate value for a specific application?
[Edit March 25 2021] In an attempt to minimize burden on our users, my approach is to read the current setting using "defaults read X AppleWindowTabbingMode" from within our software. If it isn't set to manual, the software offers to run "defaults write X AppleWindowTabbingMode manual" and suggests that the user needs to restart the software afterwards.
In an attempt to get the bundle ID programmatically, I tried:
URL u = Main.class.getProtectionDomain().getCodeSource().getLocation().toURI().toURL();
URL canonicalHome = new URL(u.toString() + "jnlp");
appID = "com.oracle.jnlp-" + canonicalHome.hashCode();
but this doesn't seem to get me the right value. As far as I can tell, webstart actually uses something called JavaAppletPlugin which uses a 'java' executable, rather than 'javaws'. Any thoughts on how to programmatically get the right value?