24

Background info:

I have this java framework that is meant to run external scripts. To do this, I use a combination of a classloader and the system java compiler to compile .java "script" files that DO NOT exist on my project's build path. All of this works, compiler black magic and all.

The inherent complication with externally loaded code is the difficulty to debug. I have addressed this by using the remote debugging feature of the java runtime.

So, I have a debug configuration that attaches to my executable jar, which has the directory with the external java scripts on the source lookup path. This actually WORKED for a while.Actually, it never worked properly, I just had scripts accidentally on my build path. Confusingly enough I can put breakpoints in the scripts, and the debugger actually STOPS there (consistent line number, -verbose:class logging and all). Understanding how eclipse finds the source files is something that would help, though. The majority of eclipse documentation is comprised of user manuals, after all.

WHAT I SUSPECTED was that I had accidentally duplicated certain script files, and thus confused the source lookup with an out-of-sync source file. This is not the case, I have since removed the duplicated files and eclipse still is unable to find the source.

What I've tried

  • double, triple, quadruple checked the source lookup paths ensuring it includes every relevant directory
  • enabled/disabled search subfolders
  • enabled/disabled search duplicates
  • using an absolute path to the directory instead of a relative workspace path

Workaround

The only workaround here is to add the script files onto the build path of the project, which is unacceptable for me.

What I'm doing now

I am slowly crawling my way through the eclipse open source project base repository looking for the answer. Eclipse, as it turns out, is a pretty big project.

Question

Can anyone provide an accurate algorithmic representation of how the Eclipse source lookup works?

Knowing this, I could possibly figure out a way to force the Eclipse debugger to use the correct path using reflection. As far as I know, there is no technical limitation that prevents dynamically compiled code from being debugged. I know this because my breakpoints are suspending my threads as I expect them to, the source code just doesn't seem to want to load :(

Related research: It seems that this might be linked with how the class is defined with a null CodeSource location, but apparently the proper thing to do when dynamically compiling code into memory is to give the null arg... the question still stands how/why this matters to eclipse's debugger.

Update 4/22 3:30: So I pursued the CodeSource solution linked above. Now, I am seeing that my class IS being loaded from the proper file path location with the -verbose:class switch, but the source lookup is still failing. Breakpoints are still properly caught, but I am greeted with the familiar Source not found red lettering.

Updated 5/6 3:15: I pursued the javap solution discussed in Andrew's answer. Turns out, the source file attribute in my .class bytecode exactly matches a file that WOULD exist on my source lookup path. This confuses me, because this hints towards folder hierarchy having an influence on the source lookup. However, I have created "phantom" package hierarchies representing the "true" packages(as defined at the top of my .java files) and moving my source files to those folders, but the source lookup is still failing when I add those paths to my source lookup path. Any additional insight as to what additional factors play into the source lookup would be huge.

13
  • Do you get any errormessages? Maybe in the Errorlog View or Console View? Did you edit the Source Lookup in the Debug window via rightclick -> Edit Source Lookup during debugging?
    – Calon
    Commented Apr 22, 2014 at 9:30
  • @calon that's part of the trouble, Eclipse tells nothing besides "source not found"... All the paths are on the lookup path, though.
    – TTT
    Commented Apr 22, 2014 at 12:59
  • updated the question with some additional research, still haven't gotten very far though
    – TTT
    Commented Apr 22, 2014 at 13:31
  • 1) Are you sure the compiled code matches the class and package names in the source files? Tricks with the compiler might rename classes. 2) Are you sure Eclipse is running with the right classloaders? It might think that the sources are for alien classes that happen to have the same name as the classes in the remote debugging session. Commented Apr 22, 2014 at 13:51
  • @LorenzoGatti I am pretty confident that the class name is matching, I pursued the linked CodeSource possible solution and now I am seeing that my class file(qualified name and all) is stated as being loaded from the location that I expect it to be loaded. This was accomplished by editing the my logic inside of my overloaded getClassLoader() method and getJavaFileForOutput() in my extended ForwardingJavaFileManager<JavaFileManager> to support feeding the returned classLoader the proper CodeSource URL location.
    – TTT
    Commented Apr 22, 2014 at 19:23

2 Answers 2

8
+100

I have a bit of experience in this area, having done some work on debugging dynamically compiled groovy scripts though the JDT. I never got this to work perfectly, and I think it is mostly a limitation of the JDT, which was never designed to handle dynamically compiled code.

TLDR: My guess is that your dynamically compiled scripts have an incorrect source file attribute in the byte code. This attribute is set in the class file by the compiler. See https://en.wikipedia.org/wiki/Java_class_file

Your confusion, I think, is that the debugger correctly stops at breakpoints you set in the scripts, but the IDE cannot load the source. This is confusing of course, but there is a good explanation for this.

Breakpoints are actually handled by the VM and the VM keeps track of them through a fully qualified name and a line number. This allows breakpoints to be hit regardless of which classloader loads the class file, but it can lead to some confusion if multiple class files are loaded through different classloaders with the same qualified name, but different source code. This algorithm for determining when to stop the VM has nothing to do with actually looking for source code when the VM stops.

Looking for source code is handled by the IDE. Since even in the statically compiled world, the source file name may not match with the class name (inner classes, anonymous classes, etc). The class name cannot be used to look up the source file.

Here is a simplification of what the IDE does when it stops at a breakpoint:

  1. Find the class file at that breakpoint
  2. Get the source attribute
  3. Find a source file in the source lookup path that matches the name of the source attribute
  4. Use some heuristics if multiple source files of same name are found (I think this would be the ranking in the source lookup tab)
  5. Return the most appropriate source file.

(Caveat, I think that the source attribute is only the simple name of the source file (ie- no directory), so I think the IDE converts the package name to a directory structure as part of the lookup, but I might be wrong about that).

So, the lookup will fail if your dynamically compiled script does not have a proper source attribute. You can check this theory by looking at the byte code. You will have to somehow compile a script and save the bits to disk. Then you can run javap -v myScript on it. I would bet that this is the problem. I have seen this happen before in other dynamically compiled languages.

9
  • Holy crap, with this line : "I think the IDE converts the package name to a directory structure as part of the lookup", I think you've struck gold!! AFK right now, but the funny thing about scripts is that they often do not reside in the same exact folder structure as your package name. Obviously this is not actually an option for me, but confirming this would be awesome info. I will introduce the concept of saving the compiled bytes to file... I just have one problem : how?
    – TTT
    Commented Apr 28, 2014 at 5:03
  • We might be able to reach a conclusion as to how to do this here. Finally.
    – TTT
    Commented Apr 28, 2014 at 5:13
  • Just kidding, I don't need to know how anymore. I'm actually really stupid. Lets see where this gets me... Where would the source attribute be?
    – TTT
    Commented Apr 28, 2014 at 5:22
  • javap -v should show you all the attributes. There are many more decompilers that are prettier that you can use. Commented Apr 29, 2014 at 4:41
  • I will bounty you and accept this answer when I get to trying this out
    – TTT
    Commented Apr 29, 2014 at 6:10
0

I've had similar but less complex problem. It was caused by using two different compilers. You wrote that you use system java compiler in your case, make sure your Eclipse uses same JDK and system compiler include debug info in compiled classes by using -g:vars parameters.

4
  • 1
    This does not really answer the question. If you have a different question, you can ask it by clicking Ask Question. You can also add a bounty to draw more attention to this question once you have enough reputation. - From Review
    – aschipfl
    Commented Mar 19, 2016 at 1:33
  • I don't have different question. I was also spending to much time on checking source lookup when problem was about compiler and it's parameters. I didn't see in above text that rpg711 rise the issue of compiler and it's debug parameters. Which i didn't know exists and I am sure they not obvious for everyone. Commented Mar 19, 2016 at 9:37
  • Thanks for the input but this question was asked two years ago, it has already been resolved, please keep things relevant on SO :-)
    – TTT
    Commented Mar 23, 2016 at 20:16
  • You don't posted resolution for this problem. If I would see my ansfer few days back I would save lot off time. Regards. Commented Mar 25, 2016 at 9:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.