0

I have an NPAPI plugin which makes use of SetWindowsHookEx under the hood to do some screen scraping. This plugin seems to run successfully on most machines I've tested. However, on one machine (32-bit Windows 7), when I'm running the plugin under Chrome (26.0.1410.43m), the call to SetWindowsHookEx crashes the process in which the plugin is hosted. It's not that SetWindowsHookEx returns an error - it simply crashes the process. And again, this only happens on one machine: it works fine on all the others we've tested.

But here's the weird thing. As you may know, Chrome has several ways of locating plugins. Normally, our setup.exe creates some registry entries that point to the plugin, like so (from the NSIS installer):

WriteRegStr HKLM "Software\MozillaPlugins\@alanta.com/WinVncCtl\" "Path" "$INSTDIR\npWinVnc.dll"
WriteRegStr HKLM "Software\MozillaPlugins\@alanta.com/WinVncCtl\" "ProductName" "Alanta Remote Desktop Server"
WriteRegStr HKLM "Software\MozillaPlugins\@alanta.com/WinVncCtl\MimeTypes\application/x-alanta-vnc" "Description" "Alanta's VNC Server NPAPI Plugin"

When Chrome loads the plugin from that location, it crashes on the SetWindowsHookEx call.

However, if I uninstall the plugin, then copy exactly the same DLL to %ProgramFiles%\Mozilla Firefox\Plugins\, then Chrome loads and runs the plugin just fine, and the call to SetWindowsHookEx() succeeds.

For what it's worth, here's the actual (pretty bog-standard) call:

// Add the CallWnd hook
hCallWndHook = SetWindowsHookEx(
                WH_CALLWNDPROC,                 // Hook in before msg reaches app
                (HOOKPROC) CallWndProc,         // Hook procedure
                hInstance,                      // This DLL instance
                0L                              // Hook in to all apps
               //   GetCurrentThreadId()            // DEBUG : HOOK ONLY WinVNC
                );

The debugger indicates that all the parameters are pointing to legitimate, valid things.

Any suggestions for troubleshooting this? Anybody aware of any weirdnesses in SetWindowsHookEx that might show up in this scenario?

EDIT: The CallWndProc looks like this:

LRESULT CALLBACK CallWndProc(int nCode, WPARAM wParam, LPARAM lParam)
{
  // Do we have to handle this message?
  if (nCode == HC_ACTION)
  {
      // Process the hook if the WinVNC thread ID is valid
      if (vnc_thread_id)
      {
          CWPSTRUCT *cwpStruct = (CWPSTRUCT *) lParam;
          HookHandle(cwpStruct->message, cwpStruct->hwnd, cwpStruct->wParam, cwpStruct->lParam);
      }
  }

   // Call the next handler in the chain
   return CallNextHookEx (hCallWndHook, nCode, wParam, lParam);
}

Breakpoints in the CallWndProc don't seem to get hit, so I suspect that the crash is happening during the setting of the hook rather than during its later processing.

2
  • You should not need to cast the pointer to your callback function to HOOKPROC. It should have that type already. If it doesn't, the compiler is trying to warn you of a potential problem. Don't just shut it up with a cast. Apr 2, 2013 at 0:07
  • @CodyGray - Good suggestion. I'm not sure why the cast was in there. The CallWndProc seems to have the correct signature, and everything compiles just fine without the cast. I added the code for the CallWndProc to my question.
    – Ken Smith
    Apr 2, 2013 at 0:10

2 Answers 2

0

If the same DLL behaves differently when placed in a different directory, it suggests this is changing the order in which plugins load.

(The directory a DLL loads from does not affect the search path used to locate and load its dependencies)

I would certainly take a look at other plugins being loaded on the same system and see if their presence is triggering (or perhaps even responsible for) your crash.

After that, I suggest uninstalling and reinstalling Chrome, possibly in a new directory, in case one of the Chrome binaries is corrupted or subverted.

1
  • Thanks for the suggestions. I tried uninstalling and reinstalling Chrome, to no effect, except that now, it crashes when I load it from either location. There are other plugins installed, of course, but I don't think that any of them are getting loaded, because the I have the --plugin-startup-dialog option set, and it's only showing a dialog box for this particular plugin. I used the SysInternals Process Monitor to check file activity, and saw that the only DLLs that were getting loaded were the ones I was expecting.
    – Ken Smith
    Apr 2, 2013 at 3:18
0

Turns out that it was an aggressive security program. Apparently its heuristics allowed some DLLs through while not allowing others. It was killing the hosting process as soon as it saw the call to SetWindowsHookEx(). Replacing it with something reasonable fixed the problem.

2
  • Interesting, glad to see that you figured out the cause. I was wondering when I read your question why you need to install a global hook in order to do screen scraping. Installing a global hook should activate a security monitor. At the very least, it's questionable behavior for a web browser plug-in. Apr 2, 2013 at 4:31
  • Good questions all. I didn't write this part of the code - I've just adapted the UltraVNC server for our own uses - so I'm not sure about the other ways to do this.
    – Ken Smith
    Apr 2, 2013 at 5:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.