7

The page I'm looking at contains :

<div id='1'> <p> text 1 <h1> text 2 </h1> text 3 <p> text 4 </p> </p> </div>

I want to get all the text in the div, except for the text that is in the <h>. (I want to get "text 1","text 3" and "text 4") There may be a few <h> elements, or none at all. And there may be a few <p> elements, even one inside the other, or none.

I thought to do this by getting all the html source of the div, and using a regex to remove the <h> elements. But selenium.get_text does not return the html, just the text (all of it!).

I know I can use selenium.get_html_source and then look for the element I need with a regex, but that looks like a waste since selenium knows how to find the element.

Does anyone have a better solution? Thanks :)

4 Answers 4

9

The following code will give you the HTML in the div element:

sel = selenium('localhost', 4444, browser, my_url)
html = sel.get_eval("this.browserbot.getCurrentWindow().document.getElementById('1').innerHTML")

then you can use BeautifulSoup to parse it and extract what you really want.

I hope it helps

3
  • sorry, I'm new in this site... You meant clicking on the v, right ?
    – Rivka
    Commented Nov 30, 2009 at 8:17
  • No problem. Thanks. I spend some times a few weeks ago on a similar problem and I am happy to know that it fixed yours too.
    – luc
    Commented Nov 30, 2009 at 8:35
  • And what about getting it directly from the WebDriver, for example you have wd = webdriver.Firefox(), and I want to get this from the wd object
    – eLRuLL
    Commented Mar 1, 2013 at 13:22
4

Use xpath. From selenium.py:

Without an explicit locator prefix, Selenium uses the following default strategies:

  • \**dom**\ , for locators starting with "document."
  • \**xpath**\ , for locators starting with "//"
  • \**identifier**\ , otherwise

In your case, you could try

selenium.get_text("//div[@id='1']/descendant::*[not(self::h1)]")

You can learn more about xpath here.

P.S. I don't know if there's good HTML documentation available for python-selenium, but I haven't found any; on the other hand, the docstrings of the selenium.py file seem to constitute comprehensive documentation. So I'd suggest looking up the source to get a better understanding of how it works.

1

What about using jQuery?

Edit:

First you have to add the required .JS files, for that go to www.jQuery.com.

Then all you need to do is call a simple jQuery selector:

alert($("div#1").html());
1
  • I don't know jQuery. Can yo give me an example? Thanks!
    – Rivka
    Commented Nov 29, 2009 at 18:08
0

The selected answer does not work in Python 3 at the time of writing. Instead use this:

from selenium import webdriver

wd = webdriver.Firefox()
wd.get(url)
return wd.execute_script('return window.document.getElementById('1').innerHTML')

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.