Questions tagged [mechanize]

Mechanize is a library for automated web browsing originally developed for Perl. There are now also Python and Ruby implementations.

mechanize
Filter by
Sorted by
Tagged with
2 votes
1 answer
8k views

Getting error "getaddrinfo: No such host is known. (Socke tError)" with mechanize gem

I tried the below code: require 'mechanize' agent = Mechanize.new{|a| a.ssl_version, a.verify_mode = 'SSLv3', OpenSSL::SSL::VERIFY_NONE} page = agent.get "https://gegsltraining.aravo.com/" page=...
1 vote
1 answer
140 views

Confusion with mechanize installation

I want to install Mechanize, but it looks like there are two gems. Should I need to install both "mechanize-downloader" and "mechanize", or only "mechanize"? gem install mechanize-downloader gem ...
2 votes
2 answers
2k views

Can I use Mechanize to download files with .docx and .xlsx,.txt extensions?

I browsed the documentation Mechanize. The below is for .pdf only. require 'mechanize' agent = Mechanize.new agent.pluggable_parser.pdf = Mechanize::FileSaver agent.get 'http://example.com/foo.pdf' ...
0 votes
1 answer
1k views

How to click the Submit button which has no id and name using mechanize?

Find the below HTML code: <input type="hidden" name="TARGET" value="www.example.com"/> <noscript><input type="submit" value="Resume"/></noscript> </form> How to click ...
0 votes
1 answer
57 views

Getting the error "nil:NilClass (No MethodError)"

Please find the part of a code: button = Login_form.button_with(:name => 'Submit') loggedin_page = Login_form.submit(button) bizz_hierarchy_form=loggedin_page.form('login_showSplashForm') button =...
0 votes
1 answer
188 views

Can the description mentioned code using Selenium-driver be implemented in mechanize?

Could anyone help me by giving small tips mentioning how the below can be written in Mechanize? I am totally new to the Gem Mechanize. require "rubygems" require "selenium-webdriver" driver = ...
0 votes
2 answers
52 views

how would I search the `/aems/dic/list` from a list of anchors?

I have the below code which is a part of an html: <td><a href="http://youtube.com">YouTube</a></td> <td><a data-category="news" href=http://kathack.com/party/aems/dic/...
0 votes
1 answer
212 views

python mechanize blank download or how to do it in casperjs

I am downloading information for a research project from a site that uses ajax to load URLs and does not allow serial downloading. I am dumping the urls from casperjs into a file I read and use ...
1 vote
2 answers
3k views

How to set Referer using python mechanize?

I'm writing python program to logging into a website programmatically. Now I'm having problem with handling cookies. Below are my code: import mechanize cj = mechanize.CookieJar() browser = mechanize....
0 votes
1 answer
1k views

How to get page description with ruby mechanize?

How do I get page descriptions with Mechanize? The Pismo gem has this functionality, but it's worse than the Mechanize gem. Mechanize finds image tags, for example. I didn't find in the Mechanize ...
2 votes
2 answers
2k views

Headless browser for Python with multithreading, javascript, and Windows support

I've tried the following with no avail Mechanize/Twill. No Javascript support Selenium. Not headless Ghost. No Windows support Anyone got any suggestions?
0 votes
2 answers
3k views

How to select radio buttons and submit a form

This is the HTML: <form method="post" action="./xxxxx" name="form1"> <input type="hidden" name="requestKey" value="52032a76cf53340b07052fd143feddc4a9d8f18130c80ec86003ab56e0d91b36"/> ...
4 votes
3 answers
2k views

How to login a website using python module "mechanize" with a cookies.txt exported by the chrome extension "cookies.txt export"?

I have looked through similar questions in stackoverflow yet seemingly no answers measures up. Now I have a cookies.txt in hands which is exported by a chrome extension named "cookies.txt". I can ...
0 votes
2 answers
64 views

Ruby Convert Date

I have a page that is returning this as a date new Date(1357106400000) (which I believe is Javascript). How do I convert that using Ruby. I've tried: Date.new(1357106400000) DateTime.new(...
0 votes
1 answer
245 views

File Upload with Perl Mechanize Firefox

I'm using WWW::Mechanize::Firefox and having trouble with upload forms. Can anyone tell me how I would use that module to submit a file to a form like this? <form action="http://www.cs.tut.fi/cgi-...
0 votes
1 answer
885 views

Mechanize send a POST to populate data on a page

I have a site that sends a POST to populate some data on a page. I usually look at the POST in Charles Proxy and pass the parameters like so: bot.post('https://www.google.com?', { "parameter" ...
1 vote
3 answers
1k views

Click an image which has a specific name

How do I click an image like below using Python mechanize? <a href="..."><img name="next" id="next" src="..."></a> I know the name and id of the image I want to click to. I need to ...
0 votes
1 answer
583 views

Error with Mechanize and Beautifulsoup httplib.InvalidURL: nonnumeric port: '' (Python)

I'm going through a list of URLs and opening them with my script, using Mechanize/BeautifulSoup. However I am getting this error: File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/...
2 votes
2 answers
256 views

Unexplained Inconsistency when Downloading an XLS file with Ruby Mechanize after redirect

I have a script that visits fcc.gov, then clicks a link which triggers a download: require "mechanize" docket_number = "12-268" #"96-128" url = "http://apps.fcc.gov/ecfs/comment_search/execute?...
0 votes
3 answers
166 views

Web Scraping Returning Empty String

I wrote some code to return and display the five most recent posts on this website. Yet when I run the code using a for loop, an empty string is returned. The code is below: require 'rubygems' ...
3 votes
1 answer
3k views

How do I scrape URLs from a JavaScript 'onclick window.open' event?

I am trying to scrape URLs from a page that uses JavaScript. Instead of having links on the page, they created onClick events for a number of table rows, whereby, when you click the row, it takes you ...
0 votes
1 answer
187 views

Perl, using Mechanize-Firefox, how can I expand a frame whilst remaining within the larger frameset.

I'm scraping a website that uses frames extensively. Within each frame is a piece of javascript like so: function CheckUniqName(uniq) { if (top.name != uniq) { top.window.location.assign("<a ...
0 votes
1 answer
204 views

Download an image from a website into a CouchDB attachment through Rails without local storage?

I'm trying to scrape some websites and store the images as CouchDB attachments. In order to expedite the process I don't want to store the images locally at all. The relevant gems I'm already using ...
0 votes
1 answer
109 views

Mechanize and invisible search form

I'm trying to perform search on some website using Mechanize but I can't submit a search form because mechanize does not see any forms. page.form returns nil and page = agent.get returns just {forms}&...
3 votes
2 answers
995 views

Ruby Mechanize Outlook Web Access

I am trying to use ruby mechanize to access specific emails from my outlook web access account. I am using the following code. require 'mechanize' require 'logger' a = Mechanize.new ...
2 votes
1 answer
317 views

Ruby Mechanize - can't get specific selectlist

I know this is working on other form/selectlist I've tried. This particular one doesn't appear to be ajax. Can anyone shed light on this? # encoding: UTF-8 require 'rubygems' require 'mechanize' ...
0 votes
1 answer
819 views

Filling and submitting textarea with Mechanize in Python

Okay, so I'm using Mechanize to log into Google groups and manually add a member to the group. I've been able to successfully log in and navigate to the appropriate page. On this page there is a ...
0 votes
1 answer
156 views

Problems scraping pages with JavaScript function in python

I'm stuck with a python problem, look. I have to scrap a page that has JS functions, but that's not the real problem, the real problem is that the information I need is provided by the function.. So ...
2 votes
2 answers
2k views

Python Mechanize to check if a server is available

I'm trying to write a script which will read a file containing some urls and then open a browser instance using mechanize module. I'm just wondering how I can do so if some url does not exist or if ...
4 votes
2 answers
4k views

Ignore Iconv::IllegalSequence while using Ruby WWW::Mechanize

I've encountered the Iconv::IllegalSequence error on some web pages when using mechanize lib. Is there a way to make mechanize just omit ill encoded characters and return the "cut" page? I'm aware of ...
1 vote
1 answer
112 views

Server doesn't see the cookies

What're the reasons that the server doesn't actually recognize the cookie I set using WWW::Mechanize in the code below? #!/usr/bin/perl use strict; use warnings; use HTTP::Cookies; use WWW::...
4 votes
3 answers
417 views

Autofill if name = hash

I am trying to autofill a text box(multiple boxes) in a form using mechanize in python, but the name of the box(es) is a hash, so I can't automate the input like br.form['name'] = 'blah' since the ...
1 vote
0 answers
181 views

Net::SSL is breaking my Sys::SigAction timeout_call function

After using Sys::SigAction timeout_call for timeouts from the answer in my previous question (note: this time im using WWW::Mechanize instead of WWW::Mechanize::Timed), my 60 second timeout alarms ...
1 vote
2 answers
201 views

Trying to subclass mechanize's Browser and create a login method

so that this way all I have to do is type browser = MyBrowser() browser.login() to get my python scripts to log in in the future. Here's what I have so far: import mechanize class MyBrowser(...
1 vote
1 answer
89 views

How to extract the URL from the last column of of a table, where the row contains certain words, using Mechanize

I'm trying to automate some tedious web tasks for an internal ticketing system I maintain. There is a page, with an automatically generated table full of mail server configurations. Each row is for a ...
1 vote
1 answer
3k views

Using mechanize to check for div with similar but different names

Currently I'm doing the following: if( firstTemp == true ) total = doc.xpath("//div[@class='pricing condense']").text else total = doc.xpath("//div[@class='pricing ']").text end I'm ...
0 votes
0 answers
85 views

Rails not adding new models to DB via rake script, can't figure out why

The Rake Script desc "Update departments and classes listing" task :update_classes => :environment do require 'rubygems' require 'mechanize' require 'nokogiri' def getSubject(agent, subject) #...
11 votes
4 answers
6k views

Force python mechanize/urllib2 to only use A requests?

Here is a related question but I could not figure out how to apply the answer to mechanize/urllib2: how to force python httplib library to use only A requests Basically, given this simple code: #!/...
0 votes
1 answer
166 views

Mechanize: Only following links with certain element in URL

Learning Mechanize: I am wanting to crawl through a news site and only follow the links that have a ?sid in the link.. I cannot seem to find much on Mechanize documentation. import mechanize br = ...
0 votes
0 answers
72 views

installing mechanize on VPS

root@cowboomer:/home/ryan/public_html# gem install mechanize -V GET http://rubygems.org/latest_specs.4.8.gz 302 Moved Temporarily GET http://production.s3.rubygems.org/latest_specs.4.8.gz 304 Not ...
0 votes
0 answers
201 views

What exactly does 'window.location.replace' do / how to replicate it in mechanize?

I'm interfacing with a site that implements delaying a page-load with client-side JavaScript. Basically, a form is submitted on PageA.asp, and instead of the data going to PageB.asp, it goes to PageC....
0 votes
1 answer
941 views

Mechanize and Python not handling cookies properly

I have a Python script using mechanize browser which logs into a self hosted Wordpress blog, navigates to a different page after the automatic redirect to the dashboard to automate several builtin ...
0 votes
3 answers
1k views

match table row id's with a common prefix

This might be merely a syntax question. I am unclear how to match only table rows whose id begins with rowId_ agent = Mechanize.new pageC1 = agent.get("/customStrategyScreener!list.action") The ...
1 vote
2 answers
4k views

How to submit javascript form with Mechanize?

I have mechanize script that is done logging in. After log in. The page shows a redirect first before going into the main logged in page. Executing redirect() brings me back to the login page. Why? ...
1 vote
1 answer
331 views

Py Mechanize submit more data using the auth'ed user

I'm trying to use Mechanize to login a website. Thanks to del for the corrections. import re import mechanize login_url = 'login.aspx' def login(id, username, password): br = mechanize.Browser()...
0 votes
0 answers
1k views

Why would mechanize .open() / .read() / .close() return None/have socket internals be None?

I'm getting occasional AttributeErrors with code of the following sort. I set a mechanize instance up with: self.mech = mechanize.Browser(factory=mechanize.RobustFactory()) self.cj = mechanize....
0 votes
1 answer
3k views

Logging into the Steam website

I'm looking to make a script that'll log into the SteamPowered.com website, and then navigate to the account page. I wrote a script that takes the sales from that page and just adds it up. The issue ...
1 vote
0 answers
152 views

Is is possible to POST to a URL with Mechanize without opening it first?

I would like POST to a URL with some form fields filled in with Mechanize in Python. Most code examples like this one: import mechanize url = "http://duckduckgo.com/html" br = mechanize.Browser() br....
3 votes
1 answer
464 views

Get around a 404 with mechanize

I'm creating a Python script that would read a file of URLs, but I know not all of them will work. I'm trying to figure out how to get around this and make it read the next line of the file, instead ...
0 votes
2 answers
2k views

(Rails) Can't get Mechanize to correctly read a web xml file

I have to read xml files that are accessibles through http with authentication. That's why I use mechanize. My problem is that I can't get mechanize to recognize these XML files so I can use .find ...

1
36 37
38
39 40
51