Questions tagged [mechanize]
Mechanize is a library for automated web browsing originally developed for Perl, there are now also Python and Ruby implementations.
2,526
questions
0
votes
0
answers
12
views
Mechanize cannot find input tag outside form
I'm trying to use Mechanize with python to access a webpage (a price searcher), then search for a specific item on "search bar", but the problem is that theinput tag is not inside a form tag....
0
votes
1
answer
39
views
Automating Web form Filling using Selenium or Mechanize Python with Pandas Dataframe
Automating Web form Filling using Selenium or Mechanize Python with Pandas Dataframe.
I am looping through a three columns of pandas dataframe to auto-complete the form for me and submit. Below is the ...
0
votes
0
answers
24
views
Can't find form input inside div (mechanize)
I am trying to log into Facebook (https://m.facebook.com/login.php), but the email and password <input>'s are inside divs, so I can't do it. When using for x in br.forms(): print(x) (br = ...
0
votes
0
answers
26
views
Mechanize how to fill a rich text area
i'm trying to fill a form with python and mechanize but a mandatory rich text block is mandatory but doesn't appear in the form with mechanize. My code is the following:
import mechanize
#Reach the ...
0
votes
0
answers
25
views
Error in using mechanize to download file from website(control not found error)
import urllib
import csv
import numpy as np
import codecs
from bs4 import BeautifulSoup
from itertools import *
import mechanize
import webbrowser
import re
from urllib.error import HTTPError
import ...
1
vote
2
answers
45
views
Why is mechanize not installing properly via pip on RPi? (python 3.9)
I can't get to successfully install the package 'mechanize' on a Raspberry Pi (so, ARM chip) with Debian Bullseye, python 3.9 in a virtualenv.
When I look in the virtualenv's sitepackages folder, ...
0
votes
1
answer
23
views
Unable to click download button with web scraper
for whatever reason mechanize is unable to click this export button to automate downloading a public government csv file, i'm trying to automate downloading a fishing report, does anyone have any ...
0
votes
0
answers
11
views
Python mechanize redirection destination leads to new redirects
I'm new to mechanize and now need to scrape a webpage that requires authentication to view the content. I am able to log in with mechanize separately (with browser.open(login link)), but the login ...
0
votes
0
answers
15
views
How do I Modify a Password List to Work Properly With mechanize.Browser - Python
The issue I am having with my script is this, only the last password in my world-list is been check by the script.
This is my script bellow.
from gettext import find
from operator import truediv
...
1
vote
0
answers
25
views
How can I open text file with mechanize browser in Python - Password List
The issue I am having with my code is, only the last password in the text file is been tried by the script, but if I put the correct password in the last line it login successful
again I want if the ...
0
votes
0
answers
29
views
How can I get a contain part of an HTML element by ID or NAME - Python - browser or BeautifulSoup
from gettext import find
import mechanize
browser = mechanize.Browser()
browser.set_handle_robots(False)
cookies = mechanize.CookieJar()
browser.set_cookiejar(cookies)
browser.addheaders = [('...
0
votes
1
answer
32
views
Structure of HTML Form Data
I know this is stupid question but how does payload of this form data be formatted. I can't make request more than one time so i can't see real request send.
I have tried same on local html file and ...
1
vote
0
answers
8
views
python mechanize submit response.read() doesn't return correct page?
all! I am trying to do some data processing on a publicly accessible database (property tax records, I know, not very glamorous!). The code runs perfectly, but it doesn't return the right page. When I ...
0
votes
0
answers
125
views
How to bypass Google recaptcha in a form using Mechanize
I need to fill a form on a webpage and bypass its captcha.
using this library I'm able to solve the captcha and get g-recaptcha-response stored in a variable. now the question is, how to pass the ...
0
votes
0
answers
11
views
Why does Mechanize throw a NoMethodError for a variable @mech that I never instantiated?
Trying to scrape some data from an internal within a rails application and I am trying to use Mechanize. The simple test ruby app worked fine, but now when implemented in rails (currently sits in a ...
0
votes
0
answers
62
views
Login multiple times when web scraping
So I am trying to Scrape all SubPages of a Page with a Login page.
I have the Site https://mainsite.org where I have to log in first before getting all the links on the Site. All the Links are passed ...
0
votes
0
answers
19
views
Python Mechanize-Plugin omits content of <b>-brackets in response
I am trying to extract text from a server-generated results page to a former query. The website is built in PHP, so I'm using the mechanize browser to interact with the page from Python.
Problem: ...
0
votes
0
answers
27
views
Get Dynamiclly loaded Source Code from python using mechanize and bs4
I want to get source code of page that loads from javascript actually that page is linkedin profile page and i want to get job and education details.
I'm not using selenium i don't want browser window ...
0
votes
0
answers
48
views
Logging in to Spiegel.de with Pyhon and Mechanize
Im attempting to login to Spiegel.de with my personal Logindata and retrieve a website behind the paywall. I believe the login works as the forms are correct but any retrieved website throws please ...
0
votes
0
answers
7
views
Framework/workflow for an efficient way to fill a form with mechanize
I'm trying to fill a few millions of forms with Mechanize.
Before, I try to do it, I would like to understand what are the general steps to make it work, and not go it some preventable errors.
For ...
0
votes
0
answers
19
views
Pythan mechanize library
import mechanize
br = mechanize.Broweser()
br.open("url")
br.select_form(nr=0)
Problem:
It is showing an error formnotfounderror
: no form matching nr 0
If a want for Instagram what ...
0
votes
0
answers
35
views
Ruby Mechanize: captcha image returns black square with red cross
I'm trying to get the base64 metadata from a captcha image inside HTML img tag, like this one: CaptchaImage.
But instead, I'm always getting a black square with a red cross like this: WrongImage.
Code ...
0
votes
0
answers
31
views
Use Mechanize to input text to Shakespeare translator, and get the translated text
I am attempting to make a discord bot where the bot responds to a message with the shakespearian english translated version using this website: https://lingojam.com/EnglishtoShakespearean
I believe I ...
0
votes
1
answer
34
views
How to edit page code with python mechanize?
I have some code on page like this:
<div class="tabs1_cnt" id="pupil_tabs_content_2189609">
<div class="pupil_tabs_content_item tabs1_cb" id="...
0
votes
1
answer
110
views
Ruby/Mechanize workaround for Let’s Encrypt Root Certificate Expiration and OpenSSL 1.0.2
With the expiration of Let's Encrypt's root certificate DST Root CA X3, my ruby scripts using OpenSSL to connect to servers that use this certificate are returning the error:
/usr/local/Cellar/ruby/2....
1
vote
1
answer
125
views
Mechanize::ResponseCodeError (404 => Net::HTTPNotFound unhandled response):
Trying to scrape images from https://en.wikipedia.org/ website using mechanize gem. I am getting Mechanize::ResponseCodeError (404 => Net::HTTPNotFound for https://upload.wikimedia.org/wikipedia/...
1
vote
2
answers
82
views
How to scrape icon link of image using mechanize gem
I have a url where I have to scrape all images using mechanize gem, but some image url's are in rel=icon.
I have to get the image from this url:
<link rel="icon" href="https://...
0
votes
0
answers
208
views
How to calculate content-size of a svg+xml image
I am trying to calculate image size after scraping. While scraping I am getting some svg image files. I don't have an idea to calculate svg image size.
This is the code I used to calculate content ...
0
votes
0
answers
20
views
Mechanize with python submit giving error page
I wanted to scrape data from this website -> https://inputsurvey.dacnet.nic.in/districtsummary_sizeclass.aspx There are a few dropdown menus and submit button.
I want to do it iteratively to get ...
0
votes
0
answers
42
views
How to submit, wait and get response from mechanize?
I want to submit a form and get response from a website.
import mechanize
br = mechanize.Browser()
br.open("http://opig.stats.ox.ac.uk/webapps/newsabdab/sabpred/humab")
for f in br.forms():
...
0
votes
1
answer
55
views
Using ruby mechanize on forms not in <form> tag?
So I'm trying to web scrape a site which has what I would consider a form - several inputs to specialize search such as <select>s and text inputs - that is not within <form> tags.
Rather, ...
0
votes
0
answers
39
views
Mechanize not finding URLs on webpage
I'm trying to use mechanize to scrape and download some files. When I get to the final page mechanize is only finding one URL on the page and it does not seem to even be a valid URL. When visiting the ...
1
vote
0
answers
131
views
Python: fastest way to write/download online images to disk?
How I can speed up the speed at which my program writes an online image to disk? Right now ~150KB takes ~8 seconds to download, 3 seconds for opening the image online and 5 seconds to write it.
...
2
votes
1
answer
89
views
save directly to file and get filename using WWW::Mechanize
I would normally save to a file using this:
$mech->save_content($mech->response->filename);
But due to some big files which cause "Out of memory" I have to use this instead:
$mech-&...
3
votes
1
answer
45
views
Why is youtube not giving me video links and only giving me their resource pages?
For some reason youtube only gives me their resource pages instead of giving me their video links
which is what I am looking for.
code:
import mechanize
import urllib
import urllib.parse as urlparse
...
0
votes
0
answers
58
views
Mechanize instance can't .open a specific website
I'm trying to use the mechanize .open method on a URL so I can eventually login and scrape this website. I've tried using this code on other websites and they work, but for some reason I'm getting the ...
1
vote
0
answers
65
views
How to fill a form input field using its ID rather than the form name with mechanize?
I am trying to automate geocoding with python. There is a simple website that does this, but I don't want my user to manually visit the website. I want to automatically take his adress and convert it ...
0
votes
0
answers
46
views
Getting 403 error when running Python mechanize within docker container
I've built a Python script with the module mechanize which scrapes data from a website. When I run the code on my python environment on my localhost, everything works fine. Now i've started to ...
0
votes
1
answer
86
views
python web scraping issues with mechanize
I am trying to scrape web results from the website: https://promedmail.org/promed-posts/
I have followed beutifulsoup. mechanical soup and mechanize so far unable to scrape the search results.
import ...
0
votes
1
answer
64
views
Can't login using mechanize from python. What am I doing wrong?
I'm trying to use mechanize in python to login to this site: https://login.haaretz.co.il/
On the surface, it looks like a 2-phase login process, same as google, but following receipts for google login ...
1
vote
1
answer
99
views
Sending custom cookies (and viewing the ones set naturally) with Perl's WWW::Mechanize
I'm trying to scrape a particularly troublesome website. Though all the parameters match and the referrer matches, I see different results when perl runs it than when I watch dev tools.
When I do a ...
1
vote
0
answers
169
views
Node has gone away in WWW::Mechanize::Chrome
WWW::Mechanize::Chrome loses somehow the node, when calling a different method like ->find_all_links()
No node with given id found
-32000 at /usr/local/share/perl/5.20.2/Chrome/DevToolsProtocol/...
1
vote
1
answer
55
views
Is it possible to get a checkbox with Mechanize other than name attribute?
I have been trying scraping with Ruby and Mechanize.
The websites I saw say that the way to get a checkbox and check it is below.
form.checkbox_with(:name => 'name').check
However, the checkbox ...
1
vote
1
answer
577
views
How to edit a facebook post in Python?
I have a post on my fb page which I need to update several times a day with data elaborated in a python script. I tried using Selenium, but it gets often stuck when saving the post hence the script ...
1
vote
1
answer
24
views
How to use mechanize package inside a function to log into website?
I am using mechanize to log into a website and scrape it with beautifulsoap. While I got it working without using functions, I don't know how to put the login-functionality into a function and then ...
5
votes
1
answer
834
views
How to fill and submit a form using python
I am filling a form of a web page with the help of mechanize module but I am getting error when I run my code.
I just want to fill the form and submit it successfully.
My attempt :
code snippet from ...
-2
votes
1
answer
35
views
How to upload file using mechanize python?
I need to upload a file(txt) in a form like this:
the structure of the add button is as follows:
after upload I have to send the file with the send button:
the structure of the send button is as ...
1
vote
3
answers
233
views
Perl: Getting UTF-8 file with Mechanize content_file
Creating a perl script (Strawberry Perl v5.32.0 on W10) to download my Google Calendar. Google provides a 'private' (no login required) url to a 'basic.ics' file.
When opening this url in my browser (...
1
vote
2
answers
7k
views
Python click button on web page without selenium
So there is a button on a webpage that downloads a csv file after clicking it. In the past I have used selenium to do this, but given the current application of this script being ran on databricks I'd ...
-1
votes
1
answer
350
views
How to fill JavaScript form using Python?
I want to use Python to fill this form.
I tried using Mechanize but this is a Microsoft Form which uses JavaScript and has no form tag and no GET/POST URL. Maybe BeautifulSoup/Selenium can do this, ...