Questions tagged [html-content-extraction]
Techniques for predicting/detecting certain article text and extracting it from a particular document.
210
questions
0
votes
0
answers
12
views
I am getting an error when trying to extract content using justext
I am trying to extract content from a url using justext.
My code is as follows:
import requests
import justext
url = 'https://yoursoccerhome.com/what-is-a-cap-in-soccer-the-meaning-and-history-of-the-...
0
votes
1
answer
144
views
Extract some information in a pdf embedded in a web page using python and requests
I am trying to extract some information in a pdf embedded in a web page using python and requests, And this is exactly the sentence I want to reach « Sciences de la vie et de l’environnement ».
image
...
1
vote
1
answer
275
views
.NET 5 HttpClient cannot GET html content page - http 500
I'm trying to use HttpClient to get html content of a page.
to try the method I tested with the google URL, and it's working, I receive the content of my html page.
but with url I want, impossible to ...
0
votes
1
answer
92
views
how to get href and text content by html Content
I am want to get content and url including all other td data.
my code:
$context = stream_context_create(
array(
"http" => array(
"header" => "...
-1
votes
2
answers
54
views
Lstrip and Rstrip won't work, need help removing text from an output in Python 3
The output is part of a list. When I try to figure out the output's type using type() it returns : <class 'bs4.element.Tag'>.
I am trying to remove everything to the left of "href" and ...
2
votes
0
answers
662
views
Extracting Headings/Chapters and related paragraphs separately from PDF file in Python 3.7
My task is to fetch chapter-wise content from pdf file separately so that i can store into database.
So far, i tried regex and tried to split but that only gives me chapter number but didn't help me ...
0
votes
1
answer
303
views
Rvest scraping webpage content returned from html_text()
I am attempting to scrape (dynamic?) content from a webpage using the rvest package. I understand that dynamic content should require the use of tools such as Selenium or PhantomJS.
However my ...
0
votes
3
answers
351
views
How to get the Email/Text entered into a field when there is no "value" attribute or text in the code snippet for Java Selenium
I want to Get the email id entered "arjun_123@gmail.com" using java code.
I tried the below codes, but it always returned null.
What I really need is, I want to check whether email is successfully ...
0
votes
0
answers
376
views
How to extract date from multiple transaction receipts in python having no pattern
I have multiple transaction receipts and am trying to extract the invoice amount from each of these receipts. The problem is that the ocr I am using is not being able to capture certain amounts from ...
1
vote
1
answer
1k
views
Extract file from http response in Azure logic app
I have an Azure function (http triggered) which returns a CSV file in response. I am calling this function from a logic app using http request action (since I need to pass authentication details) and ...
0
votes
1
answer
57
views
Can't get HTML from URL that has google script at bottom?
https://www.pickleballtournaments.com/oncourts.pl?tid=2027&venue=Main
you can view that source and see a near bottom...
Was able to get html before now there is a google script near the bottom ...
0
votes
0
answers
152
views
Apache tika converts embedded word pad file in a docx file to .bin file
I am trying to extract all the embedded files in a word file(docx) and put the embedded files in a seperate folder. I followed the example given by apache community here https://svn.apache.org/repos/...
0
votes
2
answers
130
views
How to properly get all html elements inside a table using regex in php?
So I am using regex101.com to test my string and I can't get the output I need. The sample I made can be viewed here https://regex101.com/r/YQTW4c/2.
So my regex is this:
<table class=\"datatable\...
-1
votes
2
answers
1k
views
Unable to select html element by class name in python selenium
I am trying to select I NEED THIS TEXT from the last line of the following html code and did not yet have success so far:
<div class="warn">
<div class="row container pv2">
&...
0
votes
1
answer
579
views
Extracting text from HTML with Goose (working for all but one)
I am using Goose for extracting the title and main text from various URLs. It works with most of the URLs except one specific Dutch news website. Any idea whats going wrong here?
The specific ...
0
votes
1
answer
47
views
Getting SourceCode of Website
I used the following code to get the SourceCode from a SharePoint 2010 site:
try {
WebRequest req = HttpWebRequest.Create("myLink");
req.Method = "GET";
req.Credentials = System.Net....
-1
votes
1
answer
203
views
How can I remove the enclosing tags around a piece of HTML?
I am creating a custom filter for text using the asciidoc syntax for Drupal using the customfilter module. I enclose it in [asciidoc][/asciidoc] tags and when I run it through the asciidoctor command ...
0
votes
0
answers
818
views
How I can scrape html code using javascript?
I am trying to create a client side scraper. I would like to use only javascript or jQuery that will run on the client side and fetches the html output in JSON format and displays it on my webpage.
...
0
votes
1
answer
2k
views
How to properly extract URLs from HTML code?
I have saved a website's HTML code in a .txt file on my computer. I would like to extract all URLs from this text file using the following code:
def get_net_target(page):
start_link=page.find("...
1
vote
1
answer
1k
views
HTML XPath: Extracting text mixed in with multiple level and complex tags?
related questions before:
HTML XPath: Extracting text mixed in with multiple tags?
HTML XPath: Selectively avoiding tags when extracting text
//sorry for my poor English
I'm a beginner of writing ...
1
vote
1
answer
962
views
reading web page source code in java Differs from the orginal webpage source code
I am trying to implement program to read webpage source code and save it in text file then do some operations in it but the problem when I read web page source code , there is difference between the ...
3
votes
1
answer
279
views
How to extract only main content from any web page? (without footer, menu bars,navigation bar, footer, side bar, breadcrumb)
I have extracted whole body content by using this code.
But I don't know have to Remove navigation bar, footer, side bar, breadcrumb.
Can anyone suggest me how to get this done?
foreach($dom->...
1
vote
1
answer
102
views
Best visible content extractor available
So my application needs visible content from a given URL, like just the text part, no html no header or footer data. As of now I am using beautifulsoup and boilerpipe for getting the same. But in some ...
1
vote
1
answer
853
views
extracting links with a specific class with Selenium in Python
I am trying to extract links from a infinite scroll website
It's my code for scrolling down the page
driver = webdriver.Chrome('C:\\Program Files (x86)\\Google\\Chrome\\chromedriver.exe')
...
1
vote
1
answer
1k
views
Inserting value from vba into html list
I have been able to manipulate data in a list using vba's .value method. But when trying to update a particular value it fills the value in as blank and won't let me change it. I'm trying to push a ...
0
votes
1
answer
249
views
Extract article's headline from HTML(using Boilerpipe)
Boilerpipe allows to extract just the article's text from webpage, cleaning up all the HTML mess. However, how could I extract article's headline? There is a a way to just use page's title, but it is ...
1
vote
2
answers
184
views
Django-haystack search static content
My Django 1.10 app provides a search functionality using Haystack + Elastic Search. It works great for models data, but I need to make it work for static content too (basically HTML files).
I was ...
0
votes
2
answers
90
views
HTML Extraction with Python
The issue being tackled is being unable to click and go to the next page on an HTML page. An HTML page is being accessed which displays results after your search query. At the bottom of the page, ...
0
votes
1
answer
555
views
Extract list of texts with CasperJS
I want to extract the text values out of this List:
<ul class="standardSuggestions">
<li class="">
<div id="idac">
<span class="email" id="idb7"><...
0
votes
2
answers
2k
views
Find the text nearest to an element in HTML content
I have an HTML content with specific tags which have text and images in it.
What if I am able to select an image and I want the the text nearest to an image?
<div class="topStory">
<div ...
1
vote
1
answer
2k
views
How to navigate a website and extract data with Python
I am not much of a programmer. Just learning. I want to extract (public) electoral data from my country's electoral Authority using Python. This is for academic purposes but I also want to develop my ...
0
votes
1
answer
174
views
Strip and fetch text content from each nested div on a page
I have got an HTML from the URL. What I want to achieve is fetching only plain text content inside a div. Any idea if it can be achieved.
The structure will be similar to this
<div class="first"&...
1
vote
1
answer
37
views
Identify if html contains template language code while extracting from URL
I am trying to fetch plain text content from any provided URL which has some text data. While testing the feature on one of the URLs I found some template language code present in the source.
{{if ...
1
vote
1
answer
309
views
php warning: illegal string offset ‘name’ in GetUrl.php on line 855 [duplicate]
I got error from my Apache log:
php warning: illegal string offset ‘name’ in GetUrl.php on line 855
Here is the page code:
function find_header_by_name($header_name) {
if (!$this - > ...
1
vote
2
answers
621
views
Extracting Links in Perl using TreeBuilder
I'm working on a script to extract a bunch of information into one HTML file. I'm having some difficulty extracting ONLY a specific set of links from the page in question, however.
Here is a rough ...
0
votes
2
answers
607
views
Using WWW::Mechanize to scrape multiple pages under a directory - Perl
I'm working on a project to site scrape every interview found here into an HTML ready document to be later dumped into a DB which will automatically update our website with the latest content. You can ...
0
votes
1
answer
730
views
lxml text_content can not get whole text
As title says, div[@class="posts-cont"] contain a lot of sentences, but text_content() seems only get the first sentence splited by <br>...
>>>import requests
>>>from lxml ...
2
votes
1
answer
2k
views
Extraction of main content of an article (JavaScript)
I'm writing a program that reads a general HTML "article" page (Wikipedia, NY Times, Yahoo News, ect). From that page I want to strip away all of the "noise" (ads, header bars.. anything that isn't ...
0
votes
1
answer
2k
views
Parsing HTML file using Xpath in JAVA
I have a Java code that could read the source of a URL and save to a file(source.html) and now from the saved page, I want to extract some value using XPath. Suppose I want to read the price - //div [@...
1
vote
0
answers
255
views
What is the best regular expression or other simple ways to extract an article content from a webpage in HTML or PHP source?
There are many scripts extracts articles from html pages. If using regular expression to get the only main article from html or PHP page source, what is the best regular expressions to get only the ...
0
votes
2
answers
2k
views
Use PHP to extract HTML code between special tags into an array
Using the sample code below, I want to extract all code between each <<BEGIN>> and <<END>> tag and append the extracted code to an array for further processing later on.
<?...
2
votes
1
answer
4k
views
parsing HTML in swift
Can anyone help me out with this one:
I have a HTTP page formatted this way:
<ul class="ms-spbTree" id="_ul">
<li id="XXX\$username">Person0
<ul id="XXX\$username_ul">
...
0
votes
1
answer
558
views
get the URL to the last page of a webpage
$doc = new DOMDocument();
libxml_use_internal_errors(true);
@$doc->loadHTMLFile('http://www.mudah.my/Malaysia/Electronics-3000/directd-for-sale?o=1&q=directd&th=1');
$xpath = new DOMXPath(...
0
votes
1
answer
63
views
Identifying which JS functions and CSS is used for a component in a web page
I have been asked to re-factor a website containing 50 html page. Site is using EmberJS and using plain html. I am not aware of EmberJS. Since code is spread across multiple JavaScript, CSS, HTML, and ...
0
votes
2
answers
1k
views
RCurl getURLContent detect content type through final redirect
This is a followup question to RCurl getURL with loop - link to a PDF kills looping :
I have the following getURL command:
require(RCurl)
#set a bunch of options for curl
options(RCurlOptions = list(...
1
vote
2
answers
564
views
Extract specific part of URL from string
I need to extract only parts of a URL with PHP but I am struggling to the set point where the extraction should stop. I used a regex to extract the entire URL from a longer string like this:
$regex = ...
1
vote
3
answers
2k
views
Unable to show Json html content data in textview in android
Right now i am trying to display images and texts from one html content in text-view in android. Actually i am getting those html contents from json,but the help of below code i can only able to show ...
0
votes
4
answers
1k
views
Awk only processes first line of input file? Extract attribute values from HTML elements
I have a huge text file filled with HTML attributes. I only want the value of the tag.
Ex:
<option value="API" datatype="string" datatype_value="0">API</option>
<option value="...
0
votes
1
answer
1k
views
Extract elements by tag or class name from DOM
I try use class PHP Simple HTML DOM Parser for extract images by class name but i can´t get this
I use this wth this class :
$file=file_get_contents("https://play.google.com/store/apps/details?id=...
0
votes
1
answer
53
views
extracting information from html using php
i have a page containing this scope of information.
I'm searching a way to extract the piece of information from 'exchange with'. as you see it folds with a section tag without any property an the ...