Questions tagged [stormcrawler]
StormCrawler is an open source project providing a collection of resources for building low-latency, scalable web crawlers based on Apache Storm.
215
questions
0
votes
1
answer
17
views
How to store custom metatags in elasticsearch index from a website using stormcrawler
I am crawling intranet websites using stormcrawler(v 2.10) and storing data on Elasticsearch (v 7.8.0). Using kibana for visualization. Intranet pages have custom meta tags as below
<meta name=&...
3
votes
0
answers
64
views
KryoException: Buffer underflow error in Apache Storm and Storm-Crawler
I have been encountering a recurring issue during the deployment of a new version of my topology in Storm-Crawler, and I am seeking assistance in understanding and resolving the problem.
Error: Upon ...
0
votes
1
answer
21
views
How do I set log level in stormcrawler/storm local?
I am running stormcrawler 2.8 in local mode (storm local...). I wish to wind down the amount of logging, changing the level to WARN or ERROR.
I have tried editing the storm worker.XML but that does ...
0
votes
1
answer
44
views
Storm Crawler to fetch urls with query string
I am new to storm crawler. I could configure storm crawler to fetch and parse the url "https://pubmed.ncbi.nlm.nih.gov/18926286/". But, my need is to crawl https://pubmed.ncbi.nlm.nih.gov/...
0
votes
1
answer
30
views
StormCrawler: urlfrontier.StatusUpdaterBolt performance bottleneck
we are running a basic StormCrawler topology, which which we want to opt for high throughput.
Topology:
FrontierSpout
Fetcher
JSoupParser
DummyIndexer
WarcBolt
urlfrontier.StatusUpdaterBolt
However, ...
0
votes
1
answer
17
views
StormCrawler - Metadata fields not being persisted
I have a topology with a spout that emits a tuple to the status stream and is picked up by the StatusUpdaterBolt, which in turn write data to an elasticsearch index.
The spout emits a tuple with a ...
0
votes
2
answers
34
views
Logging DEBUG messages in Stormcrawler
after looking at this previous question on logging in the StormCrawler, it was not completely clear to me how to enable the logging of DEBUG messages when running the StormCrawler.
By default, I only ...
0
votes
0
answers
30
views
how to crawl on multi lingual sites?
I was exploring and using the storm crawler with Apache Solr . As a part of it I wanted to know on how I can crawl over the multi lingual sites say for example I am able to successfully crawl over the ...
0
votes
1
answer
65
views
I started web crawling using Storm Crawler but I do not know where crawled results go to? Im not using Solr or Elastic Search
The Storm Crawler started crawling data but it seems I cannot find where data is stored I need to save this data to to a database so I can connect the data to a remote server and have it indexed. ...
0
votes
1
answer
29
views
StormCrawler: setting "maxDepth": 0 prevents ES seed injection
With StormCrawler 2.3-SNAPSHOT, setting "maxDepth": 0 in the urlfilters.json prevents the seed injection into the ES index. Is that the expected behaviour? Or should it be injecting the ...
0
votes
1
answer
62
views
Problem running example topology with storm-crawler 2.3-SNAPSHOT
I'm building SC 2.3-SNAPSHOT from source and generating a project from the archetype. Then I try to run the example Flux topology. Seeds are injected properly. I can see all of them in the ES index ...
0
votes
1
answer
26
views
Replacement of ESSeedInjector in storm-crawler 2.2
I'm updating our crawler from storm-crawler 1.14 to 2.2. What is the replacement for the old ESSeedInjector?
1
vote
1
answer
49
views
What is the meaning of bucket in StormCrawler spouts?
What is the meaning of bucket in the StormCrawler project? I have seen bucket in different spouts of the project. For example, in Solr and Sql based spouts we have used it in the spouts.
0
votes
0
answers
98
views
Stormcrawler Topology class is not working as we run with es-injector.flux
I have few urls need to scrape using stormcrawler.As per [link]https://medium.com/analytics-vidhya/web-scraping-and-indexing-with-stormcrawler-and-elasticsearch-a105cb9c02ca[/link]I followed all steps ...
0
votes
1
answer
66
views
fetching image in StormCrawler without indexing them in status
I want to download all images in the web pages and feeding them to some machine learning algorithm for classification and extracting objects within those images. I do not want to index them in the ...
0
votes
1
answer
59
views
Why there is not any Bolt for storing crawl results in Stormcrawler when we are using RDBMS?
I want to use Stormcrawler with an RDBMS engines like Oracle, MySQL, or Postgres. But in the storm-crawler-sql module, we only have a SqlSpout and a StatusUpdaterBolt. We did not find any class for ...
0
votes
1
answer
187
views
How do you set up Stormcrawler to run with chromedriver instead of phantomJS?
The tutorial here describes how to set up Stormcrawler to run with phantomJS, but phantomJS doesn't seem capable of sourcing and executing outlinking javascript pages (e.g., javascript code that's ...
0
votes
1
answer
451
views
StormCrawler: Unable to canonicalize address
When running StormCrawler using the ElasticSearch archetype version 2.1.0, in combination with Apache Storm version 2.2.0 and OpenJDK 11, I get the following error when running the default topology:
...
0
votes
1
answer
33
views
Applying different parsefilters to each domain in the same topology
I am trying to crawl different websites (e-commerce websites) and extract specific information from the pages of each website (i.e. product price, quantity, date of publication, etc.).
My question is: ...
1
vote
1
answer
367
views
Stormcrawler not retrieving all text content from web page
I'm attempting to use Stormcrawler to crawl a set of pages on our website, and while it is able to retrieve and index some of the page's text, it's not capturing a large amount of other text on the ...
0
votes
1
answer
97
views
Prioritized search results?
we've been using StormCrawler with Elasticsearch to index our own websites for a couple of years. I was wondering if we can tweak the search results to have certain pages come up at the top of the ...
0
votes
1
answer
482
views
how to crawl a login protected site or page?
I want to crawl a site, which is required access to see pages. I am able to crawl guest pages, but how to crawl login protected pages?
It will be great if somebody share steps to configure or skip the ...
0
votes
1
answer
101
views
find expired domains with StormCrawler
We need to find the list of expired domains by using StormCrawler. We get http_response_status code 200 for expired domains. By in our project, we want to find these expired domains. How can I do that ...
1
vote
1
answer
49
views
Can I filter outlinks by anchor or title in stormcrawler?
I looked at the JsoupParserBolt code, anchors are added after outlinks passed filter. Is it true that if I want to filter out links by text/anchor, I have to extend JsoupParserBolt and add anchor to ...
0
votes
1
answer
28
views
find redirected domains with StormCrawler
Our customer, give us a list of domains and ask to check redirect domains in that list. Domains are something similar this:
www.domain.com,
domain.com,
subdomain1.domain.com,
subdomain1.subdomain2....
1
vote
1
answer
773
views
java.util.ConcurrentModificationException when adding some key to metadata in stormcrawler
I have added a field to metadata for transferring and persisting in the status index. The field is a List of String and its name is input_keywords. After running topology in the Strom cluster, The ...
0
votes
1
answer
146
views
StormCrawler /Elastic Search Apache Tika for parsing PDF's. Getting error when running topology
I get the following errors when I run the es-crawler.flux topology. I'm not sure what I'm doing wrong. I don't think theres are yaml errors?
**I added the Apache Tika module as an dependency in the ...
0
votes
1
answer
25
views
How to prevent issues with our server company
We are trying to use Stormcrawler to crawl grab the index page of every site that we know a domain of - politely and ignoring any where robots.txt tell us not to. We have a database of domains - ...
0
votes
1
answer
361
views
Setting up Stormcrawler and ElasticSearch to crawl our website html file and pdf documents
We are using StormCrawler and ElasticSearch to crawl our website. We followed the documentation for using ElasticSearch with StormCrawler. When we search in Kibana we do get back html files results ...
0
votes
2
answers
77
views
dealing with redirect domains in StormCrawler
I am working on StormCrawler based project. One of our requirements is finding domains which redirected to another domain. In StormCrawler, each redirected URL considered as a depth in crawling. For ...
1
vote
1
answer
100
views
crawl URLs based on their priorities in StormCrawler
I am working on a crawler based on the StormCrawler project. I have a requirement to crawl URLs based on their priorities. For example, I have two types of priority: HIGH, LOW. I want to crawl HIGH ...
-1
votes
1
answer
26
views
Stormcrawler: Storing Outlinks of crawled Pages in Elasticsearch
I would like to store the outlink Array (URLs, Anchors) in the content Index of ElasticSearch. Can i just add a new line to indexer.md.mapping for this or is it necessary to create a new parsefilter ...
0
votes
1
answer
280
views
Metadata in status index is not updating
I have the following sample sitemap with URl's and meta data. I'm able to crawl the sitemap and its metadata successfully. But when I update the metadata in sitemap.xml for the URl's, meta data is ...
1
vote
1
answer
79
views
Language detection using Apache Tika in StormCrawler
Does the Apache Tika integration for StormCrawler support language detection for a document? Is there a list of variables that Tika produces that I can include in the output of StormCrawler?
1
vote
1
answer
83
views
StormCrawler not parsing Tika metadata
When adding the Tika parser to StormCrawler, no information is extracted from this field and stored in ElasticSearch.
es-crawler.flux
includes:
- resource: true
file: "/crawler-default....
0
votes
1
answer
113
views
Getting started with StormCrawler for document crawling
I am experiencing difficulties to get started using StormCrawler using the StormCrawler+ElasticSearch archetype. On the StormCrawler website, I see two versions, namely 1x and 2x. Similarly, Apache ...
2
votes
1
answer
275
views
Storm Crawler with Java 11
Trying to update the Java version from Java 8 to Java 11 to compile and run the StromCrawler.
My question- Does Storm Crawler is supported on Java 11?.
As we I update the java version in my POM and ...
0
votes
1
answer
34
views
Customize some core Bolts and Spouts in StormCrawler based artifact
I have created an artifact based on the available archetype for StormCrawler on http://stormcrawler.net/. According to the customer requirements, we need to modify some codes in the StormCrawler SDK. ...
1
vote
1
answer
73
views
Installing StormCrawler 2.0
I want to install StormCrawler 2.0, but I am unable to locate the appropriate installation instructions. In particular, what versions of Apache Storm, Zookeeper, and Java is StormCrawler 2.0 ...
0
votes
1
answer
90
views
Customizing StormCrawler
I have installed StormCrawler including the Elasticsearch integration. I also completed the information videos found on Youtube from the creator of StormCrawler. This was a good introduction. I am ...
0
votes
1
answer
51
views
completion event of crawling all of the sub URLs for specific base URL in Storm Crawler
I am currently working on Storm Crawler based project. I need to do some processing after the completion event of the crawling of all sub URLs for that base URL. For example, I want to change a status ...
0
votes
1
answer
38
views
How can I send StormCrawler content to multiple Elasticsearch indices, based on host?
I currently have a successful StormCrawler instance crawling about 20 sites, and indexing the content to one Elasticsearch index. Is it possible, either in ES or via StormCrawler, to send each host's ...
0
votes
1
answer
65
views
Emit a custom metadata from seed URLs through all child discovered URLs for all depth
I have a Storm Crawler based project which index all contents and status in Solr collections. For each seedUrl, I have some metadata which needs to be emitted through all child of each seed URLs. For ...
0
votes
1
answer
38
views
Separation of crawl phase from processing phase in Storm Crawler
I am currently working on a Storm Crawler based project. In the current project, we modified some Bolts and Spouts of the original Storm Crawler Core artifact. For example, we changed some parts of ...
1
vote
2
answers
434
views
How to stop storing special characters in content while indexing
This is a sample document with the following points:
Pharmaceutical
Marketing
Building –
responsibilities. Â
Mass. – Aug. 13, 2020 –Â
How to remove the special characters or non ascii unicode ...
0
votes
1
answer
91
views
About the effect of parallelism in StormCrawler
I am currently working on a Storm Crawler based project. We have a fixed and limited amount of bandwidth for fetching page from the web. We have 8 worker with a large value for parallelism hint for ...
0
votes
1
answer
132
views
Using Kafka topic for feeding seeds url to Storm Crawler
We want to feed seed URLs from a Kafka topic to the StormCrawler based project.
Is there a need to change the Storm Crawler?
0
votes
1
answer
46
views
Is there any systematic way to turn on or turn off some Bolt in StormCrawler?
I have developed a StormCrawler project that has multiple additional Bolts in that topology. My crawler should work 7 X 24 without any downtime. So I can not restart the crawler and change the ...
0
votes
2
answers
378
views
How can i debug the the docker container(storm crawler) which is written in java in vs code?
I am unable to get how can i debug the docker container(which is running storm crawler) in the vs code? I tried looking for https://code.visualstudio.com/docs/containers/debug-common
and other https://...
1
vote
1
answer
130
views
Using Stormcrawler for crawling specific subdirectories
I would like to be able crawl very specific sub-directories for a given website.
For example:
On the website www.world.com there maybe multiple sub-directories /world or /bye. These in-turn may ...