Questions tagged [stormcrawler]

StormCrawler is an open source project providing a collection of resources for building low-latency, scalable web crawlers based on Apache Storm.

Filter by
Sorted by
Tagged with
0 votes
1 answer
17 views

How to store custom metatags in elasticsearch index from a website using stormcrawler

I am crawling intranet websites using stormcrawler(v 2.10) and storing data on Elasticsearch (v 7.8.0). Using kibana for visualization. Intranet pages have custom meta tags as below <meta name=&...
MarioCB's user avatar
3 votes
0 answers
64 views

KryoException: Buffer underflow error in Apache Storm and Storm-Crawler

I have been encountering a recurring issue during the deployment of a new version of my topology in Storm-Crawler, and I am seeking assistance in understanding and resolving the problem. Error: Upon ...
Hamide Ahadi's user avatar
0 votes
1 answer
21 views

How do I set log level in stormcrawler/storm local?

I am running stormcrawler 2.8 in local mode (storm local...). I wish to wind down the amount of logging, changing the level to WARN or ERROR. I have tried editing the storm worker.XML but that does ...
Chris Powell's user avatar
0 votes
1 answer
44 views

Storm Crawler to fetch urls with query string

I am new to storm crawler. I could configure storm crawler to fetch and parse the url "https://pubmed.ncbi.nlm.nih.gov/18926286/". But, my need is to crawl https://pubmed.ncbi.nlm.nih.gov/...
Biju George's user avatar
0 votes
1 answer
30 views

StormCrawler: urlfrontier.StatusUpdaterBolt performance bottleneck

we are running a basic StormCrawler topology, which which we want to opt for high throughput. Topology: FrontierSpout Fetcher JSoupParser DummyIndexer WarcBolt urlfrontier.StatusUpdaterBolt However, ...
Michael Dinzinger's user avatar
0 votes
1 answer
17 views

StormCrawler - Metadata fields not being persisted

I have a topology with a spout that emits a tuple to the status stream and is picked up by the StatusUpdaterBolt, which in turn write data to an elasticsearch index. The spout emits a tuple with a ...
ndtreviv's user avatar
  • 3,483
0 votes
2 answers
34 views

Logging DEBUG messages in Stormcrawler

after looking at this previous question on logging in the StormCrawler, it was not completely clear to me how to enable the logging of DEBUG messages when running the StormCrawler. By default, I only ...
Michael Dinzinger's user avatar
0 votes
0 answers
30 views

how to crawl on multi lingual sites?

I was exploring and using the storm crawler with Apache Solr . As a part of it I wanted to know on how I can crawl over the multi lingual sites say for example I am able to successfully crawl over the ...
Ashnil Tamrakar's user avatar
0 votes
1 answer
65 views

I started web crawling using Storm Crawler but I do not know where crawled results go to? Im not using Solr or Elastic Search

The Storm Crawler started crawling data but it seems I cannot find where data is stored I need to save this data to to a database so I can connect the data to a remote server and have it indexed. ...
abls1's user avatar
  • 1
0 votes
1 answer
29 views

StormCrawler: setting "maxDepth": 0 prevents ES seed injection

With StormCrawler 2.3-SNAPSHOT, setting "maxDepth": 0 in the urlfilters.json prevents the seed injection into the ES index. Is that the expected behaviour? Or should it be injecting the ...
ejo's user avatar
  • 5
0 votes
1 answer
62 views

Problem running example topology with storm-crawler 2.3-SNAPSHOT

I'm building SC 2.3-SNAPSHOT from source and generating a project from the archetype. Then I try to run the example Flux topology. Seeds are injected properly. I can see all of them in the ES index ...
ejo's user avatar
  • 5
0 votes
1 answer
26 views

Replacement of ESSeedInjector in storm-crawler 2.2

I'm updating our crawler from storm-crawler 1.14 to 2.2. What is the replacement for the old ESSeedInjector?
ejo's user avatar
  • 5
1 vote
1 answer
49 views

What is the meaning of bucket in StormCrawler spouts?

What is the meaning of bucket in the StormCrawler project? I have seen bucket in different spouts of the project. For example, in Solr and Sql based spouts we have used it in the spouts.
aeranginkaman's user avatar
0 votes
0 answers
98 views

Stormcrawler Topology class is not working as we run with es-injector.flux

I have few urls need to scrape using stormcrawler.As per [link]https://medium.com/analytics-vidhya/web-scraping-and-indexing-with-stormcrawler-and-elasticsearch-a105cb9c02ca[/link]I followed all steps ...
operation_java's user avatar
0 votes
1 answer
66 views

fetching image in StormCrawler without indexing them in status

I want to download all images in the web pages and feeding them to some machine learning algorithm for classification and extracting objects within those images. I do not want to index them in the ...
aeranginkaman's user avatar
0 votes
1 answer
59 views

Why there is not any Bolt for storing crawl results in Stormcrawler when we are using RDBMS?

I want to use Stormcrawler with an RDBMS engines like Oracle, MySQL, or Postgres. But in the storm-crawler-sql module, we only have a SqlSpout and a StatusUpdaterBolt. We did not find any class for ...
aeranginkaman's user avatar
0 votes
1 answer
187 views

How do you set up Stormcrawler to run with chromedriver instead of phantomJS?

The tutorial here describes how to set up Stormcrawler to run with phantomJS, but phantomJS doesn't seem capable of sourcing and executing outlinking javascript pages (e.g., javascript code that's ...
Dennis's user avatar
  • 111
0 votes
1 answer
451 views

StormCrawler: Unable to canonicalize address

When running StormCrawler using the ElasticSearch archetype version 2.1.0, in combination with Apache Storm version 2.2.0 and OpenJDK 11, I get the following error when running the default topology: ...
Jan's user avatar
  • 1
0 votes
1 answer
33 views

Applying different parsefilters to each domain in the same topology

I am trying to crawl different websites (e-commerce websites) and extract specific information from the pages of each website (i.e. product price, quantity, date of publication, etc.). My question is: ...
Mohamed ET's user avatar
1 vote
1 answer
367 views

Stormcrawler not retrieving all text content from web page

I'm attempting to use Stormcrawler to crawl a set of pages on our website, and while it is able to retrieve and index some of the page's text, it's not capturing a large amount of other text on the ...
Dennis's user avatar
  • 111
0 votes
1 answer
97 views

Prioritized search results?

we've been using StormCrawler with Elasticsearch to index our own websites for a couple of years. I was wondering if we can tweak the search results to have certain pages come up at the top of the ...
Dan Cogliano's user avatar
0 votes
1 answer
482 views

how to crawl a login protected site or page?

I want to crawl a site, which is required access to see pages. I am able to crawl guest pages, but how to crawl login protected pages? It will be great if somebody share steps to configure or skip the ...
Madhava's user avatar
  • 47
0 votes
1 answer
101 views

find expired domains with StormCrawler

We need to find the list of expired domains by using StormCrawler. We get http_response_status code 200 for expired domains. By in our project, we want to find these expired domains. How can I do that ...
aeranginkaman's user avatar
1 vote
1 answer
49 views

Can I filter outlinks by anchor or title in stormcrawler?

I looked at the JsoupParserBolt code, anchors are added after outlinks passed filter. Is it true that if I want to filter out links by text/anchor, I have to extend JsoupParserBolt and add anchor to ...
user15364560's user avatar
0 votes
1 answer
28 views

find redirected domains with StormCrawler

Our customer, give us a list of domains and ask to check redirect domains in that list. Domains are something similar this: www.domain.com, domain.com, subdomain1.domain.com, subdomain1.subdomain2....
aeranginkaman's user avatar
1 vote
1 answer
773 views

java.util.ConcurrentModificationException when adding some key to metadata in stormcrawler

I have added a field to metadata for transferring and persisting in the status index. The field is a List of String and its name is input_keywords. After running topology in the Strom cluster, The ...
aeranginkaman's user avatar
0 votes
1 answer
146 views

StormCrawler /Elastic Search Apache Tika for parsing PDF's. Getting error when running topology

I get the following errors when I run the es-crawler.flux topology. I'm not sure what I'm doing wrong. I don't think theres are yaml errors? **I added the Apache Tika module as an dependency in the ...
Rlcheatham's user avatar
0 votes
1 answer
25 views

How to prevent issues with our server company

We are trying to use Stormcrawler to crawl grab the index page of every site that we know a domain of - politely and ignoring any where robots.txt tell us not to. We have a database of domains - ...
lportina's user avatar
0 votes
1 answer
361 views

Setting up Stormcrawler and ElasticSearch to crawl our website html file and pdf documents

We are using StormCrawler and ElasticSearch to crawl our website. We followed the documentation for using ElasticSearch with StormCrawler. When we search in Kibana we do get back html files results ...
Rlcheatham's user avatar
0 votes
2 answers
77 views

dealing with redirect domains in StormCrawler

I am working on StormCrawler based project. One of our requirements is finding domains which redirected to another domain. In StormCrawler, each redirected URL considered as a depth in crawling. For ...
aeranginkaman's user avatar
1 vote
1 answer
100 views

crawl URLs based on their priorities in StormCrawler

I am working on a crawler based on the StormCrawler project. I have a requirement to crawl URLs based on their priorities. For example, I have two types of priority: HIGH, LOW. I want to crawl HIGH ...
aeranginkaman's user avatar
-1 votes
1 answer
26 views

Stormcrawler: Storing Outlinks of crawled Pages in Elasticsearch

I would like to store the outlink Array (URLs, Anchors) in the content Index of ElasticSearch. Can i just add a new line to indexer.md.mapping for this or is it necessary to create a new parsefilter ...
heinzhostedt's user avatar
0 votes
1 answer
280 views

Metadata in status index is not updating

I have the following sample sitemap with URl's and meta data. I'm able to crawl the sitemap and its metadata successfully. But when I update the metadata in sitemap.xml for the URl's, meta data is ...
VenkateshPamulapati's user avatar
1 vote
1 answer
79 views

Language detection using Apache Tika in StormCrawler

Does the Apache Tika integration for StormCrawler support language detection for a document? Is there a list of variables that Tika produces that I can include in the output of StormCrawler?
Jacob's user avatar
  • 21
1 vote
1 answer
83 views

StormCrawler not parsing Tika metadata

When adding the Tika parser to StormCrawler, no information is extracted from this field and stored in ElasticSearch. es-crawler.flux includes: - resource: true file: "/crawler-default....
Jacob's user avatar
  • 21
0 votes
1 answer
113 views

Getting started with StormCrawler for document crawling

I am experiencing difficulties to get started using StormCrawler using the StormCrawler+ElasticSearch archetype. On the StormCrawler website, I see two versions, namely 1x and 2x. Similarly, Apache ...
James's user avatar
  • 1
2 votes
1 answer
275 views

Storm Crawler with Java 11

Trying to update the Java version from Java 8 to Java 11 to compile and run the StromCrawler. My question- Does Storm Crawler is supported on Java 11?. As we I update the java version in my POM and ...
Vinay S's user avatar
  • 21
0 votes
1 answer
34 views

Customize some core Bolts and Spouts in StormCrawler based artifact

I have created an artifact based on the available archetype for StormCrawler on http://stormcrawler.net/. According to the customer requirements, we need to modify some codes in the StormCrawler SDK. ...
aeranginkaman's user avatar
1 vote
1 answer
73 views

Installing StormCrawler 2.0

I want to install StormCrawler 2.0, but I am unable to locate the appropriate installation instructions. In particular, what versions of Apache Storm, Zookeeper, and Java is StormCrawler 2.0 ...
Christian's user avatar
0 votes
1 answer
90 views

Customizing StormCrawler

I have installed StormCrawler including the Elasticsearch integration. I also completed the information videos found on Youtube from the creator of StormCrawler. This was a good introduction. I am ...
Christian's user avatar
0 votes
1 answer
51 views

completion event of crawling all of the sub URLs for specific base URL in Storm Crawler

I am currently working on Storm Crawler based project. I need to do some processing after the completion event of the crawling of all sub URLs for that base URL. For example, I want to change a status ...
aeranginkaman's user avatar
0 votes
1 answer
38 views

How can I send StormCrawler content to multiple Elasticsearch indices, based on host?

I currently have a successful StormCrawler instance crawling about 20 sites, and indexing the content to one Elasticsearch index. Is it possible, either in ES or via StormCrawler, to send each host's ...
chris.k's user avatar
0 votes
1 answer
65 views

Emit a custom metadata from seed URLs through all child discovered URLs for all depth

I have a Storm Crawler based project which index all contents and status in Solr collections. For each seedUrl, I have some metadata which needs to be emitted through all child of each seed URLs. For ...
aeranginkaman's user avatar
0 votes
1 answer
38 views

Separation of crawl phase from processing phase in Storm Crawler

I am currently working on a Storm Crawler based project. In the current project, we modified some Bolts and Spouts of the original Storm Crawler Core artifact. For example, we changed some parts of ...
aeranginkaman's user avatar
1 vote
2 answers
434 views

How to stop storing special characters in content while indexing

This is a sample document with the following points: Pharmaceutical Marketing Building – responsibilities.  Mass. – Aug. 13, 2020 – How to remove the special characters or non ascii unicode ...
VenkateshPamulapati's user avatar
0 votes
1 answer
91 views

About the effect of parallelism in StormCrawler

I am currently working on a Storm Crawler based project. We have a fixed and limited amount of bandwidth for fetching page from the web. We have 8 worker with a large value for parallelism hint for ...
aeranginkaman's user avatar
0 votes
1 answer
132 views

Using Kafka topic for feeding seeds url to Storm Crawler

We want to feed seed URLs from a Kafka topic to the StormCrawler based project. Is there a need to change the Storm Crawler?
aeranginkaman's user avatar
0 votes
1 answer
46 views

Is there any systematic way to turn on or turn off some Bolt in StormCrawler?

I have developed a StormCrawler project that has multiple additional Bolts in that topology. My crawler should work 7 X 24 without any downtime. So I can not restart the crawler and change the ...
aeranginkaman's user avatar
0 votes
2 answers
378 views

How can i debug the the docker container(storm crawler) which is written in java in vs code?

I am unable to get how can i debug the docker container(which is running storm crawler) in the vs code? I tried looking for https://code.visualstudio.com/docs/containers/debug-common and other https://...
Vishal Sinha's user avatar
1 vote
1 answer
130 views

Using Stormcrawler for crawling specific subdirectories

I would like to be able crawl very specific sub-directories for a given website. For example: On the website www.world.com there maybe multiple sub-directories /world or /bye. These in-turn may ...
Ranadip Saha's user avatar

1
2 3 4 5