Tutorial: StormCrawler 1.4 + Elasticsearch + Kibana
Why Stack Overflow recommends it
Stack Overflow users link it as the walkthrough for getting StormCrawler working with Elasticsearch, and several follow it step by step before asking about version mismatches or indexing problems.
Recommended by 1 people across 1 different months, 2017 to 2017.
What it is
A 20-minute screencast by StormCrawler's lead developer showing how to bootstrap a StormCrawler 1.4 project from the Maven archetype, switch it to Elasticsearch, run the injection and crawl topologies, and set up Kibana to monitor metrics and the status index.
What people said
Recommends Julien Nioche · 2017-06 · Stack Overflow · on “Storm-crawler crawl and indexing”
Great to hear you like StormCrawler. As explained in README [LINK] and the video tutorial [this video] based on ES2.x, you should use the ES_IndexInit [LINK] script to set the mapping explicitly. It probably works without it but it would not be optimal.
Cites RajashekharC · 2017-04 · Stack Overflow · on “Issue while connecting to elasticsearch”
I have been following video blog on Storm Crawler. I am trying to create a web crawler referring the WIKI and video [this video] , but I am getting: java.lang.IllegalStateException: Received message from unsupported version: [2.0.0] minimal compatible version is: [5.0.0] I am using following: Elasticsearch : 5.3.0 version Storm: 1.0.3 version. Please help me with what versions are recommended and used by you. Appreciate your help. Thanks! Raj
Cites Ron Dobley · 2017-11 · Stack Overflow · on “Stormcrawler not indexing content with Elasticsearch”
When using Stormcrawler it is indexing to Elasticsearch, but not the content. Stormcrawler is up-to-date with 'origin/master' [LINK] Using elasticsearch-5.6.4 crawler-conf.yaml has indexer.url.fieldname: "url" indexer.text.fieldname: "content" indexer.canonical.name: "canonical" The url and title fields are indexed, but not content. I have trying to get this working by following Julien's tutorial at: [this video] Everything is working, except for the content is not being indexed into Elasticsearch. I feel like this is some small config error, but I have tried many variations with no luck. So, now I seek help. Thanks.