- 浏览: 167334 次
- 性别:
- 来自: 北京
文章分类
最新评论
-
小桔子:
u 棒棒哒!按照你的搞定了,之前搞了好久!u 棒棒哒!!! ...
Ubuntu为Tomcat启用80端口 -
u011938035:
我用的是nutch1.7,org.apache.nutch.n ...
nutch1.4 URLNormalizers 详解 -
peigang:
试试跟踪一下脚本,应该是环境变量的问题。
nutch1.4:爬虫定时抓取设置 -
zhangmj10:
你好,看这帖子是好久以前的,不知道你能不能看到。不知道能不能帮 ...
nutch1.4:爬虫定时抓取设置 -
shinide1989:
楼主你好,我正需要修改html的解析,并想把结果存为其他格 ...
nutch1.4插件开发
/** * Licensed to the Apache Software Foundation (ASF) under one or more * contributor license agreements. See the NOTICE file distributed with * this work for additional information regarding copyright ownership. * The ASF licenses this file to You under the Apache License, Version 2.0 * (the "License"); you may not use this file except in compliance with * the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ package org.apache.nutch.crawl; import java.util.*; import java.text.*; // Commons Logging imports import org.apache.commons.lang.StringUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.apache.hadoop.fs.*; import org.apache.hadoop.conf.*; import org.apache.hadoop.mapred.*; import org.apache.hadoop.util.Tool; import org.apache.hadoop.util.ToolRunner; import org.apache.nutch.parse.ParseSegment; import org.apache.nutch.indexer.solr.SolrDeleteDuplicates; import org.apache.nutch.indexer.solr.SolrIndexer; import org.apache.nutch.util.HadoopFSUtil; import org.apache.nutch.util.NutchConfiguration; import org.apache.nutch.util.NutchJob; import org.apache.nutch.fetcher.Fetcher; public class Crawl extends Configured implements Tool { public static final Logger LOG = LoggerFactory.getLogger(Crawl.class); private static String getDate() { return new SimpleDateFormat("yyyyMMddHHmmss").format (new Date(System.currentTimeMillis())); } /* Perform complete crawling and indexing (to Solr) given a set of root urls and the -solr parameter respectively. More information and Usage parameters can be found below. */ public static void main(String args[]) throws Exception { Configuration conf = NutchConfiguration.create(); int res = ToolRunner.run(conf, new Crawl(), args); System.exit(res); } public int run(String[] args) throws Exception { if (args.length < 1) { System.out.println ("Usage: Crawl <urlDir> -solr <solrURL> [-dir d] [-threads n] [-depth i] [-topN N]"); return -1; } Path rootUrlDir = null; Path dir = new Path("crawl-" + getDate()); int threads = getConf().getInt("fetcher.threads.fetch", 10); int depth = 5; long topN = Long.MAX_VALUE; String solrUrl = null; for (int i = 0; i < args.length; i++) { if ("-dir".equals(args[i])) { dir = new Path(args[i+1]); i++; } else if ("-threads".equals(args[i])) { threads = Integer.parseInt(args[i+1]); i++; } else if ("-depth".equals(args[i])) { depth = Integer.parseInt(args[i+1]); i++; } else if ("-topN".equals(args[i])) { topN = Integer.parseInt(args[i+1]); i++; } else if ("-solr".equals(args[i])) { solrUrl = args[i + 1]; i++; } else if (args[i] != null) { rootUrlDir = new Path(args[i]); } } JobConf job = new NutchJob(getConf()); if (solrUrl == null) { LOG.warn("solrUrl is not set, indexing will be skipped..."); } FileSystem fs = FileSystem.get(job); if (LOG.isInfoEnabled()) { LOG.info("crawl started in: " + dir); LOG.info("rootUrlDir = " + rootUrlDir); LOG.info("threads = " + threads); LOG.info("depth = " + depth); LOG.info("solrUrl=" + solrUrl); if (topN != Long.MAX_VALUE) LOG.info("topN = " + topN); } Path crawlDb = new Path(dir + "/crawldb"); //设置爬取库目录 Path linkDb = new Path(dir + "/linkdb"); //链接库目录 Path segments = new Path(dir + "/segments"); //段信息目录 Path indexes = new Path(dir + "/indexes"); //索引目录 Path index = new Path(dir + "/index"); //初始化各项参数 Path tmpDir = job.getLocalPath("crawl"+Path.SEPARATOR+getDate()); Injector injector = new Injector(getConf()); //URL注入器对象;数据下载入口 Generator generator = new Generator(getConf()); //生成器;生成待下载URL列表 Fetcher fetcher = new Fetcher(getConf()); //抓取器;按照HTTP协议访问互联网,获取网页数据具体内容。下载过程由下载列表和操作参数控制,直到下载完毕。 ParseSegment parseSegment = new ParseSegment(getConf()); //解析数据段;数据段(Segment)存放网络爬虫每一次抓取使用的待下载列表、已经获得的网页内容和本次内容的索引。 CrawlDb crawlDbTool = new CrawlDb(getConf()); //抓取数据库工具 LinkDb linkDbTool = new LinkDb(getConf()); //链接库工具 // initialize crawlDb injector.inject(crawlDb, rootUrlDir); int i; for (i = 0; i < depth; i++) { // generate new segment Path[] segs = generator.generate(crawlDb, segments, -1, topN, System .currentTimeMillis()); if (segs == null) { LOG.info("Stopping at depth=" + i + " - no more URLs to fetch."); break; } fetcher.fetch(segs[0], threads); // fetch it if (!Fetcher.isParsing(job)) { parseSegment.parse(segs[0]); // parse it, if needed } crawlDbTool.update(crawlDb, segs, true, true); // update crawldb } if (i > 0) { linkDbTool.invert(linkDb, segments, true, true, false); // invert links /** * 使用solr索引处理完毕的文件,索引、复制、合并 */ if (solrUrl != null) { // index, dedup & merge FileStatus[] fstats = fs.listStatus(segments, HadoopFSUtil.getPassDirectoriesFilter(fs)); SolrIndexer indexer = new SolrIndexer(getConf()); indexer.indexSolr(solrUrl, crawlDb, linkDb, Arrays.asList(HadoopFSUtil.getPaths(fstats))); SolrDeleteDuplicates dedup = new SolrDeleteDuplicates(); dedup.setConf(getConf()); dedup.dedup(solrUrl); } } else { LOG.warn("No URLs to fetch - check your seed list and URL filters."); } if (LOG.isInfoEnabled()) { LOG.info("crawl finished: " + dir); } return 0; } }
发表评论
-
Nutch1.7二次开发培训讲义
2015-09-16 15:23 1239做Nutch二次开发,开发阶段用什么操作系统都可以,只要有J ... -
nutch-default.xml 配置范例
2014-07-22 20:20 2160nutch的配置文件属性很多,需要根据实际需要详细配置。下面 ... -
nutch本地模式调试环境配置
2014-07-22 17:33 721nutch本地模式调试可以跟踪详细的爬取过程,便于调 ... -
nutch分布式调试环境配置
2014-07-17 14:35 593准备:hadoop单机模式设置,参考:http:/ ... -
nutch 正文提取流程解析
2013-05-03 17:59 1089nutch正文提取在Fatcher的run方法中进行,本文 ... -
用Eclipse开发nutch准备工作
2012-09-20 11:34 1285本文来源于:http://zettadata.blogs ... -
nutch1.4 CrawlDatum详解
2012-09-17 14:04 0nutch中CrawlDatum对象封装了爬取数据,包括爬取地 ... -
nutch1.4 分布式爬取
2012-06-19 12:02 5387从nutch1.3开始本地抓取(单机),分布式抓取(集群)所使 ... -
nutch1.4:爬虫定时抓取设置
2012-06-13 15:03 4624nutch1.4定时爬取数据配合linux定时任务可以实现nu ... -
nutch1.4 开发:增加外部jar包
2012-06-11 14:48 1539ntuch1.4开发中可能会涉及到引入外部jar包的情况,比如 ... -
nutch1.4 爬虫父页面参数传递到子页面注意事项
2012-06-02 11:51 17771、inject中以读取文件的方式传入自定义参数: d ... -
nutch1.4 Fetcher详解
2012-05-24 17:37 0org.apache.nutch.fetcher.Fetche ... -
nutch1.4 Protocol接口解析
2012-04-24 17:44 0实现Protocol接口的过滤器插件,所有插件都extends ... -
nutch1.4自定义字段开发实例
2012-04-13 19:41 0本文介绍nutch1.4中插件方式实现自定义字段,并在solr ... -
nutch1.4插件开发
2012-04-13 17:02 2836参考了不少nutch插件开 ... -
nutch1.4 解析器 ParseSegment详解
2012-04-11 15:17 1211org.apache.nutch.parse.ParseSeg ... -
nutch1.4 Generator详解
2012-03-31 15:14 0org.apache.nutch.crawl.Generato ... -
nutch1.4 ScoringFilter详解
2012-03-29 17:39 1150org.apache.nutch.scoring.Scorin ... -
nutch1.4 URLFilter详解
2012-03-29 17:16 1624org.apache.nutch.net.URLFilter接 ... -
nutch1.4 URLNormalizers 详解
2012-03-29 15:56 1667org.apache.nutch.net.URLNorm ...
相关推荐
nutch1.4帮助文档,学习nutch1.4必备,最新nutch1.4核心类解读!
nutch1.4 在windows下的安装配置环境搭建
nutch 1.4 在windows下安装配置
Nutch是一个由Java实现的,刚刚诞生开放源代码(open-source)的web搜索引擎。Nutch目前最新的版本为version1.4。这个为nutch的最新版 1.4。
apache-nutch-1.4-bin.tar.gz.part2
Nutch1[1].4_windows下eclipse配置图文详解
如果想把多次用nutch crawl获得的所有目录合并在一起。可以按以下步骤进行
apache-nutch-1.4-bin.part2
apache-nutch-1.4-bin.part1
apache-nutch-1.4-bin.tar.gz.part1
Windows下使用Eclipse配置Nutch2图文详解
nutch 爬到的CSDN数据 nutch crawlnutch 爬到的CSDN数据 nutch crawlnutch 爬到的CSDN数据 nutch crawl
Nutch 是一个开源Java 实现的搜索引擎。这里是它的安装包。
对于nutch源码的解读,让你了解nutch工作流程各个功能模块的作用
nutch虽然是开源的,但初学nutch的同志门肯定对源码比较头疼,很难看懂,本资料是对crawl源码的解析,希望对大家有用。
一、org.apache.nutch.crawl.Injector: 1,注入url.txt 2,url标准化 3,拦截url,进行正则校验(regex-urlfilter.txt) 4,对符URL标准的url进行map对构造, CrawlDatum>,在构造过程中给CrawlDatum初始化得分...
很好的一个开源搜索引擎,可以自己设计添加代码。
nutcher 是 Apache Nutch 的中文教程,在... Nutch流程控制源码详解(bin/crawl中文注释版) Nutch教程——URLNormalizer源码详解 Nutch参数配置——http.content.limit 文档截图:
一个已经部署好的 nutch1.7爬虫。...找到org.apache.nutch.crawl.Crawl 这个类。 run configuration 在 Programa argument 里面 输入 crawl urls -dir out -threads 20 -depth 2 然后就可以跑了。
1.1 Nutch 基本原理 1.1.1 Nutch 基本组成 1.1.2 Nutch 工作流程 1.2 Nutch 流程详解 1.2.1 Nutch 数据流程 1.2.2 Nutch 流程分析