- Python-文本处理
- Python-文本处理简介
- Python - 文本处理环境
- Python - 字符串不变性
- Python - 排序行
- Python - 重新格式化段落
- Python - 计算段落中的标记
- Python - 二进制 ASCII 转换
- Python - 字符串作为文件
- Python-向后读取文件
- Python - 过滤重复单词
- Python - 从文本中提取电子邮件
- Python - 从文本中提取 URL
- Python - 漂亮的打印
- Python - 文本处理状态机
- Python - 大写和翻译
- Python - 标记化
- Python - 删除停用词
- Python - 同义词和反义词
- Python - 文本翻译
- Python-单词替换
- Python-拼写检查
- Python - WordNet 接口
- Python - 语料库访问
- Python - 标记单词
- Python - 块和缝隙
- Python - 块分类
- Python-文本分类
- Python-二元组
- Python - 处理 PDF
- Python-处理Word文档
- Python - 读取 RSS 提要
- Python-情感分析
- Python - 搜索和匹配
- Python - 文本修改
- Python-文本换行
- Python-频率分布
- Python-文本摘要
- Python - 词干算法
- Python - 约束搜索
Python - 词干算法
在自然语言处理领域,我们遇到两个或多个单词具有共同词根的情况。例如,三个词——同意、同意和同意有相同的词根同意。涉及任何这些单词的搜索都应将它们视为作为根单词的同一个单词。因此,将所有单词链接到其词根就变得至关重要。NLTK 库有方法执行此链接并给出显示根词的输出。
nltk 中提供了三种最常用的词干算法。他们给出的结果略有不同。下面的示例显示了所有三种词干算法的使用及其结果。
import nltk from nltk.stem.porter import PorterStemmer from nltk.stem.lancaster import LancasterStemmer from nltk.stem import SnowballStemmer porter_stemmer = PorterStemmer() lanca_stemmer = LancasterStemmer() sb_stemmer = SnowballStemmer("english",) word_data = "Aging head of famous crime family decides to transfer his position to one of his subalterns" # First Word tokenization nltk_tokens = nltk.word_tokenize(word_data) #Next find the roots of the word print '***PorterStemmer****\n' for w_port in nltk_tokens: print "Actual: %s || Stem: %s" % (w_port,porter_stemmer.stem(w_port)) print '\n***LancasterStemmer****\n' for w_lanca in nltk_tokens: print "Actual: %s || Stem: %s" % (w_lanca,lanca_stemmer.stem(w_lanca)) print '\n***SnowballStemmer****\n' for w_snow in nltk_tokens: print "Actual: %s || Stem: %s" % (w_snow,sb_stemmer.stem(w_snow))
当我们运行上面的程序时,我们得到以下输出 -
***PorterStemmer**** Actual: Aging || Stem: age Actual: head || Stem: head Actual: of || Stem: of Actual: famous || Stem: famou Actual: crime || Stem: crime Actual: family || Stem: famili Actual: decides || Stem: decid Actual: to || Stem: to Actual: transfer || Stem: transfer Actual: his || Stem: hi Actual: position || Stem: posit Actual: to || Stem: to Actual: one || Stem: one Actual: of || Stem: of Actual: his || Stem: hi Actual: subalterns || Stem: subaltern ***LancasterStemmer**** Actual: Aging || Stem: ag Actual: head || Stem: head Actual: of || Stem: of Actual: famous || Stem: fam Actual: crime || Stem: crim Actual: family || Stem: famy Actual: decides || Stem: decid Actual: to || Stem: to Actual: transfer || Stem: transf Actual: his || Stem: his Actual: position || Stem: posit Actual: to || Stem: to Actual: one || Stem: on Actual: of || Stem: of Actual: his || Stem: his Actual: subalterns || Stem: subaltern ***SnowballStemmer**** Actual: Aging || Stem: age Actual: head || Stem: head Actual: of || Stem: of Actual: famous || Stem: famous Actual: crime || Stem: crime Actual: family || Stem: famili Actual: decides || Stem: decid Actual: to || Stem: to Actual: transfer || Stem: transfer Actual: his || Stem: his Actual: position || Stem: posit Actual: to || Stem: to Actual: one || Stem: one Actual: of || Stem: of Actual: his || Stem: his Actual: subalterns || Stem: subaltern