NLTK 토큰화, 태그

In [1]:

import nltk

In [2]:

!pip3 show nltk

Name: nltk
Version: 3.4.1
Summary: Natural Language Toolkit
Home-page: http://nltk.org/
Author: Steven Bird
Author-email: stevenbird1@gmail.com
License: Apache License, Version 2.0
Location: c:\users\whanh\appdata\local\continuum\anaconda3\lib\site-packages
Requires: six
Required-by: 

In [3]:

sentence = """At eight o'clock on Thursday morning Arthur didn't feel very good."""

nltk의 punkt 다운로드¶

안깔고 토큰화 하면 punkt를 깔라는 오류가 뜬다. 아나콘다 환경이 아닐시 안되면 https://programmers.co.kr/learn/courses/21/lessons/946#note 보고 따라할 것

In [5]:

nltk.download('punkt')

[nltk_data] Downloading package punkt to
[nltk_data]     C:\Users\whanh\AppData\Roaming\nltk_data...
[nltk_data]   Unzipping tokenizers\punkt.zip.

Out[5]:

True

sentence 토큰화¶

In [6]:

tokens = nltk.word_tokenize(sentence)
tokens 

Out[6]:

['At',
 'eight',
 "o'clock",
 'on',
 'Thursday',
 'morning',
 'Arthur',
 'did',
 "n't",
 'feel',
 'very',
 'good',
 '.']

nltk의 averaged_perceptron_tagger 다운로드¶

안깔고 태깅하면 averaged_perceptron_tagger를 깔라는 오류가 뜬다. 안되면 https://programmers.co.kr/learn/courses/21/lessons/946#note 보고 따라할 것

In [8]:

nltk.download('averaged_perceptron_tagger')

[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     C:\Users\whanh\AppData\Roaming\nltk_data...
[nltk_data]   Unzipping taggers\averaged_perceptron_tagger.zip.

Out[8]:

True

pog_tag로 테깅하기¶

In [10]:

tagged = nltk.pos_tag(tokens)
tagged

Out[10]:

[('At', 'IN'),
 ('eight', 'CD'),
 ("o'clock", 'NN'),
 ('on', 'IN'),
 ('Thursday', 'NNP'),
 ('morning', 'NN'),
 ('Arthur', 'NNP'),
 ('did', 'VBD'),
 ("n't", 'RB'),
 ('feel', 'VB'),
 ('very', 'RB'),
 ('good', 'JJ'),
 ('.', '.')]

nltk github: https://github.com/nltk/nltk_data/tree/gh-pages/packages