NLTK

NLTK is a popular library for Natural Language Processing in Python. It provides easy-to-use interfaces to many corpora and lexical resources such as WordNet. NLTK also is very easy to learn, has excellent documentation, and is widely used in academia and industry.

NLTK download and installation

With the release of NLTK 3.0, many API changes were made. In particular, the word tokenize() function of the nltk.tokenize.punkt module has been removed in 3.0.0 and replaced with the word_tokenize() function of the same module, which employs different algorithms to accomplish the same goal. It is therefore not compatible with models created using NLTK 2.0 style models (retrain your models if you want to use NLTK 3.0).

NLTK corpus

The NLTK corpus is a massive collection of resources for training and testing Natural Language Processing models. It includes over 50 corpora and lexical resources such as WordNet, along with 20+ tokenizers and ttk modules designed specifically for Finnish, French, German, Portuguese, Spanish and Russian.

NLTK wordnet

NLTK’s WordNet interface [9] is a lexical database for English. It groups English words into sets of synonyms called synsets, provides short definitions and usage examples, and records a number of relations among these synonym sets or their members. WordNet can thus be seen as a combination of dictionary and thesaurus. However, it also aims to cover a much wider range of possible senses than a thesaurus, including abstract concepts and ideas as well as physical objects. Princeton University’s WordNet is often used with NLTK.

wordtokenize

wordtokenize is a well known library for those who are looking to do NLP in Python. It is a part of the Natural Language Toolkit library. It Tokenizes words, which means it breaks up a string of text into individual words.

What is wordtokenize?

Wordtokenize is a tool that helps you break a string of text into “tokens” or small units of meaning. It’s commonly used in Natural Language Processing (NLP) applications to pre-process text data before feeding it into a machine learning algorithm.

Breaking text down into tokens is not always straightforward, as there can be multiple ways to interpret where one token starts and another ends. For example, the sentence “I’m going to the store” could be tokenized as [“I’m”, “going”, “to”, “the”, “store”] or as [“I”, “‘m”, “going”, “to”, “the”, “store”].

Which interpretation is correct depends on the context and what you want to use the tokens for. In general, though, wordtokenize does a pretty good job of automatically detecting word boundaries and returning sensible results.

How to use wordtokenize?

Basic Usage
At its simplest, you can use word_tokenize() like this:

from nltk.tokenize import word_tokenize

print(word_tokenize(“Don’t be fooled by the dark sounding name, Mr. Jone’s Orphanage is as cheery as cheery goes for a orphanage.”))
You’ll get the following output:

[‘Do’, “n’t”, ‘be’, ‘fooled’, ‘by’, ‘the’, ‘dark’, ‘sounding’, ‘name’, ‘,’, ‘Mr.’, ‘Jone’, “‘s”, ‘Orphanage’, ‘is’, ‘as’, ‘cheery’, ‘as’, ‘cheery’, ‘goes’, ‘for’, ‘a’,’orphanage’,’.’]

Examples

nltk.tokenize.punkt tokenizes words by taking various characters as delimiters. A delimiter is a character that marks the beginning or end of a word. For example, the periods in the following sentence are delimiters: “Mr. Smith goes to the store.” The word “Mr.” is not a word, but “Smith” is.

Example 1

NLTK NLTK is a leading platform for building Python programs to work with human language data. It provides easy-to-use interfaces to over 50 corpora and lexical resources such as WordNet, along with a suite of text processing libraries for classification, tokenization, stemming, tagging, parsing, and semantic reasoning, wrappers for industrial-strength NLP libraries.

Example 2

Natural Language Toolkit (NLTK)

The Natural Language Toolkit (NLTK) is a Python package for natural language processing. NLTK provides a wide variety of resources for processing human language data, including data structures, algorithms, and tutorials.

NLTK is available for a number of different platforms, including Windows, Mac OS X, and Linux. The current version of NLTK is 3.2.4.