Natural Language Processing

spaCy is designed for production use: you build and train NLP pipelines, then package them to deploy. Its trained pipelines are Python packages. In an automated build, install them with pip from a direct link instead of spaCy's download command, and put that link in your requirements.txt. In a larger code base, import the pipeline as a module, so a missing one raises an ImportError right away. Run your texts through nlp.pipe in batches, and disable the components you don't need. To train your own pipeline, run spacy train with one config file.

NLTK comes with corpora and lexical resources such as WordNet, plus libraries to tokenize, tag, parse, and classify text. Its creators wrote a book that teaches NLP with it, and you can read it online. The book also says NLTK isn't highly optimized for runtime performance, so use it to learn and experiment. The data ships separately: get it with NLTK's data downloader, and set NLTK_DATA when you install it somewhere other than the standard locations.

Stanza is designed to work across many languages, using the Universal Dependencies formalism. It's also the official Python interface to Stanford's Java CoreNLP, though its own pipeline doesn't need CoreNLP. You can list the processors to load with processors=. Pass all your documents to the pipeline at once, since a for loop over one sentence at a time is very slow. For a lot of text, run it on a GPU. To keep the pipeline from downloading anything at runtime, download the models ahead of time.

jieba segments Chinese text into words: accurate mode suits text analysis, and search engine mode cuts long words into short ones for better recall. Add your own words with jieba.load_userdict() to get higher accuracy, and for Traditional Chinese, switch to its bigger dictionary with jieba.set_dictionary(). The other picks work with it: spaCy can use jieba as its Chinese segmenter, and Stanza supports it as a tokenizer.

pypinyin matches pinyin by whole words, so it handles characters with more than one reading. It also writes zhuyin (Bopomofo) and Wade-Giles. When a wrong word split gives a wrong reading, segment the text with jieba first and pass in the list of words. For readings that are still wrong, add your own with load_phrases_dict() or load_single_dict().

OpenCC converts between Traditional and Simplified Chinese, including the regional wording of Mainland China, Taiwan, and Hong Kong. It's dictionary-based, not a language model, so the results are predictable and it runs offline. Install it with pip install opencc, then create a converter from a config file: opencc.OpenCC('s2t.json').convert(text). The config picks the direction and the variant: s2t.json gives OpenCC's standard Traditional characters, s2tw.json gives Taiwan's, and s2twp.json also swaps in Taiwan's phrases.

pangu.py inserts spaces between CJK characters and letters, digits, and symbols. Call pangu.space_text() on a string or pangu.space_file() on a file. From the command line, pangu-py -c prints the corrected text and exits with 1 when the spacing needed fixing.

Check the license of the models and data, not only the library. spaCy is MIT, but its Spanish pipelines are GPL and its Italian ones are for non-commercial use only. NLTK is Apache, and its corpora come under various licenses, each listed in its own README.