File Manipulation

Libraries for file manipulation.

Without installing anything, two Python file manipulation libraries cover paths and file types by name: pathlib and mimetypes. python-magic reads the bytes.

How to choose:

  • Building and joining paths, and listing directories: pathlib
  • A MIME type from a file name or URL: mimetypes
  • A file's type from its content, like the Unix file command: python-magic
  • Rerunning or reloading code when files change: watchfiles
  • Your own handler for each created, modified, or moved file: watchdog
  • Telling text file types apart, like source code from config files: Magika

Listed in editorial order, grouped by use case. Click a column header to re-sort the whole list.

Press / to search. Tags open their category. Click a row for details.

Search and filter

Results

Row number Tags

Paths and File Watching 3 projects

(Python standard library) A cross-platform, object-oriented path library.
Simple, modern and fast file watching and code reload in Python.
API and shell utilities to monitor file system events.

File Type Detection 3 projects

(Python standard library) Map filenames to MIME types.
A Python interface to the libmagic file type identification library.
ahupp/github.com/ahupp/python-magic /2,922 stars /
Detects file content types with a deep-learning model.
google/github.com/google/magika /18,752 stars /

File Manipulation guide

If you aren't sure which pathlib class you need, the docs say Path is most likely it: it makes a concrete path for the platform your code runs on. Join paths with the / operator, and list a directory with iterdir() or glob().

mimetypes maps a file name's extension to a MIME type, and a MIME type back to extensions. A type guess returns 2 values: a type for the Content-Type header, and an encoding like gzip for the Content-Encoding header. The type is None for a missing or unknown suffix.

python-magic wraps libmagic, which identifies file types by checking their headers, the way the Unix file command does. It's a thin wrapper, so install libmagic too, with apt-get install libmagic1 or brew install libmagic. Call magic.from_file(path, mime=True) for a MIME type, or magic.from_buffer() on bytes you already have.

watchfiles is built for file watching and code reload, and its Rust core groups changes into batches instead of firing once per file. watch() is a generator that yields sets of changes, and awatch() is the async version. To restart code when files change, use run_process() with a function or a command. From a shell, the watchfiles CLI does the same: watchfiles --filter python 'pytest --lf' src tests.

watchdog gives you an API and a shell tool for monitoring directories. Its quickstart has you subclass FileSystemEventHandler, override methods like on_created() and on_modified(), schedule the handler on an Observer, and start that thread. An observer skips subdirectories unless you pass recursive=True to schedule().

Magika runs a compact deep-learning model on a few chunks of a file, without loading large files entirely into memory. On its own test set, it outperforms existing approaches, especially on textual content types. Create a Magika() instance, then call identify_path() on a file or identify_bytes() on bytes you already have. Check result.ok before you read result.output.label or result.output.mime_type.

Know a project that belongs here?

Tell us what it does and why it stands out.