tokenizersbackend does not exist

when loading a model from hugging face hub, you may encounter this error:

ValueError: Tokenizer class TokenizersBackend does not exist or is not currently imported.

why this happens

this error occurs when a model was uploaded with transformers v5.0 (which uses the new unified tokenizer backend system) but you’re trying to load it with transformers v4.x (which doesn’t know about TokenizersBackend).

in v5.0, hugging face consolidated the tokenizer architecture. the previous “slow” (python-based) and “fast” (rust-based) tokenizer distinction was replaced with a unified backend system where TokenizersBackend is the preferred rust-based implementation.

when a model author saves a tokenizer with v5.0, the tokenizer_config.json contains:

{
  "tokenizer_class": "TokenizersBackend"
}

transformers v4.x doesn’t recognize this class, causing the error.

the fix

upgrade to transformers v5.0 or later.

option 1: install the release candidate

# pip
pip install transformers --pre

# uv
uv add transformers --prerelease=allow

# or specify minimum version
pip install "transformers>=5.0.0rc0"

option 2: install from github (latest dev)

# pip
pip install git+https://github.com/huggingface/transformers/

# uv (one-off)
uv run --with "git+https://github.com/huggingface/transformers/" python script.py

verifying the fix

check your transformers version:

import transformers
print(transformers.__version__)
# should show 5.0.0rc0, 5.0.0.dev0, or later

then load the model:

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("model-name")
model = AutoModel.from_pretrained("model-name")

example: fill-mask pipeline

with v5.0 installed, pipelines work as expected:

from transformers import pipeline

fill_mask = pipeline("fill-mask", model="mjbommar/ogbert-v1-mlm")
result = fill_mask("A contract is a legally binding <|mask|> between parties.")

print(result[0])
# {'score': 0.77, 'token': 2511, 'token_str': 'agreement',
#  'sequence': 'A contract is a legally binding agreement between parties.'}

if you cannot upgrade

if you’re stuck on v4.x and need to use a v5-format model, you have limited options:

1. contact the model author - ask them to re-upload with v4.x compatibility
2. convert locally - download the model files and manually edit tokenizer_config.json to use a v4-compatible tokenizer class (requires knowing the correct class name)
3. wait for stable release - transformers v5.0 stable should be released soon

related errors

similar errors you might see with version mismatches:

references

• transformers v5 tokenizer changes
• transformers v5 migration guide
• v5.0.0rc0 release notes