#nlp
4pages tagged with "nlp"
programming
- building a 150K-word english dictionary with llms: opengloss— synthetic encyclopedic dictionary and semantic knowledge graph — 537K sense definitions, 9.14M edges, and a live api, generated in under a week for under $1,000
- fast zero-dependency sentence splitting in python with nupunkt— pure python and rust sentence boundary detection — 91% precision, 10M+ chars/sec, zero dependencies, trained on legal text but used across domains
- tokenizer backend changes— unified tokenizer backend system eliminating the fast/slow distinction in transformers v5.0
- transformers v5.0— hugging face transformers v5.0 release overview, features, and migration guide