Reference

All functions
hftokenizers	HuggingFace Tokenizers binding for R
make_wrappers()	Generate wrapper code
post_processors_template_processing	Template Processing
pre_tokenizers_whitespace()	Whitespace pre tokenizer
tokenizer	The tokenizer class
trainers_bpe	Byte pair encoding trainer class