123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113ied 4 ½ months__ggml_vocab_test__Äpfel__ggml_vocab_test__
__ggml_vocab_test__ __ggml_vocab_test__ __ggml_vocab_test__ __ggml_vocab_test__ __ggml_vocab_test__
__ggml_vocab_test__
__ggml_vocab_test__
__ggml_vocab_test__
__ggml_vocab_test__Hello world__ggml_vocab_test__ Hello world__ggml_vocab_test__Hello World__ggml_vocab_test__ Hello World__ggml_vocab_test__ Hello World!__ggml_vocab_test__Hello, world!__ggml_vocab_test__ Hello, world!__ggml_vocab_test__ this is 🦙.cpp__ggml_vocab_test__w048 7tuijk dsdfhu__ggml_vocab_test__нещо на Български__ggml_vocab_test__កាន់តែពិសេសអាចខលចេញ__ggml_vocab_test__🚀 (normal) 😶🌫️ (multiple emojis concatenated) ✅ (only emoji that has its own token)__ggml_vocab_test__Hello__ggml_vocab_test__ Hello__ggml_vocab_test__ Hello__ggml_vocab_test__ Hello__ggml_vocab_test__ Hello__ggml_vocab_test__ Hello Hello__ggml_vocab_test__ (__ggml_vocab_test__
=__ggml_vocab_test__' era__ggml_vocab_test__Hello, y'all! How are you 😁 ?我想在apple工作1314151天～__ggml_vocab_test__!!!!!!__ggml_vocab_test__3__ggml_vocab_test__33__ggml_vocab_test__333__ggml_vocab_test__3333__ggml_vocab_test__33333__ggml_vocab_test__333333__ggml_vocab_test__3333333__ggml_vocab_test__33333333__ggml_vocab_test__333333333__ggml_vocab_test__Cửa Việt__ggml_vocab_test__ discards__ggml_vocab_test__
🚀 (normal) 😶🌫️ (multiple emojis concatenated) ✅ 🦙🦙 3 33 333 3333 33333 333333 3333333 33333333 3.3 3..3 3...3 កាន់តែពិសេសអាច😁 ?我想在apple工作1314151天～ ------======= нещо на Български ''''''```````""""......!!!!!!?????? I've been 'told he's there, 'RE you sure? 'M not sure I'll make it, 'D you like some tea? We'Ve a'lL__ggml_vocab_test__