tokenized text has special character prepended #19
Description
text = "Good night 😊"
encoded_input = tokenizer(text, return_tensors='tf')
print(tokenizer.convert_ids_to_tokens(encoded_input['input_ids'][0])) will print the line below. why are their Ġ characters in the tokenized text
['', 'Good', 'Ġnight', 'ĠðŁĺ', 'Ĭ', '']