KorAP-Tokenizer
2.2.5
KorAP-Tokenizer · DFA tokenizer with character offset output, large abbreviation tables and CMC support.
KorAP-Tokenizer · DFA tokenizer with character offset output, large abbreviation tables and CMC support.