Install

pip install bonltk

How to use

Comming soon

Todo:

• Tokenizers:
  • [ ] Hugging face tokenizers
  • [x] sentencepiece tokenizer
  • [ ] Compare above tokenizers with botok
• WordVectors:
  • [x] Word2Vec with gensim
  • [ ] Emlo
• Language Models:
  • [ ] Huggingface transformers
  • [ ] UMLFit Language model with fastai

• Text Similarity:
  • [ ] Sentence similarity using UMLFit, like in inltk
  • [ ] Implement Text similarity techniques mention in here
  • [ ] Compare all the text similarity algorithms

Resrouce links:

• UMLFit for sequence tagging
• Text Similarities : Estimate the degree of similarity between two texts