riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-31 02:02:21 +06:00
Code Issues Actions 33 Packages Projects Releases Wiki Activity
77c8f6c627
transformers/model_cards/sagorsarker/codeswitch-spaeng-lid-lince
History
Sagor Sarker 4753816e39
added bangla-bert-base model card and also modified other model cards (#7071) 
* added bangla-bert-base

* Apply suggestions from code review

Co-authored-by: Julien Chaumond <chaumond@gmail.com>
2020-09-11 15:17:25 -04:00
..
README.md added bangla-bert-base model card and also modified other model cards (#7071) 2020-09-11 15:17:25 -04:00

README.md

language datasets license tags
es
en
lince
MIT
codeswitching
spanish-english
language-identification

codeswitch-spaeng-lid-lince

This is a pretrained model for language identification of spanish-english code-mixed data used from LinCE

This model is trained for this below repository.

https://github.com/sagorbrur/codeswitch

To install codeswitch:

pip install codeswitch

Identify Language

  • Method-1

from transformers import AutoTokenizer, AutoModelForTokenClassification, pipeline

tokenizer = AutoTokenizer.from_pretrained("sagorsarker/codeswitch-spaeng-lid-lince")

model = AutoModelForTokenClassification.from_pretrained("sagorsarker/codeswitch-spaeng-lid-lince")
lid_model = pipeline('ner', model=model, tokenizer=tokenizer)

lid_model("put any spanish english code-mixed sentence")
  • Method-2
from codeswitch.codeswitch import LanguageIdentification
lid = LanguageIdentification('spa-eng') 
text = "" # your code-mixed sentence 
result = lid.identify(text)
print(result)