Sugoi

sugoi

remove async trait

NLLB-200 600M-distilled

Source

• https://huggingface.co/facebook/nllb-200-distilled-600M

Command

• ct2-transformers-converter --model facebook/nllb-200-distilled-600M --output_dir nllb-200-distilled-600m

mBART-50 many to many multilingual machine translation

Source

https://huggingface.co/facebook/mbart-large-50-many-to-many-mmt/blob/main/special_tokens_map.json

Command

ct2-transformers-converter --model facebook/mbart-large-50-many-to-many-mmt --output_dir large-many-to-many-mmt

M2M100 418M

Sources

• https://huggingface.co/facebook/m2m100_418M

Commands

ct2-transformers-converter --model facebook/m2m100_418M --output_dir 418M

JParaCrawl 3.0

Command

ct2-fairseq-converter --model_path ${src}-to-${tgt}/${size}/${size}.pretrain.pt --data_dir ${src}-to-${tgt}/${size}--source_lang {src} --target_lang ${tgt} --output_dir ${src}-${tgt}-${size}

Variables

• src=en, tgt=ja, size=small
• src=en, tgt=ja, size=base
• src=en, tgt=ja, size=big
• src= ja, tgt=en, size= small
• src= ja, tgt=en, size= base
• src= ja, tgt=en, size=big

Model Source

https://www.kecl.ntt.co.jp/icl/lirg/jparacrawl/