1. Introduction 2. Model Summary 3. Model Downloads 4. Evaluation Results 5. Chat Website & API Platform 6. How to Run Locally 7. License 8. Citation 9. Contact
功能特性
- We investigate a Multi-Token Prediction (MTP) objective and prove it beneficial to model performance.
- We design an FP8 mixed precision training framework and, for the first time, validate the feasibility and effectiveness of FP8 training on an extremely large-scale model.
- Through co-design of algorithms, frameworks, and hardware, we overcome the communication bottleneck in cross-node MoE training, nearly achieving full computation-communication overlap.
项目预览
