Its MOEReview Li: Branch-Train-Merge but MoEs now. Each layer is combined by standard moe routing with a weight that is tuned.
MOEReview Sukhbaatar: Branch-Train-MiXMOEReview Sukhbaatar: Branch-Train-MiXIts MOEReview Li: Branch-Train-Merge but MoEs now. Each layer is combined by standard moe routing with a weight that is tuned.