Split \(Q\) projection and attention out projection into experts, with one router coordinating them.
Better than MHA performanec.
MOEReview Zhang: Mixure of Attention HeadsMOEReview Zhang: Mixure of Attention HeadsSplit \(Q\) projection and attention out projection into experts, with one router coordinating them.
Better than MHA performanec.