Standard MoEs either waste computation by padding unused capacity within each expert, or drop tokens assigned to an expert when it exceeds capacity (i.e. truncate so that we don’t have to pad too much).

Method

Instead of

we do

and leverage efficient block sparse multiplication to have variably-sized experts.