Why is AI training fundamentally different?

Because AI training interleaves network (all gathers / reduces) with computation (geomm), unlike usual networking where the thing you are computing (on top in stack) is detached from networking (below in stack)

Sidebar: what’s the key hard thing of exchanges

When two things happening roughly at the same time, how does one know who did something first?

*Layer 1 switching: instead of doing switching by parsing packet headers, the optical physical layer does the routing by optical routers.