There's another reason why reassembly of ip fragments is done only at the destination host. Also keep in mind there might be multiple paths through the network so intermediate routers might not "see" all the fragmented packets.

As to why reassembly of ip is done, it's done to reconstruct the original packet before that packet is further processed. Until the packet is reassembled, it's not useful because one or more parts of it would be missing. (Also remember why fragementation is done to begin with, because the original packet couldn't not be transmitted, full size, across some network link. It's broken down into as many maximum size fragments that can be processed across a network link. Also, as Lucien noted for the extra processing load placed on the receiving host reassembling fragments, this is also true for the device the fragments them, which is why something like PMTU exists, i.e. to notify sender its packets are too large.)