This is what I know about jumbo frames
Every device in a jumbo frame enabled vlan should use jumbo frames.
Well it's obvious, if server 1 is jumbo configured and server 2 isn't, server 2 won't be able to process jumbo frames from server 1.
If server 1 it attempting to communicate to server 2 with TCP, server 2 would advertize its smaller MSS (due to non-jumbo support) and TCP should work fine.
What's not as clear (to me), for other IP communication, whether MTU fragmentation and/or PMTU would arise for end-to-end L2 communication. For instance, server 1 sends a jumbo UDP packets to server 2 on the same VLAN.
I tried to (quickly) find some information on how bridges should handle different media MTU but didn't find anything. Even if I did, and even though switches are really multiport bridges, this is the type of issue I would expect switches might not always handle correctly since they might assume media is always the same, Ethernet. (NB: not surprising, different Cisco swithes support Jumbo differently. 6500s appear to support Jumbo per port, lessor switches, Jumbo is global, see: http://www.cisco.com/en/US/products/hw/switches/ps700/products_configuration_example09186a008010edab.shtml#c10)
Certainly it would be better if all devices within the same logical L2 (VLAN) supported the same frame sizes, but the question is, is this a requirement? (At L3 interfaces, I would expect "normal" MTU and PMTU.)
Perhaps someone else can comment on pure L2?
BTW, there's appears to be a major distinction for Jumbo support based on port bandwidths. Jumbo appears to be best supported on gig (which makes sense, due to Ethernet frame timing gap requirements), but this can lead to more issues, even at L2, if you mix gig with non-gig port bandwidths across the VLAN.