Apparently this is strictly a configuraiton problem on the host. There is nothing wrong with the Cisco or Nortel routers.
Please me explain to you the data flow. Assume that the FEP sends a packet to the 1603. (The following applies to a packet from the OSA to the FEP) If the packet is too large for the 1603 to handle, the 1603 will drop the packet. As a result, the SDLC link will go down. Otherwise, the 1603 adds a DLSw header (16 bytes I think), a TCP header (normally 20 bytes), and an IP header (20 bytes). The 1603 uses TCP to tranport the packet with TCP/IP header. Depending upon your TCP setting, TCP may segment the packets into a number of different segments. Then, TCP uses IP to deliver the segment(s) to the Nortel router. If the TCP segments are too big for any routers between the DLSw peer, the router may either fragment the TCP segments or send an ICMP message, according to RFC 1191, to the 1603. The ICMP causes the 1603 to lower the TCP segment size. If any routers between the DLSw peers do not support RFC1191, the routers will drop the TCP segment. This will cause TCP detecting a gap in TCP sequence number. Eventually, TCP will disconnect the virtual circuit, which cause the peer go down.
Once all the TCP segments for the orignal SNA packet arrive at the Nortel router, the TCP stack on the Nortel router assemble the original SNA packet and put in on the token ring interface.
From the description, it looks like that the SNA packet reaches destination VTAM. VTAM finds out that the PIU is too large. If there is any problem on the IP path, I expect to see either:
1. a disconnect on the LLC2 or SDLC circuit. In other words, a DLSw circuit disconnect
or
2. a DLSw peer disconnect
Please check the modtab and find out the max RU size inside the bind. Make sure both VTAMs can handle the RU size.