The frame (layer two) is created containing a packet (layer three). The IP Addresses are contained in the packet.
At layer two, frames/switches have no knowledge of IP, only MAC addresses.
In days past (long past, at this point), different manufacturers used Ethertype to identify their frames. The applications of the time would only look at frames containing their Ethertype.
The protocol field identifies the which stack (in a multi-protocol environment, like IP and IPX) the frame contents should be handed to.
Blocking the source port doesn't work, because the originator could choose any port number to send from. They send TO a specific port that the server is listening on (like 23 for Telnet). The source port could be anything, but the destingation port for Telnet is "well known" at port 23.
INSPECT / CBAC is "deep packet inspection" ... rather than just looking for a value at a certain offset (like a source MAC or source IP), it peels the frame and peels the packet to look at the contents ... then making an assessment of the contents to decide if it's traffic to block or pass based on the PERMITs and DENYs of the configuration.
Good Luck
Scott