I don't think there's anything to ensure it is not evaluating any four bytes of timestamp's eight bytes of values. Re-transmitted segments with newer TS values are less likely to also be matched erroneously (except for maybe echo values on SYN|ACKs) but the session might have to pay the initial rto penalty.