Polygon-polygon minimum translation vector appears invalid

[Slaktus]

Calling IntersectMTV with two rotated polygons leads to strange behaviour:

• MTV is not normalized
• MTV orientation may be incorrect depending on the orientation of the collision
• Depending on orientation of intersection, the MTV only gets calculated if poly2 completely contains poly1, or at least it appears the center point of poly1 must be contained by poly2

https://imgur.com/a/CSnDjWi

Let me know what more information you need, if any. I'm using a modified version of 4ed1885 adapted to use Unity.Mathematics types and math functions. I'm also avoiding allocations by reusing cached objects rather than constructing new ones, but other than that, the code should be the same.

[Slaktus]

Better animation showing problem more clearly:
https://imgur.com/a/WSQGNmb

The cyan line is the unnormalized minimum translation vector not multiplied by the magnitude.

[Tjstretchalot]

Thank you for the detailed report! It's surprising that this is happening with triangles and rectangles. Would it be possible to dump the location of each polygon (the vertices + offset is enough) at a moment you know you have the wrong result?

[Slaktus]

Hey,

Sorry, but I've already rewritten the function, so that'll be a bit of a hassle.

If there's no error on your side, then the bugs might have been introduced by some over-optimistic optimization on my end.

[Tjstretchalot]

I'll leave this up in the hope that I will have an opportunity at some point to test some polygons by eyeballing reasonable numbers for what was in the video.