Optimize. No need to test everything against everything. Make buckets for overlapping collision layers. Possibility to have static colliders, that never move and don't collide with each other.