Allow instance to have both v4 and v6 ephemeral IP at create time

[askfongjojo]

I tried creating this dual-stack instance with two ephemeral IP addresses and got the following validation errors:

oxide --insecure --profile local2-admin  api "/v1/instances?project=v6" --method POST --input - <<EOF
> {
>    "name":"dual-stack",
>    "description":"v4 and v6",
>    "hostname":"vm",
>    "memory":8589934592,
>    "ncpus":2,
>    "disks":[
>       {
>          "type":"attach",
>          "name":"focal1"
>       }
>    ],
>    "network_interfaces":{
>       "type":"default_dual_stack"
>    },
> "external_ips": [
>     {
>       "type": "ephemeral",
>       "pool_selector": {"ip_version": "v4", "type": "auto"}
>     },
>     {
>       "type": "ephemeral",
>       "pool_selector": {"ip_version": "v6", "type": "auto"}
>     }
>    ],
>    "start":false
> }
> EOF
error; status code: 400 Bad Request
{
  "error_code": "InvalidRequest",
  "message": "An instance may not have more than 1 ephemeral IP address",
  "request_id": "b14c4e45-301b-4cce-9b73-a12ecf56bdaa"
}
error

[bnaecker]

This error comes from here:

omicron/nexus/src/app/instance.rs

Lines 649 to 660 in 357a09e

	if params
	.external_ips
	.iter()
	.filter(|v| matches!(v, params::ExternalIpCreate::Ephemeral { .. }))
	.count()
	> MAX_EPHEMERAL_IPS_PER_INSTANCE
	{
	return Err(Error::invalid_request(&format!(
	"An instance may not have more than {} ephemeral IP address",
	MAX_EPHEMERAL_IPS_PER_INSTANCE,
	)));
	}

We're checking the total number of Ephemeral addresses is less than a maximum, but we don't distinguish between IP versions. We should allow this number per IP version.

[bnaecker]

Ok, so this is actually related to a discussion I had in chat with @david-crespo recently. Because you can create an Ephemeral IP from just a reference to an IP Pool, we actually don't know which IP versions the client is asking for here. So the best we can do is cap this at 2 total requests for an Ephemeral IP here.

There's no great solution here, but we have (at least) two bad options:

• Limit the total number of Ephemeral IPs requested to 2. In the instance create saga, if we allocate a second Ephemeral, check that it's of a different version. If we have too many, fail the saga action and unwind the whole thing.
• We could use the escape hatch I added in Support dual-stack NICs in the public API #9508, the CollectionAttach::attach_resource_with_update_condition() method. That lets users add an anything-goes validation CTE to the query that does the attaching. If that validation CTE fails, the whole attach fails. We could put this "at most 1 Ephemeral of an IP version" constraint in there. Right now, the constraint is checking that we have a VPC-private IP address of the same IP version as the public address we're trying to attach.