Arp request sent on the 'left' namespace

[un-lock-able]

I'm now confused on the rules linux follows to choose to send arp request for a certain ip. Now in rattan, the left name space has ip 10.1.1.1/32 and right has ip 10.2.1.1/32. I have created another interface with ip 10.3.1.1/16 on the right namespace.

From my understanding of arp, arp is used for searching for MAC of ips that that are in the same subnet with the host, and because the ip of the left interface has ip 10.1.1.1/32, any ip that is different from 10.1.1.1 should be considered on a different subnet, and when sending to these ips, linux should never send arp requests to try to figure out the MAC of the target, but instead send the packets according to the routing table to let its router to determine where the packet should go to. So when I try to ping 10.3.1.1 using 10.1.1.1, it should not send a arp to try to find the MAC of 10.3.1.1; instead, it should know that the target is not on the same subnet and send the packet to 10.2.1.1.

However, in my experiment, wireshark showed that pinging 10.3.1.1 using ip 10.1.1.1 will cause 10.1.1.1 to send arp request to try to find out 10.3.1.1's MAC, which is not what I expected. Where have I gone wrong?

[BobAnkh]

cc @Centaurus99

[minhuw]

What does the routing table looks like on the left namespace? You can check it with ip route show.

[minhuw]

Another thing I'm interested in is the MAC address of packets sent to 10.3.1.1. If it's MAC address of NIC with IP address 10.3.1.1. It seems that Linux treat them like in the same subnet. You could also tcpdump packets received on 10.2.1.1 to see whether packets are directly sent to 10.3.1.1.

[un-lock-able]

I've done some more experiments on this, and I think that I may have come up with an explanation and some solution.

Currently, the routing table in the left namespace is

❯ ip route
default dev vL1-L-5C9iUV proto static

and the interfaces are

❯ ip addr 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host proto kernel_lo 
       valid_lft forever preferred_lft forever
266: vL0-L-5C9iUV@if267: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 38:7e:58:e7:01:01 brd ff:ff:ff:ff:ff:ff link-netns ns-rattan-5C9iUV
    inet 192.168.1.1/32 brd 192.168.1.1 scope global vL0-L-5C9iUV
       valid_lft forever preferred_lft forever
    inet6 fe80::3a7e:58ff:fee7:101/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
272: vL1-L-5C9iUV@if273: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 38:7e:58:01:01:01 brd ff:ff:ff:ff:ff:ff link-netns ns-rattan-5C9iUV
    inet 10.1.1.1/32 brd 10.1.1.1 scope global vL1-L-5C9iUV
       valid_lft forever preferred_lft forever
    inet6 fe80::3a7e:58ff:fe01:101/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

and in on the other side, the 'right' namespace, the routes are

❯ ip route
default via 183.172.xx.xx dev wlp5s0 proto dhcp src 183.172.xx.xx metric 600 
10.1.1.1 dev vR1-R-3CjePN proto static 
10.3.0.0/16 dev br0 proto kernel scope link src 10.3.1.1 
183.172.xx.xx/21 dev wlp5s0 proto kernel scope link src 183.172.xx.xx metric 600 

❯ ip addr      
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
<links used to connect to internet omitted>
12: praguerecv@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether 7e:99:39:5c:59:5c brd ff:ff:ff:ff:ff:ff link-netns praguens
    inet6 fe80::7c99:39ff:fe5c:595c/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
14: cubicrecv@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether 56:51:25:72:d5:e6 brd ff:ff:ff:ff:ff:ff link-netns cubicns
    inet6 fe80::5451:25ff:fe72:d5e6/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
16: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a6:fd:57:f4:82:71 brd ff:ff:ff:ff:ff:ff
    inet 10.3.1.1/16 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::a4fd:57ff:fef4:8271/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
269: vR0-R-5C9iUV@if268: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 38:7e:58:e7:02:01 brd ff:ff:ff:ff:ff:ff link-netns ns-rattan-5C9iUV
    inet 192.168.2.1/32 brd 192.168.2.1 scope global vR0-R-5C9iUV
       valid_lft forever preferred_lft forever
    inet6 fe80::9c12:f3ff:fe00:1cbd/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
271: vR1-R-5C9iUV@if270: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 38:7e:58:02:01:01 brd ff:ff:ff:ff:ff:ff link-netns ns-rattan-5C9iUV
    inet 10.2.1.1/32 brd 10.2.1.1 scope global vR1-R-5C9iUV
       valid_lft forever preferred_lft forever
    inet6 fe80::9837:e4ff:fe19:6ecb/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

The IP 10.3.1.1, which in my case is the IP of the bridge br0, has a MAC different from other interfaces.

I tried to clear the routing table and reset it to the following in the left namespace, and now it can successefully ping 10.3.1.1.

❯ ip route                                                        
default via 10.2.1.1 dev vL1-L-5C9iUV proto static 
10.2.1.1 dev vL1-L-5C9iUV proto static scope link

My assumption is that, by specifying a default route with only the device name, linux assums that all IPs are reachable directly through that interface, and thus it tries to find the actual MAC address of the target by using ARP. My modified version of routing table, in contrast, tells linux that only 10.2.1.1 is directly reachable through the network interface, and all other IPs should be sent to 10.2.1.1 and routed by 10.2.1.1. I have tried to modify rattan itself to let it create route table in this way, in branch route-env.

[un-lock-able]

I'm now even more confused: there could be different number of left veth and right veth, when that happens, what should our routing strategy be?

[Centaurus99]

Perhaps this part should be determined by the actual usage method. For now, we can temporarily maintain the current strategy, only using the first veth of left and right netns for routing. Users can modify the routes themselves.

rattan/rattan-core/src/env.rs

Lines 444 to 472 in 712b922

	debug!("Set default route for left namespace");
	add_route_with_netns(
	None,
	None,
	left_veth_pairs[1].left.index,
	left_netns.clone(),
	)?;
	debug!("Set default route for right namespace");
	match config.mode {
	StdNetEnvMode::Compatible => {
	for left_veth in left_veth_pairs.iter().skip(1) {
	add_route_with_netns(
	left_veth.left.ip_addr,
	None,
	right_veth_pairs[1].right.index,
	right_netns.clone(),
	)?;
	}
	}
	_ => {
	add_route_with_netns(
	None,
	None,
	right_veth_pairs[1].right.index,
	right_netns.clone(),
	)?;
	}
	}

[Centaurus99]

Additionally, we want any veth in the left netns to directly access any veth in the right netns when manually specifying the network interface, without needing to send an ARP request. Therefore, the previously mentioned add_arp_entry part may not be removable.

[un-lock-able]

Yes, I have created another commit to create similar route table in the right namespace, letting the default route be the first veth pair. Maybe I should also create a PR from this?

[Centaurus99]

Yeah. Thanks for your effort!