Not able to bring docker and flannel interfaces on the same subnet

[LalatenduMohanty]

I am following https://github.com/scollier/SATraining/blob/master/configFlannel.md for setting up flannel.
But after doing the configuration as per the SATraining docs, I am not able to bring docker and flannel interfaces in to the same subnet.
To fix the issue, I noticed that the step that involve instructing systemd to import and read the /run/flannel/subnet.env for /usr/lib/systemd/system/docker.service is missing as compared to http://www.colliernotes.com/2015/01/flannel-and-docker-on-fedora-getting.html

However in Fedora 21 Atomic "root" does not privilege to modify to /usr/lib/systemd/system/docker.service.
I tried copying the the docker systemd unit file i.e. /usr/lib/systemd/system/docker.service to /etc/systemd/system/ and modify it there. However it is still not working for me.

[eparis]

Any modification of unit files is wrong. colliernotes is out of date. You should be following the notes in the SATraining. What's the actual problem (after you undo whatever .service file changes you did in /etc)

[LalatenduMohanty]

@eparis the issue is after doing the configuration as per the SATraining docs, I am not able to bring docker and flannel interfaces in to the same subnet.

I have updated the issue as I did not mention it clearly in the issue.

[eparis]

Any modification of .service files is wrong. You should delete any files you created in /etc/systemd/system/.

There is a problem where docker will not delete/recreate the docker0 interface if docker came up and created docker0 with the wrong addresses. That is why we called for a reboot after configuring and enabling flannel. (Although it's possible, as the doc says) to stop docker, delete docker0, and restart docker.

After a reboot, can you send copies of the state of the system? What interfaces exist? What's in /run/flannel/* etc?

[LalatenduMohanty]

I had performed a reboot after the initial configuration (enabling flannel). But it did not help. Here are the information after the reboot.

#################################
bash-4.3# cd /run/flannel/
-bash-4.3# ls
subnet.env
-bash-4.3# cat subnet.env
FLANNEL_SUBNET=18.0.29.1/24
FLANNEL_MTU=1450
#################################

-bash-4.3# ifconfig
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.42.1 netmask 255.255.0.0 broadcast 0.0.0.0
ether 56:84:7a:fe:97:99 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.70.35.229 netmask 255.255.254.0 broadcast 10.70.35.255
inet6 2620:52:0:4622:5054:ff:feac:1b19 prefixlen 64 scopeid 0x0
inet6 fe80::5054:ff:feac:1b19 prefixlen 64 scopeid 0x20
ether 52:54:00:ac:1b:19 txqueuelen 1000 (Ethernet)
RX packets 1715164 bytes 86594227 (82.5 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 940785 bytes 61785636 (58.9 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 18.0.29.0 netmask 255.255.0.0 broadcast 0.0.0.0
inet6 fe80::c47:a8ff:fea2:4c4e prefixlen 64 scopeid 0x20
ether 0e:47:a8:a2:4c:4e txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 7 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 0 (Local Loopback)
RX packets 920658 bytes 48480114 (46.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 920658 bytes 48480114 (46.2 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

[eparis]

What version of flannel? Did you edit /usr/lib/systemd/system/flanneld.service ? Did you create a new flanneld.service in /etc? You'll notice the service is supposed to have:

[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
Before=docker.service

[Service]
Type=notify
EnvironmentFile=/etc/sysconfig/flanneld
EnvironmentFile=-/etc/sysconfig/docker-network
ExecStart=/usr/bin/flanneld -etcd-endpoints=${FLANNEL_ETCD} -etcd-prefix=${FLANNEL_ETCD_KEY} $FLANNEL_OPTIONS
ExecStartPost=/usr/libexec/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker

[Install]
RequiredBy=docker.service

Which will create /run/flannel/docker. But you don't have a /run/flannel/docker. Which leads me to believe you didn't undo all of the .service changes you made (or you are running some version of etcd other than RHEL7/recent Fedora)

[LalatenduMohanty]

I have not modified /usr/lib/systemd/system/flanneld.service. The flanneld version is "0.2.0". I have not created a new flanneld service. But I have modified the /etc/sysconfig/flanneld as per the guide.

I am using Fedora-Cloud-Atomic-20141203-21.x86_64.qcow2 . Also I don't see run/flannel/docker file. I had not done any change except creating a docker service file in etc/systemd/system/, which I have deleted after your first comment.

-bash-4.3# flanneld --version
0.2.0
########################################################
-bash-4.3# cat /usr/lib/systemd/system/flanneld.service
#Systemd unit file for running flanneld

[Unit]
Description=Flanneld overlay address etcd agent
After=network.target

[Service]
Type=notify
EnvironmentFile=/etc/sysconfig/flanneld
ExecStart=/usr/bin/flanneld -etcd-endpoints=${FLANNEL_ETCD} -etcd-prefix=${FLANNEL_ETCD_KEY} ${FLANNEL_OPTIONS}

[Install]
WantedBy=docker.service

#####################################################################

-bash-4.3# cat /etc/sysconfig/flanneld

Flanneld configuration options

etcd url location. Point this to the server where etcd runs

FLANNEL_ETCD="http://10.70.35.229:4001"

etcd config key. This is the configuration key that flannel queries

For address range assignment

FLANNEL_ETCD_KEY="/coreos.com/network"

Any additional options that you want to pass

FLANNEL_OPTIONS="eth0"

[eparis]

f21 flannel needs updated :-( I'll work on it now.

[scollier]

This has been open for a while. I just tested on Atomic 7.1.2. Would you mind having another look with a newer etcd, flannel, docker and k8s?

[LalatenduMohanty]

@scollier yeah works fine with CentOS 7, I will also check the latest upstream bits.