Docker connection issues. #46
Description
The set-up is as follows:
- two docker containers on a server
- training a model on a different server using the malmo instances in the docker containers runing on the first server.
The two containers are using the malmo:latest Dockerfile with no modification. I am launching the containers and mapping the application port to the host port of the server like this:
docker run -p 10011:10000 malmo:latest
docker run -p 10012:10000 malmo:latest
also tried to the same result:
docker run -p host_ip:10011:10000 malmo:latest
docker run -p host_ip:10012:10000 malmo:latest
From a separate server I am trying to train a model and connect it the same way as if the malmo instances would have been local, but specifying the host ip of the dokcers and the host ports. So my clients list for the Challenger and TrainingAgent would look like this: [('10.18.28.14', 10011), ('10.18.28.14', 10012)].
The issue I have is that while the initial handshake between the two instances does happen, I eventually get some errors when one of the malmo instances tries to connect to some port it shouldn't have access to.
Also, this setup is working fine when the dockers are local, on the same server as the model I am training.
Here is the sample output of the first docker:
[15:26:31] [Thread-15/INFO] [STDOUT]: [com.microsoft.Malmo.Utils.TCPInputPoller:run:191]: Listening for messages on port 10000
[15:26:31] [Client thread/INFO] [STDOUT]: [com.microsoft.Malmo.StateMachine:queueStateChange:143]: CLIENT request state: DORMANT
[15:26:31] [Client thread/INFO] [STDOUT]: [com.microsoft.Malmo.StateMachine:setState:101]: CLIENT enter state: DORMANT
[15:26:44] [Thread-17/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine$1:onCommand:443]: Received from 10.18.28.17:
[15:26:44] [Thread-17/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine$1:onCommand:444]: MALMO_REQUEST_CLIENT:0.21.0:20000:
[15:26:44] [Thread-17/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine:isReserved:154]: ==== RES: - -1500910004819
[15:26:44] [Thread-17/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine$1:reply:428]: REPLYING WITH: MALMOOK
[15:26:44] [Thread-19/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine$1:onCommand:443]: Received from 10.18.28.17:
[15:26:44] [Thread-19/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine$1:onCommand:444]: <?xml version="1.0" encoding="UTF-8" standalone="no" ?><MissionInit xmlns="http://ProjectMalmo.microsoft.com" PlatformVersion="0.21.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://ProjectMalmo.microsoft.com MissionInit.xsd"><Mission><About><Summary>Catch the pig!</Summary></About><ModSettings><MsPerTick>1</MsPerTick></ModSettings><ServerSection><ServerInitialConditions><Time><StartTime>6000</StartTime><AllowPassageOfTime>false</AllowPassageOfTime></Time><Weather>clear</Weather><AllowSpawning>false</AllowSpawning></ServerInitialConditions><ServerHandlers><FlatWorldGenerator destroyAfterUse="true" forceReset="false" generatorString="3;minecraft:bedrock,2*minecraft:dirt,minecraft:grass;1;village" seed=""/><DrawingDecorator><DrawCuboid type="air" x1="-10" x2="10" y1="4" y2="45" z1="-10" z2="10"/><DrawLine type="sand" x1="1" x2="7" y1="3" y2="3" z1="0" z2="0"/><DrawLine type="fence" x1="1" x2="7" y1="4" y2="4" z1="0" z2="0"/><DrawLine type="sand" x1="1" x2="7" y1="3" y2="3" z1="
[15:26:45] [Thread-19/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine$1:reply:428]: REPLYING WITH: MALMOOK
[15:26:45] [Client thread/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine$DormantEpisode:checkForMissionCommand:764]: Mission received: Catch the pig!
[15:26:45] [Client thread/INFO] [STDOUT]: [com.microsoft.Malmo.StateMachine:queueStateChange:143]: CLIENT request state: CREATING_HANDLERS
[15:26:45] [Client thread/INFO] [STDOUT]: [com.microsoft.Malmo.StateMachine:setState:101]: CLIENT enter state: CREATING_HANDLERS
[15:26:45] [Thread-20/INFO] [STDOUT]: [com.microsoft.Malmo.Utils.TCPInputPoller:run:191]: Listening for messages on port 10656
[15:26:45] [Client thread/INFO] [STDOUT]: [com.microsoft.Malmo.StateMachine:queueStateChange:143]: CLIENT request state: EVALUATING_WORLD_REQUIREMENTS
[15:26:45] [Client thread/INFO] [STDOUT]: [com.microsoft.Malmo.StateMachine:setState:101]: CLIENT enter state: EVALUATING_WORLD_REQUIREMENTS
[15:26:45] [Client thread/INFO] [STDOUT]: [com.microsoft.Malmo.StateMachine:queueStateChange:143]: CLIENT request state: CREATING_NEW_WORLD
[15:26:45] [Client thread/INFO] [STDOUT]: [com.microsoft.Malmo.StateMachine:setState:101]: CLIENT enter state: CREATING_NEW_WORLD
[15:26:46] [Server thread/INFO]: Starting integrated minecraft server version 1.8
[15:26:46] [Server thread/INFO]: Generating keypair
[15:26:46] [Server thread/INFO]: Converting map!
[15:26:46] [Server thread/INFO]: Scanning folders...
[15:26:46] [Server thread/INFO]: Total conversion count is 0
[15:26:46] [Server thread/INFO] [FML]: Injecting existing block and item data into this server instance
[15:26:46] [Server thread/INFO] [FML]: Applying holder lookups
[15:26:46] [Server thread/INFO] [FML]: Holder lookups applied
[15:26:46] [Server thread/INFO] [FML]: Loading dimension 0 (TEMP_10000_48960987-6f36-4612-893f-a25dca36a97f) (net.minecraft.server.integrated.IntegratedServer@5e783ed9)
[15:26:48] [Server thread/INFO] [FML]: Loading dimension 1 (TEMP_10000_48960987-6f36-4612-893f-a25dca36a97f) (net.minecraft.server.integrated.IntegratedServer@5e783ed9)
[15:26:48] [Server thread/INFO] [FML]: Loading dimension -1 (TEMP_10000_48960987-6f36-4612-893f-a25dca36a97f) (net.minecraft.server.integrated.IntegratedServer@5e783ed9)
[15:26:48] [Server thread/INFO]: Preparing start region for level 0
[15:26:48] [Server thread/INFO]: Changing view distance to 6, from 10
[15:26:49] [Thread-23/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine$1:onCommand:443]: Received from 10.18.28.17:
[15:26:49] [Thread-23/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine$1:onCommand:444]: MALMO_FIND_SERVER
[15:26:49] [Thread-23/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine$1:reply:428]: REPLYING WITH: MALMONOSERVERYET
[15:26:50] [Server thread/INFO] [STDOUT]: [com.microsoft.Malmo.StateMachine:queueStateChange:143]: SERVER request state: WAITING_FOR_MOD_READY
[15:26:50] [Server thread/INFO] [STDOUT]: [com.microsoft.Malmo.StateMachine:queueStateChange:143]: CLIENT request state: WAITING_FOR_SERVER_READY
[15:26:50] [Client thread/INFO] [STDOUT]: [com.microsoft.Malmo.StateMachine:setState:101]: CLIENT enter state: WAITING_FOR_SERVER_READY
[15:26:50] [Client thread/INFO]: Started on 39584
[15:26:50] [Server thread/INFO] [STDOUT]: [com.microsoft.Malmo.StateMachine:setState:101]: SERVER enter state: WAITING_FOR_MOD_READY
[15:26:50] [Netty Local Client IO #0/INFO] [FML]: Server protocol version 2
[15:26:50] [Netty Server IO #1/INFO] [FML]: Client protocol version 2
[15:26:50] [Netty Server IO #1/INFO] [FML]: Client attempting to join with 4 mods : FML@8.0.99.99,malmomod@0.21.0,Forge@11.14.3.1543,mcp@9.05
[15:26:50] [Netty Local Client IO #0/INFO] [FML]: [Netty Local Client IO #0] Client side modded connection established
[15:26:50] [Server thread/INFO] [STDOUT]: [com.microsoft.Malmo.StateMachine:queueStateChange:143]: SERVER request state: DORMANT
[15:26:50] [Server thread/INFO] [STDOUT]: [com.microsoft.Malmo.StateMachine:setState:101]: SERVER enter state: DORMANT
[15:26:50] [Server thread/INFO] [STDOUT]: [com.microsoft.Malmo.Server.ServerStateMachine$DormantEpisode:execute:468]: INCOMING MISSION: Received MissionInit directly through ServerStateMachine constructor.
[15:26:50] [Server thread/INFO] [STDOUT]: [com.microsoft.Malmo.Server.ServerStateMachine$DormantEpisode:onReceiveMissionInit:499]: Mission received: Catch the pig!
[15:26:50] [Server thread/INFO]: Received mission: ?9Catch the pig!
[15:26:50] [Server thread/INFO]: Source: ?a10.18.28.17
[15:26:50] [Server thread/INFO] [STDOUT]: [com.microsoft.Malmo.StateMachine:queueStateChange:143]: SERVER request state: BUILDING_WORLD
[15:26:50] [Server thread/INFO] [FML]: [Server thread] Server side modded connection established
[15:26:50] [Server thread/INFO]: Agent_1[local:E:6515ccef] logged in with entity id 0 at (647.5, 4.0, 446.5)
[15:26:50] [Server thread/INFO]: Agent_1 joined the game
[15:26:50] [Server thread/INFO] [STDOUT]: [com.microsoft.Malmo.StateMachine:setState:101]: SERVER enter state: BUILDING_WORLD
[15:26:50] [Thread-26/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine$1:onCommand:443]: Received from 10.18.28.17:
[15:26:50] [Thread-26/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine$1:onCommand:444]: MALMO_FIND_SERVER
[15:26:50] [Thread-26/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine$1:reply:428]: REPLYING WITH: MALMOS10.18.28.14:39584
[15:26:51] [Server thread/INFO] [STDOUT]: [com.microsoft.Malmo.StateMachine:queueStateChange:143]: SERVER request state: WAITING_FOR_AGENTS_TO_ASSEMBLE
[15:26:51] [Server thread/INFO] [STDOUT]: [com.microsoft.Malmo.StateMachine:setState:101]: SERVER enter state: WAITING_FOR_AGENTS_TO_ASSEMBLE
[15:26:51] [Server thread/INFO] [STDOUT]: [com.microsoft.Malmo.Server.ServerStateMachine$WaitingForAgentsEpisode:execute:878]: Experiment requires:
[15:26:51] [Server thread/INFO] [STDOUT]: [com.microsoft.Malmo.Server.ServerStateMachine$WaitingForAgentsEpisode:execute:881]: >>>> Agent_1
[15:26:51] [Server thread/INFO] [STDOUT]: [com.microsoft.Malmo.Server.ServerStateMachine$WaitingForAgentsEpisode:execute:881]: >>>> Agent_2
[15:26:52] [Client thread/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine$WaitingForServerEpisode:onClientTick:923]: ***Telling server we are ready - Agent_1
[15:26:52] [Client thread/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine$WaitingForServerEpisode:onClientTick:923]: ***Telling server we are ready - Agent_1
[15:26:53] [Client thread/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine$WaitingForServerEpisode:onClientTick:923]: ***Telling server we are ready - Agent_1
And the second, the one that errors out:
[15:26:19] [Thread-15/INFO] [STDOUT]: [com.microsoft.Malmo.Utils.TCPInputPoller:run:191]: Listening for messages on port 10000
[15:26:19] [Client thread/INFO] [STDOUT]: [com.microsoft.Malmo.StateMachine:queueStateChange:143]: CLIENT request state: DORMANT
[15:26:19] [Client thread/INFO] [STDOUT]: [com.microsoft.Malmo.StateMachine:setState:101]: CLIENT enter state: DORMANT
[15:26:44] [Thread-17/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine$1:onCommand:443]: Received from 10.18.28.17:
[15:26:44] [Thread-17/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine$1:onCommand:444]: MALMO_REQUEST_CLIENT:0.21.0:20000:
[15:26:44] [Thread-17/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine:isReserved:154]: ==== RES: - -1500910004830
[15:26:44] [Thread-17/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine$1:reply:428]: REPLYING WITH: MALMOOK
[15:26:49] [Thread-19/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine$1:onCommand:443]: Received from 10.18.28.17:
[15:26:49] [Thread-19/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine$1:onCommand:444]: MALMO_FIND_SERVER
[15:26:49] [Thread-19/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine$1:reply:428]: REPLYING WITH: MALMONOSERVER
[15:26:50] [Thread-21/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine$1:onCommand:443]: Received from 10.18.28.17:
[15:26:50] [Thread-21/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine$1:onCommand:444]: <?xml version="1.0" encoding="UTF-8" standalone="no" ?><MissionInit xmlns="http://ProjectMalmo.microsoft.com" PlatformVersion="0.21.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://ProjectMalmo.microsoft.com MissionInit.xsd"><Mission><About><Summary>Catch the pig!</Summary></About><ModSettings><MsPerTick>1</MsPerTick></ModSettings><ServerSection><ServerInitialConditions><Time><StartTime>6000</StartTime><AllowPassageOfTime>false</AllowPassageOfTime></Time><Weather>clear</Weather><AllowSpawning>false</AllowSpawning></ServerInitialConditions><ServerHandlers><FlatWorldGenerator destroyAfterUse="true" forceReset="false" generatorString="3;minecraft:bedrock,2*minecraft:dirt,minecraft:grass;1;village" seed=""/><DrawingDecorator><DrawCuboid type="air" x1="-10" x2="10" y1="4" y2="45" z1="-10" z2="10"/><DrawLine type="sand" x1="1" x2="7" y1="3" y2="3" z1="0" z2="0"/><DrawLine type="fence" x1="1" x2="7" y1="4" y2="4" z1="0" z2="0"/><DrawLine type="sand" x1="1" x2="7" y1="3" y2="3" z1="
[15:26:51] [Thread-21/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine$1:reply:428]: REPLYING WITH: MALMOOK
[15:26:51] [Client thread/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine$DormantEpisode:checkForMissionCommand:764]: Mission received: Catch the pig!
[15:26:51] [Client thread/INFO] [STDOUT]: [com.microsoft.Malmo.StateMachine:queueStateChange:143]: CLIENT request state: CREATING_HANDLERS
[15:26:51] [Client thread/INFO] [STDOUT]: [com.microsoft.Malmo.StateMachine:setState:101]: CLIENT enter state: CREATING_HANDLERS
[15:26:51] [Thread-22/INFO] [STDOUT]: [com.microsoft.Malmo.Utils.TCPInputPoller:run:191]: Listening for messages on port 10112
[15:26:51] [Client thread/INFO] [STDOUT]: [com.microsoft.Malmo.StateMachine:queueStateChange:143]: CLIENT request state: WAITING_FOR_SERVER_READY
[15:26:51] [Client thread/INFO] [STDOUT]: [com.microsoft.Malmo.StateMachine:setState:101]: CLIENT enter state: WAITING_FOR_SERVER_READY
[15:26:51] [Client thread/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine$WaitingForServerEpisode:execute:992]: We should be joining 10.18.28.14:39584
[15:26:52] [Client thread/INFO]: Connecting to 10.18.28.14, 39584
[15:26:52] [Server Connector #1/ERROR]: Couldn't connect to server
java.net.ConnectException: Connection refused: /10.18.28.14:39584
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_131]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:1.8.0_131]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:191) ~[NioSocketChannel.class:4.0.15.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:255) ~[AbstractNioChannel$AbstractNioUnsafe.class:4.0.15.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:502) ~[NioEventLoop.class:4.0.15.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:452) ~[NioEventLoop.class:4.0.15.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:346) ~[NioEventLoop.class:4.0.15.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101) ~[SingleThreadEventExecutor$2.class:4.0.15.Final]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_131]
[15:26:52] [Client thread/INFO] [STDOUT]: [com.microsoft.Malmo.StateMachine:queueStateChange:143]: CLIENT request state: ERROR_CANNOT_CONNECT_TO_SERVER
[15:26:52] [Client thread/INFO] [STDOUT]: [com.microsoft.Malmo.StateMachine:setState:101]: CLIENT enter state: ERROR_CANNOT_CONNECT_TO_SERVER
[15:26:52] [Client thread/INFO] [STDOUT]: [com.microsoft.Malmo.Client.ClientStateMachine$MissionEndedEpisode:sendMissionEnded:1915]: Sending mission ended message to 10.18.28.17:10173.
[15:26:52] [Client thread/INFO] [STDOUT]: [com.microsoft.Malmo.StateMachine:queueStateChange:143]: CLIENT request state: DORMANT
[15:26:52] [Client thread/INFO] [STDOUT]: [com.microsoft.Malmo.StateMachine:setState:101]: CLIENT enter state: DORMANT
[15:26:52] [Client thread/INFO] [STDOUT]: [com.microsoft.Malmo.Utils.TCPInputPoller:clearCommands:127]: JETTISONING 0 COMMANDS