Delay of connection to vm console

[SviridoffA]

problem

On the first launch, the connection is established immediately. But after some time delay starting to grow. With a cluster running for 7 days, connection delay can reach approximately 4 seconds, and continues to increase over time. But after restarting Cloudstack agent on the KVM host where the virtual machine we’re connecting to is located, the delay disappears and the connection becomes instant again. The problem returns over time.

versions

ACS - 4.19.2 , 4.19.3,4.20.2 (cloudstack agent with the same version), issue was also observed in version 4.19.1
Hypervisor - KVM
Storage - Linstor
Network - Bridges (vlan isolation)

The steps to reproduce the bug

Just let the cluster run for 7-14 days, then try checking the delay. Restarting the agent will temporarily resolve the issue.

What to do about it?

It would be great if this issue could be resolved, as we’d like to avoid restarting the agents via cron.

[weizhouapache]

@SviridoffA
does the agent mean the cloudstack-agent service on kvm host? or the cloud service in CPVM ?

[SviridoffA]

Hi @weizhouapache , I mean restarting cloudstack-agent service on kvm host.

[SviridoffA]

Some additional information. All infrastracture build on ubuntu 24.04.
Configuration of kvm host network described on picture below.

Management ip configured on cloudbr0, all other traffic types goes directly on bond0. Agent creates guest network bridges directly on bond0, and same for public network. And as a result, bridge cloudbr1 is not used now. I tried to set public.network.device and guest.network.device to cloudbr1, But cloudstack-agent ignoring it. But it doesn't matter, everything works and it suits me.

I just added this here in order to demonstrate my configuration and exclude possible mistakes in it.

[weizhouapache]

@SviridoffA
just to confirm, does 4 seconds mean the wait time when you open the vm console, or the delay of mouse or keyboard events in the vm console ?

[SviridoffA]

@weizhouapache
It's delay from click on console icon to open browser tab with console. Over the past month, I’ve been recording delays of up to 8 seconds. I restarted the agent on all nodes on the day this report was created. GitHub shows that two weeks have passed, and this time there is an increase in delay, but it’s less than usual. Currently, this delay is about 2 seconds. This might be related to the fact that I haven’t been very active in the UI over the past few weeks. Yesterday, I restarted agents on 2 of my 4 nodes, and for this 2 nodes browser tab opens instantly, while on the ones where I didn’t restart, it takes about two seconds.

[SviridoffA]

@weizhouapache
This might not be a problem if some kind of loader was present instead of the icon on click. Currently, some users don’t understand that browser is waiting for a response. They end up clicking multiple times. When the browser finally receives the response, user gets several tabs open simultaneously, some of which are non-functional, and only the last-requested tab works. If needed, I can wait for a more significant delay and record a short video demonstrating how the console opens and how restarting of agents affects this behavior.

[weizhouapache]

@weizhouapache It's delay from click on console icon to open browser tab with console. Over the past month, I’ve been recording delays of up to 8 seconds. I restarted the agent on all nodes on the day this report was created. GitHub shows that two weeks have passed, and this time there is an increase in delay, but it’s less than usual. Currently, this delay is about 2 seconds. This might be related to the fact that I haven’t been very active in the UI over the past few weeks. Yesterday, I restarted agents on 2 of my 4 nodes, and for this 2 nodes browser tab opens instantly, while on the ones where I didn’t restart, it takes about two seconds.

@SviridoffA
can you open a vm console, and search the agent.log by keyword "GetVncPort" ?

If the agent.log does not contain DEBUG info, you need to enable it

[SviridoffA]

@SviridoffA can you open a vm console, and search the agent.log by keyword "GetVncPort" ?

If the agent.log does not contain DEBUG info, you need to enable it

@weizhouapache I connected to the VM console, but unfortunately I couldn’t find any records for that keyword in /var/log/cloud.log. The default logging level is likely INFO. Could you please advise me on how to properly enable DEBUG logging? Unfortunately, I can’t find where this is configured. Is it in /usr/local/cloud/systemvm/conf/agent.properties or somewhere else?

Currently, in the log file, I see that Java exceptions are occurring periodically in recent days. They look like this:

2025-04-14 08:36:40,728 ERROR [vnc.network.NioSocket] (Thread-14042:null) Error reading from socket channel: newPosition < 0: (-1 < 0)
java.lang.IllegalArgumentException: newPosition < 0: (-1 < 0)
        at java.base/java.nio.Buffer.createPositionException(Buffer.java:318)
        at java.base/java.nio.Buffer.position(Buffer.java:293)
        at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1094)
        at com.cloud.consoleproxy.vnc.network.NioSocket.readFromSocketChannel(NioSocket.java:105)
        at com.cloud.consoleproxy.vnc.network.NioSocketInputStream.readBytesToBuffer(NioSocketInputStream.java:63)
        at com.cloud.consoleproxy.vnc.network.NioSocketInputStream.rearrangeBufferToFitSize(NioSocketInputStream.java:81)
        at com.cloud.consoleproxy.vnc.network.NioSocketInputStream.getReadBytesAvailableToFitSize(NioSocketInputStream.java:36)
        at com.cloud.consoleproxy.vnc.network.NioSocketSSLEngineManager.read(NioSocketSSLEngineManager.java:145)
        at com.cloud.consoleproxy.vnc.network.NioSocketTLSInputStream.readFromSSLEngineManager(NioSocketTLSInputStream.java:38)
        at com.cloud.consoleproxy.vnc.network.NioSocketTLSInputStream.rearrangeBufferToFitSize(NioSocketTLSInputStream.java:63)
        at com.cloud.consoleproxy.vnc.network.NioSocketInputStream.getReadBytesAvailableToFitSize(NioSocketInputStream.java:36)
        at com.cloud.consoleproxy.vnc.network.NioSocketInputStream.readAndCopyUnsignedInteger(NioSocketInputStream.java:95)
        at com.cloud.consoleproxy.vnc.network.NioSocketInputStream.readServerInit(NioSocketInputStream.java:134)
        at com.cloud.consoleproxy.vnc.network.NioSocketHandlerImpl.readServerInit(NioSocketHandlerImpl.java:94)
        at com.cloud.consoleproxy.vnc.NoVncClient.readServerInit(NoVncClient.java:497)
        at com.cloud.consoleproxy.ConsoleProxyNoVncClient.initialisationPhase(ConsoleProxyNoVncClient.java:247)
        at com.cloud.consoleproxy.ConsoleProxyNoVncClient.authenticateVNCServerThroughNioSocket(ConsoleProxyNoVncClient.java:233)
        at com.cloud.consoleproxy.ConsoleProxyNoVncClient.authenticateToVNCServer(ConsoleProxyNoVncClient.java:189)
        at com.cloud.consoleproxy.ConsoleProxyNoVncClient$1.run(ConsoleProxyNoVncClient.java:108)
        at java.base/java.lang.Thread.run(Thread.java:829

[weizhouapache]

@SviridoffA can you open a vm console, and search the agent.log by keyword "GetVncPort" ?
If the agent.log does not contain DEBUG info, you need to enable it

@weizhouapache I connected to the VM console, but unfortunately I couldn’t find any records for that keyword in /var/log/cloud.log. The default logging level is likely INFO. Could you please advise me on how to properly enable DEBUG logging? Unfortunately, I can’t find where this is configured. Is it in /usr/local/cloud/systemvm/conf/agent.properties or somewhere else?

can you search the agent.log on the kvm host where the vm is running on ?

[SviridoffA]

can you search the agent.log on the kvm host where the vm is running on ?

I opened KVM node and found the virtual machine that’s running on it. I connected to console of this vm several times. Then I went to /var/log/cloudstack/agent/agent.log and didn’t find a single line that contained word “GetVncPort.” After that, I went to /etc/cloudstack/agent/log4j-cloud.xml and changed the following values:

<!-- ======================= -->
   <!-- Setup the Root category -->
   <!-- ======================= -->

   <root>
      <level value="DEBUG"/>
      <appender-ref ref="CONSOLE"/>
      <appender-ref ref="FILE"/>
   </root>

<!-- ================ -->
   <!-- Limit categories -->
   <!-- ================ -->

   <category name="com.cloud">
     <priority value="DEBUG"/>
   </category>

<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/" debug="true">

After that, I restarted cloudstack-agent. I connected to the VM console several more times and still didn’t find “GetVncPort”. Also, I don’t see any messages with the DEBUG level. Only INFO and WARN.
I ran tail -f and I don’t see agent.log reacting to my console connections.