[6.1.2/6.1.3/6.2.0-rc.*] Entire sidebar does not update in real time.

[iostat]

Description:

The entire sidebar containing all your conversations, etc. does not update when messages are sent, received, or viewed, nor when layout options are changed. Notification sounds still play, and the room you are in receives and displays the message as expected.

Steps to reproduce / Expected / Actual :

I've condensed these three sections since there are multiple issues all related to the sidebar updating and have slightly different manifestations:

1. Be in a channel, send a message to that channel -- if your sidebar layout is sent to extended you expect to see that message, but don't.
2. Receive a message in a channel/DM/thread, and observe that it doesn't show up in the sidebar until you refresh the web app (Cmd-R/F5)
3. Receive a message in a channel/DM/thread that you are not currently viewing -- you expect an unread notification to show up -- it will show up once you refresh the page
4. If you had an unread message upon refreshing the page and then view it (or mark as read via dropdown), the unread notification doesn't clear until the page is refreshed
5. Change your layout or grouping options, and observe nothing happening when you click those options. Refresh the page and observe your settings changes take effect.

Server Setup Information:

• Version of Rocket.Chat Server: 6.1.3 (happened in 6.1.2 as well, also 6.2.0-rc.1)
• Operating System: Linux
• Deployment Method: Docker in Kubernetes
• Number of Running Instances: 1 (happens with multiple instances as well)
• DB Replicaset Oplog: Enabled (occurs regardless if USE_NATIVE_OPLOG=true or IGNORE_CHANGE_STREAM=true are set)
• NodeJS Version: 14.21.2 - x64
• MongoDB Version: 5.0.16 (wiredTiger) -- ReplicaSet with 3 nodes -- all appear to be synced up and healthy.

Client Setup Information

• Desktop App or Browser Version: Confirmed on desktop app 3.9.3 as well relatively recent versions of Firefox, Chrome, Brave (colleagues report same symptoms)
• Operating System: MacOS / Windows / Linux

Additional context

Relevant logs:

Server:

The server appears to show an occasional error about auto-translate being disabled occasionally:

{"level":50,"time":"2023-04-17T20:02:21.016Z","pid":1,"hostname":"app-g2b7m","name":"System","msg":"Exception while invoking method autoTranslate.getSupportedLanguages","err":{"type":"errorClass","message":"Auto-Translate is disabled [error-autotranslate-disabled]","stack":"Error: Auto-Translate is disabled [error-autotranslate-disabled]\n    at MethodInvocation.autoTranslate.getSupportedLanguages (app/autotranslate/server/methods/getSupportedLanguages.ts:11:10)\n    at MethodInvocation.methodsMap.<computed> (app/lib/server/lib/debug.js:74:34)\n    at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1902:12)\n    at packages/ddp-server/livedata_server.js:1820:15\n    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1329:12)\n    at packages/ddp-server/livedata_server.js:1818:36\n    at new Promise (<anonymous>)\n    at Server.applyAsync (packages/ddp-server/livedata_server.js:1817:12)\n    at Server.apply (packages/ddp-server/livedata_server.js:1756:26)\n    at Server.call (packages/ddp-server/livedata_server.js:1738:17)\n    at Object.post (app/api/server/v1/misc.ts:554:27)\n    at app/api/server/api.js:463:96\n    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1329:12)\n    at Object._internalRouteActionHandler [as action] (app/api/server/api.js:463:39)\n    at Route._callEndpoint (packages/rocketchat_restivus/lib/route.coffee:150:32)\n    at packages/rocketchat_restivus/lib/route.coffee:59:33\n    at packages/simple_json-routes.js:100:9\n => awaited here:\n    at Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:60:12)\n    at Server.apply (packages/ddp-server/livedata_server.js:1769:22)\n    at Server.call (packages/ddp-server/livedata_server.js:1738:17)\n    at Object.post (app/api/server/v1/misc.ts:554:27)\n    at app/api/server/api.js:463:96\n    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1329:12)\n    at Object._internalRouteActionHandler [as action] (app/api/server/api.js:463:39)\n    at Route._callEndpoint (packages/rocketchat_restivus/lib/route.coffee:150:32)\n    at packages/rocketchat_restivus/lib/route.coffee:59:33\n    at packages/simple_json-routes.js:100:9","isClientSafe":true,"error":"error-autotranslate-disabled","reason":"Auto-Translate is disabled","errorType":"Meteor.Error"},"msg":"Auto-Translate is disabled [error-autotranslate-disabled]"}

Browser

When changing layout options the following error is printed to the console. The network tab shows the corresponding API calls succeeding with a 200 OK:

Uncaught TypeError: can't convert undefined to object
    _modify https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1413
    _modify https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1413
    _modifyAndNotify https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1413
    update https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1413
    _eachPossiblyMatchingDoc https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1413
    _eachPossiblyMatchingDoc https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1413
    update https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1413
    update https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1433
    n https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1523
    n https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1525
    upsert https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1433
    e https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1525
    withValue https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1
    directOp https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1525
    e https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1525
    n https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1525
    h https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1636
    emit https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1515
    emit https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1515
    t https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1515
    emit https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1515
    emit https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1515
    setupDdpConnection https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1515
    onmessage https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    forEachCallback https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    onmessage https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    dispatchEvent https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    _dispatchMessage https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    _didMessage https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    onmessage https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    dispatchEvent https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    onchunk https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    emit https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    onreadystatechange https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    EventHandlerNonNull* https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    XHRCorsObject https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    setTimeout handler*e/n.delay https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    XHRCorsObject https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    re https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    _scheduleRecv https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    onclose https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    dispatchEvent https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    onfinish https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    emit https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    onreadystatechange https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    EventHandlerNonNull* https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    XHRCorsObject https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    setTimeout handler*e/n.delay https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    XHRCorsObject https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    re https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    _scheduleRecv https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    onclose https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    dispatchEvent https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    onfinish https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    emit https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    onreadystatechange https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    EventHandlerNonNull* https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    XHRCorsObject https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    setTimeout handler*e/n.delay https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    XHRCorsObject https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    re https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    _scheduleRecv https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    onclose https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    dispatchEvent https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    onfinish https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    emit https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    onreadystatechange https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    EventHandlerNonNull* https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    XHRCorsObject https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    setTimeout handler*e/n.delay https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    XHRCorsObject https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    re https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    _scheduleRecv https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    onclose https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    dispatchEvent https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    onfinish https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393
    emit https://our.self-hosted.rocket.chat/5f55c7c3b80c898801b800bd83a41990908ca260.js?meteor_js_resource=true:1393

No relevant errors appear to be printed in the other scenarios

[iostat]

Tried 6.2.0-rc.1 as well, no luck

[iostat]

In 6.2.0-rc.1, I did a bit of digging in the minified code, that error appears to be thrown at the Object.keys call somewhere in this minified blob:

if(!v._isPlainObject(t))throw _("Modifier must be an object");t=o.clone(t);const n=g(t),i=n?o.clone(e):t;if(n){if(Object.keys(t).forEach((e=>{const n=r.isInsert&&"$setOnInsert"===e,o=E[n?"$set":e],s=t[e];if(!o)throw _("Invalid modifier specified ".concat(e));

Seems like it's coming from Meteor/MiniMongo, looks a lot like a minfied version of: https://github.com/meteor/meteor/blob/f070c254cb319b419bbb05276e67a1df13cced9c/packages/minimongo/local_collection.js#L1130-L1160 with the failing Object.keys being at Line 1142

[iostat]

^ that's happening after a user.setPreferences API call, which seems to return successfully

{"user":{"_id":"q6rNNZKxZYwD9RpaS","settings":{"preferences":{"sidebarViewMode":"extended","sidebarSortby":"alphabetical","autoImageLoad":true,"collapseMediaByDefault":false,"convertAsciiEmoji":true,"desktopNotificationDuration":0,"desktopNotifications":"all","dontAskAgainList":[],"emailNotificationMode":"nothing","enableAutoAway":true,"hideFlexTab":false,"hideRoles":false,"hideUsernames":false,"highlights":[],"idleTimeLimit":300,"language":"","messageViewMode":0,"muteFocusedConversations":true,"newMessageNotification":"chime","newRoomNotification":"door","notificationsSoundVolume":100,"saveMobileBandwidth":false,"sendOnEnter":"normal","unreadAlert":true,"useEmojis":true,"clockMode":2,"displayAvatars":true,"pushNotifications":"all","themeAppearence":"light","sidebarShowUnread":true,"sidebarShowFavorites":false,"sidebarGroupByType":true,"desktopNotificationRequireInteraction":true,"sidebarDisplayAvatar":true}}},"success":true}

[iostat]

Sometimes refreshing the page when viewing a DM gives me a "You are not authorized to view this page" and an error-invalid-user Promise rejection in console. However, navigating to the DM from the sidebar opens it up just fine. Not sure if any of this is related

[iostat]

I ran a local instance on my machine against our prod MongoDBs, so I could see unminified stack traces: Here's what I get when I try to change the grouping/sorting options (symptom number 5) Here's the backtrace, confirming it's coming from minimongo...:

Uncaught TypeError: can't convert undefined to object
    _modify local_collection.js:1152
    _modify local_collection.js:1142
    _modifyAndNotify local_collection.js:543
    update local_collection.js:421
    _eachPossiblyMatchingDoc local_collection.js:514
    _eachPossiblyMatchingDoc local_collection.js:510
    update local_collection.js:415
    update collection.js:674
    method dispatch_run-as-user.js:333
    upsert collection.js:748
    cancel userData.ts:73
    emit ev.js:10
    emit ev.js:10
    Streamer client.js:64
    emit ev.js:10
    emit ev.js:10
    setupDdpConnection client.js:26
    onmessage browser.js:185
    forEachCallback common.js:30
    onmessage browser.js:184
    dispatchEvent sockjs-0.3.4.js:87
    _dispatchMessage sockjs-0.3.4.js:1078
    _didMessage sockjs-0.3.4.js:1138
    onmessage sockjs-0.3.4.js:1285
    websocket sockjs-0.3.4.js:1284
    _try_next_protocol sockjs-0.3.4.js:1205
    _didClose sockjs-0.3.4.js:1113
    onfinish sockjs-0.3.4.js:1037
    emit sockjs-0.3.4.js:151
    onfinish sockjs-0.3.4.js:2028
    emit sockjs-0.3.4.js:151
    onreadystatechange sockjs-0.3.4.js:854
    _start sockjs-0.3.4.js:830
    XHRLocalObject sockjs-0.3.4.js:894
    setTimeout handler*module/utils.delay sockjs-0.3.4.js:405
    XHRLocalObject sockjs-0.3.4.js:893
    doXhr sockjs-0.3.4.js:2011
    InfoReceiver sockjs-0.3.4.js:1998
    setTimeout handler*module/utils.delay sockjs-0.3.4.js:405
    InfoReceiver sockjs-0.3.4.js:1998
    createInfoReceiver sockjs-0.3.4.js:2086
    SockJS sockjs-0.3.4.js:1025
    _launchConnection browser.js:172
    allowConnection index.js:5
    initEncryptedSession ecdh.ts:29
    module ecdh.ts:67
    fileEvaluate modules-runtime.js:346
    require modules-runtime.js:248
    moduleLink modules.js:365
    module app.js:345945
    fileEvaluate modules-runtime.js:346
    require modules-runtime.js:248
    require modules-runtime.js:268
    <anonymous> app.js:357587
[local_collection.js:1152:13](meteor://%F0%9F%92%BBapp/packages/minimongo/local_collection.js)
​```

[iostat]

Also, still happening in 6.2.0-rc.5

[iostat]

I did try to downgrade mongo feature compatibility version to 4.4, but the app refused to start due to

[An error occurred when creating an index for collection "users: An equivalent index already exists with the same name but different options. Requested index: { v: 2, unique: true, key: { username: 1 }, name: "username_1", sparse: true }, existing index: { v: 2, unique: true, key: { username: 1 }, name: "username_1", sparse: 1 }]

I'm assuming that's because at some point it created the indices at 5.0 feature compat version. Is it safe to drop the indices and try to restart the app with the mongo RS in feature compat 4.4? Hoping not to lose any user/message data if I do so, obviously...

[fdjohnston]

This is also happening for me on a brand new install. App is basically unusable as the sidebar does not update at all on new messages, status changes, etc. Forcing a hard-refresh redraws the sidebar properly.

Installation details:

Linux

Version
6.2.5

Deployment ID
wnZr2Evw22B9Hndp6

Apps Engine Version
1.39.1

Node Version
v14.21.3

Database Migration
294 (June 20, 2023 9:04 PM)

MongoDB
6.0.6-5 / wiredTiger (oplog Enabled)

Commit Details
HEAD: (0d0fa6f)
Branch: HEAD

[Sayan4444]

I guess the issue is solved, if not a video will be very helpful