Rocket 0.60.x fails to start - no prebuilt fibers available for node v8.x

[reetp]

On trying to restart after manual upgrade from 59.x I could see rocket failing to start in the logs.

A manual start showed:

[root@test Rocket.Chat]# node main.js
module.js:540
throw err;
^

Error: Cannot find module 'fibers'
at Function.Module._resolveFilename (module.js:538:15)
at Function.Module._load (module.js:468:25)
at Module.require (module.js:587:17)
at require (internal/module.js:11:18)
at Object. (/opt/Rocket.Chat/programs/server/boot.js:1:75)
at Module._compile (module.js:643:30)
at Object.Module._extensions..js (module.js:654:10)
at Module.load (module.js:556:32)
at tryModuleLoad (module.js:499:12)
at Function.Module._load (module.js:491:3)

I tried to install fibers

npm install fibers

No package available. Could not build locally.

Looking at the fibers page it seems that there are prebuilt packages up to node v6.x

After that it requires a manual build. So the upgrade to node v8 breaks the install.

The server installation does not have any build packages on it so I am unable to build fibers locally (and probably won't have sufficiently new build packages to be able to do so either). I am also not keen on having build packages n a production server. So the devs decision to leap from node 4.x to 8.x breaks the install.

https://www.npmjs.com/package/fibers
"If you are running NodeJS version 4.x, 5.x, or 6.x on Linux, OS X, or Windows (7 or later) then you should be able to install fibers from npm just fine. If you are running an older (or newer) version of node or some other operating system you will have to compile fibers on your system."

Server Setup Information:

• Version of Rocket.Chat Server: 0.60.4-rc.1
• Operating System: CentOS 6
• Deployment Method(snap/docker/tar/etc): tar
• Number of Running Instances: 1
• DB Replicaset Oplog:
• Node Version: 8.9.4
  [root@test Rocket.Chat]# uname -a
  Linux test 2.6.32-696.18.7.el6.x86_64 Refactored and reorganised entire css/less files and logic. #1 SMP Thu Jan 4 17:31:22 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Steps to Reproduce:

Manually update
Try to run main.js

Expected behavior:

Rocket runs

Actual behaviour:

Rocket fails to start as fibers is not installed

NPM install fibers

Install log

[root@test Rocket.Chat]# npm install fibers

fibers@2.0.0 install /opt/Rocket.Chat/node_modules/fibers
node build.js || nodejs build.js

linux-x64-57 exists; testing
Problem with the binary; manual build incoming
make: Entering directory /opt/Rocket.Chat/node_modules/fibers/build' CXX(target) Release/obj.target/fibers/src/fibers.o make: g++: Command not found make: *** [Release/obj.target/fibers/src/fibers.o] Error 127 make: Leaving directory /opt/Rocket.Chat/node_modules/fibers/build'
gyp ERR! build error
gyp ERR! stack Error: make failed with exit code: 2
gyp ERR! stack at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:258:23)
gyp ERR! stack at emitTwo (events.js:126:13)
gyp ERR! stack at ChildProcess.emit (events.js:214:7)
gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:198:12)
gyp ERR! System Linux 2.6.32-696.18.7.el6.x86_64
gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild" "--release"
gyp ERR! cwd /opt/Rocket.Chat/node_modules/fibers
gyp ERR! node -v v8.9.4
gyp ERR! node-gyp -v v3.6.2
gyp ERR! not ok
node-gyp exited with code: 1
Please make sure you are using a supported platform and node version. If you
would like to compile fibers on this machine please make sure you have setup your
build environment--
Windows + OS X instructions here: https://github.com/nodejs/node-gyp
Ubuntu users please run: sudo apt-get install g++ build-essential
Alpine users please run: sudo apk add python make g++
sh: nodejs: command not found
npm WARN enoent ENOENT: no such file or directory, open '/opt/Rocket.Chat/package.json'
npm WARN Rocket.Chat No description
npm WARN Rocket.Chat No repository field.
npm WARN Rocket.Chat No README data
npm WARN Rocket.Chat No license field.

npm ERR! file sh
npm ERR! code ELIFECYCLE
npm ERR! errno ENOENT
npm ERR! syscall spawn
npm ERR! fibers@2.0.0 install: node build.js || nodejs build.js
npm ERR! spawn ENOENT
npm ERR!
npm ERR! Failed at the fibers@2.0.0 install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2018-01-09T16_22_31_833Z-debug.log

As a test I attempted to run it using node v6.x
npm -g install fibers future
cd /opt/Rocket.Chat/programs/server
npm install fibers future

node main.js

Log:
.....
Will load cache for rocketchat_permissions
74 records load from rocketchat_permissions
Will load cache for rocketchat_roles
11 records load from rocketchat_roles
CROWD ➔ info Disabling CROWD user sync
CustomOAuth ➔ debug Init CustomOAuth dolphin { serverURL: '',
authorizePath: '/m/oauth2/auth/',
tokenPath: '/m/oauth2/token/',
identityPath: '/m/oauth2/api/me/',
scope: 'basic',
addAutopublishFields:
{ forLoggedInUser: [ 'services.dolphin' ],
forOtherUsers: [ 'services.dolphin.name' ] } }
CustomOAuth ➔ debug Init CustomOAuth drupal { serverURL: '',
identityPath: '/oauth2/UserInfo',
authorizePath: '/oauth2/authorize',
tokenPath: '/oauth2/token',
scope: 'openid email profile offline_access',
tokenSentVia: 'payload',
usernameField: 'preferred_username',
mergeUsers: true,
addAutopublishFields:
{ forLoggedInUser: [ 'services.drupal' ],
forOtherUsers: [ 'services.drupal.name' ] } }
CustomOAuth ➔ debug Init CustomOAuth github_enterprise { serverURL: '',
identityPath: '/api/v3/user',
authorizePath: '/login/oauth/authorize',
tokenPath: '/login/oauth/access_token',
addAutopublishFields:
{ forLoggedInUser: [ 'services.github-enterprise' ],
forOtherUsers: [ 'services.github-enterprise.username' ] } }
CAS ➔ info Disabling CAS login service
CustomOAuth ➔ debug Init CustomOAuth gitlab { serverURL: 'https://gitlab.com',
identityPath: '/api/v3/user',
scope: 'api',
addAutopublishFields:
{ forLoggedInUser: [ 'services.gitlab' ],
forOtherUsers: [ 'services.gitlab.username' ] } }
packages/meteor.js:1254
Fiber(runWithEnvironment).run();
^

TypeError: Object.getOwnPropertyDescriptors is not a function
at insert (packages/mongo/collection.js:435:12)
at ModelsBaseDb.insert (packages/rocketchat:lib/server/models/_BaseDb.js:245:33)
at insert (packages/rocketchat:lib/server/models/_Base.js:95:19)
at EventEmitter.RocketChat.sendMessage (packages/rocketchat:lib/server/functions/sendMessage.js:51:45)
at ErrorHandler.trackError (packages/rocketchat:error-handler/server/lib/RocketChat.ErrorHandler.js:58:15)
at Object.Meteor._debug (packages/rocketchat:error-handler/server/lib/RocketChat.ErrorHandler.js:34:9)
at onException (packages/meteor.js:1222:14)
at runWithEnvironment (packages/meteor.js:1245:9)

[reetp]

This possibly refers:
#9167

One wonders why would I want a build system installed on my production server?

Prior versions installed without having to build anything. Now it seems you need a load of build dependencies just to install a module.

This is a retrograde step IMHO. I will struggle to upgrade/deploy Rocket now.

[dariosusman]

Indeed. Same issue here. I had to struggle with the nodejs upgrade.
I also have this error when running:

Exception in callback of async function: Error: Cannot find module '/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/rocketchat_google-vision/node_modules/grpc/src/node/extension_binary/node-v57-linux-x64/grpc_node.node'
    at Function.Module._resolveFilename (module.js:536:15)
    at Function.Module._load (module.js:466:25)
    at Module.require (module.js:579:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/rocketchat_google-vision/node_modules/grpc/src/node/src/grpc_extension.js:30:15)
    at Module._compile (module.js:635:30)
    at Object.Module._extensions..js (module.js:646:10)
    at Module.load (module.js:554:32)
    at tryModuleLoad (module.js:497:12)
    at Function.Module._load (module.js:489:3)
    at Module.require (module.js:579:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/rocketchat_google-vision/node_modules/grpc/src/node/src/client.js:38:12)
    at Module._compile (module.js:635:30)
    at Object.Module._extensions..js (module.js:646:10)
    at Module.load (module.js:554:32)
    at tryModuleLoad (module.js:497:12)
    at Function.Module._load (module.js:489:3)
    at Module.require (module.js:579:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/rocketchat_google-vision/node_modules/grpc/src/node/index.js:30:14)
    at Module._compile (module.js:635:30)
    at Object.Module._extensions..js (module.js:646:10)
    at Module.load (module.js:554:32)
    at tryModuleLoad (module.js:497:12)
    at Function.Module._load (module.js:489:3)
    at Module.require (module.js:579:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/rocketchat_google-vision/node_modules/@google-cloud/common-grpc/src/service.js:26:12)
    at Module._compile (module.js:635:30)
    at Object.Module._extensions..js (module.js:646:10)
    at Module.load (module.js:554:32)
    at tryModuleLoad (module.js:497:12)
    at Function.Module._load (module.js:489:3)
    at Module.require (module.js:579:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/rocketchat_google-vision/node_modules/@google-cloud/common-grpc/src/operation.js:36:15)
    at Module._compile (module.js:635:30)
    at Object.Module._extensions..js (module.js:646:10)
    at Module.load (module.js:554:32)
    at tryModuleLoad (module.js:497:12)
    at Function.Module._load (module.js:489:3)
    at Module.require (module.js:579:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/rocketchat_google-vision/node_modules/@google-cloud/common-grpc/src/index.js:26:21)
    at Module._compile (module.js:635:30)
    at Object.Module._extensions..js (module.js:646:10)
    at Module.load (module.js:554:32)
    at tryModuleLoad (module.js:497:12)
    at Function.Module._load (module.js:489:3)
    at Module.require (module.js:579:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/rocketchat_google-vision/node_modules/@google-cloud/vision/src/index.js:26:18)
    at Module._compile (module.js:635:30)
    at Object.Module._extensions..js (module.js:646:10)
    at Module.load (module.js:554:32)
    at tryModuleLoad (module.js:497:12)
    at Function.Module._load (module.js:489:3)
    at Module.require (module.js:579:17)
    at require (internal/module.js:11:18)
    at Object.require (/opt/Rocket.Chat/programs/server/boot.js:221:20)
    at new GoogleVision (packages/rocketchat:google-vision/server/googlevision.js:4:21)
    at googlevision.js (packages/rocketchat:google-vision/server/googlevision.js:153:27)
    at fileEvaluate (packages/modules-runtime.js:333:9)
    at require (packages/modules-runtime.js:228:16)
    at /opt/Rocket.Chat/programs/server/packages/rocketchat_google-vision.js:418:1
    at /opt/Rocket.Chat/programs/server/packages/rocketchat_google-vision.js:425:3
    at /opt/Rocket.Chat/programs/server/boot.js:339:34
    at Array.forEach (<anonymous>)
    at Function._.each._.forEach (/opt/Rocket.Chat/programs/server/node_modules/underscore/underscore.js:79:11)
    at /opt/Rocket.Chat/programs/server/boot.js:158:5
    at /opt/Rocket.Chat/programs/server/boot.js:388:5
    at Function.run (/opt/Rocket.Chat/programs/server/profile.js:510:12)
    at /opt/Rocket.Chat/programs/server/boot.js:387:11

At this stage, I'm rolling back to an old instance snapshot. You cannot update from rocket.chat-0.59.3 to
rocket.chat-0.60.3 - there's no proper documentation for anything.

[geekgonecrazy]

@rodrigok any ideas on these?

[reetp]

@dariosusman yes documentation is horrendous, particularly when making such fundamental changes.

Testing seems to be leaving a lot to be desired as well.

[reetp]

@geekgonecrazy FYI I maintain a build for Koozali SME server and this is a big problem for us.

Any suggestions on how to build the fibers module on a separate dev box and include it in the rpm I build ? Is that even possible? (Besides the issue of whether it will actually build on CentOS 6)

[RyszardK]

CentOS/RHEL 6/7 - no prebuilt binaries for fibers@2.0.0

CentOS/RHEL 7 - one can build according to this guide

CentOS/RHEL 6 - build fails because gcc is "old":

$ gcc --version
gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-18)

Workaround for CentOS/RHEL 6:

1. build fibers@2.0.0 on CentOS/RHEL 7 with this guide
2. on your CentOS/RHEL 6 server force install fibers@2.0.0:

cd $(Rocket.Chat)/programs/server
npm -f i fibers@2.0.0

3. copy binary (fibers.ndoe) built on CentOS/RHEL 7 to $(Rocket.Chat)/programs/server/node_modules/fibers/bin/linux-x64-57/
4. start Rocket.Chat (worked for me anyway)

[reetp]

@RyszardK Thanks. Yes I guessed that CentOS 6 might be 'too old', despite being supported until 30 Nov 2020 (nearly 2 years)

I don't have a CentOS 7 box, and I am not overly keen on 'force installing' stuff on stable production servers. Might be fine for me, but not sure my users would be too happy.

What a mess. Doesn't inspire confidence.

[reetp]

Hmmm. I have tried to build Fibers on my Trusty desktop as follows:

Get latest node: https://tecadmin.net/install-latest-nodejs-npm-on-ubuntu/

curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -

sudo apt-get install nodejs

sudo apt-get install g++ build-essential

sudo npm -g install n node-gyp

git clone git://github.com/laverdet/node-fibers.git

cd node-fibers

npm install

File is here:

~/git/node-fibers/bin/linux-x64-57/fibers.node

On the Rocket.Chat testbox run:

cd /opt/Rocket.Chat

npm -f i fibers@2.0.0

Answer 'Yes I am sure I want to do something rash and stupid on my nice stable production server'

On the build box:

scp ~/git/node-fibers/bin/linux-x64-57/fibers.node testbox://opt/Rocket.Chat/programs/server/node_modules/fibers/bin/linux-x64-57/

I then test start main.js but it fails again (......screams into the void):

packages/meteor.js:1254
Fiber(runWithEnvironment).run();
^

TypeError: Object.getOwnPropertyDescriptors is not a function
at insert (packages/mongo/collection.js:435:12)
at ModelsBaseDb.insert (packages/rocketchat:lib/server/models/_BaseDb.js:245:33)
at insert (packages/rocketchat:lib/server/models/_Base.js:95:19)
at EventEmitter.RocketChat.sendMessage (packages/rocketchat:lib/server/functions/sendMessage.js:51:45)
at ErrorHandler.trackError (packages/rocketchat:error-handler/server/lib/RocketChat.ErrorHandler.js:58:15)
at Object.Meteor._debug (packages/rocketchat:error-handler/server/lib/RocketChat.ErrorHandler.js:34:9)
at onException (packages/meteor.js:1222:14)
at runWithEnvironment (packages/meteor.js:1245:9)

Don't say I didn't try.... Scratching head and starting to wonder about the logic of 'latest means greatest'

Can someone also explain the -48 and -57 versions ??

[dariosusman]

As a sysadmin, I cannot waste hours like I did trying to figure out what nodejs dependencies are missing. That should have been solved or otherwise documented. I simply rolled back from a snapshot and got done.
A stable release does not become a development environment.

[rodrigok]

@reetp Can someone also explain the -48 and -57 versions ?? means you are trying to run some native code using a different version from the one use to build it.

@dariosusman Unfortunately we don't have a test environment with all OS and versions, would awesome to have it but I don't know how we could have that, we are open for suggestions.

We always run tests for all commits, PRs and release, we run the build, run it on our official server while in release candidate (open.rocket.chat), but almost everything is based in Ubuntu or Debian.

Can you guys confirm if this solution meteor/meteor#9269 (comment) solves the problem?

If yes we can add that to our documentation.

We know it could increase the steps to install in CentOS but it's wasn't our decision, we always try to keep updated with our framework and use the new NodeJS versions to bring more security and performance to our project.

Thanks for your patience and hope you can help us to fix this or even improve the documentation.

[RyszardK]

@rodrigok

Can you guys confirm if this solution meteor/meteor#9269 (comment) solves the problem?

Doesn't work for me - no g++ or build-essential packages for RedHatEnterpriseLinux 6.

[reetp]

Seems that g++ might be available via SCL. See:

https://unix.stackexchange.com/questions/102354/install-g-on-rhel-6#102355

https://superuser.com/questions/590808/yum-install-gcc-g-doesnt-work-anymore-in-centos-6-4

However I ran the suggested workaround above on ubuntu. It built the module but rocket failed to start and no idea why... maybe not seeing the module correctly but no indications.

@rodrigok I don't expect you to have test VMs for every linux flavour but certainly for the major distros. It really isn't hard with VMs in this day and age. You have dox on CentOS installation. It should be checked.

So effectively now if we run something you consider non standard (and you don't define a standard) we are cut off overnight?

"We know it could increase the steps to install in CentOS but it's wasn't our decision"

That makes no sense at all..... it suggests you did test and knew, but ignored it.

Exasperating, and no sensible solution in sight.

[reetp]

Sorry for double post... my app crashed!

[reetp]

OK, this will build fibers.node on CentOS 6 using SCL (7 should be similar) but I haven't tried to install it yet.... that's my next step.

Install node 8 packages

https://nodejs.org/en/download/package-manager/#enterprise-linux-and-fedora

curl --silent --location https://rpm.nodesource.com/setup_8.x | sudo bash -
sudo yum install -y nodejs

Install dev tools
https://www.softwarecollections.org/en/scls/rhscl/devtoolset-6/

sudo yum install centos-release-scl
sudo yum install devtoolset-6

scl enable devtoolset-6 bash

cd ~/git
git clone git://github.com/laverdet/node-fibers.git
cd node-fibers
npm install

File builds in:

~/git/node-fibers/bin/linux-x64-57/fiber.node