Aborted when run #1229
Description
when I run, I only got aborted
~$ /usr/bin/mongod --config /data/data/tokumx-cluster/shard2/tokumx.conf
Mon Jun 8 22:41:47.280 warning: *****************************
Mon Jun 8 22:41:47.280 warning: No value set for expireOplogDays, using default of 14 days.
Mon Jun 8 22:41:47.280 warning: *****************************
Mon Jun 8 22:41:47.280 nssize is a deprecated parameter
all output going to: /data/data/tokumx-cluster/shard2/log/shard2.log
Aborted
log is below:
***** SERVER RESTARTED *****
Mon Jun 8 22:41:47.285 [initandlisten] TokuMX starting : pid=20477 port=27002 dbpath=/data/data/tokumx-cluster/shard2/data 64-bit host=tokumx01
Mon Jun 8 22:41:47.285 [initandlisten] TokuMX mongod server v2.0.1-mongodb-2.4.10, using TokuKV rev 39fefcc4d2b1421ea09a9e41b130a64b230583b7
Mon Jun 8 22:41:47.285 [initandlisten] git version: d454707
Mon Jun 8 22:41:47.285 [initandlisten] build info: Linux 62691929ac33 3.11.0-26-generic #45-Ubuntu SMP Tue Jul 15 04:02:06 UTC 2014 x86_64 GNU/Linux BOOST_LIB_VERSION=1_49
Mon Jun 8 22:41:47.285 [initandlisten] options: { config: "/data/data/tokumx-cluster/shard2/tokumx.conf", dbpath: "/data/data/tokumx-cluster/shard2/data", logappend: "true", logpath: "/data/data/tokumx-cluster/shard2/log/shard2.log", pluginsDir: "/usr/lib/tokumx/plugins", port: 27002, replSet: "shard2", shardsvr: "true" }
Mon Jun 8 22:41:47.285 [initandlisten] [tokumx] startup
Mon Jun 8 22:41:47 2015 TokuFT recovery starting in env /data/data/tokumx-cluster/shard2/data
Mon Jun 8 22:41:47 2015 TokuFT recovery failed -30975
Mon Jun 8 22:41:47.402 [initandlisten] Assertion: 16762:Automatic environment recovery failed. There may be data corruption.
0xc4ced3 0x880f0b 0x882075 0x87a600 0x87ad9f 0x87b5bd 0x7b0cca 0x7b13a8 0x79b89a 0x7f68963cfead 0x7adcc9
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x23) [0xc4ced3]
/usr/bin/mongod(_ZN5mongo7storage21MsgAssertionExceptionC2EiRKSs+0x9b) [0x880f0b]
/usr/bin/mongod(_ZN5mongo7storage23DataCorruptionExceptionC2EiRKSs+0xa5) [0x882075]
/usr/bin/mongod(_ZN5mongo7storage16handle_ydb_errorEi+0x6e0) [0x87a600]
/usr/bin/mongod(_ZN5mongo7storage22handle_ydb_error_fatalEi+0xf) [0x87ad9f]
/usr/bin/mongod(_ZN5mongo7storage7startupEPNS_16TxnCompleteHooksEPNS0_14UpdateCallbackE+0x5bd) [0x87b5bd]
/usr/bin/mongod(_ZN5mongo14_initAndListenEi+0x34a) [0x7b0cca]
/usr/bin/mongod(_ZN5mongo13initAndListenEi+0x18) [0x7b13a8]
/usr/bin/mongod(main+0x29a) [0x79b89a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd) [0x7f68963cfead]
/usr/bin/mongod() [0x7adcc9]
Mon Jun 8 22:41:47.408 [initandlisten] fatal error 16762: Automatic environment recovery failed. There may be data corruption.
Mon Jun 8 22:41:47.408 [initandlisten] 16762 Automatic environment recovery failed. There may be data corruption.
Mon Jun 8 22:41:47.408 [initandlisten] Fatal Assertion 16762
0xc4ced3 0xacd20c 0x87af86 0x87b5bd 0x7b0cca 0x7b13a8 0x79b89a 0x7f68963cfead 0x7adcc9
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x23) [0xc4ced3]
/usr/bin/mongod(_ZN5mongo13fassertFailedEi+0x4c) [0xacd20c]
/usr/bin/mongod(_ZN5mongo7storage22handle_ydb_error_fatalEi+0x1f6) [0x87af86]
/usr/bin/mongod(_ZN5mongo7storage7startupEPNS_16TxnCompleteHooksEPNS0_14UpdateCallbackE+0x5bd) [0x87b5bd]
/usr/bin/mongod(_ZN5mongo14_initAndListenEi+0x34a) [0x7b0cca]
/usr/bin/mongod(_ZN5mongo13initAndListenEi+0x18) [0x7b13a8]
/usr/bin/mongod(main+0x29a) [0x79b89a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd) [0x7f68963cfead]
/usr/bin/mongod() [0x7adcc9]
Mon Jun 8 22:41:47.412 [initandlisten]
***aborting after fassert() failure
Mon Jun 8 22:41:47.412
Mon Jun 8 22:41:47.412 ================================================================================
Mon Jun 8 22:41:47.412 Fatal error detected
Mon Jun 8 22:41:47.412 ================================================================================
Mon Jun 8 22:41:47.412
Mon Jun 8 22:41:47.412 About to gather debugging information, please include all of the following along
Mon Jun 8 22:41:47.412 with logs from other servers in the cluster in a bug report.
Mon Jun 8 22:41:47.412
Mon Jun 8 22:41:47.412 --------------------------------------------------------------------------------
Mon Jun 8 22:41:47.412 Version info:
Mon Jun 8 22:41:47.412
Mon Jun 8 22:41:47.412 tokumxVersion: 2.0.1
Mon Jun 8 22:41:47.412 gitVersion: d454707
Mon Jun 8 22:41:47.412 tokukvVersion: 39fefcc4d2b1421ea09a9e41b130a64b230583b7
Mon Jun 8 22:41:47.412 sysInfo: Linux 62691929ac33 3.11.0-26-generic #45-Ubuntu SMP Tue Jul 15 04:02:06 UTC 2014 x86_64 GNU/Linux BOOST_LIB_VERSION=1_49
Mon Jun 8 22:41:47.412 loaderFlags:
Mon Jun 8 22:41:47.412 compilerFlags: -fPIC -fno-strict-aliasing -ggdb -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -pipe -Wnon-virtual-dtor -Woverloaded-virtual -Wno-unused-local-typedefs -fno-builtin-memcmp -O3
Mon Jun 8 22:41:47.412 debug: false
Mon Jun 8 22:41:47.412
Mon Jun 8 22:41:47.412 --------------------------------------------------------------------------------
Mon Jun 8 22:41:47.412 Simple stacktrace:
Mon Jun 8 22:41:47.412
0xc4ced3 0x989157 0x98c5eb 0x7ae1d7 0x7f68963e31e0 0x7f68963e3165 0x7f68963e63e0 0xacd247 0x87af86 0x87b5bd 0x7b0cca 0x7b13a8 0x79b89a 0x7f68963cfead 0x7adcc9
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x23) [0xc4ced3]
/usr/bin/mongod() [0x989157]
/usr/bin/mongod(_ZN5mongo13dumpCrashInfoERKNS_10StringDataE+0x2b) [0x98c5eb]
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0xa7) [0x7ae1d7]
/lib/x86_64-linux-gnu/libc.so.6(+0x321e0) [0x7f68963e31e0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f68963e3165]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x180) [0x7f68963e63e0]
/usr/bin/mongod(_ZN5mongo13fassertFailedEi+0x87) [0xacd247]
/usr/bin/mongod(_ZN5mongo7storage22handle_ydb_error_fatalEi+0x1f6) [0x87af86]
/usr/bin/mongod(_ZN5mongo7storage7startupEPNS_16TxnCompleteHooksEPNS0_14UpdateCallbackE+0x5bd) [0x87b5bd]
/usr/bin/mongod(_ZN5mongo14_initAndListenEi+0x34a) [0x7b0cca]
/usr/bin/mongod(_ZN5mongo13initAndListenEi+0x18) [0x7b13a8]
/usr/bin/mongod(main+0x29a) [0x79b89a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd) [0x7f68963cfead]
/usr/bin/mongod() [0x7adcc9]
Mon Jun 8 22:41:47.417
Mon Jun 8 22:41:47.417 --------------------------------------------------------------------------------
Mon Jun 8 22:41:47.417 TokuKV engine backtrace:
Mon Jun 8 22:41:47.417
Mon Jun 8 22:41:47.417 Backtrace: (Note: toku_do_assert=0x0x7f6897947d90)
Mon Jun 8 22:41:47.422 /usr/lib/tokumx/libtokuportability.so(Z27db_env_do_backtrace_errfuncPFvPKviPKczES0+0x1f) [0x7f6897949a9f]
Mon Jun 8 22:41:47.422 /usr/lib/tokumx/libtokufractaltree.so(+0x14bdd) [0x7f6897b63bdd]
Mon Jun 8 22:41:47.422 /usr/bin/mongod() [0x989228]
Mon Jun 8 22:41:47.422 /usr/bin/mongod(_ZN5mongo13dumpCrashInfoERKNS_10StringDataE+0x2b) [0x98c5eb]
Mon Jun 8 22:41:47.422 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0xa7) [0x7ae1d7]
Mon Jun 8 22:41:47.422 /lib/x86_64-linux-gnu/libc.so.6(+0x321e0) [0x7f68963e31e0]
Mon Jun 8 22:41:47.422 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f68963e3165]
Mon Jun 8 22:41:47.422 /lib/x86_64-linux-gnu/libc.so.6(abort+0x180) [0x7f68963e63e0]
Mon Jun 8 22:41:47.422 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0x87) [0xacd247]
Mon Jun 8 22:41:47.422 /usr/bin/mongod(_ZN5mongo7storage22handle_ydb_error_fatalEi+0x1f6) [0x87af86]
Mon Jun 8 22:41:47.422 /usr/bin/mongod(_ZN5mongo7storage7startupEPNS_16TxnCompleteHooksEPNS0_14UpdateCallbackE+0x5bd) [0x87b5bd]
Mon Jun 8 22:41:47.422 /usr/bin/mongod(_ZN5mongo14_initAndListenEi+0x34a) [0x7b0cca]
Mon Jun 8 22:41:47.422 /usr/bin/mongod(_ZN5mongo13initAndListenEi+0x18) [0x7b13a8]
Mon Jun 8 22:41:47.422 /usr/bin/mongod(main+0x29a) [0x79b89a]
Mon Jun 8 22:41:47.422 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd) [0x7f68963cfead]
Mon Jun 8 22:41:47.422 /usr/bin/mongod() [0x7adcc9]
Mon Jun 8 22:41:47.422 Engine status function not available
Mon Jun 8 22:41:47.422 Memory usage:
Arena 0:
system bytes = 0
in use bytes = 0
Total (incl. mmap):
system bytes = 0
in use bytes = 0
max mmap regions = 0
max mmap bytes = 0
Backtrace: (Note: toku_do_assert=0x0x7f6897947d90)
/usr/lib/tokumx/libtokuportability.so(_Z19db_env_do_backtraceP8_IO_FILE+0x1b)[0x7f68979476bb]
/usr/bin/mongod[0x989228]
/usr/bin/mongod(_ZN5mongo13dumpCrashInfoERKNS_10StringDataE+0x2b)[0x98c5eb]
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0xa7)[0x7ae1d7]
/lib/x86_64-linux-gnu/libc.so.6(+0x321e0)[0x7f68963e31e0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7f68963e3165]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x180)[0x7f68963e63e0]
/usr/bin/mongod(_ZN5mongo13fassertFailedEi+0x87)[0xacd247]
/usr/bin/mongod(_ZN5mongo7storage22handle_ydb_error_fatalEi+0x1f6)[0x87af86]
/usr/bin/mongod(_ZN5mongo7storage7startupEPNS_16TxnCompleteHooksEPNS0_14UpdateCallbackE+0x5bd)[0x87b5bd]
/usr/bin/mongod(_ZN5mongo14_initAndListenEi+0x34a)[0x7b0cca]
/usr/bin/mongod(_ZN5mongo13initAndListenEi+0x18)[0x7b13a8]
/usr/bin/mongod(main+0x29a)[0x79b89a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x7f68963cfead]
/usr/bin/mongod[0x7adcc9]
Engine status function not available
Memory usage:
Arena 0:
system bytes = 0
in use bytes = 0
Total (incl. mmap):
system bytes = 0
in use bytes = 0
max mmap regions = 0
max mmap bytes = 0
Mon Jun 8 22:41:47.427
Mon Jun 8 22:41:47.427 --------------------------------------------------------------------------------
Mon Jun 8 22:41:47.427 Crash reason:
Mon Jun 8 22:41:47.427
Mon Jun 8 22:41:47.427 Got signal: 6 (Aborted).
Mon Jun 8 22:41:47.427
Mon Jun 8 22:41:47.427 --------------------------------------------------------------------------------
Mon Jun 8 22:41:47.427 Process info:
Mon Jun 8 22:41:47.427
Mon Jun 8 22:41:47.427 OS: Linux x86_64
Mon Jun 8 22:41:47.427 NCPU: 24
Mon Jun 8 22:41:47.427 VIRT: 824 MB
Mon Jun 8 22:41:47.427 RES: 824 MB
Mon Jun 8 22:41:47.427 PHYS: 129155 MB
Mon Jun 8 22:41:47.427 RLIMIT_CORE: 0 (soft), unlimited (hard)
Mon Jun 8 22:41:47.427 RLIMIT_CPU: unlimited (soft), unlimited (hard)
Mon Jun 8 22:41:47.427 RLIMIT_DATA: unlimited (soft), unlimited (hard)
Mon Jun 8 22:41:47.427 RLIMIT_FSIZE: unlimited (soft), unlimited (hard)
Mon Jun 8 22:41:47.427 RLIMIT_NOFILE: 655360 (soft), 655360 (hard)
Mon Jun 8 22:41:47.427 RLIMIT_STACK: 8388608 (soft), unlimited (hard)
Mon Jun 8 22:41:47.427 RLIMIT_AS: unlimited (soft), unlimited (hard)
Mon Jun 8 22:41:47.427 _SC_OPEN_MAX: 655360
Mon Jun 8 22:41:47.427 _SC_PAGESIZE: 4096
Mon Jun 8 22:41:47.427 _SC_PHYS_PAGES: 33063826
Mon Jun 8 22:41:47.427 _SC_AVPHYS_PAGES: 17045999
Mon Jun 8 22:41:47.428 _SC_NPROCESSORS_CONF: 24
Mon Jun 8 22:41:47.428 _SC_NPROCESSORS_ONLN: 24
Mon Jun 8 22:41:47.428
Mon Jun 8 22:41:47.428 --------------------------------------------------------------------------------
Mon Jun 8 22:41:47.428 Parsed server options:
Mon Jun 8 22:41:47.428
Mon Jun 8 22:41:47.428 config: "/data/data/tokumx-cluster/shard2/tokumx.conf"
Mon Jun 8 22:41:47.428 dbpath: "/data/data/tokumx-cluster/shard2/data"
Mon Jun 8 22:41:47.428 logappend: "true"
Mon Jun 8 22:41:47.428 logpath: "/data/data/tokumx-cluster/shard2/log/shard2.log"
Mon Jun 8 22:41:47.428 pluginsDir: "/usr/lib/tokumx/plugins"
Mon Jun 8 22:41:47.428 port: 27002
Mon Jun 8 22:41:47.428 replSet: "shard2"
Mon Jun 8 22:41:47.428 shardsvr: "true"
Mon Jun 8 22:41:47.428
Mon Jun 8 22:41:47.428 --------------------------------------------------------------------------------
Mon Jun 8 22:41:47.428 Filesystem information:
Mon Jun 8 22:41:47.428
Mon Jun 8 22:41:47.428 Information for dbpath "/data/data/tokumx-cluster/shard2/data":
Mon Jun 8 22:41:47.428
Mon Jun 8 22:41:47.428 type magic: 0x0000EF53
Mon Jun 8 22:41:47.428 type: ext2/3/4
Mon Jun 8 22:41:47.428 bsize: 4096
Mon Jun 8 22:41:47.428 blocks: 274983766
Mon Jun 8 22:41:47.428 bfree: 262571347
Mon Jun 8 22:41:47.428 bavail: 248597216
Mon Jun 8 22:41:47.428
Mon Jun 8 22:41:47.428 --------------------------------------------------------------------------------
Mon Jun 8 22:41:47.428 Current operations in progress:
Mon Jun 8 22:41:47.428
Mon Jun 8 22:41:47.428 { opid: 0, active: false, op: "none", ns: "", query: {}, desc: "initandlisten", threadId: "0x7f68983d5800", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: {}, timeAcquiringMicros: {} } }
Mon Jun 8 22:41:47.428
Mon Jun 8 22:41:47.428 --------------------------------------------------------------------------------
Mon Jun 8 22:41:47.428 OpDebug info:
Mon Jun 8 22:41:47.428
Mon Jun 8 22:41:47.428 none keyUpdates:0 0ms
Mon Jun 8 22:41:47.428