「MySQL」- InnoDB: Assertion failure in thread 139783858161408 in file btr0pcur.cc line 432

  CREATED BY JENKINSBOT

问题描述

数据库所在主机磁盘发生损坏。在修复后重新启动数据库服务,在启动时产生如下错误:

2019-12-16 05:16:43 7f21f7360700  InnoDB: Assertion failure in thread 139783858161408 in file btr0pcur.cc line 432
InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) == buf_block_get_page_no(btr_pcur_get_block(cursor))
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
21:16:43 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=1
max_threads=151
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68109 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x2879060
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f21f735fe18 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x8e82e5]
/usr/sbin/mysqld(handle_fatal_signal+0x494)[0x669714]
/lib64/libpthread.so.0(+0xf7e0)[0x7f2221b2d7e0]
/lib64/libc.so.6(gsignal+0x35)[0x7f22205cd495]
/lib64/libc.so.6(abort+0x175)[0x7f22205cec75]
/usr/sbin/mysqld[0xa94d83]
/usr/sbin/mysqld[0xa37514]
/usr/sbin/mysqld[0xa3a54b]
/usr/sbin/mysqld[0x99882c]
/usr/sbin/mysqld(_ZN7handler11ha_rnd_nextEPh+0x65)[0x5a8615]
/usr/sbin/mysqld(_Z13rr_sequentialP11READ_RECORD+0x1e)[0x81c8fe]
/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x179)[0x6c6989]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x3da)[0x6c59ea]
/usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_P10SQL_I_ListI8st_orderESB_S7_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x250)[0x70afc0]
/usr/sbin/mysqld(_Z13handle_selectP3THDP13select_resultm+0x19f)[0x70b85f]
/usr/sbin/mysqld[0x6e50bd]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x27ee)[0x6e8b3e]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x370)[0x6eb540]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x112e)[0x6ec77e]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0xcf)[0x6b815f]
/usr/sbin/mysqld(handle_one_connection+0x47)[0x6b8287]
/usr/sbin/mysqld(pfs_spawn_thread+0x12d)[0x98409d]
/lib64/libpthread.so.0(+0x7aa1)[0x7f2221b25aa1]
/lib64/libc.so.6(clone+0x6d)[0x7f2220683bcd]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f21c8006c90): is an invalid pointer
Connection ID (thread ID): 2
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2019-12-16 05:16:44 28933 [Note] Plugin 'FEDERATED' is disabled.
2019-12-16 05:16:44 28933 [Note] InnoDB: Using atomics to ref count buffer pool pages
2019-12-16 05:16:44 28933 [Note] InnoDB: The InnoDB memory heap is disabled
2019-12-16 05:16:44 28933 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-12-16 05:16:44 28933 [Note] InnoDB: Memory barrier is not used
2019-12-16 05:16:44 28933 [Note] InnoDB: Compressed tables use zlib 1.2.11
2019-12-16 05:16:44 28933 [Note] InnoDB: Using Linux native AIO
2019-12-16 05:16:44 28933 [Note] InnoDB: Not using CPU crc32 instructions
2019-12-16 05:16:44 28933 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2019-12-16 05:16:44 28933 [Note] InnoDB: Completed initialization of buffer pool
2019-12-16 05:16:44 28933 [Note] InnoDB: Highest supported file format is Barracuda.
2019-12-16 05:16:44 28933 [Note] InnoDB: The log sequence numbers 905825208157 and 905825208157 in ibdata files do not match the log sequence number 905825347990 in the ib_logfiles!
2019-12-16 05:16:44 28933 [Note] InnoDB: Database was not shutdown normally!
2019-12-16 05:16:44 28933 [Note] InnoDB: Starting crash recovery.
2019-12-16 05:16:44 28933 [Note] InnoDB: Reading tablespace information from the .ibd files...
2019-12-16 05:16:44 28933 [Note] InnoDB: Restoring possible half-written data pages
2019-12-16 05:16:44 28933 [Note] InnoDB: from the doublewrite buffer...
2019-12-16 05:16:44 28933 [Note] InnoDB: 128 rollback segment(s) are active.
2019-12-16 05:16:44 28933 [Note] InnoDB: Waiting for purge to start
2019-12-16 05:16:44 28933 [Note] InnoDB: 5.6.45 started; log sequence number 905825347990
2019-12-16 05:16:44 28933 [Note] Server hostname (bind-address): '*'; port: 3306
2019-12-16 05:16:44 28933 [Note] IPv6 is available.
2019-12-16 05:16:44 28933 [Note]   - '::' resolves to '::';
2019-12-16 05:16:44 28933 [Note] Server socket created on IP: '::'.
2019-12-16 05:16:44 28933 [Note] Event Scheduler: Loaded 0 events
2019-12-16 05:16:44 28933 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.45'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)

根据错误提示,可能是InnoDB表发生损坏,并且数据库在不断重启。

问题原因

未知,以目前知识水平只能推测是InnoDB表损坏。

解决办法

既然没有找到原因,自然也谈不上解决办法。但是问题最后还是解决了,下面是误打误撞的过程:

由于数据库在不断重启,按照「14.21.2 Forcing InnoDB Recovery」指南,设置innodb_force_recovery=1后,数据库可以启动,不再自动重启。

对表进行CHECK TABLE ...操作,有三张表会导致数据库重启。由此推测这三张表存在问题,可能发生损坏。

尝试导出数据,但是不成功。即使设置innodb_force_recovery=6后,依旧无法导出数据。使用SELECT * FROM DB_NAME语句,会导致数据库重启。

接下来是神奇的地方:(1)在SELECT * ... LIMIT时,查询部分数据可以成功,但是达到某一条数据后失败。(1)在SELECT * ... ORDER BY DESC LIMIT时,查询部分数据可以成功,同样到达某条数据时失败。(3)但是这两个语句能够查询出完成数据集(可以说不能“越过”某条数据,在某条数据之后将会发生错误)。(4)如果SELECT WHERE ID = "<id>"语句,可以将所有数据查询出来。

接下来还有惊喜地方:(1)设置innodb_force_recovery=0后,数据库能够启动,而且不会一直重启。(1)我尝试使用OPTIMIZE TABLE tbl;命令,可以修复表。

附加说明

而实际上不是并不是磁盘损坏,而是文件系统损坏。

相关链接

Assertion failure in thread 139768130299648 in file btr0pcur.c line 430

参考文献

14.21.2 Forcing InnoDB Recovery
How can I fix database corruption on an InnoDB table?