2

I have an issue with my db. It crashed this morning without any explaination but i know that some indexes are corrupted. I tried to restart it with the command mcsadmin restartSystem but I got a message which said a stuff like "api called failed"

So I stopped it and tried to restart it by using stopSystem and startSystem but it failed. So I stopped all by using shutdownSystem y and I have rebooted all my servers to reset all my environments.

Now, each time I check process status, I get:

Process             Module    Status            Last Status Change        Process ID
------------------  ------    ---------------   ------------------------  ----------
ProcessMonitor      um1       ACTIVE            Thu Jan  9 14:02:57 2020        9533
ServerMonitor       um1       ACTIVE            Thu Jan  9 14:16:21 2020       43455
DBRMWorkerNode      um1       ACTIVE            Thu Jan  9 14:16:22 2020       43542
ExeMgr              um1       ACTIVE            Thu Jan  9 14:16:26 2020       43737
DDLProc             um1       ACTIVE            Thu Jan  9 14:16:30 2020       43882
DMLProc             um1       ACTIVE            Thu Jan  9 14:16:34 2020       44093
mysqld              um1       MAN_OFFLINE       Thu Jan  9 14:16:39 2020

ProcessMonitor      pm1       ACTIVE            Thu Jan  9 14:01:36 2020        9502
ProcessManager      pm1       ACTIVE            Thu Jan  9 14:01:42 2020        9919
DBRMControllerNode  pm1       ACTIVE            Thu Jan  9 14:16:17 2020       24950
ServerMonitor       pm1       ACTIVE            Thu Jan  9 14:16:19 2020       24975
DBRMWorkerNode      pm1       ACTIVE            Thu Jan  9 14:16:19 2020       25009
PrimProc            pm1       ACTIVE            Thu Jan  9 14:16:23 2020       25074
WriteEngineServer   pm1       ACTIVE            Thu Jan  9 14:16:24 2020       25105

By consequences, when I try to call mcsmysql, i get a wonderful:

ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/usr/local/mariadb/columnstore/mysql/lib/mysql/mysql.sock' (111)

and this message in err.log that I don't manage to understand:

2020-01-09 14:30:01 0 [Warning] /usr/local/mariadb/columnstore/mysql//bin/mysqld: unknown variable 'loose-server_audit_syslog_info=MyColumnStoreClusterRcc'
2020-01-09 14:30:01 0 [Note] Server socket created on IP: '::'.
2020-01-09 14:30:01 0 [Note] Reading of all Master_info entries succeeded
2020-01-09 14:30:01 0 [Note] Added new Master_info '' to hash table
2020-01-09 14:30:01 0 [Note] /usr/local/mariadb/columnstore/mysql//bin/mysqld: ready for connections.
Version: '10.3.16-MariaDB-log'  socket: '/usr/local/mariadb/columnstore/mysql/lib/mysql/mysql.sock'  port: 3306  Columnstore 1.2.5-1
mysys/stacktrace.c:270(my_print_stacktrace)[0x55eefe5ab279]
sql/signal_handler.cc:209(handle_fatal_signal)[0x55eefe0e3b6f]
sigaction.c:0(__restore_rt)[0x7fc2c36945d0]
:0(__GI_raise)[0x7fc2c15712c7]
:0(__GI_abort)[0x7fc2c15729b8]
4.8.2/ostream:530(std::basic_ostream<char, std::char_traits<char> >& std::operator<< <std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*) [clone .part.9])[0x55eefde60428]
page/page0page.cc:865(page_copy_rec_list_start(buf_block_t*, buf_block_t*, unsigned char*, dict_index_t*, mtr_t*))[0x55eefe2d1c88]
btr/btr0btr.cc:3750(btr_compress(btr_cur_t*, unsigned long, mtr_t*))[0x55eefe39bb76]
btr/btr0cur.cc:5411(btr_cur_compress_if_useful(btr_cur_t*, unsigned long, mtr_t*))[0x55eefe3ab63e]
btr/btr0cur.cc:5843(btr_cur_pessimistic_delete(dberr_t*, unsigned long, btr_cur_t*, unsigned long, bool, mtr_t*))[0x55eefe3aea72]
row/row0purge.cc:153(row_purge_remove_clust_if_poss_low(purge_node_t*, unsigned long))[0x55eefe3227be]
row/row0purge.cc:198(row_purge_remove_clust_if_poss)[0x55eefe3243c9]
row/row0purge.cc:1261(row_purge)[0x55eefe325526]
que/que0que.cc:1042(que_thr_step)[0x55eefe2e546f]
include/sync0types.h:1125(my_atomic_addlint)[0x55eefe365120]
srv/srv0srv.cc:2617(srv_do_purge)[0x55eefe34babb]
pthread_create.c:0(start_thread)[0x7fc2c368cdd5]
/lib64/libc.so.6(clone+0x6d)[0x7fc2c163902d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0): is an invalid pointer
Connection ID (thread ID): 1
Status: NOT_KILLED

How can I solve it without reinstall or delete my data?

thx, Mathieu

2 Answers 2

1

Just a wild guess but have you checked on your disk space?

1

Thx jerlich for your comment. It was effectively the original issue which provoked the crash. We have finally managed to solve it, but, in fact, this current issue was cause by the fact that some indexes of our innodb tables were corrupt due to the crash so we had to do a recovery. To did it we have followed this method:

  • Stop the system with mcsadmin shutdownSystem.
  • Kill the mysql process.
  • edit /etc/my.cnf set the value to the variable innodb_force_recovery =1 (increment 1 by 1 if not works).
  • restart mysql.
  • dump our innodb tables.
  • drop those tables (take columnstore has to be stop or else you'll also loose your mcs tables).
  • stop mysql.
  • comment innodb_force_recovery in /etc/my.cnf
  • restart mysql.
  • import dump.
  • restart system with mcsadmin restartSystem.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.