backup

MySQL Online Backup with xtrabackup

My last post about backups was a while ago. Although it is still valid, it’s not advisable to use it as daily backup. For that we can use XtraBackup.

With XtraBackup, you can take online backups without interrupt the server workload(it will require a small lock to take binlog coordinates).
Today I’ll show how to take a complete backup.

Install:

To install it, I advise you to use Yum / Apt-get repos:

Centos / Redhat:

sudo yum install http://www.percona.com/downloads/percona-release/redhat/0.1-4/percona-release-0.1-4.noarch.rpm
sudo yum install percona-xtrabackup-24

Debian / Ubuntu

wget https://repo.percona.com/apt/percona-release_0.1-4.$(lsb_release -sc)_all.deb
sudo dpkg -i percona-release_0.1-4.$(lsb_release -sc)_all.deb
sudo apt-get update
sudo apt-get install percona-xtrabackup-24

Backup:

To take the backup, we will use innobackupex script:

innobackupex --user=root --password='Passw0rd!' /backups/

The script will produce some log messages, we need to verify if it ends with :

...
170429 21:07:12 completed OK!

Parallel:

Xtrabackup allows the copy of the tables to be done in parallel, you can set the number of threads by using the –parallel option:

innobackupex --user=root --password='Passw0rd!' --parallel=8 /backups/

We can observe that each threads receives a number which is displayed within []:

170429 21:12:27 [05] Copying ./ibdata1 to /backups/2017-04-29_21-12-27/ibdata1
170429 21:12:27 [06] Copying ./mysql/plugin.ibd to /backups/2017-04-29_21-12-27/mysql/plugin.ibd
170429 21:12:27 [06]        ...done
170429 21:12:27 [02] Copying ./mysql/servers.ibd to /backups/2017-04-29_21-12-27/mysql/servers.ibd
170429 21:12:27 [02]        ...done
170429 21:12:27 [03] Copying ./mysql/help_topic.ibd to /backups/2017-04-29_21-12-27/mysql/help_topic.ibd
170429 21:12:27 [07] Copying ./mysql/help_category.ibd to /backups/2017-04-29_21-12-27/mysql/help_category.ibd
170429 21:12:27 [07]        ...done

Compress:

Xtrabackup also allows you to compress your backup using –compress and –compress-threads (normally set to the same number of –parallel threads):

innobackupex --user=root --password='Passw0rd!' --parallel=8 --compress --compress-threads=8 /backups/

For example, a backup that takes 702M of disk space, now takes only 387M:

702M	/backups/2017-04-29_21-12-27
387M	/backups/2017-04-29_21-15-53

Restoring:

To restore the backup, we need to (1) decompress it with –decompress option in case we have compressed it before (we will require qpress for that) and (2) apply the log of transactions that happened while the backup was been taken (Basically a crash recovery that InnoDB does when MySQL starts):

innobackupex --decompress /backups/2017-04-29_21-18-04/
innobackupex --apply-log --use-memory=4G /backups/2017-04-29_21-18-04

To speed up the –apply-log process, we can configure –use-memory, which will behave like InnoDB Buffer Pool.

All .qp files will remain on the directory, we need to manually remove then. We can do it using below command:

find /backups/2017-04-29_21-18-04  -name "*.qp" -exec rm -f {} \;

Now we only need to copy the restored backup to the MySQL datadir and configure mysql as owner and group owner of all files in the folder.

That is it for today. Remember, if you don’t test your backups, YOU DON’T HAVE A BACKUP!

MySQL 5.6 Replication with GTID – Global Transaction ID

Hi guys, Early February Oracle released the new version of MySQL named 5.6, one of the enhancements is the GTID (Global Transaction ID)

GTID is an unique identifier which will be added at each transaction, and will be very useful on the slave. remember before we needed to set MASTER_LOG_FILE and MASTER_LOG_POS, now we don’t need them anymore.

Let’s see some new variables which we need to add to our cnf file:
gtid-mode : It will enable GTID, in order to this function work, we need to turn on log-bin and log-slave-updates
enforce-gtid-consistency : It will guarantee that only allowed command will be executed ( more information here)

Basicly, is only this what we need to enable GTID, for this tutorial I will use 2 virtual machines, black (master – 192.168.1.110) and white (slave – 192.168.1.111), see bellow my cnf file:

black.cnf

[mysqld]
...
binlog-format=MIXED
log-bin
log-slave-updates=true
gtid-mode=on 
enforce-gtid-consistency=true 
master-info-repository=TABLE
relay-log-info-repository=TABLE
sync-master-info=1
binlog-checksum=CRC32
master-verify-checksum=1
server-id=1
report-host=black

white.cnf

[mysqld]
...
binlog-format=MIXED
log-slave-updates=true
log-bin
gtid-mode=on 
enforce-gtid-consistency=true 
master-info-repository=TABLE
relay-log-info-repository=TABLE
sync-master-info=1
slave-parallel-workers=2
binlog-checksum=CRC32
master-verify-checksum=1
slave-sql-verify-checksum=1
binlog-rows-query-log_events=1
server-id=2
report-host=white

Next steep, we need to create the replication user on master (black):

GRANT REPLICATION SLAVE ON *.* TO replication@192.168.1.111 IDENTIFIED BY 'reppwd';
FLUSH PRIVILEGES;

In case you serve already has data, you can follow this post to create a backup from your master, note that now when we raw the SHOW MASTER STATUS; we see a new field named Executed_Gtid_Set, and then restore the backup on slave

Let’s start the replication, the very good stuff in here is that we don’t need to worry about the binary log and position anymore, is just simple as that with GTID:

CHANGE MASTER TO MASTER_HOST='192.168.1.110', MASTER_USER='replication', MASTER_PASSWORD='reppwd', MASTER_AUTO_POSITION=1;
START SLAVE;

At this point, you should have your replication running with GTID, lets see :

white>show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.1.110
                  Master_User: replication
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: black-bin.000002
          Read_Master_Log_Pos: 936
               Relay_Log_File: white-relay-bin.000003
                Relay_Log_Pos: 1146
        Relay_Master_Log_File: black-bin.000002
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 936
              Relay_Log_Space: 1560
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 1
                  Master_UUID: 2671c08b-7cf0-11e2-ac39-00163ebee7c2
             Master_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: 2671c08b-7cf0-11e2-ac39-00163ebee7c2:1-4
            Executed_Gtid_Set: 2671c08b-7cf0-11e2-ac39-00163ebee7c2:1-4
                Auto_Position: 1
1 row in set (0.00 sec)

That Retrieved_Gtid_Set and Executed_Gtid_Set displays the transaction that we read from master, and the transaction that we already executed.

Let’s check if the commands executed on master are replied on slave:

black> CREATE DATABASE gtid_test;
Query OK, 1 row affected (0.00 sec)

black> use gtid_test;
Database changed
black> CREATE TABLE test (id INT NOT NULL PRIMARY KEY);
Query OK, 0 rows affected (0.00 sec)

black> INSERT INTO test (id) VALUES (1), (2), (3), (4), (5);
Query OK, 5 rows affected (0.00 sec)
Records: 5  Duplicates: 0  Warnings: 0

black> 

Now let’s check it on slave:

white> USE gtid_test;
Database changed
white> SELECT * FROM test;
+----+
| id |
+----+
|  1 |
|  2 |
|  3 |
|  4 |
|  5 |
+----+
5 rows in set (0.00 sec)

white>

Done, we have our replication running with Global Transaction ID (GTID)

Consistent backup – snapshot

Hi guys, today let’s learn about how to have a consistent backup (snapshot)
First of all, in what situations do we use a snapshot?

1. Lets say that your production server now will have a replica, how do you do the first load of data in this slave? what was the master bin log position when you started the backup, during the backup process, does anyone wrote any query to db?

2. In case you want to implement an incremental backup strategy, you can take a snapshot once a week and in case you need to restore you server, you just restore the snapshot and apply the binary logs.

Then, let’s start.

To grantee this data integrity we will need 2 sessions open on master, first one to lock all databases, second one to do the copy

Then, let’s go:

Session 1:

FLUSH TABLES WITH READ LOCK;
SHOW MASTER STATUS;

You will receive an output like this:

mysql> SHOW MASTER STATUS;
+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000001 |      107 |              |                  |
+------------------+----------+--------------+------------------+
1 row in set (0.00 sec)

IMPORTANT: LEAVE THIS SESSION OPEN DURING ALL PROCESS, IF YOU CLOSE THIS SESSION, THE DATABASE LOCK WILL BE RELEASED

To do the backup, we will use mysqldump:
Session 2:

mysqldump -u usuario -p --all-databases > all_dbs.sql

After the backup is completed, make sure it ends with:
— Dump completed on xxxx-xx-xx xx:xx:xx

Done, we now ca release the lock on session 1:

UNLOCK TABLES;

Is this, we have a snapshot of our database.