October 22, 2012 - by Derek Crudgington
Starting with our newer machine images we've introduced a few customer specific utilities to help you in the SmartOS world: Quickbackup, Smtools, and Dtracetools. These are packages updated frequently in which anyone can install and use immediately to reap the benefits and make your life easier.
First we'll go over the basics of Quickbackup which is only available for Percona, MongoDB images and comes pre-installed.
Quickbackup is a command line tool wrapped around Percona Xtrabackup which provides a very simplistic, one command approach to backing up your data so you don't have to do any of the messy behind the scene configuration for Xtrabackup. When you provision a Percona image Quickbackup is already installed for you and you can begin backing up your MySQL data manually or on a schedule.
To take a backup manually:
# quickbackup backup * Backup for /backups/2012-10-12_19-57-41-backup.tar.gz * Backup completed successfully * Rotation completed successfully
Backups are stored at /backups with the proper time stamp. If you want to view the details of the backup you can view the Quickbackup log file at /var/log/mysql/quickbackup.log.
To restore a backup first shut down MySQL and then run the restore:
# svcadm disable mysql:percona
Verify it is down:
# svcs mysql:percona STATE STIME FMRI disabled 14:19:21 svc:/network/mysql:percona
Restore by passing in the path to the back up file:
# quickbackup restore /backups/2011-04-16_20-20-00-backup.tar.gz * Restore from /backups/2011-04-16_20-20-00-backup.tar.gz * Restore completed successfully.
To take a backup on a schedule:
For backing up on a schedule Quickbackup is already installed as a SMF service which properties must be configured to specify when you want to backup. By default the service is set to backup every night at midnight but comes disabled by default. To enable this all you need to do is enable the service:
# svcadm enable mysql-quickbackup
Your backups will begin collecting at /backups every night at midnight. To set the service to perform a backup every 15 minutes of every hour Monday, Tuesday, Wednesday, Thursday, and Friday, you would use the following configuration commands to Quickbackup:
# svccfg -s mysql-quickbackup setprop quickbackup/minute = 0,15,30,45 # svccfg -s mysql-quickbackup setprop quickbackup/hour = all # svccfg -s mysql-quickbackup setprop quickbackup/day = mon,tue,wed,thur,fri
After setting the configuration you must refresh and restart the service for changes to take:
# svcadm refresh mysql-quickbackup # svcadm restart mysql-quickbackup # svcs mysql-quickbackup STATE STIME FMRI enabled Apr_13 svc:/network/mysql-quickbackup
Quickbackup is highly configurable. To view the configuration settings:
# svcprop mysql-quickbackup | grep "quickbackup/" quickbackup/day astring all quickbackup/expiredays astring 3 quickbackup/hour astring 0 quickbackup/minute astring 0 quickbackup/username astring qb-xxxxxxxx quickbackup/password astring xxxxxxxx quickbackup/backupdir astring /backups quickbackup/stream astring xbstream
The only settings you should be concerned about are day, hour, minute, expiredays (age of backup files to be rotated), backupdir (specify where to store backups), and stream. The stream method is set to xbstream by default which allows super fast backups. In our testing we have done 10G backups in less than a minute!
Documentation for Quickbackup (Percona) can be found at http://wiki.joyent.com/wiki/display/jpc2/Joyent+Quickbackup+for+Percona+MySQL+Server
Quickbackup for MongoDB is very similar to the Percona version, only that it is powered on the back end by mongodump and mongorestore. The service comes installed by default on MongoDB images which has the SMF name svc:/pkgsrc/quickbackup-mongodb and provides backing up on a schedule. All of the commands and configuration are the same as above.
To take a backup manually:
# quickbackup-mongodb backup * Backup for /backups/2012-10-12_19-57-41-backup.tar.gz * Backup completed successfully * Rotation completed successfully
To restore a backup the MongoDB service must be running, then you can run the restore:
# svcs pkgsrc/quickbackup-mongodb STATE STIME FMRI enabled 21:11:00 svc:/pkgsrc/quickbackup-mongodb:default
Then specify which backup to restore:
# quickbackup-mongodb restore /backups/2012-03-20_20-59-48-backup.tar.gz * Restore from /backups/2012-03-20_20-59-48-backup.tar.gz * Restore completed successfully
To view the details of the backups and restores see /var/log/mongodb/quickbackup-mongodb.log. For further configuration on Quickbackup for MongoDB, see http://wiki.joyent.com/wiki/display/jpc2/Joyent+Quickbackup+for+MongoDB
Smtools is a collection of utilities (available in pkgsrc 2012Q1, 2012Q2, package 'smtools') that can help answer questions, and make tasks much more simple in your Image such as:
On newer Images smtools comes installed by default, but can be installed through pkgsrc:
# pkgin install smtools
Tools are installed into /opt/local/bin and prefixed with sm-*. Let's see an example of some of them:
# sm-summary * Gathering SmartMachine summary.. SM UUID xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx SM ID 2 Hostname getest1.local SmartOS build joyent_20120718T044524Z Image percona 1.6.0 Pkgsrc http://pkgsrc.joyent.com/sdc6/2012Q1/i386/All Processes 26 Memory (RSS) Cap 16384M Memory (RSS) Used 4137M Memory (RSS) Free 12247M Swap Cap 32768M Swap Used 9519M /tmp Used 12K Disk Quota 482G % Disk Used 2%
# sm-list-timezones * Listing timezones for xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx.. Africa/Abidjan Africa/Accra Africa/Addis_Ababa Africa/Algiers Africa/Asmara Africa/Asmera Africa/Bamako Africa/Bangui Africa/Banjul …
# sm-set-timezone Greenwich * Setting timezone for xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx.. * Changed to timezone to Greenwich. You will need to reboot.
# sm-meminfo rss * Gathering rss memory infomation for xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx.. Memory cap: 16.0 GiB Memory used: 4.0 GiB Memory free: 11.9 GiB
# sm-set-hostname testblah * Setting hostname for xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx.. * Changed hostname to testblah. You will need to reboot.
# sm-install-wordpress /home/admin * Downloading latest wordpress. * Configuring wordpress. * Created MySQL database wp_xxxxxxxx. * Created MySQL user wp_xxxxxxxxxx pass xxxxxxxxxxxxxx with access to wp_xxxxxxxx from anywhere. * Visit http://x.x.x.x/wp-admin/install.php to finish the install.
# sm-list-dbs mysql * Listing MySQL databases for xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx.. information_schema mgtest mysql performance_schema wp_a1d5001f
# sm-create-db mysql testdb1 * Created MySQL database testdb1.
# sm-tune-mysql * Tuning MySQL, this can take up to 60 seconds.. Your tuned configuration is at /mysql/my.cnf.tuned There is more to just copying this file over your old one and restarting. Review the configuration file first to make any necessary changes, and because 'innodb_log_file_size' has likely been modified you need to remove the old log files first before starting. If you don't do this properly then MySQL won't start.
# sm-rebuild-pkgsrc This script upgrades userland software in /opt/local, using packages from the following package set: http://pkgsrc.joyent.com/sdc6/2012Q2/x86_64 In the process, your /opt/local path will be backed up, new bootstrap image will be downloaded and packages that you had installed, will be re-installed from the new package set. Consequently, if you had any software or files installed under /opt/local that did not come from pkgsrc packages (when this machine was provisioned, or installed later on), it will be removed and only preserved in the backup compressed tarball. ? You do want to proceed with initial checks? y/N
See http://wiki.joyent.com/wiki/display/jpc2/SmartMachine+Tools+Package for more examples of Smtools.
The Dtracetools is a pkgsrc package available since 2011Q4 which contains many (over 100!) DTrace scripts you can use to trace and debug issues in your Image. Dtracetools is available by default on all Images, but can be installed through pkgsrc with:
# pkgin install dtracetools
The tools will be installed at /opt/local/bin with the prefix dtrace-*. There are many useful Dscripts which provide tracing of PHP, Ruby, Perl, Python, Bash, Tcl, MongoDB, MySQL, and more. Below are a few examples:
# dtrace-mongocmd.d -p `pgrep -x mongod` dtrace: script '/opt/local/bin/dtrace-mongocmd.d' matched 2 probes CPU ID FUNCTION:NAME 6 71046 :tick-1s 6 71046 :tick-1s 6 71046 :tick-1s mongo cmds/s: 18 6 71046 :tick-1s mongo cmds/s: 0 6 71046 :tick-1s mongo cmds/s: 0 6 71046 :tick-1s mongo cmds/s: 0 6 71046 :tick-1s mongo cmds/s: 0 6 71046 :tick-1s mongo cmds/s: 0 6 71046 :tick-1s mongo cmds/s: 18 6 71046 :tick-1s mongo cmds/s: 0 6 71046 :tick-1s mongo cmds/s: 0 ^C
# dtrace-mysql_pid_slow.d -p `pgrep -x mysqld` 1 Tracing... Min query time: 1000000 ns. TIME(ms) CPU(ms) QUERY 91 3 show databases\0 27 0 select * from mysql.user\0 ^C
# dtrace-mysql_pid_latency.d -p `pgrep -x mysqld` Tracing PID 10630... Hit Ctrl-C to end. MySQL queries/second: 1; query latency (ns): value ------------- Distribution ------------- count 131072 | 0 262144 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1 524288 | 0 ^C MySQL queries/second: 1; query latency (ns): value ------------- Distribution ------------- count 65536 | 0 131072 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1 262144 | 0
# dtrace-mysql_pid_avg.d -p `pgrep -x mysqld` Tracing PID 10630... TIME QUERIES 1+sec_Qs AVG(ms) MAX(ms) 2012 Oct 19 15:23:07 1 0 0 0 2012 Oct 19 15:23:08 1 0 0 0 2012 Oct 19 15:23:09 0 0 0 0 ^C
# dtrace-php_funccalls.d Tracing... Hit Ctrl-C to end. ^C FILE FUNC CALLS blah.php blastoff 4 blah.php launch 4 blah.php one 4 blah.php sleep 4 blah.php three 4 blah.php two 4
# dtrace-php_flow.d TIME(us) -- FUNC 20578568252193 <- sleep 20578568252218 -> launch 20578568252229 -> three 20578568252269 -> two 20578568252282 -> one 20578568252296 -> blastoff 20578568252314 <- blastoff 20578568252320 <- one 20578568252324 <- two 20578568252329 <- three 20578568252333 <- launch 20578568252341 -> sleep
As you can see we can easily get information from our applications using the Dtracetools package. This package is still under extreme development where some scripts may need minor tweaking. We are constantly updating http://wiki.joyent.com/wiki/display/jpc2/Collecting+Data+with+DTraceTools with updates as it develops.