Configuring NFS in SmartOS
In the past, I've written up configuring NFS in Solaris, FreeBSD, and Linux. Now I'll turn my focus to SmartOS. Briefly, NFS (network file system) provides access to remote filesystems which appear similar to local resources on client hosts. The following focuses on NFS server and client configurations in SmartOS.
Our host details are:
HOST (server): cn112 (172.25.0.49)HOST (CN (compute node): cn110 (----)HOST (SmartMachine VM): vm100 (172.25.0.100)PROMPT: HOST [0]OS: SmartOS 20130321T213641ZSmartMachine IMAGE: smartos-base64-1.9.1NOTE: The following should apply equally well to previous versions of SmartOS.
Before explaining any logic or reasoning, below is only command we really need for our SmartOS NFS server node (cn112) and the two commands for our SmartMachine client VM (vm100):
cn112 [0] /usr/sbin/zfs create -p -o quota=1.5g -o \> sharenfs='rw=@172.25.0.0/24:@172.25.1.0/25,ro=@172.25.1.128/25,root=172.25.0.100' \> zones/nfs/c9fd4db6-5da0-4b0d-8c7a-87c55b10caffvm100 [0] /usr/sbin/mount -Fnfs -o rw,intr \> 172.25.0.49:/zones/nfs/c9fd4db6-5da0-4b0d-8c7a-87c55b10caff /mnt/nfs-poolvm100 [0] /opt/local/bin/grep nfs-pool /etc/vfstab172.25.0.49:/zones/nfs/c9fd4db6-5da0-4b0d-8c7a-87c55b10caff - /mnt/nfs-pool nfs - yes -
In the following setup, I include necessary network configuration details since neither the server nor the client are currently configured for our internal NFS segment. Without further ado, on our NFS server we have no VMs, either SmartMachines or KVM instances, since we are not using this host as a traditional CN. Also, wehave 2 network interfaces, though only one (the admin NIC) iscurrently plumbed and addressed:
cn112 [0] /usr/sbin/vmadm listUUID TYPE RAM STATE ALIAScn112 [0] /usr/sbin/ifconfig -alo0: flags=2001000849 mtu 8232 index 1 inet 127.0.0.1 netmask ff000000e1000g0: flags=1100943 mtu 1500 index 2 inet 192.168.56.112 netmask ffffff00 broadcast 192.168.56.255 ether 8:0:27:24:cf:6alo0: flags=2002000849 mtu 8252 index 1 inet6 ::1/128cn112 [0] /usr/sbin/dladm show-phys -mLINK SLOT ADDRESS INUSE CLIENTe1000g0 primary 8:0:27:24:cf:6a yes e1000g0e1000g1 primary 8:0:27:77:c1:cc yes e1000g1cn112 [0] /usr/bin/sysinfo | /usr/bin/json "Network Interfaces"{ "e1000g0": { "MAC Address": "08:00:27:24:cf:6a", "ip4addr": "192.168.56.112", "Link Status": "up", "NIC Names": [ "admin" ] }, "e1000g1": { "MAC Address": "08:00:27:77:c1:cc", "ip4addr": "", "Link Status": "up", "NIC Names": [ "nfs" ] }}cn112 [0]
Our second NIC, "e1000g1" is on the same LAN segment presented to VMs by our other CNs. Below, I've configured the interfaceand updated our configuration for retention through reboot:
cn112 [0] /usr/bin/ifconfig e1000g1 plumb 172.25.0.49 netmask 255.255.255.0 broadcast + upcn112 [0] /usr/bin/grep nfs /usbkey/confignfs_nic=8:0:27:77:c1:ccnfs0_ip=172.25.0.49nfs0_netmask=255.255.255.0nfs0_network=...nfs0_gateway=172.25.0.49cn112 [0]
Over on our CN (cn110), we can see that we have 3 VMs configured, 2 of which are currently running. (The one we care about is "vm100".) We can also see that only one interface is plumbed in the global zone (GZ), that we have 3 NICs (admin, world, and vmint), and only the "world" NIC (e1000g1) is currently in use by VMs:
cn110 [0] /usr/sbin/vmadm listUUID TYPE RAM STATE ALIAS56800183-ab62-4db4-9d39-1fdc9a7e607a OS 256 stopped vm102c9fd4db6-5da0-4b0d-8c7a-87c55b10caff OS 256 running vm100e669fd30-7c81-463d-99bc-7989f9e8af78 OS 256 running vm101cn110 [0] ifconfig -alo0: flags=2001000849 mtu 8232 index 1 inet 127.0.0.1 netmask ff000000e1000g0: flags=1100943 mtu 1500 index 2 inet 192.168.56.110 netmask ffffff00 broadcast 192.168.56.255 ether 8:0:27:20:64:b9lo0: flags=2002000849 mtu 8252 index 1 inet6 ::1/128cn110 [0] /usr/sbin/dladm show-phys -mLINK SLOT ADDRESS INUSE CLIENTe1000g0 primary 8:0:27:20:64:b9 yes e1000g0e1000g1 primary 8:0:27:67:1e:93 yes e1000g1e1000g2 primary 8:0:27:16:b9:c4 no --cn110 [0] /usr/sbin/dladm show-vnicLINK OVER SPEED MACADDRESS MACADDRTYPE VID ZONEnet0 e1000g1 0 72:53:2c:7d:77:17 fixed 0 e669fd30-7c81-463d-99bc-7989f9e8af78net0 e1000g1 0 e2:c9:ae:f6:82:1a fixed 0 c9fd4db6-5da0-4b0d-8c7a-87c55b10caffcn110 [0] /usr/sbin/sysinfo | /usr/bin/json "Network Interfaces"{ "e1000g0": { "MAC Address": "08:00:27:20:64:b9", "ip4addr": "192.168.56.110", "Link Status": "up", "NIC Names": [ "admin" ] }, "e1000g1": { "MAC Address": "08:00:27:67:1e:93", "ip4addr": "", "Link Status": "up", "NIC Names": [ "world" ] }, "e1000g2": { "MAC Address": "08:00:27:16:b9:c4", "ip4addr": "", "Link Status": "unknown", "NIC Names": [ "vmint" ] }}cn110 [0]
NIC e1000g2 (tagged as "vmint") on "cn110" resides on the same LAN segment as e1000g1 (nfs) on "cn112". Also, verifying the above regarding "vm100", we can see only one configured virtual NIC (vNIC), which is attached to our "world" interface:
cn110 [0] /usr/sbin/vmadm get c9fd4db6-5da0-4b0d-8c7a-87c55b10caff | /usr/bin/json nics[ { "interface": "net0", "mac": "e2:c9:ae:f6:82:1a", "nic_tag": "world", "gateway": "10.0.22.254", "primary": true, "ip": "10.0.22.100", "netmask": "255.255.255.0" }]cn110 [0]
In order to use the NFS services to be shared out from "cn112", we'll need to add a new vNIC to "vm100". Below, we add the new vNIC to "vm100", and verify it was configured:
cn110 [0] /usr/bin/cat /var/tmp/vm100.addnic{ "add_nics": [ { "physical": "net1", "index": 1, "nic_tag": "vmint", "ip": "172.25.0.100", "netmask": 255.255.255.0" } ]}cn110 [0] /usr/sbin/vmadm update c9fd4db6-5da0-4b0d-8c7a-87c55b10caff < /var/tmp/vm100.addnicSuccessfully updated c9fd4db6-5da0-4b0d-8c7a-87c55b10caffcn110 [0] /usr/sbin/vmadm get c9fd4db6-5da0-4b0d-8c7a-87c55b10caff | /usr/bin/json nics[ { "interface": "net0", "mac": "e2:c9:ae:f6:82:1a", "nic_tag": "world", "gateway": "10.0.22.254", "primary": true, "ip": "10.0.22.100", "netmask": "255.255.255.0" }, { "interface": "net1", "mac": "c2:9c:ab:c3:de:4d", "nic_tag": "vmint", "ip": "172.25.0.100", "netmask": "255.255.255.0" }]cn110 [0]
As seen below, while the configuration for "vm100's" vNIC is in place, the vNIC is not yet created or usable until after the "vm100" SmartMachine is rebooted:
cn110 [0] /usr/sbin/dladm show-vnicLINK OVER SPEED MACADDRESS MACADDRTYPE VID ZONEnet0 e1000g1 0 72:53:2c:7d:77:17 fixed 0 e669fd30-7c81-463d-99bc-7989f9e8af78net0 e1000g1 0 e2:c9:ae:f6:82:1a fixed 0 c9fd4db6-5da0-4b0d-8c7a-87c55b10caffcn110 [0] /usr/sbin/vmadm reboot c9fd4db6-5da0-4b0d-8c7a-87c55b10caffSuccessfully completed reboot for c9fd4db6-5da0-4b0d-8c7a-87c55b10caffcn110 [0] /usr/sbin/vmadm list -vUUID TYPE RAM STATE ALIAS56800183-ab62-4db4-9d39-1fdc9a7e607a OS 256 stopped vm102c9fd4db6-5da0-4b0d-8c7a-87c55b10caff OS 256 running vm100e669fd30-7c81-463d-99bc-7989f9e8af78 OS 256 running vm101cn110 [0] /usr/sbin/vmadm get c9fd4db6-5da0-4b0d-8c7a-87c55b10caff | /usr/bin/json nics[ { "interface": "net0", "mac": "e2:c9:ae:f6:82:1a", "nic_tag": "world", "gateway": "10.0.22.254", "primary": true, "ip": "10.0.22.100", "netmask": "255.255.255.0" }, { "interface": "net1", "mac": "c2:9c:ab:c3:de:4d", "nic_tag": "vmint", "ip": "172.25.0.100", "netmask": "255.255.255.0" }]cn110 [0] /usr/sbin/dladm show-vnicLINK OVER SPEED MACADDRESS MACADDRTYPE VID ZONEnet0 e1000g1 0 72:53:2c:7d:77:17 fixed 0 e669fd30-7c81-463d-99bc-7989f9e8af78net0 e1000g1 0 e2:c9:ae:f6:82:1a fixed 0 c9fd4db6-5da0-4b0d-8c7a-87c55b10caffnet1 e1000g2 0 c2:9c:ab:c3:de:4d fixed 0 c9fd4db6-5da0-4b0d-8c7a-87c55b10caffcn110 [0]
Back on our NFS server (cn112), we check our zpool "zones" to determine how much space we have before setting up an NFS share:
cn112 [0] /usr/sbin/zfs list -r zonesNAME USED AVAIL REFER MOUNTPOINTzones 4.32G 11.3G 654K /zoneszones/cf7e2f40-9276-11e2-af9a-0bad2233fb0b 323M 11.3G 323M /zones/cf7e2f40-9276-11e2-af9a-0bad2233fb0bzones/config 38K 11.3G 38K legacyzones/cores 31K 10.0G 31K /zones/global/coreszones/dump 838M 11.3G 838M -zones/opt 957K 11.3G 957K legacyzones/swap 3.11G 14.4G 16K -zones/usbkey 127K 11.3G 127K legacyzones/var 73.3M 11.3G 73.3M legacycn112 [0] /usr/sbin/imgadm listUUID NAME VERSION OS PUBLISHEDcf7e2f40-9276-11e2-af9a-0bad2233fb0b base64 1.9.1 smartos 2013-03-22T17:11:29Zcn112 [0]
(Of note above, we see ZFS dataset "zones/cf7e2f40-9276-11e2-af9a-0bad2233fb0b" already exists and is related to image "smartos-base64-1.9.1". Given the function of this node, that dataset is currently irrelevant.) We now get to set up our first NFS share. By default, SmartOS gives us a "zones" zpool consisting of all disk space and within this zpool, I've created our NFS share dataset:
cn112 [0] /usr/sbin/zfs create -p -o quota=1.5g -o \> sharenfs='rw=@172.25.0.0/24:@172.25.1.0/25,ro=@172.25.1.128/25,root=172.25.0.100' \> zones/nfs/c9fd4db6-5da0-4b0d-8c7a-87c55b10caffcn112 [0] /usr/sbin/zfs get sharenfs zones/nfs/c9fd4db6-5da0-4b0d-8c7a-87c55b10caffNAME PROPERTY \ VALUE SOURCEzones/nfs/c9fd4db6-5da0-4b0d-8c7a-87c55b10caff sharenfs \ rw=@172.25.0.0/24:@172.25.1.0/25,ro=@172.25.1.128/25,root=172.25.0.100 localcn112 [0]
The shared dataset, zones/nfs/c9fd4db6-5da0-4b0d-8c7a-87c55b10caff, is so named as its intended usage is by "vm100" on "cn110", thus the end portion is "vm100's" UUID. (You can name your's however you'd like.) Since "/etc" on "cn112" is transient in nature, we didn't bother with files under "/etc/dfs" like we could otherwise do under Solaris. With this being SmartOS, we instead can use ZFS' native NFS handling via "sharenfs", which adheres to the same nomenclature and conventions as the traditional "share_nfs" command. Our SmartMachine, vm100, will have full root access to the share given the options set above. Also, we've set a quota to restrict the share to no more than 1.5 GB in size. After executing the above 'zfs create' command, our NFS share is immediately available as seen below:
cn112 [0] /usr/sbin/share-@zones/nfs/c9 /zones/nfs/c9fd4db6-5da0-4b0d-8c7a-87c55b10caff \ sec=sys,rw=@172.25.0.0/24:@172.25.1.0/25,ro=@172.25.1.128/25,root=172.25.0.100 ""cn112 [0] /usr/sbin/zpool history zones | /usr/bin/grep create | tail -12013-05-06.01:17:56 zfs create -p -o quota=1.5g -o \ sharenfs=rw=@172.25.0.0/24:@172.25.1.0/25,ro=@172.25.1.128/25,root=172.25.0.100 \ zones/nfs/c9fd4db6-5da0-4b0d-8c7a-87c55b10caffcn112 [0] /usr/bin/head -3 /var/svc/log/network-nfs-server:default.log[ May 6 01:17:51 Enabled. ][ May 6 01:17:52 Executing start method ("/lib/svc/method/nfs-server start"). ][ May 6 01:17:52 Method "start" exited with status 0. ]cn112 [0]
Notably, we can see in our SMF log for our NFS server service that method "nfs-server" started just seconds before the 'zfs create' command was executed. This wasn't due to any errant command that I didn't show; it was handled for us due to the "sharenfs" option we passed to 'zfs create'.
Back on our CN (cn110), I've logged into "vm100", created a mountpoint for our new share to be mounted to, and mounted the share to thesystem:
cn110 [0] /usr/sbin/zlogin -l root c9fd4db6-5da0-4b0d-8c7a-87c55b10caff[Connected to zone 'c9fd4db6-5da0-4b0d-8c7a-87c55b10caff' pts/4]Last login: Mon May 6 00:39:27 on pts/3 __ . . _| |_ | .-. . . .-. :--. |-|_ _| ;| || |(.-' | | | |__| `--' `-' `;-| `-' ' ' `-' / ; SmartMachine (base64 1.9.1) `-' http://wiki.joyent.com/jpc2/SmartMachine+Basevm100 [0] /opt/local/bin/mkdir /mnt/nfs-poolvm100 [0] /usr/sbin/mount -Fnfs -o rw,intr \> 172.25.0.49:/zones/nfs/c9fd4db6-5da0-4b0d-8c7a-87c55b10caff /mnt/nfs-poolvm100 [0] /opt/local/bin/df -h /mnt/nfs-poolFilesystem Size Used Avail Use% Mounted on172.25.0.49:/zones/nfs/c9fd4db6-5da0-4b0d-8c7a-87c55b10caff 1.5G 31K 1.5G 1% /mnt/nfs-poolvm100 [0] /opt/local/bin/whoamirootvm100 [0] /opt/local/bin/touch /mnt/nfs-pool/myfilevm100 [0] /usr/bin/truss -vlstat -tlstat ls -l /mnt/nfs-pool/myfilelstat("/mnt/nfs-pool/myfile", 0x0045F020) = 0 d=0x0000022200000003 i=8 m=0100644 l=1 u=0 g=0 sz=0 at = May 6 01:57:52 UTC 2013 [ 1367805472.207230095 ] mt = May 6 01:57:52 UTC 2013 [ 1367805472.207230266 ] ct = May 6 01:57:52 UTC 2013 [ 1367805472.207246365 ] bsz=8192 blks=1 fs=nfs-rw-r--r-- 1 root root 0 May 6 01:57 /mnt/nfs-pool/myfilevm100 [0]cn112 [0] /usr/bin/truss -vlstat -tlstat ls -l /zones/nfs/c9fd4db6-5da0-4b0d-8c7a-87c55b10cafflstat64("/zones/nfs/c9fd4db6-5da0-4b0d-8c7a-87c55b10caff/myfile", 0x08047660) = 0 d=0x0169000A i=8 m=0100644 l=1 u=0 g=0 sz=0 at = May 6 01:57:52 UTC 2013 [ 1367805472.207230095 ] mt = May 6 01:57:52 UTC 2013 [ 1367805472.207230266 ] ct = May 6 01:57:52 UTC 2013 [ 1367805472.207246365 ] bsz=131072 blks=1 fs=zfs-rw-r--r-- 1 root root 0 May 6 01:57 /zones/nfs/c9fd4db6-5da0-4b0d-8c7a-87c55b10caff/myfilecn112 [0]
A quick test above via 'touch' also details that we can create files on the new share from our VM as root. To further validate, I've also checked the same file from the server side. Now, on to test our configuration retention on the server side. Below, I've unmounted the share from "vm100" and rebooted our NFS server. Post reboot, I've checked our configured NICs and verified that our NFS share is again presented for client access via 'share':
vm100 [0] /usr/sbin/umount /mnt/nfs-poolvm100 [0]cn112 [0] /usr/sbin/init 6troy@glados [0] ssh -l root cn112Password:Last login: Mon May 6 00:13:28 2013 from 192.168.56.1- SmartOS Live Image v0.147+ build: 20130321T213641Zcn112 [0] /usr/sbin/ifconfig -alo0: flags=2001000849 mtu 8232 index 1 inet 127.0.0.1 netmask ff000000e1000g0: flags=1100943 mtu 1500 index 2 inet 192.168.56.112 netmask ffffff00 broadcast 192.168.56.255 ether 8:0:27:24:cf:6anfs0: flags=1100843 mtu 1500 index 3 inet 172.25.0.49 netmask ffffff00 broadcast 172.25.0.255 ether 2:8:20:a6:87:71lo0: flags=2002000849 mtu 8252 index 1 inet6 ::1/128cn112 [0] /usr/sbin/dladm show-phys -mLINK SLOT ADDRESS INUSE CLIENTe1000g0 primary 8:0:27:24:cf:6a yes e1000g0e1000g1 primary 8:0:27:77:c1:cc yes e1000g1cn112 [0] /usr/sbin/dladm show-link nfs0LINK CLASS MTU STATE BRIDGE OVERnfs0 vnic 1500 up -- e1000g1cn112 [0] /usr/sbin/dladm show-vnicLINK OVER SPEED MACADDRESS MACADDRTYPE VID ZONEnfs0 e1000g1 1000 2:8:20:a6:87:71 random 0 --cn112 [0] /usr/sbin/sysinfo | /usr/bin/json "Network Interfaces"{ "e1000g0": { "MAC Address": "08:00:27:24:cf:6a", "ip4addr": "192.168.56.112", "Link Status": "up", "NIC Names": [ "admin" ] }, "e1000g1": { "MAC Address": "08:00:27:77:c1:cc", "ip4addr": "", "Link Status": "up", "NIC Names": [ "nfs" ] }}cn112 [0] /usr/sbin/share-@zones/nfs/c9 /zones/nfs/c9fd4db6-5da0-4b0d-8c7a-87c55b10caff sec=sys,rw=@172.25.0.0/24:@172.25.1.0/25,ro=@172.25.1.128/25,root=172.25.0.100 ""cn112 [0]
As seen below, we can immediately again mount up the NFS share and see no data has been lost:
vm100 [0] /usr/sbin/mount -Fnfs -o rw,intr \> 172.25.0.49:/zones/nfs/c9fd4db6-5da0-4b0d-8c7a-87c55b10caff /mnt/nfs-poolvm100 [0] /opt/local/bin/df -h /mnt/nfs-poolFilesystem Size Used Avail Use% Mounted on172.25.0.49:/zones/nfs/c9fd4db6-5da0-4b0d-8c7a-87c55b10caff 1.5G 31K 1.5G 1% /mnt/nfs-poolvm100 [0] /opt/local/bin/ls /mnt/nfs-poolmyfilevm100 [0]
Now let's retain the share mount point by updating "/etc/vfstab" on "vm100". To test configuration retention here, also below,I've logged out, rebooted the SmartMachine via 'vmadm' and logged back in:
vm100 [0] /opt/local/bin/grep nfs-pool /etc/vfstab172.25.0.49:/zones/nfs/c9fd4db6-5da0-4b0d-8c7a-87c55b10caff - /mnt/nfs-pool nfs - yes rw,intrvm100 [0] exit[Connection to zone 'c9fd4db6-5da0-4b0d-8c7a-87c55b10caff' pts/4 closed]cn110 [0] /usr/sbin/vmadm reboot c9fd4db6-5da0-4b0d-8c7a-87c55b10caffSuccessfully completed reboot for c9fd4db6-5da0-4b0d-8c7a-87c55b10caffcn110 [0] /usr/sbin/zlogin -l root c9fd4db6-5da0-4b0d-8c7a-87c55b10caff[Connected to zone 'c9fd4db6-5da0-4b0d-8c7a-87c55b10caff' pts/4]Last login: Mon May 6 01:31:09 on pts/5 __ . . _| |_ | .-. . . .-. :--. |-|_ _| ;| || |(.-' | | | |__| `--' `-' `;-| `-' ' ' `-' / ; SmartMachine (base64 1.9.1) `-' http://wiki.joyent.com/jpc2/SmartMachine+Basevm100 [0] /opt/local/bin/df -h /mnt/nfs-poolFilesystem Size Used Avail Use% Mounted on172.25.0.49:/zones/nfs/c9fd4db6-5da0-4b0d-8c7a-87c55b10caff 1.5G 31K 1.5G 1% /mnt/nfs-poolvm100 [0]
For our verification, I've simply ran 'df' against our share mount point. At this point, our work is now complete.
Post written by Troy Dietrich