Kebe Says - Dan McDonald's Blog

So recently I acquired @danmcd on Twitter. It was a long time coming. I was relatively late in early-adopting twitter: late spring 2009. By then someone else had claimed the handle danmcd, to my chagrin.

I was chagrined (in 2009) because I’ve been danmcd at SOMEWHERE since 1988. First .edus, even a .gov and .mil, and of course a series of .coms including my own kebe.com

(Who and/or what is Kebe might be another blog post in and of itself. In the meantime, this answer will suffice:

)

Names are important. Especially in the virtual world, they establish not only presence, but often place as well. I ended up being @kebesays on twitter for a long time. Luckily, Twitter makes handle-swapping relatively easy, so anyone who was following @kebesays got moved over to @danmcd without issue. I still keep /* XXX KEBE SAYS … at the top, because if you see that in my code, it indicates work-in-progress issues; and aren’t we all works-in-progress?

Speaking of names and places: one name and one place that has been associated with Triton and SmartOS - Joyent - will no longer be associated with SmartOS or Triton. Samsung has decided to use other in-house technology for their future, and that work will continue with Joyent. SmartOS and Triton are being spun off to MNX Solutions, where I will be continuing SmartOS development. See the MNX Triton FAQ and my email for more.

Oh and yes, I’ll get to be ‘danmcd‘ at MNX as well.

So what happened?

We’ve introduced a requested feature in SmartOS: the ability to select a platform image from loader(4), aka OS-8231.

To enable this feature, you must (using example bootable pool bootpool):

• Update BOTH the boot bits and the Platform Image to this release. Normally piadm(1M) updates both, so please use either latest or another ISO-using installation.
• Once booted to this PI, utter piadm activate 20210812T031946Z OR install another ISO-using installation (even if you never use it) to have the new piadm(1M) generate the /bootpool/boot/os/ directory the new modifications to loader require.

This represents a minor flag day because an older piadm(1M) will not update an existing /bootpool/boot/os/ directory. The PI-selection menus live in /bootpool/boot/os/, and will remain in an inconsistent state with older PIs using piadm(1M). It is safe to remove /bootpool/boot/os/ if you wish, as the activated (default) PI always boots correctly modulo actual /bootpool/boot/ corruption regardless.

So Tell Me about the Internals and the os/ Directory!

There were two SmartOS repositories that had changes. The first changeset was in illumos-joyent’s loader(5) Forth files. Alongside some additional support routines, the crux of the change is this addition to the main Joyent loader menu:

\
\ If available, load the "Platform Image Selection" option.
\
try-include /os/pi.rc

If the piadm(1M)-generated file /bootpool/os/pi.rc does not exist, the Joyent loader menu appears as it did prior to this fix.

The os/ Directory and illumos Needing platform/

The os/ directory in a bootable pool’s bootpool/boot filesystem contains directories of Platform Image stamps and the aforementioned pi.rc file.

[root@smartos-efi ~]# piadm list
PI STAMP               BOOTABLE FILESYSTEM            BOOT IMAGE NOW  NEXT 
20210715T010227Z       bootpool/boot                  available  no   no  
20210805T161859Z       bootpool/boot                  available  no   no  
20210812T031946Z       bootpool/boot                  next       yes  yes 
[root@smartos-efi ~]# ls /bootpool/boot/os
20210715T010227Z  20210805T161859Z  pi.rc
[root@smartos-efi ~]# 

Each PI stamp directory contains a single platform symbolic link up to the platform-STAMP directory that contains the PI.

[root@smartos-efi ~]# ls -lt /bootpool/boot/os/20210805T161859Z
total 1
lrwxrwxrwx   1 root     root          31 Aug 12 14:41 platform -> ../../platform-20210805T161859Z
[root@smartos-efi ~]#

The Triton Head Node loader menu has a pointer to the “prior Platform Image” has the explicit path of …/os/STAMP/platform contain the platform image. It was a design mistake of the original standalone SmartOS to not lay out platform image in this manner, but given that piadm(1M) must generate the pi.rc file anyway, it is not much more difficult to add symbolic-link construction as well.

The pi.rc File

The pi.rc file includes an additional menu item for the main Joyent loader screen:

It also contains up to three pages of platform images to choose from. Here’s an example of page 1 of 3:

The default PI is on every page, and up to five (5) additional PIs can appear per page. This means 16 PIs (default + 3 * 5) can be offered on a loader screen. Every time a platform image is activated, deleted, or added, the piadm(1M) command regenerates the entire os/ directory, including pi.rc.

So How and Why Do I Use This?

• Temporarily revert to and older Platform Image may be useful to check for regressions or to isolate behavior to a specific release.
• Developers can use *just* platform-image installations (platform-yyyymmddThhmmssZ.tgz to test their new builds without making the bootable pool unusable.

The piadm list output indicates being booted into a non-default PI by its NOW column:

PI STAMP               BOOTABLE FILESYSTEM            BOOT IMAGE NOW  NEXT 
20210114T041228Z       zones/boot                     available  no   no  
20210114T163038Z       zones/boot                     available  no   no  
20210211T055122Z       zones/boot                     none       no   no  
20210211T163919Z       zones/boot                     none       no   no  
20210224T232633Z       zones/boot                     available  no   no  
20210225T124034Z       zones/boot                     none       no   no  
20210226T213821Z       zones/boot                     none       no   no  
20210311T001742Z       zones/boot                     available  no   no  
20210325T002528Z       zones/boot                     available  no   no  
20210422T002312Z       zones/boot                     available  no   no  
20210520T001536Z       zones/boot                     available  no   no  
20210617T001230Z       zones/boot                     available  no   no  
20210701T204427Z       zones/boot                     available  no   no  
20210715T010227Z       zones/boot                     available  no   no  
20210729T002724Z       zones/boot                     available  no   no  
20210804T003855Z       zones/boot                     available  no   no  
20210805T161859Z       zones/boot                     available  yes  no  
20210812T031946Z       zones/boot                     next       no   yes 

In the above example, the SmartOS machine is booted into 20210805T161859Z, but its default is 20210812T031946Z. It would also look this way if piadm activate 20210812T031946Z was just invoked, as the semantics are the same.

Pardon my latency

Well, at least I’m staying on track for single-digit blog posts in a year. :)

Okay, seriously, 2020’s pandemic-and-other-chaos tends to distract. Also, I did actually have a few things worth my attention.

RFD 176

The second half of 2020 at work has been primarily about RFD 176 – weaning SmartOS and Triton off of the requirement for a USB key. Phases I (standalone SmartOS) and II (Triton Compute Node) are finished. Phase III (Triton Head Node) is coming along nicely, thanks to real-world testing on Equinix Metal (nee Packet), and I hope to have a dedicated blog post about our work in this space coming in the first quarter 2021.

Follow our progress in the rfd176 branches of smartos-live and sdc-headnode.

Twins & College

My twins are US High School seniors, meaning they’re off to college/university next fall, modulo pandemic-and-other-chaos. This means applications, a little stress, and generally folding in pandemic-and-other-chaos issues into the normal flow of things as well. Out of respect for their privacy and autonomy, I’ll stop here to avoid details each of them can spill on their own terms.

On 2021

Both “distractions” mentioned above will continue into 2021, so I apologize in advance for any lack of content here for my half-dozen readers. You can follow me on any of the socials mentioned on the right, because I’ll post there if the spirit moves me (especially on issues of the moment).

Booting from a zpool

The most recent published biweekly release of SmartOS has a new feature I authored: the ability to manage and boot SmartOS-bootable ZFS pools.

A few people read about this feature, and jumped to the conclusion that the SmartOS boot philosophy, enumerated here:

• The "/" filesystem is on a ramdisk
• The "/usr" filesystem is read-only
• All of the useful state is stored on the zones ZFS pool.

were suddenly thrown out the window. Nope.

This change is the first phase in a plan to not depend on ISO images or USB sticks for SmartOS, or Triton, to boot.

The primary thrust of this specific SmartOS change was to allow installation-time enabling of a bootable zones pool. The SmartOS installer now allows one to specify a bootable pool, either one created during the "create my special pools" shell escape, or just by specifying zones.

A secondary thrust of this change was to allow running SmartOS deployments to upgrade their zones pools to be BIOS bootable (if the pool structure allows booting), OR to create a new pool with new devices (and use zpool create -B) to be dedicated to boot. For example:

smartos# zpool create -f -B standalone c3t0d0
smartos# piadm bootable -e standalone
smartos# piadm bootable
standalone                     ==> BIOS and UEFI
zones                          ==> non-bootable
smartos# 

Under the covers

(NOTE: Edited 3 May 2023 to change "1M" man page refs to "8".)

Most of what’s above can be gleaned from the manual page. This section will discuss what the layout of a bootable pool actually looks like, and how the piadm(8) command sets things up, and expects things to BE set up.

Bootable pool basics

The piadm bootable command will indicate if a pool is bootable at all via the setting of the bootfs property on the pool. That gets you the BIOS bootability check, which admittedly is an assumption. The UEFI check happens by finding the disks s0 slice, and seeing if it’s formatted as pcfs, and if the proper EFI System Partition boot file is present.

bootfs layout

For standalone SmartOS booting, bootfs is supposed to be mounted on "/" with the pathname equal to the bootfs name. By convention, we prefer POOL/boot. Let’s take a look:

smartos# piadm bootable
zones                          ==> BIOS and UEFI
smartos# piadm list
PI STAMP           BOOTABLE FILESYSTEM            BOOT BITS?   NOW   NEXT  
20200810T185749Z   zones/boot                     none         yes   yes  
20200813T030805Z   zones/boot                     next         no    no   
smartos# cd /zones/boot
smartos# ls -lt
total 9
lrwxrwxrwx   1 root     root          27 Aug 25 15:58 platform -> ./platform-20200810T185749Z
lrwxrwxrwx   1 root     root          23 Aug 25 15:58 boot -> ./boot-20200813T030805Z
drwxr-xr-x   3 root     root           3 Aug 14 16:10 etc
drwxr-xr-x   4 root     root          15 Aug 13 06:07 boot-20200813T030805Z
drwxr-xr-x   4 root     root           5 Aug 13 06:07 platform-20200813T030805Z
drwxr-xr-x   4 1345     staff          5 Aug 10 20:30 platform-20200810T185749Z
smartos# 

Notice that the Platform Image stamp 20200810T185749Z is currently booted, and will be booted the next time. Notice, however, that there are no “BOOT BITS”, also known as the Boot Image, for 20200810T185749Z, and instead the 20200813T030805Z boot bits are employed? This allows a SmartOS bootable pool to update just the Platform Image (ala Triton) without altering loader. If one utters piadm activate 20200813T030805Z, then things will change:

smartos# piadm activate 20200813T030805Z
smartos# piadm list
PI STAMP           BOOTABLE FILESYSTEM            BOOT BITS?   NOW   NEXT  
20200810T185749Z   zones/boot                     none         yes   no   
20200813T030805Z   zones/boot                     next         no    yes  
smartos# ls -lt
total 9
lrwxrwxrwx   1 root     root          27 Sep  2 00:25 platform -> ./platform-20200813T030805Z
lrwxrwxrwx   1 root     root          23 Sep  2 00:25 boot -> ./boot-20200813T030805Z
drwxr-xr-x   3 root     root           3 Aug 14 16:10 etc
drwxr-xr-x   4 root     root          15 Aug 13 06:07 boot-20200813T030805Z
drwxr-xr-x   4 root     root           5 Aug 13 06:07 platform-20200813T030805Z
drwxr-xr-x   4 1345     staff          5 Aug 10 20:30 platform-20200810T185749Z
smartos# 

piadm(8) manipulates symbolic links in the boot filesystem to set versions of both the Boot Image (i.e. loader) and the Platform Image.

Let's throw out the first pitch.

I've moved my blog over from blogspot to here at kebe.com. I've recently upgraded the hardware for my Home Data Center (the subject of a future post), and while running the Blahg software doesn't require ANY sort of hardware upgrade, I figured since I had the patient open I'd make the change now.

Yes it's been almost five years since last I blogged. Let's see, since the release of OmniOS r151016, I've:

• Cut r151018, r151020, and r151022.
• Got RIFfed from OmniTI.
• Watched OmniOS become OmniOSce with great success.
• Got hired at Joyent and made more contributions to illumos via SmartOS.
• Tons more I either wouldn't blog about, or just plain forgot to mention.

So I'm here now, and maybe I'll pick up again? The most prolific year I had blogging was 2007 with 11 posts, with 2011 being 2nd place with 10. Not even sure if I *HAVE* a half-dozen readers anymore, but now I have far more control over the platform (and the truly wonderful software I'm using).

While Blahg supports comments, I've disabled them for now. I might re-enabled them down the road, but for now, you can find me on one of the two socials on the right and comment there.