Kebe Says - Dan McDonald's Blog

NOTE: A version of this was sent to the racoon2-users alias also.

I've been spending some of my time bringing up racoon2 (an IKEv2 and IKEv1 daemon) on OpenSolaris.

Because of vast differences in PF_KEY implementations between OpenSolaris and other OS kernels, I've spent my racoon2 time actually getting IKEv1 to work first, instead of IKEv2. Right now, what's working is:

• IKEv1 initiates and derives IPsec SAs for single-algorithm IPsec policies.

That's it! IKEv1 responder needs work, as does all of IKEv2, as does work
for multiple-choice of algorithms. But there's enough change in there to say
something now.

ARCHITECTURAL DIFFERENCES

The most noteworthy change in the OpenSolaris work so far is that literally there's no spmd (a separate IPsec SPD daemon racoon2 uses) required for now. This is because:

• We don't have the indirection between ACQUIREs and the appropriate policy entry. Our extended ACQUIREs contain everything needed to construct a proposal. There's no SPD consultation required with an OpenSolaris ACQUIRE.
• Our responder-side logic uses inverse-ACQUIRE, which will provide the same structure as ACQUIRE w.r.t. proposal construction. This is the closest we get to needing something like spmd, and given its syntactic equality to an extended ACQUIRE, we can use it on rekeying if the responder initiates the next time.

If spmd serves another purpose, we will revisit it. As it stands, however, I cannot see us using it.

CODE DIFFERENCES

In OpenSolaris, we use the "webrev" tool to generate easy-to-review web pages with diffs of all varieties. The webrev for what I have so far in racoon2 is available at:

http://cr.opensolaris.org/~danmcd/racoon2-opensolaris/

Feel free to make comments or suggestions about what I've done.

A little over a year ago, Bart Smaalders blogged about his new home server. Subsequently Bill built a similarly-configured one. (I thought that he had blogged about his too, but he hadn't.)

I'd been toying with the idea of following in Bill's and Bart's footsteps for some time. A recent influx allowed me to upgrade lots of home technology (including a new Penryn-powered MacBook Pro), and finally allowed me to build out what I like to think of as my home data center. I mention f''(Bart's...) because this box really is the second-derivative of Bart's original box (with Bill's being the first-derivative).

And the starting lineup for this box is:

• An AMD Opteron Model 185 - I was lucky enough to stumble across one of these. 2 cores of 2.6GHz AMD64 goodness.
• A Tyan S2866 - I bought the one with two Ethernet ports - one nVidia (nge) and one Broadcom (bge). It has audio too, but I haven't tested it as I've my Macs for such things. It has all of the goodies Bart mentioned, but I *think* that the SATA might be native now. (Please comment if you know.)
• 2GB ECC RAM - with room for two more if need be.
• A two-port old Intel Pro Ethernet 10/100 - good thing the driver (iprb) for this is now open-source. I'll explain why I need four Ethernet ports in a bit.
• Two Western Digitial "green" 750GB SATA drives Each drive has 32GB root partitions (yes that's large, until Indiana matures, though, I'll stick with UFS roots), 4 GB swap (for core dumps), and the remaining large areas combine to make one mirrored ZFS pool with ~700 decimal GB of storage.
• A cheap MSI nVidia 8400GS - It's more than enough to drive my 1920x1200 display.
• An overkill Antec 850W power supply - obtained for only $100 from the carcass of CompUSA.
• A Lian Li U60 case - My brother-in-law, who has years in the trenches of PC care, feeding, and repair, recommended Lian Li to me. It has all the space I need and more for drives, and its fan layout is pretty comprehensive. Since this box lives in my office, noise isn't that much of an issue.
• OpenSolaris build 83 - While I'm pumped about what's going on with Indiana it's still under development, and I want something a bit more stable.

So why four ethernet ports (covering three drivers)? Well, like Indiana, Crossbow is exciting, but not yet integrated into the main OpenSolaris tree. I do, however, very much like the idea of Virtual Network Machines and I'll be using these four ports to build three such machines on this server using prerequisite-to-Crossbow IP Instances. Two ports will form the router zone. The router will also be a firewall, and maybe an IPsec remote-access server too. With Tunnel Reform in place, I can let my or my wife's notebook Macs access our internal home network from any location. One port will be the public web server, and assuming Comcast doesn't screw things up too badly on their business-class install, the new home of www.kebe.com. The last port will be the internal-server and global-zone/administrative station. All of that ZFS space needs to be accessible from somewhere, right?

I'd like to thank Bart and Bill for the hardware inspiration, and to my friends in OpenSolaris networking for offering up something I can exploit immediately to create my three machines in one OpenSolaris install. I'll keep y'all informed about how things are going.

Oh my am I exhausted! I hoped to have most of the text of this completed before my flight got back to Manchester last night, but that didn't happen.

I keep telling people I know that Michigan is a hardware school (in spite of having some great software people - see my post from Monday). We Solaris developers at the Sun table were brutally reminded of this yesterday. Lots of EE's with Verilog and/or VHDL experience. Many of them asking about architecture and/or verification, but a surprising number who have never heard of SPARC, the UltraSPARC T1 (aka. Niagara), or that they can see the entire source for the Niagara with OpenSPARC. Almost every business card of mine I handed out to folks had the word, "OpenSPARC" on the back so they could Google it later.

We also tried to make sure everyone had OpenSolaris disks. There are four binary distributions of OpenSolaris on that set of disks: Solaris Express Community Edition (see the previous link) - Sun's current OpenSolaris vehicle, Nexenta - which is probably going to be one of the more comfortable ones for Ubuntu Linux users to land in, Belenix - which is optimized for Live CD use, and Schillix, which was the first non-Sun distribution of OpenSolaris, by Joerg Schilling of "cdrecord" fame. I hope some of the students went home and had success playing with OpenSolaris. You all should visit opensolaris.org and engage the community discussions with your feedback and questions.

I mentioned Monday about how much like a geezer I felt. I had more of that yesterday not only saying, "Class of '91" a few times, but also when Professor Quentin Stout visited our table. My only graduate-level class I took at U. of M. was his Parallel Algorithms class in the fall of 1990 (during Football/Marching Band season). Back in the day it was all theory - we discussed how to partition problems using the abstract PRAM (Parallel Random Access Machine). It was the ONLY parallel ANYTHING class offered when I had an available slot. This was when shared-memory multiprocessors were experiments or startups (anyone remember the BBN Butterfly, the Sequent Balance, or the Encore Multimax?). I mentioned to Prof. Stout I took his class back then. He then proceeded to tell me how the class is far more practical now. He told me all about stuff like OpenMP, and other high-level constructs that as a systems' programmer I just don't get to use all that much. I still, however, felt pretty smart for seeing the future back in 1990. I hope I have as good luck 17 years later.

Anyway, I had a great time in Ann Arbor, and I hope to get back there sooner rather than later. If anyone who visited our table is reading this, leave a comment, and don't be afraid to be honest. :)

I mentioned I was going to be at the University of Michigan's Engineering career fair, and here I am!

I got in yesterday (Sunday) afternoon, and did some things to re-orient myself. I visited my fraternity house first, and quickly, because rush began that night. In some ways things hadn't changed a bit - the house is still there and the rooms have the same names (my old room with a skylight window is still called Lighthouse). In other ways, they had - the TV is bigger and flatter, half of 'em had laptops, and the basement was being seriously renovated. The guys were pretty mellow, probably because of all of the post-beating-of-Penn-State celebrations. I then wandered around campus, eating dinner at Krazy Jim's Blimpyburger, where they give you burgers made of small, ground-that-day, patties. Yum!

When I flew in, the woman next to me on the plane explained the phenomenon she experienced when taking one of her kids to her alma mater. It all felt intimately familiar to her, even modulo some new buildings, but then she suddenly realized she was an old fart wandering campus. My kids aren't old enough to be shopping colleges yet, but I definitely felt the combination of familiarity and age. I saw buildings with new names, old names on new buildings, and just plain new buildings (esp. at North Campus). 20 years ago I was a freshman, now I'm literally old enough to be a father to a student in the incoming class of 2011.

This morning, I tagged along with Kais Belgaied as he visted some Computer Science faculty and grad students here. Our first visit was with Professor Z. Morley Mao, who's a new professor here. She has a lot of great ideas on how to exploit the Crossbow project for aiding intrusion detection (and mitigation), among other interesting ideas. We then talked to two other professors, Atul Prakash and Thomas Wenisch, and a few students as well. I remember Prof. Prakash from my time at Michigan (1987-1991), but the other two are new Assistant Professors. I'm confident from what I saw that U. of M.'s CSE division of EECS is going to be strong for a continuing number of years.

[Edit from Wednesday]Shoot! I forgot I also visited my old theory professor, Kevin Compton. He's a very good teacher, and helps even the most clueless undergrads (hem hem). He told me he's teaching a very popular undergraduate cryptography class, which is just too-cool, IMHO.

This evening several of us (Kais, Eric Kustarz, Bill and Sherry Moore, and I) gave a breezy tech talk about various goodies in OpenSolaris that we work on. We also had very yummy Pizza House pizza. Pizza House was "established 1986", which means it wasn't all that old when I was there, but it was good enough to have our host recommend it.

I'm now back in my hotel, squeezing packets over a flaky, but free, wifi. Tomorrow we will be spending the whole day at the table, taking resumes and answering questions. If one of you four readers of this blog is a U. of M. student, you don't have to wear a suit when visiting us. :)

Solaris 10 Update 4 (or as marketing calls it, Solaris 10 08/07) contains some backported goodies we've had in Nevada/OpenSolaris for a while.

IPsec Tunnel Reform was one of the first big pieces of code to be dropped into the S10u4 codebase. It shores up our interoperability story, so you can now start constructing VPNs that tell IKE to negotiation Tunnel-Mode (as opposed to IP-in-IP transport mode). Tunnels themselves are still network interfaces, but their IPsec configuration is now wholly in the purview of ipsecconf(1M). Modulo IKE (which we still OEM part of), we developed Tunnel Reform in the open with OpenSolaris.

Also new for S10u4 is IP Instances. Before u4, you could create non-global zones, but their network management (e.g. ifconfig(1M)) had to be done from the global zone. With u4, one can create a unique instance zone which gives the zone its own complete TCP/IP stack. The global zone needs to only assign a GLDv3-compatible interface to the zone (e.g. bge, nge, e1000g) to give it a unique IP Instance. You could have a single box be your router/firewall/NAT, your web server, and who knows what else, all while keeping those functions out of the fully-privileged global zone. It makes me think about upgrading to business-class Internet service at home, building my own box like Bart did and getting a few extra Ethernet ports.

Oh, and if you want to do it all with less ethernet ports, check out OpenSolaris's Crossbow and its VNIC abstraction!

Have fun moving your network bits in new and interesting ways!

As of OpenSolaris build 73, the way we do IPsec NAT-Traversal changes for the cleaner.

Before this build, IPsec NAT-Traversal was performed by pushing a STREAMS module on top of an open UDP socket. This module (nattymod) would either strip UDP headers out of ESP-in-UDP packets, or strip the "0-SPI" marker (four bytes of zeroes) before passing the datagram up to the application.

This method worked, but it had some flaws, including the implicit setting of certain socket options (UDP_INCLHDR) that would then potentially be blocked from applications that actually required them. Also, nattymod did not perform the insertion of the 0-SPI automatically, the application was stuck doing that on its own. And while FireEngine merged TCP in to IP for S10, we needed to wait for one of the earlier builds of OpenSolaris to get the UDP equivalent.

With the new NAT-Traversal scheme, a key management application (like our closed-source in.iked(1M)) that wishes to aid in NAT-Traversal simply sets a new socket option: UDP_NAT_T_ENDPOINT. If this option is set, the following things happen:

• On inbound packets, the first four bytes after the UDP header are inspected.
• If there are less than four byte, the packet is dropped, and assumed to be a NAT-T keepalive.
• If the four-byte are all zeros (i.e. the 0-SPI), they are stripped and regular UDP processing occurs.
• Otherwise, the UDP header is stripped and the packet is shuffled off the IPsec's ESP for processing.
• On outbound packets
• The 0-SPI is inserted between the UDP header and the user-generated data.
• ESP will send ESP-in-UDP by itself depending on the Security Association's properties.

This will help anyone who wants to port their open-source IKE or other key management application to Solaris deal with the possibility of NAT boxes.

---

And on a related note, this will be mentioned during Nicolas Droux's OpenSolaris Networking for Developers talk next week at Sun Tech Days in Boston. I'll be there too, talking about S10 and OpenSolaris security features, as well as being in the audience for Nicolas's talk.

The whole reason I was reading e-mail on a Sunday was not to look for telnetd exploits.

I was logged in because Team IPsec runs its punchin IPsec remote-access server (sometimes called a VPN server, but I hate that term because it's pushed by too many middlebox vendors) which was having routing problems.

As stated before, Solaris implements tunnels as point-to-point interfaces. For a remote-access server like we have in punchin, this means every external IP address gets a tunnel interface. (Until we had Tunnel Reform, this meant only one client per external IP address, which messed up NATs for multiple clients.) A tunnel interface has two addresses - a local one and a remote one. The local one can be shared with other tunnels or even with a different local interface (like the local ethernet). Such interfaces are called unnumbered interfaces.

A remote access server does forward packets, and is therefore by definition a router. One of our servers just swapped out Zebra (from older OpenSolaris/Nevada build) to Quagga. We use Quagga's OSPF to learn the topology of the Sun internal network (the SWAN).

As clients "punch out", their tunnel gets destroyed. Now each of these tunnels shares the same local IP address with our ethernet to the SWAN. Unfortunately, these "interface down" events confuse Quagga, and suddenly all of my punchin clients can't move bits to the internal network anymore.

There is a workaround, and that's to assign a different local IP address than the one that is directly connected to the SWAN for use with all of the client tunnels. It's not that painful, as I only lose one out of 256 possible client addresses (our engineering ones only have a /24 from which to allocate client addresses). Still, as an esteemed colleague said, "I hope that's not the *whole* solution."

It isn't, and I would like to ask the Quagga community (as I've already asked our local routing folks, Paul Jakma and Alan Maguire) to make sure that Quagga and its routing protocols play nicely with unnumbered interfaces. It'll allow me to plumb tunnels until I'm all out of address space! :)

This entry brought to you by the Technorati tags routing, Solaris, and OpenSolaris.

I don't have a tag for general Security because dammit, I'm still a networking person who works on security! (UPDATE: I'm wiser now in 2020 and have added one.)

Anyway, you've seen elsewhere about how Alan H. turned around the S10 fix as quickly as he could. I'm going to tell you how Alan already found this:

D 1.67  07/02/11 19:46:41 danmcd        90 89   00009/00010/04896
6523815 LARGE vulnerability in telnetd

when he went to file a bug that'd already been putback into Nevada/OpenSolaris.

The best place to see what happened is to visit the OpenSolaris discussions, especially this thread.

I was reading e-mail on a Sunday because of an operations problem I was having with one of our punchin IPsec remote access servers. (I'll discuss the problem, a routing one, in a followup entry later today.) I found the initial note and read the PDF file to which "skunsul" so graciously provided a link. MAN I was embarassed. After trying it on some lab machines and my laptop, I brought up the in.telnetd source (at the line number provided by Kingcope). My first approach was to verify the content of the $USER environment variable fed to in.telnetd. I compiled-and-ran the fix, which seemed to work. Great! Time to find some code reviewers.

My only regret about this was not putting the review on security-discuss@opensolaris.org or networking-discuss@opensolaris.org. I'll try better next time, especially for something that was announced on an opensolaris list initially. Anyway, two reviewers (OpenSolaris board member and well-known Sun Good Guy Casper Dik, and crypto framework expert Krishna Yenduri) suggested that login(1) is already getopt-compliant, and that I should just pass "--" between the rest of the arguments and the contents of $USER, no matter how *&^$-ed up it is. Because it was a Sunday, I didn't get rapid turnaround on e-mail replies. This is why the putback didn't happen until six hours after I'd read the note from skunsul. Krishna also recommended (in the spirit of open development) that I place the diffs on the very thread, and I did just that.

Anyone I know here who happened to have seen the initial note would've jumped on this in the same way - please don't think I did something others wouldn't do. My point is - this is the first security exploit reported to us via OpenSolaris, and I think the "Open" part of OpenSolaris helped out the code, as well as Sun's customers.

This entry brought to you by the Technorati tags Solaris and OpenSolaris.

Hey everyone!

The IPsec Tunnel Reform project's code review is now underway. Take a look and see what it took to bring up IPsec Tunnel-Mode processing in a world where tunnels are not actions from a policy, but rather a first-class network interface (or at least after Clearview it will be).

Highlights for administrators include:

• Augmentiations to ipsecconf(1m) to specify a tunnel interface's policy, whether it's S9-style IP-in-IP transport mode, or RFC 2401-compliant Tunnel Mode.


• No changes to IKE configuration.


• You can configure tunnel security without ifconfig(1m) using just ipsecconf(1m). We put all IPsec policy in ipsecconf(1m) and let ifconfig manage interfaces (and route(1m) manage routing).


• Additions to ipseckey(1m) for manual tunnel-mode SA configuration, or monitoring of kernel interactions with Key Management.


• Better interoperability with everyone else's Tunnel Mode IPsec.

Highlights for OpenSolaris-hackers include:

• New per-tunnel policy structure: ipsec_tun_pol_t, which instantiates the existing policy-head per tunnel.


• Getting rid of IRE_DB_REQ messages for SA addition/updates. This improves SA-adding performance and reduces the complexity of the ESP and AH modules.


• New PF_KEY and PF_POLICY messages to reflect Tunnel Mode.


• Shifting of tunnel IPsec policy enforcment from the lower-instance of IP to "tun" itself. (NOTE: This will change again when we merge with Clearview.)

Share your comments on tref-discuss, and let us know what you think!

This entry brought to you by the Technorati tags IPsec, Solaris, and OpenSolaris.

NOTE: Links here point to docs that no longer exist. Maybe the the Internet Archive might have 'em?

IPsec in Solaris has one missing piece, and we're about to put it in place.

The IPsec Tunnel Reform project aims to give Solaris and OpenSolaris an RFC 2401-compliant tunnel-mode implementation.

There's a lot of changes in the source base, some of which aren't open sourced (IKE), but most of which are in existing OpenSolaris code. The project page has a webrev showing the changes thus far. We're trying to be more open in our development processes here in the Solaris group, and showing you Tunnel Reform before we've finished it, AND before we've started major test efforts, is Team IPsec's own way of contributing to this openness.

Think of the source snapshot as a "Code Preview" instead of a "Code Review". There's a newly-rewhacked design document there too, and we'd like you to look at it and discuss it on the OpenSolaris communities or the tref-discuss@opensolaris.org mailing list.

And once we're done with this, we can think about RFC 4301 (2401's replacement) and friends, more zones support, SMF-izing things, giving TX labelled SA support... :)

---

This entry brought to you by the Technorati tags IPsec, Solaris, and OpenSolaris.