Verdict: DragonFly BSD
You want to tinker with genuinely new ideas in kernel design, and HAMMER2 sounds like an invitation rather than a threat. DragonFly forked from FreeBSD in 2003 over an argument about how SMP scaling should actually work, then went and built its answer. It is not just a FreeBSD variant anymore. It is a different OS with different ideas.
What you get
Section titled “What you get”- HAMMER2: a filesystem with snapshots, dedup, compression, and clustering built from scratch by people who knew what they wanted and weren’t afraid to rebuild everything.
- A kernel architecture designed around message passing and per-CPU data instead of global locks. SMP that actually scales instead of giving up at 16 cores.
- A small community where you are not a number. Contributions matter. Bug reports get responses. The people who wrote the kernel answer their own mailing list.
- Novel ideas. Not ports of Linux concepts. Not rehashes of BSD traditions. Different. Interesting. Sometimes half-finished, but always purposeful.
- A developers-first approach. Documentation exists if you read code. Man pages exist if you compile the kernel. You are building alongside people who care about correctness, not users.
What will hurt
Section titled “What will hurt”- x86_64 only. That pile of exotic hardware you wanted to resurrect belongs on NetBSD. DragonFly made a choice and lives with it.
- Small community means small package coverage. Popular software works. Obscure software might not be packaged. You might be the first.
- Some of the interesting ideas are perpetually in progress. HAMMER2 clustering. Work scheduler optimizations. VFS work. You signed up for a journey, and it is a journey.
- Fewer eyes on code mean fewer bug reports mean bugs stick around longer. But when they are fixed, they are fixed thoroughly.
Getting help
Section titled “Getting help”- DragonFly mailing lists - users@, kernel@, and commits@.
- DragonFly documentation - built by the developers, not volunteers.
- IRC: irc.libera.chat, #dragonflybsd.
- Read commit messages. The developers explain their thinking in the commit logs.
First steps
Section titled “First steps”- HAMMER2 is the right choice. Use it. This is the whole point.
- Read the HAMMER2 design document. Seriously. It explains a lot of why DragonFly does what it does.
- Expect to read kernel source if things break. That is the trade.
- Join the mailing lists. The community is small but engaged. You might actually influence development.
- This is not for production servers. This is for people who like OS internals.
Where to start
Section titled “Where to start”Download from dragonflybsd.org, install with HAMMER2, and go read the HAMMER2 design document before you even boot. You are not here to run an OS. You are here to understand one.