Our sister company PlanetNode sells cloud. The question that comes with that is not whether you virtualise, but on what. OpenStack can do it. So can OpenNebula. And a handful of other platforms that promise roughly the same.
It became Proxmox VE — not because the others are no good, but because each of those platforms has its own flavour, and Proxmox's suits the way we work. OpenStack is powerful, but it brings along a stack that is a full-time maintenance job in itself; we wanted a hypervisor with an API that does what you expect, and to spend our time on the product around it rather than on the platform beneath it.
Because that is where the real work is. A hypervisor does not make a cloud: something has to know which images exist, which IP addresses are free, which customer ordered what, and how a fresh machine finds out who it is. That is exactly where we built our own — on top of components we never want to make ourselves.
The split came out as follows: Proxmox VE for virtualisation, NetBox as the source of truth for IPAM, Packer for building images and cloud-init for the first boot inside the guest. All open source, all mature. Around that we built three things ourselves: a template pipeline, a metadata service and a provisioning layer in our customer portal.
- The build pipeline ships the same sealed image to every cluster; the template name is the sync key.
- After a paid order the portal clones the right template and configures the machine through the Proxmox API.
- On first boot the server asks who it is — hostname, keys, user-data.
- The metadata service fetches the answer from the portal over a signed internal channel; no copy is stored anywhere.
This is the story of those three layers. Not because everyone should build their own cloud, but because the trade-off inside it — what do you buy in, what do you build yourself — is exactly the trade-off we make for customers as well.
What a portal has to do that a panel does not
There are off-the-shelf hosting panels, and they take a great deal of work off your hands. They also determine your product, though: the panel knows what a VPS looks like, how an order runs and which fields a customer fills in.
We wanted three things that do not fit in there. A network design of our own choosing, because we run our own network and keep IPAM in one place. A single administration, in which an order, an invoice and a running machine are the same process rather than three systems you reconcile by hand. And our own pace in the OS catalogue: when a new distribution release appears, we want to be able to build, test and roll it out ourselves.
Layer 1: templates you can trust
Every virtual server starts from a template. We build those with Packer on a separate build machine, deliberately outside the production clusters. The result is an ordinary disk image, and that is precisely the point: every cluster gets exactly the same artefact and treats it identically — whether Ceph, ZFS or LVM sits underneath.
A few choices that keep the maintenance bearable:
One build for all families. There is no separate pipeline for Debian, Ubuntu, Rocky and Alpine. There is one parameterised build; per family we only record the official cloud image plus its checksum. Adding a new distribution means a variables file, not a new pipeline.
Date-based versions. Templates are named tpl-<family>-v<year>.<month>.<patch>. That name is the only key clusters synchronise on — internal IDs may differ per cluster, because you never want to depend on those. Distribution is idempotent: it replaces by name, refuses to touch anything that is not a template, and you can rerun it per cluster without having to think about it.
Images are sealed. Before closing up, the SSH host keys are removed, /etc/machine-id is emptied, the cloud-init state is wiped and the build user is deleted. That is not tidiness but necessity, and it is worth knowing exactly what goes wrong if you skip it:
- Shared host keys give every server from that template the same SSH identity. A customer connecting to the wrong machine gets no warning, because the fingerprint matches — and anyone who ever gets hold of the image can impersonate every server cloned from it.
- A shared machine ID leaks all the way into your network layer: by default systemd derives the DHCPv6 DUID from it, so two machines request a lease with the same identity and contest each other's address. On top of that, journald and most monitoring agents use it as a unique key — you see two servers as one host.
- Leftover cloud-init state is the quietest problem: on first boot cloud-init sees that it has "already run" and skips the entire configuration step. The machine then comes up with the hostname and keys from build time, without a single error message.
Checks before rollout. The build validates itself, verifies inside the image that our cloud-init datasource actually loads, and can start a test clone that checks via the guest agent whether everything works as intended. A template that does not get through that is not rolled out.
The catalogue itself lives in the customer portal. New templates appear there as a draft; someone deliberately activates them and designates a default. Outdated versions are assigned a successor instead of disappearing — existing machines are left untouched, only new installations follow the successor chain. That way "rolling out the new Debian" is a decision, not a side effect of a successful build.
Layer 2: our own metadata service, and why
This is where it gets interesting. When a virtual server boots for the first time, it has to know who it is: hostname, SSH keys, any start-up scripts. In the cloud world that happens through a metadata service — the address 169.254.169.254 you find at every large provider. The machine asks who it is itself.
Proxmox has its own mechanism for this, but that mainly supplies network details: IP, gateway, DNS. For hostname, keys and user data you have to fall back on files on the hypervisor. That works, but it means customer data ends up copied onto your nodes — and that updating it becomes file management on production hosts. For data that differs per customer and changes as you go, that is precisely the wrong place.
So we built our own metadata service: a small daemon in Rust on every hypervisor node. Towards the guest it speaks the familiar protocol that every cloud image already knows. Inwards it stores nothing — it fetches the answer from the customer portal, over an internal channel that is signed per node and protected against replay. The portal remains the single source of truth; a change takes effect immediately because there is no copy anywhere that can go stale.
The most important design principle lies in how the daemon determines who is calling: based on where the connection comes from, never on what the machine claims about itself. Let the guest say "I am server X" and server A can request server B's keys. Identity should come from the infrastructure, not from the request.
Two cloud-init lessons that cost us hours
Anyone writing their own cloud-init datasource is guaranteed to run into these two. Neither is stated clearly anywhere in the documentation, so here they are:
1. The detection script does not know your datasource. At start-up cloud-init runs a detection step that decides which sources are tried at all. It only knows the built-in sources; yours is silently dropped from the list, without an error. And as soon as it sees a Proxmox configuration disk, it rewrites the list to that source only. The fix is not to improve your datasource, but to configure the list so that the detection step disables itself.
2. The first boot stage fixes your choice. Cloud-init runs in stages: one before the network, then one after it. The datasource chosen in the first stage applies for the whole boot — a source that only reports in after the network is never looked at again. That means your datasource has to register in the early stage, and therefore has to bring up networking temporarily itself in order to reach the metadata service. The built-in EC2 variant does exactly the same; that you have to copy it is something you only discover by watching it fail.
Both lessons produced the same insight: our datasource takes the network from what Proxmox provides — that stays Proxmox's responsibility — and the identity from our own service. If the metadata service goes down, the machine simply boots on what was already there. No single component may be a single point of failure for "does the server come up".
Layer 3: from order to running server
The last layer is the provisioning, and it lives in the customer portal — a Laravel application that is shop, administration and control panel all at once.
Once an order is paid, a job goes into the queue that runs through six steps: reserve an address (from NetBox, under a lock so that two simultaneous orders never get the same IP), generate an identity, clone the machine from the right template on that cluster, configure, start, and verify that the guest agent responds.
- Addressfrom NetBox
- IdentitySMBIOS UUID
- Clonefrom the template
- Configurenetwork & specs
- Bootfirst start
- Verifyguest agent
Reservation happens under a per-prefix lock: two simultaneous orders never get the same IP.
The machine ID is recorded before the machine exists — a crash leaves at most a record without a machine, never the other way around.
Every step is idempotent: a retry resumes where it failed instead of starting over.
Two properties make the difference between "usually works" and "always works":
Every step is idempotent and independently restartable. If the job falls over halfway, the retry picks up where it went wrong instead of starting again. Half-executed orders are the most expensive kind of failure there is — they cost manual work as well as trust.
Bookkeeping before action. A machine ID is claimed and recorded before the machine is created. If the process crashes between those two moments, at worst there is a record without a machine — something you can clean up. The other way round, a machine that is registered nowhere is an orphan that nobody will find again. In processes like this you always choose the failure direction you can clean up.
The clean-up logic is explicit: reserved addresses only become final on success, and a periodic check clears out reservations that no longer belong to anything.
What it delivers
From order to a running, reachable server without a human being involved — with IPAM, invoicing and machine management in one administration. New distribution releases are a scheduled build plus one click to make them active. And because almost everything is standard open source, we can repair, explain and replace every layer ourselves.
That last point is not ideology but a practical choice. If customers entrust infrastructure to you, you have to be able to explain how it works, and when something breaks you must not depend on someone else's support department. Everything we built ourselves — the pipeline, the metadata service, the provisioning — has deliberately been kept thin: it is the glue between mature components, not a rebuild of them.
The same approach, for your environment
This architecture was built for PlanetNode, but the building blocks come from the same toolkit we use for customers: software that connects what does not connect by itself, networks we design and manage ourselves, and infrastructure in Dutch data centres.
The common thread is always the same question, the same one at the heart of our webshop integration: what do you buy off the shelf, and where do you build your own? Our answer is consistent — buy the foundation, build the glue. Standard components where they fit, bespoke software precisely where your process differs from the average.
Curious what that means for your situation? Schedule a technical intake — and we will work out together where that line belongs.