OpenStack Compute node / VMs on K8s¶
This document describes approaches for implementing OpenStack Compute node and running VMs on top of K8s from the prospective of “Hypervisor Pod”. It includes overview of currently selected approach, future steps and alternative solutions.
Potential solutions¶
This section consists of a list of potential solutions for implementing OpenStack VMs on topc K8s. This solutions the case when Neutron ML2 OVS used for OpenStack networking. Pros and Cons listed for each solution. Not all possible solutions (with all combinations of pod-container topologies) listed here, only part that makes sense or needed to show transition from bad to good options.
1. Everything in one pod¶
Pods and containers¶
Hypervisor / Compute / Networking Pod:
- QEMU / KVM / Libvirt
- OVS DB
- OVS vswitchd
- nova-compute
- neutron-ovs-agent
Pros & cons¶
Pros:
- One pod will represent the whole OpenStack compute node (very minor advantage)
Cons:
- It’s impossible to make upgrade of any service without killing virtual machines
- All containers will have the same characteristics such as net=host, user, volumes and etc.
2. Libvirt and VMs baremetal, OpenStack part in one pod¶
Pods and containers¶
Baremetal (not in containers):
1. QEMU / KVM / Libvirt 4. OVS (DB and vswitchd)
Compute / Networking Pod:
- nova-compute
- neutron-ovs-agent
Pros & cons¶
Pros:
- Restart of docker and docker containers will not affect running VMs as libvirt running on baremetal
- Docker and docker containers downtime will not affect networking as OVS running on host, only new rules will not be passed to the host
Cons:
- External orchestration required to for managing Libvirt on baremetal, especially for upgrades
- It’s impossible to update nova without neutron and vice versa.
3. Libvirt and VMs baremetal, pod per OpenStack process¶
Pods and containers¶
Baremetal (not in containers):
- QEMU / KVM / Libvirt
- OVS DB / vswitchd
Compute Pod:
- nova-compute
Networking Pod:
- neutron-ovs-agent
Pros & cons¶
Same as option number 3, but it’s possible to upgrade nova and neutron separately.
4. Libvirt and VMs in one pod, pod per OpenStack service¶
Notes¶
It’s a primary approach and it’s currently implemented in Fuel CCP. Libvirt upgrade in such case could only be done by evacuating virtual machines from the host first, but, for example, nova-compute could be upgraded in place.
Pods and containers¶
Hypervisor pod:
- QEMU / KVM / Libvirt
OVS DB pod:
- OVS DB
OVS vswitchd pod:
- OVS vswitchd
Compute Pod:
- nova-compute
Networking pod:
- neutron-ovs-agent
Pros & cons¶
Pros:
- No external orchestration required for compute node provisioning
- All OpenStack parts and dependencies are managed through K8s in such case, so it’s possible to upgrade any service including libvirt, nova, neutron and ovs without external orchestration, just through the K8s API
Cons:
- Docker or docker containers downtime will affect running VMs or networking
5. Libvirt in pod w/ host pid, pod per OpenStack service, VMs outside of containers¶
Notes¶
It’s a “next step” approach based on Pros & Cond. It should be investigated in details and stability should be verified. If there will be no issues than it should become regerence approach of OPenStack VMs deployment on K8s. Potentially, another level of improvements needed to avoid affecting networking when docker or docker containers restarted.
Pods and containers¶
Hypervisor pod:
- QEMU / KVM / Libvirt (using host pid)
OVS DB pod:
- OVS DB
OVS vswitchd pod:
- OVS vswitchd
Compute Pod:
- nova-compute
Networking pod:
- neutron-ovs-agent
Pros & cons¶
Same as option number 4, but improved to not affect virtual machines when docker or docker containers restart.
Conclusion¶
Option number 4 is currently selected as implementation design for Fuel CCP, while as end goal we’d like to achieve approach where restarting docker and docker containers will not affect running virtual machines. In future, we’ll need to evaluate additional improvements to guarantee that K8s and docker downtime doesn’t affect running VMs.