Ask HN: What's a good Linux OS and setup to build a dev “network” on my laptop?

15 points by zodzedzi 2 years ago | 14 comments

I have a new lightweight laptop with 16 GB of RAM, decent SSD space, and an i7 CPU.

And I would like to setup a few systems on it, to have "everything" with me as I move around while disconnected.

I'm looking for recommendations and ideas into how to set this up.

Examples of these systems would be:

a) A headless server with a dev stack: runs a build workflow, a web server, etc.

b) A stable "production" version of server (a)

c) A "desktop" dev environment: IDE, language/tools-rich

d) A "basics" desktop environment for non-tech work: office tools, maybe some media editing (even blender 3d if feeling lucky)

They won't all be running simultaneously.

I haven't worked on the OS systems and platforms level since a past life. I don't know much about virtualization/containers/etc, but I'm good with networking fundamentals, and I'm willing to learn any old/new tech.

Ideally:

1. The host OS is minimal (i.e. can only keep the core packages I need), and driver-friendly so the core hardware (compute/graphics) is efficiently accessible to each guest OS.

2. I can network the different "machines" together; so the dev desktop can use the dev server's API, and the basics desktop accesses the apps on the production server in its browser.

3. I can pull data from each system out into the host OS for data backups.

4. I can restrict network/internet access from/to any of those systems, from the host OS.

5. All systems are open-source, can work completely disconnected and do not phone home in order to function.

Thankful for any experience you can share.

jefurii 2 years ago |

I've been doing my work in VMs running on my local for a long time now and it works great. I got started because I wanted my development environment to be a close match for my stage and production.

For a long time I ran VirtualBox guests on a laptop host running Debian or Ubuntu. VB is great especially if you're getting started with virtual machines, it has a GUI and things are easy to understand. I used a host-only network to keep traffic inside my laptop.

If you're going to edit code inside a VM you'll need an IDE that can handle it. Emacs with tramp has served me well for years. IIRC with VSCode you have to install a server on the VM.

I do web development so I've never felt the need to run desktops inside VMs. It's nice to have some separation between work and stuff like HN but Gnome workspaces gives me just enough.

A couple years ago I levelled up and switched to KVM on a headless Debian box. This system uses bridge networking with DHCP from my router, and I use Tailscale to access my VMs from outside my LAN. You can use virt-manager (which also works for VirtualBox) but I mostly use the libvirt CLI tools and cockpit/cockpit-machines if I need something a little more visual. Caveat: I don't know how KVM would handle suspend on a laptop.

LinuxBender 2 years ago |

It sounds like you are describing QubesOS [1] I've used it with 16 GB of ram just fine provided each VM does not need a lot of ram. Here [2] are some screenshots and show how networks and firewalls are presented and a video walk-through [3]

[1] - https://www.qubes-os.org/intro/

[2] - https://www.qubes-os.org/screenshots/

[3] - https://www.youtube.com/live/hkWWz3xGqS8?feature=share&t=849

aborsy 2 years ago | |

How trustworthy is QubesOS?

LinuxBender 2 years ago | | |

Do you mean trustworthy in terms of the developers not being compromized by state actors, or do you mean in terms of security model and design?

In terms of security model it uses Xen to virtualize VM's and the default network isolation is decent enough for most people wanting network segmentation. The system administrator can either open up or further lock down the networking and firewall(s) to change how restricted VM's are from one another or the internet or the local networks. Their default implementation assumes that one may want to use a ram-only stateless or ephemeral OS to access Tor and another VM that can only talk to the Tor VM to a specific IP/port to prevent leakage. For other security scenarios people would have to create their own firewall rules and networking so that learning exercise would be on each person.

In terms of state actors there is really no way to answer that for any distribution as developers can be compelled into silence through assorted fear inducing tactics and lawful intercept can be a series of subtle design weaknesses that would not be spotted by the best external developers. These subtle design weaknesses can be a combination of OS libraries, combinations of hidden CPU registers and known x86 works as designed flaws.

blablabla123 2 years ago |

For an off-the-shelve solution there is Qubes but from what I've heard it's probably quite a pain to use unless you're comfortable debugging Linux desktop and driver problems in particular. Generally KVM/libvirt is nice because it's fast and can be set up transparently and networking can be customized in any possible way. (It might also be possible to get GPU Passthrough working with KVM but I think this only works if you have 2 GPUs)

dyingkneepad 2 years ago |

It really depends on why you want to have separate environments. What's the thing you're trying to avoid here?

Anything that requires graphics is much more efficiently done in the Bare Metal machine. You can have a separate user for those or some other kind of abstraction that's not a virtual machine.

You can also use Network Namespaces directly, using "ip netns" or even cgroups. That way you can run every single program natively, but have multiple network environments for them.

You can go for the jails-style containers where you also run everything natively but on a different chroot, without having a Kernel running on top of a Kernel. For this I recommend creating a chroot with debootstrap or febootstrap and then launching it with systemd-nspawn.

You can also have specific applications running on their own "jail" with /usr/bin/firejail.

Now if you want even stricter separation you'll have to go with full virtualization (guest Kernel running on top of Host Kernel). This is much worse in terms of efficiency of used resources, but gives you the most separation from one thing to another. For that you can use virt-manager or just Virtualbox.

robert_foss 2 years ago |

I think Debian for all of these usecases. Or Ubuntu.

gitgud 2 years ago |

If I understand the question correctly, you’d like to build a networked system entirely on a single machine.

If this is the case, then I’d recommend trying docker-compose. It’s quite an easy little system to get up and running, just need a single file to declare different containers and their relationships. Then just run “docker-compose up/down” to spin up and spin down the network.

The beauty of this is that it can run on extremely barebones Linux distributions, as it’s all container based.

It’s a great starting point anyway, and I’m sure there’s many alternatives now that use a similar api

mindcrash 2 years ago |

Have you considered devcontainers?

Its use results in carrying entire development environments with you, and because it is using containerization these will not clutter your host OS.

Using DevPod (https://devpod.sh/) you are not locked into Visual Studio or Visual Studio Code, but you can use whatever tool you want.

IMO this kind of setup will provide a much better DX than running a bunch of VMs eating away the resources of your laptop.

entropicgravity 2 years ago |

I use Linux Mint for a similar host set up. Its worth it to move away from virtualBox to the open source versions as I did recently, KVM etc.

There are quite a few paper cuts in setting it up the first time. I recall it took about five hours before I finally had my first VM in the right place, with right size of disk space and ram and the version of linux I wanted. But after the learning curve it becomes very easy. Honest :)

wesapien 2 years ago |

I've been using Fedora since 30 and it has been a very good OS. Install the dnf group for virtualization and you got a KVM host that let's you play with VMs and LXC. There is also a dnf group for containers.

iam-TJ 2 years ago |

I have something similar and have had for several years. Just needs a recent-ish kernel and decent distro!

I try to avoid virtual machines unless there is an absolute need to virtualise the hardware as well - in most cases single kernel, multiple containers is preferred.

Specifically, I use Debian with systemd-nspawn unprivileged (user namespaced) containers (managed by machinectl), defined and built using mkosi as BTRFS subvolumes (a BTRFS mounted on /var/lib/machines/ allows nspawn to create copy-on-write containers from a 'template' container).

The nice thing about nspawn is being able to create simple declarative configurations with private namespaces. With a private network namespace it has the concept of 'zones' which are containers that share the same (private) bridged network. systemd-networkd has built-in configs for both the host and containers to do automatic configuration.

I use this to create per-domain (e.g. example.com) zones with multiple function-specific containers (one for router/firewall, another for DHCP and DNS, others for single-sign-on, database, web, mail, and whatever else is required). I use veth to connect the zone selectively to the host interface public bridge interface providing external access if necessary.

This allows 'development' 'test' and 'production' domains using mkosi to define and build each repeatedly as needed. In the non-public domains the zone can be configured to use the same public IP addresses as production will use without it leaking outside the zone.

I use IPv6 only in the zones with both ULAs and global prefixes and DNS64/NAT64 in the 'router' container if the zone needs to make outbound connections (to IPv4 destinations).

I use netfilter rules applied to each zone's bridge interface to impose firewall restrictions.

Using veth pairs one can connect two containers (or zones) directly rather than using the common zone bridge interface.

I do dev work on the host itself; if you want to partition application usage I'd suggest keeping it simple by using multiple user accounts, one per 'task' so $USER configs and so on are all kept separate.

In the 'development' containers I do things like bind-mounting the base source-code directory, or project-specific directory, into the container. I do all source-code work on the host but build and test in the appropriate container.

For example I have 'linux-builder' container where I do out-of-tree kernel builds for multiple architectures. The container has the required crossbuild tools installed. Recently this has been used to build latest mainline kernel for amd64, arm64, and powerpc64 (for the Xbox 360!).

rvz 2 years ago |

Windows Subsystem for Linux.

Best 'Linux distro' for Windows.