# YOLO COLO Capsul
## Testing
1. Go to https://yolo.servers.coop
2. Click login and enter a valid email address
3. Check your email (including spam 😾) for a login link, click it
4. Click "Account Balance", then "Add funds with Credit/Debit"
5. Enter some amount ($20?) and click through
6. Use these card details:
- Number: 4242424242424242
- Expiry: any future data
- CVV: any 3 digits
- Email: any
- Name: any
7. Click "SSH Public Keys"
8. Upload an SSH key
9. Click "Capsuls", "Create one"
10. Choose any available size, select Debian 10 image, choose your SSH key, click "Create"
11. Wait a few seconds. You probably need to reload the page once it completes, to display the IP
12. SSH `user@<ip address>`
### Testing notes
1. [FEATURE] Support additional currencies
2. [FEATURE] Disable BTCpay UI if backend not configured
3. [FEATURE] Make Cyberia support email address configurable
4. [BUG] Capsul SSH data not displaying on details page (SSH host key fingerprints, known hosts file)
- ~~3wc: think this is fixed by adding openssh-client to the Docker image, building now..~~
- ~~3wc: update: that still didn't work. seems the host can't communicate with the guests on their public IPs?~~
- mirsal: Fixed temporarily with a couple iptables rules, they are not persistent though so they would need to either be persisted somehow or re-added every time the host is rebooted or the network configuration is updated
5. [FUUTURE] Add IPv6 support (in progress)
6. ~~[BUG] IP address reporting breaks when VMs also have an IPv6 address~~
- ~~See https://git.autonomic.zone/3wordchant/capsul-flask/pulls/9 for a fix~~
Uncategorised:
- (mjray) - the text is very thin and low-contrast, especially links, but that may be the screen I am working on in today’s bright sun - well, and my defective eyes
- 3wc: tried to improve these with the re-skin, widened the gap between the bg and fg colours
- (mjray) - ssh public key upload seems to be a textarea not a file upload.
## V2, dedi
Local:
```
cd /path/to/autonomic/infrastructure
$EDITOR hosts/inventory
# add yolocolo server with default SSH port
ansible-playbook actions/newvps.yml -l yolocolo
$EDITOR hosts/inventory
# change yolocolo SSH port to 222
ansible-playbook actions/swarm.yml -l yolocolo
```
Add DNS records, `yolocolo 1800 IN A <IP>`, `*.yolocolo 1800 IN CNAME yolocolo`
SSH to the server, then:
```
sudo apt install libvirt-daemon-system virtinst git dnsmasq qemu qemu-kvm
sudo mkdir -p /var/www /tank/{vm,img,config}
sudo mkdir -p /tank/img/debian/11
cd !$
sudo wget https://cloud.debian.org/images/cloud/bullseye/20220310-944/debian-11-genericcloud-amd64-20220310-944.qcow2 -O root.img.qcow2
```
~~sudo wget https://cloud.debian.org/images/cloud/buster/20201023-432/debian-10-genericcloud-amd64-20201023-432.qcow2 -O root.img.qcow2~~ → that was oldstable
Then, back on your local machine:
```
abra server add yolocolo.doesthisthing.work <username> 222
abra server yolocolo.doesthisthing.work init
abra app new traefik --server yolocolo.doesthisthing.work --domain traefik.yolocolo.doesthisthing.work --app-name traefik_yolocolo_doesthisthing_work
abra app new capsul --server yolocolo.doesthisthing.work --domain yolocolo.doesthisthing.work --app-name capsul_yolocolo_doesthisthing_work
abra app traefik_yolocolo_doesthisthing_work deploy
abra app capsul_yolocolo_doesthisthing_work config
abra app capsul_yolocolo_doesthisthing_work deploy
```
Add to `/tank/config/cyberia-cloudinit.yml` on the host:
```
#cloud-config
package_upgrade: true
packages:
- curl
- wget
- git
- vim
bootcmd:
- echo 'iface ens3 inet6 dhcp' >> /etc/network/interfaces.d/50-cloud-init
- ifdown ens3; ifup ens3
final_message: "The system is finally up, after $UPTIME seconds"
users:
- name: user
groups: sudo
sudo: ['ALL=(ALL) NOPASSWD:ALL']
ssh-authorized-keys:
```
Network definition:
(make sure `dev` is the main ethernet interface, and `public3` and `bridge3` match what's hardcoded in Capsul)
```
<network>
<name>public3</name>
<forward mode="route" dev="enp3s0"/>
<bridge name="virbr3" />
<ip address="148.251.164.81" netmask="255.255.255.240">
<dhcp>
<range start="148.251.164.82" end="148.251.164.94"/>
</dhcp>
</ip>
<ip family='ipv6' address='2a01:4f8:192:2d3:fe::' prefix='96'>
<dhcp>
<range start='2a01:4f8:192:2d3:fe::1000' end='2a01:4f8:192:2d3:fe::2000'/>
</dhcp>
</ip>
</network>
```
```
sudo virsh net-define /tmp/public3.xml
sudo virsh net-start public3
```
Plus:
- Sign up for a Stripe account and put the test API public and private keys into the Capsul .env file
- Get a DB shell, and:
- `DELETE FROM host_network WHERE network_name IN ('public1', 'public2')`
- `UPDATE host_network SET ...` (fix IP addresses)
### Disk size increase
```
sudo cp capsul-vltnn1oemu.qcow2 capsul-vltnn1oemu.backup.qcow2
sudo qemu-img resize capsul-vltnn1oemu.qcow2 +125G
sudo virt-resize --expand /dev/sda1 capsul-vltnn1oemu.backup.qcow2 capsul-vltnn1oemu.qcow2
```
### Getting host key scanning to work
Put the following in `/etc/libvirt/hooks/network`
```
#!/bin/sh
VIRT_PUBLIC_NET=public3
VIRT_IF_NAME=virbr3
DOCKER_GWBRIDGE=docker_gwbridge
if [ "${1}" = "$VIRT_PUBLIC_NET" -a "${2}" = "started" ]; then
/sbin/iptables -I FORWARD -i $VIRT_IF_NAME -o $DOCKER_GWBRIDGE -m state --state RELATED,ESTABLISHED -j ACCEPT
/sbin/iptables -I FORWARD -o $VIRT_IF_NAME -i $DOCKER_GWBRIDGE -j ACCEPT
fi
if [ "${1}" = "$VIRT_PUBLIC_NET" -a "${2}" = "stopped" ]; then
/sbin/iptables -D FORWARD -i $VIRT_IF_NAME -o $DOCKER_GWBRIDGE -m state --state RELATED,ESTABLISHED -j ACCEPT
/sbin/iptables -D FORWARD -o $VIRT_IF_NAME -i $DOCKER_GWBRIDGE -j ACCEPT
fi
```
Then make it executable with `chmod +x /etc/libvirt/hooks/network`
## V1, VPS
```
sudo apt install libvirt-daemon-system virtinst git dnsmasq qemu qemu-kvm python3-venv git build-essential python3-dev libpg-dev libjpeg-dev postgresql-server
sudo -u postgres createuser -P capsul
sudo -u postgres createdb -O capsul capsul
sudo mkdir -p /var/www /tank/{vm,img,config}
sudo mkdir -p /tank/img/debian/10
cd !$
sudo wget https://cloud.debian.org/images/cloud/buster/20201023-432/debian-10-genericcloud-amd64-20201023-432.qcow2 -O root.img.qcow2
sudo adduser --system --home /var/www capsul
cd /var/www/capsul
sudo -u capsul -s
git clone https://git.autonomic.zone/3wordchant/capsul.git
python3 -m venv venv
source venv/bin/activate
cd capsul
pip install pipenv cppy
pipenv install
pipenv run gunicorn --bind 0.0.0.0:5000 -k gevent --worker-connections 1000 app:app
```
```
# /tank/config/cyberia-cloudinit.yml
#cloud-config
package_upgrade: true
packages:
- curl
- wget
- git
- vim
hostname: web01
fqdn: web01.example.com
final_message: "The system is finally up, after $UPTIME seconds"
users:
- name: user
groups: sudo
sudo: ['ALL=(ALL) NOPASSWD:ALL']
ssh-authorized-keys:
```
## Issues
- public network not working
- `capsul` user doesn't have permission to create, can't see network created by root
- TODO: research if there's a way to make the root-created network shared
- ~~missing upstream `cyberia-cloudinit.yml`, unsure how close this version is~~
- spicy hot changes to capsul-flask:
- reduce memory reservation from 20GB to 0.5 GB 🙃
- fetch free memory info from /proc/meminfo instead of ZFS
- TODO: ask about patching this
- don't use kvm (see below)
- KVM doesnt seem to work on Hetzner cloud instance
- ~~500 error on capsul create, think this happens even with SPOKE_MODEL=mock~~
## Scratch
https://stackoverflow.com/questions/48422001/how-to-launch-qemu-kvm-from-inside-a-docker-container
libvirt-client
IPtables:
```
# Generated by xtables-save v1.8.2 on Mon Jun 1 18:48:22 2020
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -i virbr1 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr1 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr1 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr1 -p tcp -m tcp --dport 67 -j ACCEPT
# accept on virbrs
-A FORWARD -i virbr1 -o virbr1 -j ACCEPT
-A FORWARD -i virbr1 -o virbr2 -j ACCEPT
# enp3s0 -> virt networks
-A FORWARD -d 69.61.2.160/27 -i enp3s0 -o virbr1 -j ACCEPT
# virt networks -> enp3s0
-A FORWARD -s 69.61.2.160/27 -i virbr1 -o enp3s0 -j ACCEPT
# reject all else
-A FORWARD -i virbr1 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -o virbr1 -j REJECT --reject-with icmp-port-unreachable
-A OUTPUT -o virbr1 -p udp -m udp --dport 68 -j ACCEPT
COMMIT
# Completed on Mon Jun 1 18:48:22 2020
# Generated by xtables-save v1.8.2 on Mon Jun 1 18:48:22 2020
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING -o virbr1 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
COMMIT
# Completed on Mon Jun 1 18:48:22 2020
```