Self Hosted Developers Guide - Part 1: Infrastructure

In this series, I'll be documenting my journey to self hosting my core development tools and infrastructure from git hosting, CI/CD, backups, container orchestration, and more. This article will cover the infrastructure I've set up for my development environment and continues integration.

With the recent turning of the economy and several companies doing layoffs, I’ve been thinking about the impending change of pricing models for the tools I use. Particularly the free tooling that I’ve so enjoyed using and have come accustomed to. As a matter of hedging my bets, I’d like to move my private, business related development tools to a self-hosted stack where I can control costs.

In this series I’m going to try and document my journey to self-hosting my core development tools and infrastructure with the following workflows in mind:

• Cloud Git hosting (ala GitHub)
• Continuous Integration (CI) / Continuous Deployment (CD)
• Publish Artifacts (binaries, docker images, etc)
• Automated Deployments (to staging, production, etc)
• PR Previews (if we’re lucky!)

I think the PR Previews is going to be the most difficult part of this journey, but we shall see where we end up! For the time being we can start with our core infrastructure. This will give us a sound foundation to build from in the coming articles.

Why These Choices

Gitea

Gitea is a lightweight, self-hosted Git hosting solution. It’s written in Go and is a fork of Gogs. It’s a great alternative to GitHub, GitLab, and BitBucket. Specifically I chose Gitea for this because it:

• integrates very well with Drone.io
• has a light runtime footprint
• is easy to setup and configure
• has a UI very similar to GitHub
• Supports package registries (e.g. Docker, NPM, etc)

However there are some alternatives to consider if you’re not a fan of Gitea. Some of the alternatives include:

• Gogs
• OneDev
• Gitlab

Drone.io

Drone.io is a lightweight CI/CD solution. It’s written in Go and provides a very simple YAML based configuration for your pipelines. It’s a great alternative to Jenkins, CircleCI, and TravisCI. Specifically I chose Drone.io for this because:

• I have used it in the past
• It was easy to setup and configure
• The FOSS fork (woodpecker) does not seem to be 100% production ready

Alternatives to Consider

• Jenkins
• Woodpecker

Once Woodpecker hits v1.0 I’ll will probably switch to that, especially because it has compatibility with the Drone.io YAML configuration so I can reuse my pipelines.

Deploying Containers

We’ll skip the Virtual Machine and Docker setup portions of the stack and focus on the specifics of the tools we’re using. As such, before you start make sure you have the following ready to go:

• A Virtual Machine running linux (I’m using Debian 10)
• Docker and Docker Compose installed

After you have you environment setup, we’ll need a docker-compose file to orchestrate our containers. Specifically we’ll be deploying

1. Gitea
2. Drone.io
3. Drone.io Agent

There is a small chicken-egg problem we will run into with Drone and Gitea, but it’s fairly easy to solve once we get there.

Use the following docker-compose and create a file called docker-compose.yml in ~/docker/gitea (or wherever you want to store your docker-compose files).

Be sure to update the following environment variables:

• USER_UID and USER_GID to match your user’s UID and GID
• ROOT_URL to match the URL you’ll be using to access Gitea (e.g. http://10.10.10.101:3000)
• DRONE_SERVER_HOST to match the URL you’ll be using to access Drone.io (e.g. http://10.10.10.101:8000)
• DRONE_RPC_SECRET to a secure random password or key

We’ll set DRONE_GITEA_CLIENT_ID and DRONE_GITEA_CLIENT_SECRET later on in the guide.

---
version: "3.7"
services:
  # The Gitea Git Hosting Service
  #  - We're going to mount the data directory to a local directory
  #    inside of the VM so we can backup the ~/docker/gitea/data/* directory in a future post
  #  - We're going to expose the UI on port 3000
  #  - We're going to expose the SSH port on port 2222
  gitea:
    image: gitea/gitea:1.18
    container_name: gitea
    networks:
      - web
      - gitea
    volumes:
      - ~/docker/gitea/data/gitea:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "3000:3000" # UI
      - "2222:22" # SSH
    environment:
      - USER_UID=1101
      - USER_GID=1101
      - DISABLE_REGISTRATION=true
      - ROOT_URL=http://gitea:3000
    restart: unless-stopped

  # The Drone.io CI/CD Service
  #  - We're going to mount the data directory to a local directory
  #    inside of the VM so we can backup the ~/docker/drone/data/* directory
  #  - We're going to expose the UI on port 8000
  drone:
    image: drone/drone:2
    container_name: drone
    networks:
      - web
      - gitea
    volumes:
      - drone:/data
    ports:
      - "8000:80"
    environment:
      - DRONE_GITEA_SERVER=gitea:3000
      - DRONE_GITEA_CLIENT_ID=abc123 # ~(-1) We'll Get These Later
      - DRONE_GITEA_CLIENT_SECRET=abc123 # ~(-1) We'll Get These Later
      - DRONE_RPC_SECRET=my-secret-rpc-password
      - DRONE_SERVER_HOST=http://drone:80
      - DRONE_SERVER_PROTO=https # ~(-1) http or https
    restart: always

  # The Drone.io Agent
  #  - We're going to mount the docker socket so we can run docker containers
  #    inside of the drone agent
  #  - We're going to set the capacity to 2
  drone-runner:
    image: drone/drone-runner-docker:1
    container_name: drone-runner
    networks:
      - web
      - gitea
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      # Corresponds to DRONE_RPC_SECRET in drone service
      - DRONE_RPC_SECRET=my-secret-rpc-password
      - DRONE_RPC_PROTO=http
      - DRONE_RPC_HOST=drone:80
      - DRONE_RUNNER_CAPACITY=2
      - DRONE_RUNNER_NAME=drone-runner
    restart: always

networks:
  web:
    external: True
  gitea:
    external: False

Now that we have our docker-compose file, we can start our containers.

docker-compose up -d

Connecting Drone to Gitea

1. Navigate to your Gitea instance (e.g. http://10.10.10.101:3000) and run through the initial setup-process. Once you’ve done that you’ll need to create a new OAuth2 Application in Gitea.

2. Navigate to /user/settings/applications and click the “Manage OAuth2 Applications” section and fill out the form.

Note that the redirect URL requires the /login path

3. Submit your form and you’ll be presented with a Client ID and Client Secret. Copy these values and paste them into your docker-compose.yml file.

DRONE_GITEA_CLIENT_ID = Client ID

DRONE_GITEA_CLIENT_SECRET = Client Secret

4. Restart your Drone.io container

docker-compose restart drone

5. Navigate to your Drone.io instance (e.g. http://10.10.10.101:8000) and sign-in. You should be redirected to you Gitea instance to authorize Drone.io to access your account.

If login is successful, you’ll be redirected back to Drone.io and you should be able to see your repositories (if any) and start writing pipelines!

And That’s It!

Now that we have our infrastructure setup, we can start writing some pipelines and automating our development process. In the next post we’ll be looking at how to setup a basic pipeline for a Go application and how to use Drone.io to build and deploy our application.