Self Hosted Developers Guide - Part 1: Infrastructure
In this series, I'll be documenting my journey to self hosting my core development tools and infrastructure from git hosting, CI/CD, backups, container orchestration, and more. This article will cover the infrastructure I've set up for my development environment and continues integration.
Use Gitea for Git hosting
Use Drone.io (or woodpecker) for CI/CD
Use Docker Compose for container orchestration
With the recent turning of the economy and several companies doing layoffs, I’ve been thinking about the impending change of pricing models for the tools I use. Particularly the free tooling that I’ve so enjoyed using and have come accustomed to. As a matter of hedging my bets, I’d like to move my private, business related development tools to a self-hosted stack where I can control costs.
In this series I’m going to try and document my journey to self-hosting my core development tools and infrastructure with the following workflows in mind:
- Cloud Git hosting (ala GitHub)
- Continuous Integration (CI) / Continuous Deployment (CD)
- Publish Artifacts (binaries, docker images, etc)
- Automated Deployments (to staging, production, etc)
- PR Previews (if we’re lucky!)
I think the PR Previews is going to be the most difficult part of this journey, but we shall see where we end up! For the time being we can start with our core infrastructure. This will give us a sound foundation to build from in the coming articles.
Why These Choices
Gitea is a lightweight, self-hosted Git hosting solution. It’s written in Go and is a fork of Gogs. It’s a great alternative to GitHub, GitLab, and BitBucket. Specifically I chose Gitea for this because it:
- integrates very well with Drone.io
- has a light runtime footprint
- is easy to setup and configure
- has a UI very similar to GitHub
- Supports package registries (e.g. Docker, NPM, etc)
However there are some alternatives to consider if you’re not a fan of Gitea. Some of the alternatives include:
Drone.io is a lightweight CI/CD solution. It’s written in Go and provides a very simple YAML based configuration for your pipelines. It’s a great alternative to Jenkins, CircleCI, and TravisCI. Specifically I chose Drone.io for this because:
- I have used it in the past
- It was easy to setup and configure
- The FOSS fork (woodpecker) does not seem to be 100% production ready
Alternatives to Consider
Once Woodpecker hits v1.0 I’ll will probably switch to that, especially because it has compatibility with the Drone.io YAML configuration so I can reuse my pipelines.
We’ll skip the Virtual Machine and Docker setup portions of the stack and focus on the specifics of the tools we’re using. As such, before you start make sure you have the following ready to go:
- A Virtual Machine running linux (I’m using Debian 10)
- Docker and Docker Compose installed
After you have you environment setup, we’ll need a docker-compose file to orchestrate our containers. Specifically we’ll be deploying
- Drone.io Agent
There is a small chicken-egg problem we will run into with Drone and Gitea, but it’s fairly easy to solve once we get there.
Use the following docker-compose and create a file called
~/docker/gitea (or wherever you want to store your docker-compose files).
Be sure to update the following environment variables:
USER_GIDto match your user’s UID and GID
ROOT_URLto match the URL you’ll be using to access Gitea (e.g. http://10.10.10.101:3000)
DRONE_SERVER_HOSTto match the URL you’ll be using to access Drone.io (e.g. http://10.10.10.101:8000)
DRONE_RPC_SECRETto a secure random password or key
DRONE_GITEA_CLIENT_SECRET later on in the guide.
--- version: "3.7" services: # The Gitea Git Hosting Service # - We're going to mount the data directory to a local directory # inside of the VM so we can backup the ~/docker/gitea/data/* directory in a future post # - We're going to expose the UI on port 3000 # - We're going to expose the SSH port on port 2222 gitea: image: gitea/gitea:1.18 container_name: gitea networks: - web - gitea volumes: - ~/docker/gitea/data/gitea:/data - /etc/timezone:/etc/timezone:ro - /etc/localtime:/etc/localtime:ro ports: - "3000:3000" # UI - "2222:22" # SSH environment: - USER_UID=1101 - USER_GID=1101 - DISABLE_REGISTRATION=true - ROOT_URL=http://gitea:3000 restart: unless-stopped # The Drone.io CI/CD Service # - We're going to mount the data directory to a local directory # inside of the VM so we can backup the ~/docker/drone/data/* directory # - We're going to expose the UI on port 8000 drone: image: drone/drone:2 container_name: drone networks: - web - gitea volumes: - drone:/data ports: - "8000:80" environment: - DRONE_GITEA_SERVER=gitea:3000 - DRONE_GITEA_CLIENT_ID=abc123 # ~(-1) We'll Get These Later - DRONE_GITEA_CLIENT_SECRET=abc123 # ~(-1) We'll Get These Later - DRONE_RPC_SECRET=my-secret-rpc-password - DRONE_SERVER_HOST=http://drone:80 - DRONE_SERVER_PROTO=https # ~(-1) http or https restart: always # The Drone.io Agent # - We're going to mount the docker socket so we can run docker containers # inside of the drone agent # - We're going to set the capacity to 2 drone-runner: image: drone/drone-runner-docker:1 container_name: drone-runner networks: - web - gitea volumes: - /var/run/docker.sock:/var/run/docker.sock environment: # Corresponds to DRONE_RPC_SECRET in drone service - DRONE_RPC_SECRET=my-secret-rpc-password - DRONE_RPC_PROTO=http - DRONE_RPC_HOST=drone:80 - DRONE_RUNNER_CAPACITY=2 - DRONE_RUNNER_NAME=drone-runner restart: always networks: web: external: True gitea: external: False
Now that we have our docker-compose file, we can start our containers.
docker-compose up -d
Connecting Drone to Gitea
Navigate to your Gitea instance (e.g. http://10.10.10.101:3000) and run through the initial setup-process. Once you’ve done that you’ll need to create a new OAuth2 Application in Gitea.
/user/settings/applicationsand click the “Manage OAuth2 Applications” section and fill out the form.
Note that the redirect URL requires the
- Submit your form and you’ll be presented with a Client ID and Client Secret. Copy these values and paste them into your
DRONE_GITEA_CLIENT_ID = Client ID
DRONE_GITEA_CLIENT_SECRET = Client Secret
- Restart your Drone.io container
docker-compose restart drone
- Navigate to your Drone.io instance (e.g. http://10.10.10.101:8000) and sign-in. You should be redirected to you Gitea instance to authorize Drone.io to access your account.
If login is successful, you’ll be redirected back to Drone.io and you should be able to see your repositories (if any) and start writing pipelines!
And That’s It!
Now that we have our infrastructure setup, we can start writing some pipelines and automating our development process. In the next post we’ll be looking at how to setup a basic pipeline for a Go application and how to use Drone.io to build and deploy our application.