Run on Virtual Machine/Bare Metal

This tutorial goes through the procedure of cluster setup to achieve high availablity and scalability when OneDev is running on virtual machine/bare metal.

Set Up External Database

Set up an external database to be used by OneDev. List of databases supported by OneDev:

1. MySQL 5.0 or higher
2. SQL Server 2008 or higher
3. PostgreSQL 9.0 or higher
4. MariaDB 5.0 or higher

Run Multiple OneDev Servers

1. Run multiple OneDev servers on multiple machines, and configure them to connect to the external database above

2. Servers talk to each other via cluster_ip specified in conf/server.properties. If left empty, OneDev will detect the ip address automatically

3. If you want to run multiple servers on same machine for a test setup, make sure to edit conf/server.properties use different value for http_port, ssh_port and cluster_port

4. Make sure various ports defined in conf/server.properties can be accessed from other OneDev servers and cluster load balancer (will configure later)

Request a Trial Subscription

High availability and scalability requires a subscription to use. Switch to menu item Administration / Subscription Management, request a trial subscription key and install it into OneDev if you do not have a subscription yet:

Configure Project Replicas

Switch to menu item Administration / High Availability & Scalability (will be available after trial subscription is activated above), and you will see two OneDev servers in the cluster. Let's change project replicas to 2 and click the button Save Settings & Redistribute Projects:

This tells OneDev to maintain two replicas of each project on different servers. Now create a new project, and you can check the replica status like below:

Do something with the project such as pushing files or running builds, you will see that the replica is update to date. In case you need to find projects without enough replicas, or with outdated replicas, run below project query:

without enough replicas or has outdated replicas 

Configure Front End Load Balancer

A front end load balancer should also be configured to serve as single access point of the cluster, either for users or for CI/CD agents. We will set up Nginx as load balancer.

1. Set up load balancing for http protcol

   To set up load balancing for http protocol, add an upstream directive in http directive and add entry for each OneDev server in the cluster

   upstream backend {
     ip_hash; # use this if you can not use session sticky which is only available in Nginx+
     server onedev_server1:6610;
     server onedev_server2:6610;
   }

   Then configure the virtual host to use this upstream:

   server {
     listen 80;
     listen [::]:80;
   
     server_name onedev.example.com;
   
     # no size limit of uploaded file
     client_max_body_size 0; 
   
     location /wicket/websocket {
         proxy_pass http://backend/wicket/websocket;
         proxy_http_version 1.1;
         proxy_set_header Upgrade $http_upgrade;
         proxy_set_header Connection "upgrade";                
         proxy_read_timeout 1800s; 
     }
   
     location /~server {
         proxy_pass http://backend/~server;
         proxy_http_version 1.1;
         proxy_set_header Upgrade $http_upgrade;
         proxy_set_header Connection "upgrade";
     }
   
     location /~api/streaming {
         proxy_pass http://backend/~api/streaming;
         proxy_buffering off;
     }
   
     location / {
         proxy_pass http://backend/;
     }
   }

2. Set up load balancing for SSH protocol

   OneDev repositories can also be accessed via SSH protocol. To load balancing SSH access, Nginx needs to have stream module loaded (this is default on Ubuntu). Add below as root directive of nginx.conf (do not put inside http directive):

   stream {
     upstream ssh {
       server onedev_server1:6611;
       server onedev_server2:6611;
     }
     server {
       listen 22; # use a different port if port 22 is occupied
       proxy_pass ssh;
     }
   }

3. Update server url and ssh root url in OneDev system settings to point to the load balancer

Now you should be able to access OneDev web UI via the dns name. Repositories can also be access over http/ssh protocol via the dns name. When one server fails, traffic will be failed over to another server automatically. If you have agents to run CI/CD jobs, they should also be configured to connect to OneDev dns name to benefit from load balancing and fail over.

caution

1. OneDev can be configured to send alert email via Administratio / Alert Settings when a server is down abnormally. After fixing the issue, make sure to redistribute projects again if new server is added or existing server is removed
2. This tutorial does not include HA setup of database. Refer to database specific guide if you need to do that