Skip to main content

High Availability and Scalability

note

This feature is available in enterprise edition only. Try free for 30 days

Multiple OneDev servers connecting to same database will form a cluster automatically. With this cluster, you can distribute different projects to different servers for horizontal scaling, as well as replicate project repositories and artifacts for high availability.

Connect to Same External Database​

Firstable we need to configure different OneDev servers to connect to same database. So the default internal database will not work here. Refer to this procedure to switch to an external database if necessary

caution

Do not use loopback address (localhost, 127.0.0.1, etc.) in database connection url even if database server resides on same machine as OneDev. Otherwise, OneDev servers running on other machine will complain with error Invalid servers detected in cluster: loopback address should not be used

Set up Cluster IP and Ports​

OneDev servers need to talk to each other for load distribution and data replication via property cluster_ip, cluster_port, http_port and ssh_port defined in conf/server.properties:

cluster_ip​

Other servers will connect to this server via this ip address. It is left empty by default, and OneDev will detect the ip address automatically. In case other servers can not connect to auto-detected ip address, you will need to specify it explicitly

cluster_port​

Other servers will connect to cluster_ip and this port for cluster management. Default value of this property is 5710

http_port​

Other servers and front end will connect to cluster_ip and this port to relay web requests and git requests from end users. Other servers also use this port for data replication. Default value of this property is 6610

ssh_port​

Other servers and front end will connect to cluster_ip and this port to relay git requests over SSH protocol. Default value of this property is 6611

info

These properties can also be configured via environment variables with same name

Project Replicas and Distribution​

When a new OneDev server is added to the cluster, only new projects may get distributed to it. Existing projects need to be redistributed manually via menu Administration / High Availability & Scalability:

distribute projects

You may also configure number of project replicas here. If a project has multiple replicas, only one replica will be active, and others will be backups. If the server hosting the active replica is down, one of the backup replica on other server will be active to continue the service. You may check replica status of a project like below:

replica status

note

OneDev may take some time to redistribute your projects. The time taking depends on how many projects you have, size of your projects (repositories, artifacts etc), and how powerful is your machine

You may use below project query to find out projects without enough replicas, or with outdated replicas:

without enough replicas or has outdated replicas

Configure Front End Load Balancer​

A front end load balancer should also be configured to server as single access point of the cluster, either for users or for CI/CD agents. When deployed in Kubernetes, you may simply configure the service type to be LoadBalancer, or use ingress to route traffic. For other installation types, it is suggested to install Nginx to serve as load balancer.

Nginx Load Balancing for Http Protcol​

To set up load balancing for http protocol, add an upstream directive in http directive and add entry for each OneDev server in the cluster

upstream backend {
  ip_hash; # use this if you can not use session sticky which is only available in Nginx+
  server onedev_server1:6610;
  server onedev_server2:6610;
}

Then configure the virtual host to use this upstream:

server {
  listen 80;
  listen [::]:80;

  server_name onedev.example.com;

  # no size limit of uploaded file
  client_max_body_size 0; 

  location /wicket/websocket {
      proxy_pass http://backend/wicket/websocket;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";                
      proxy_read_timeout 1800s; 
  }

  location /~server {
      proxy_pass http://backend/~server;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
  }

  location / {
      proxy_pass http://backend/;
  }
}

Make sure to update server url at OneDev side via menu administration / system setting with the server name specified in server directive above. Users and CI/CD agents accessing this server name will get the traffic load balanced, and can also fail over to working servers if one server fails

Nginx Load Lalancing for SSH Protocol​

OneDev repositories can also be accessed via SSH protocol. To load balancing SSH access, Nginx needs to have stream module loaded (this is default on Ubuntu). Add below as root directive of nginx.conf (do not put inside http directive):

stream {
  upstream ssh {
    server onedev_server1:6611;
    server onedev_server2:6611;
  }
  server {
    listen 22; # use a different port if port 22 is occupied
    proxy_pass ssh;
  }
}

Then update SSH root url at OneDev side via menu administration / system setting.

Configure Alert Setting​

When a server is shutdown abnormally, OneDev will show an alert notifying you to take action:

alert

In a production system, you should also configure alert setting to send important events via email:

alert setting

If a new server is added or existing server is removed finally, make sure to redistribute projects to satisfy project replica requirement

Database High Availability & Scalability​

Database is external to OneDev, and you will need to follow database specific guide to set up high availability and scalability if desired.

Future Reading​

Tutorials available to guide how to set up HA in docker and Kubernetes