overseer

Monitorable, gracefully restarting, self-upgrading binaries in Go (golang)

The main goal of this project is to facilitate the creation of self-upgrading binaries which play nice with standard process managers, secondly it should expose a simple API with reasonable defaults for maximum user enjoyment.

⚠️ This is beta software. Do not use in production. Consider the API unstable. Please report any issues as you encounter them.

Features

• Simple
• Works with process managers
• Graceful, zero-down time restarts
• Easy self-upgrading binaries

Install

go get github.com/jpillora/overseer

Quick example

This program works with process managers, supports graceful, zero-down time restarts and self-upgrades its own binary.

package main

import (
	"fmt"
	"log"
	"net/http"
	"time"

	"github.com/jpillora/overseer"
	"github.com/jpillora/overseer/fetcher"
)

//create another main() to run the overseer process
//and then convert your old main() into a 'prog(state)'
func main() {
	overseer.Run(overseer.Config{
		Program: prog,
		Address: ":3000",
		Fetcher: &fetcher.HTTP{
			URL:      "http://localhost:4000/binaries/myapp",
			Interval: 1 * time.Second,
		},
	})
}

//prog(state) runs in a child process
func prog(state overseer.State) {
	log.Printf("app (%s) listening...", state.ID)
	http.Handle("/", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintf(w, "app (%s) says hello\n", state.ID)
	}))
	http.Serve(state.Listener, nil)
}

How it works

• overseer uses the main process to check for and install upgrades and a child process to run Program.
• The main process retrieves the files of the listeners described by Address/es.
• The child process is provided with these files which is converted into a Listener/s for the Program to consume.
• All child process pipes are connected back to the main process.
• All signals received on the main process are forwarded through to the child process.
• Fetcher runs in a goroutine and checks for updates at preconfigured interval. When Fetcher returns a valid binary stream (io.Reader), the master process saves it to a temporary location, verifies it, replaces the current binary and initiates a graceful restart.
• The fetcher.HTTP accepts a URL, it polls this URL with HEAD requests and until it detects a change. On change, we GET the URL and stream it back out to overseer. See also fetcher.S3.
• Once a binary is received, it is run with a simple echo token to confirm it is a overseer binary.
• Except for scheduled upgrades, the child process exiting will cause the main process to exit with the same code. So, overseer is not a process manager.

See Configuration options here and the runtime State available to your program here.

More examples

• See the example/ directory and run example.sh, you should see the following output:

  $ cd example/
  $ sh example.sh
  serving . on port 5002
  BUILT APP (1)
  RUNNING APP
  app#1 (1cd8b9928d44b0a6e89df40574b8b6d20a417679) listening...
  app#1 (1cd8b9928d44b0a6e89df40574b8b6d20a417679) says hello
  app#1 (1cd8b9928d44b0a6e89df40574b8b6d20a417679) says hello
  BUILT APP (2)
  app#2 (b9b251f1be6d0cc423ef921f107cb4fc52f760b3) listening...
  app#2 (b9b251f1be6d0cc423ef921f107cb4fc52f760b3) says hello
  app#2 (b9b251f1be6d0cc423ef921f107cb4fc52f760b3) says hello
  app#1 (1cd8b9928d44b0a6e89df40574b8b6d20a417679) says hello
  app#1 (1cd8b9928d44b0a6e89df40574b8b6d20a417679) exiting...
  BUILT APP (3)
  app#3 (248f80ea049c835e7e3714b7169c539d3a4d6131) listening...
  app#3 (248f80ea049c835e7e3714b7169c539d3a4d6131) says hello
  app#3 (248f80ea049c835e7e3714b7169c539d3a4d6131) says hello
  app#2 (b9b251f1be6d0cc423ef921f107cb4fc52f760b3) says hello
  app#2 (b9b251f1be6d0cc423ef921f107cb4fc52f760b3) exiting...
  app#3 (248f80ea049c835e7e3714b7169c539d3a4d6131) says hello
  

• Only use graceful restarts

  func main() {
  	overseer.Run(overseer.Config{
  		Program: prog,
  		Address: ":3000",
  	})
  }
  

  Send main a SIGUSR2 (Config.RestartSignal) to manually trigger a restart

• Only use auto-upgrades, no restarts

  func main() {
  	overseer.Run(overseer.Config{
  		Program: prog,
  		NoRestartAfterFetch: true
  		Fetcher: &fetcher.HTTP{
  			URL:      "http://localhost:4000/binaries/myapp",
  			Interval: 1 * time.Second,
  		},
  	})
  }
  

  Your binary will be upgraded though it will require manual restart from the user, suitable for creating self-upgrading command-line applications.

Known issues

• The master process's overseer.Config cannot be changed via an upgrade, the master process must be restarted.
• Currently shells out to mv for moving files because mv handles cross-partition moves unlike os.Rename.
• Addresses can only be changed by restarting the main process.
• Only supported on darwin and linux.

More documentation

• Core overseer package
• Common fetcher.Interface
• HTTP fetcher type
• S3 fetcher type

Docker

1. Compile your overseerable app to a /path/on/docker/host/dir/app

2. Then run it with:

   docker run -d -v /path/on/docker/host/dir/:/home/ -w /home/ debian  -w /home/app
   

3. For testing, swap out -d (daemonize) for --rm -it (remove on exit, input, terminal)

4. app can use the current working directory as storage

5. If the OS doesn't ship with TLS certs, you can mount them from the host with -v /etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt

TODO

• Tests! The test suite should drive an:
  • HTTP client for application version/uptime testing
  • HTTP server for application upgrades
  • overseer binary process
• HTTP fetcher long-polling
• SCP fetcher (connect to a server, poll path)
• Github fetcher (given a repo, poll releases)
• etcd fetcher (given a cluster, watch key)
• overseer CLI tool (TODO)
• overseer package
  • Execute and verify calculated delta updates with https://github.com/kr/binarydist
  • Omaha client support