Monitorable, gracefully restarting, self-upgrading binaries in Go (golang)
 
 
Go to file
Jaime Pillora 2e4decb22e add change detection to file fetcher, cleanup code, fix spellings, swap example code to use file fetcher 2017-08-24 20:24:47 +10:00
cmd rename to overseer, allow manual restarts with USR2, fetcher now optional, added fetcher init method 2016-02-09 14:46:45 +11:00
example add change detection to file fetcher, cleanup code, fix spellings, swap example code to use file fetcher 2017-08-24 20:24:47 +10:00
fetcher add change detection to file fetcher, cleanup code, fix spellings, swap example code to use file fetcher 2017-08-24 20:24:47 +10:00
.gitignore fixed on linux with mv 2016-02-09 00:20:50 +11:00
CONTRIBUTING.md clarify signing 2016-03-06 22:52:24 +11:00
LICENSE rewrite complete, mostly working now 2016-02-08 12:06:54 +11:00
README.md expose overseer sanity check (closes #9), remove beta warning 2016-05-25 15:53:56 +10:00
graceful.go added github fetcher, added auto-extraction of gzip archives (gzip encoded responses are already supported) 2016-06-21 23:09:12 +10:00
overseer.go add change detection to file fetcher, cleanup code, fix spellings, swap example code to use file fetcher 2017-08-24 20:24:47 +10:00
proc_master.go add change detection to file fetcher, cleanup code, fix spellings, swap example code to use file fetcher 2017-08-24 20:24:47 +10:00
proc_slave.go add change detection to file fetcher, cleanup code, fix spellings, swap example code to use file fetcher 2017-08-24 20:24:47 +10:00
sys_posix.go Chmod and Chown may not be supported 2016-04-01 22:37:28 +02:00
sys_posix_mv.go Added many more docs. Added an extra log level: warn, which is on by default. Normal operation are logs, unexcepted events are warnings. Added IDs to slave logs. 2016-02-13 17:11:17 +11:00
sys_unsupported.go Chmod and Chown may not be supported 2016-04-01 22:37:28 +02:00
sys_windows.go add change detection to file fetcher, cleanup code, fix spellings, swap example code to use file fetcher 2017-08-24 20:24:47 +10:00

README.md

overseer

GoDoc

overseer is a package for creating monitorable, gracefully restarting, self-upgrading binaries in Go (golang). The main goal of this project is to facilitate the creation of self-upgrading binaries which play nice with standard process managers, secondly it should expose a small and simple API with reasonable defaults.

overseer diagram

Commonly, graceful restarts are performed by the active process (dark blue) closing its listeners and passing these matching listening socket files (green) over to a newly started process. This restart causes any foreground process monitoring to incorrectly detect a program crash. overseer attempts to solve this by using a small process to perform this socket file exchange and proxying signals and exit code from the active process.

Features

  • Simple
  • Works with process managers
  • Graceful, zero-down time restarts
  • Easy self-upgrading binaries

Install

go get github.com/jpillora/overseer

Quick example

This program works with process managers, supports graceful, zero-down time restarts and self-upgrades its own binary.

package main

import (
	"fmt"
	"log"
	"net/http"
	"time"

	"github.com/jpillora/overseer"
	"github.com/jpillora/overseer/fetcher"
)

//create another main() to run the overseer process
//and then convert your old main() into a 'prog(state)'
func main() {
	overseer.Run(overseer.Config{
		Program: prog,
		Address: ":3000",
		Fetcher: &fetcher.HTTP{
			URL:      "http://localhost:4000/binaries/myapp",
			Interval: 1 * time.Second,
		},
	})
}

//prog(state) runs in a child process
func prog(state overseer.State) {
	log.Printf("app (%s) listening...", state.ID)
	http.Handle("/", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintf(w, "app (%s) says hello\n", state.ID)
	}))
	http.Serve(state.Listener, nil)
}

How it works:

  • overseer uses the main process to check for and install upgrades and a child process to run Program.
  • The main process retrieves the files of the listeners described by Address/es.
  • The child process is provided with these files which is converted into a Listener/s for the Program to consume.
  • All child process pipes are connected back to the main process.
  • All signals received on the main process are forwarded through to the child process.
  • Fetcher runs in a goroutine and checks for updates at preconfigured interval. When Fetcher returns a valid binary stream (io.Reader), the master process saves it to a temporary location, verifies it, replaces the current binary and initiates a graceful restart.
  • The fetcher.HTTP accepts a URL, it polls this URL with HEAD requests and until it detects a change. On change, we GET the URL and stream it back out to overseer. See also fetcher.S3.
  • Once a binary is received, it is run with a simple echo token to confirm it is a overseer binary.
  • Except for scheduled restarts, the active child process exiting will cause the main process to exit with the same code. So, overseer is not a process manager.

See Configuration options here and the runtime State available to your program here.

More examples

  • See the example/ directory and run example.sh, you should see the following output:

    $ cd example/
    $ sh example.sh
    serving . on port 5002
    BUILT APP (1)
    RUNNING APP
    app#1 (1cd8b9928d44b0a6e89df40574b8b6d20a417679) listening...
    app#1 (1cd8b9928d44b0a6e89df40574b8b6d20a417679) says hello
    app#1 (1cd8b9928d44b0a6e89df40574b8b6d20a417679) says hello
    BUILT APP (2)
    app#2 (b9b251f1be6d0cc423ef921f107cb4fc52f760b3) listening...
    app#2 (b9b251f1be6d0cc423ef921f107cb4fc52f760b3) says hello
    app#2 (b9b251f1be6d0cc423ef921f107cb4fc52f760b3) says hello
    app#1 (1cd8b9928d44b0a6e89df40574b8b6d20a417679) says hello
    app#1 (1cd8b9928d44b0a6e89df40574b8b6d20a417679) exiting...
    BUILT APP (3)
    app#3 (248f80ea049c835e7e3714b7169c539d3a4d6131) listening...
    app#3 (248f80ea049c835e7e3714b7169c539d3a4d6131) says hello
    app#3 (248f80ea049c835e7e3714b7169c539d3a4d6131) says hello
    app#2 (b9b251f1be6d0cc423ef921f107cb4fc52f760b3) says hello
    app#2 (b9b251f1be6d0cc423ef921f107cb4fc52f760b3) exiting...
    app#3 (248f80ea049c835e7e3714b7169c539d3a4d6131) says hello
    

    Note: app#1 stays running until the last request is closed.

  • Only use graceful restarts:

    func main() {
    	overseer.Run(overseer.Config{
    		Program: prog,
    		Address: ":3000",
    	})
    }
    

    Send main a SIGUSR2 (Config.RestartSignal) to manually trigger a restart

  • Only use auto-upgrades, no restarts

    func main() {
    	overseer.Run(overseer.Config{
    		Program: prog,
    		NoRestart: true,
    		Fetcher: &fetcher.HTTP{
    			URL:      "http://localhost:4000/binaries/myapp",
    			Interval: 1 * time.Second,
    		},
    	})
    }
    

    Your binary will be upgraded though it will require manual restart from the user, suitable for creating self-upgrading command-line applications.

  • Multi-platform binaries using a dynamic fetch URL

    func main() {
    	overseer.Run(overseer.Config{
    		Program: prog,
    		Fetcher: &fetcher.HTTP{
    			URL: "http://localhost:4000/binaries/app-"+runtime.GOOS+"-"+runtime.GOARCH,
    			//e.g.http://localhost:4000/binaries/app-linux-amd64
    		},
    	})
    }
    

Known issues

  • The master process's overseer.Config cannot be changed via an upgrade, the master process must be restarted.
    • Therefore, Addresses can only be changed by restarting the main process.
  • Currently shells out to mv for moving files because mv handles cross-partition moves unlike os.Rename.
  • Only supported on darwin and linux.
  • Package init() functions will run twice on start, once in the main process and once in the child process.

More documentation

Third-party Fetchers

Docker

  1. Compile your overseerable app to a /path/on/docker/host/dir/app

  2. Then run it with:

    #run the app inside a standard Debian container
    docker run -d -v /path/on/docker/host/dir/:/home/ -w /home/ debian /home/app
    
  3. For testing, swap out -d (daemonize) for --rm -it (remove on exit, input, terminal)

  4. app should mount its parent directory as a volume in order to store the latest binaries on the host

  5. If the OS doesn't ship with TLS certs, you can mount them from the host with -v /etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt

Contributing

See CONTRIBUTING.md