Return out of scale handler when hitting an error

Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>

LICENSE

@@ -1,6 +1,8 @@
 MIT License
 
-Copyright (c) 2019 Alex Ellis
+Copyright (c) 2020 Alex Ellis
+Copyright (c) 2020 OpenFaaS Ltd
+Copyright (c) 2020 OpenFaas Author(s)
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.md

@@ -1,39 +1,35 @@
-# faasd - lightweight OSS serverless 🐳
+# faasd - Serverless for everyone else
+
+faasd is built for everyone else, for those who have no desire to manage expensive infrastructure.
 
 [![Build Status](https://travis-ci.com/openfaas/faasd.svg?branch=master)](https://travis-ci.com/openfaas/faasd)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 [![OpenFaaS](https://img.shields.io/badge/openfaas-serverless-blue.svg)](https://www.openfaas.com)
 ![Downloads](https://img.shields.io/github/downloads/openfaas/faasd/total)
 
-faasd is the same OpenFaaS experience and ecosystem, but without Kubernetes. Functions and microservices can be deployed anywhere with reduced overheads whilst retaining the portability of containers and cloud-native tooling such as containerd and CNI.
+faasd is [OpenFaaS](https://github.com/openfaas/) reimagined, but without the cost and complexity of Kubernetes. It runs on a single host with very modest requirements, making it fast and easy to manage. Under the hood it uses [containerd](https://containerd.io/) and [Container Networking Interface (CNI)](https://github.com/containernetworking/cni) along with the same core OpenFaaS components from the main project.
+
+## When should you use faasd over OpenFaaS on Kubernetes?
+
+* You have a cost sensitive project - run faasd on a 5-10 USD VPS or on your Raspberry Pi
+* When you just need a few functions or microservices, without the cost of a cluster
+* When you don't have the bandwidth to learn or manage Kubernetes
+* To deploy embedded apps in IoT and edge use-cases
+* To shrink-wrap applications for use with a customer or client
+
+faasd does not create the same maintenance burden you'll find with maintaining, upgrading, and securing a Kubernetes cluster. You can deploy it and walk away, in the worst case, just deploy a new VM and deploy your functions again.
 
 ## About faasd
 
 * is a single Golang binary
-* can be set-up and left alone to run your applications
-* is multi-arch, so works on Intel `x86_64` and ARM out the box
 * uses the same core components and ecosystem of OpenFaaS
+* is multi-arch, so works on Intel `x86_64` and ARM out the box
+* can be set-up and left alone to run your applications
 
 ![demo](https://pbs.twimg.com/media/EPNQz00W4AEwDxM?format=jpg&name=small)
 
 > Demo of faasd running in KVM
 
-## What does faasd deploy?
-
-* faasd - itself, and its [faas-provider](https://github.com/openfaas/faas-provider) for containerd - CRUD for functions and services, implements the OpenFaaS REST API
-* [Prometheus](https://github.com/prometheus/prometheus) - for monitoring of services, metrics, scaling and dashboards
-* [OpenFaaS Gateway](https://github.com/openfaas/faas/tree/master/gateway) - the UI portal, CLI, and other OpenFaaS tooling can talk to this.
-* [OpenFaaS queue-worker for NATS](https://github.com/openfaas/nats-queue-worker) - run your invocations in the background without adding any code. See also: [asynchronous invocations](https://docs.openfaas.com/reference/triggers/#async-nats-streaming)
-* [NATS](https://nats.io) for asynchronous processing and queues
-
-You'll also need:
-
-* [CNI](https://github.com/containernetworking/plugins)
-* [containerd](https://github.com/containerd/containerd)
-* [runc](https://github.com/opencontainers/runc)
-
-You can use the standard [faas-cli](https://github.com/openfaas/faas-cli) along with pre-packaged functions from *the Function Store*, or build your own using any OpenFaaS template.
-
 ## Tutorials
 
 ### Get started on DigitalOcean, or any other IaaS
@@ -42,9 +38,9 @@ If your IaaS supports `user_data` aka "cloud-init", then this guide is for you.
 
 * [Build a Serverless appliance with faasd](https://blog.alexellis.io/deploy-serverless-faasd-with-cloud-init/)
 
-### Run locally on MacOS, Linux, or Windows with Multipass.run
+### Run locally on MacOS, Linux, or Windows with multipass
 
-* [Get up and running with your own faasd installation on your Mac/Ubuntu or Windows with cloud-config](https://gist.github.com/alexellis/6d297e678c9243d326c151028a3ad7b9)
+* [Get up and running with your own faasd installation on your Mac/Ubuntu or Windows with cloud-config](/docs/MULTIPASS.md)
 
 ### Get started on armhf / Raspberry Pi
 
@@ -60,6 +56,8 @@ Automate everything within < 60 seconds and get a public URL and IP address back
 
 * [Provision faasd on DigitalOcean with built-in TLS support](docs/bootstrap/digitalocean-terraform/README.md)
 
+## Operational concerns
+
 ### A note on private repos / registries
 
 To use private image repos, `~/.docker/config.json` needs to be copied to `/var/lib/faasd/.docker/config.json`.
@@ -91,6 +89,81 @@ journalctl -t openfaas-fn:figlet -f &
 echo logs | faas-cli invoke figlet
 ```
 
+### Logs for the core services
+
+Core services as defined in the docker-compose.yaml file are deployed as containers by faasd.
+
+View the logs for a component by giving its NAME:
+
+```bash
+journalctl -t default:NAME
+
+journalctl -t default:gateway
+
+journalctl -t default:queue-worker
+```
+
+You can also use `-f` to follow the logs, or `--lines` to tail a number of lines, or `--since` to give a timeframe.
+
+### Exposing core services
+
+The OpenFaaS stack is made up of several core services including NATS and Prometheus. You can expose these through the `docker-compose.yaml` file located at `/var/lib/faasd`.
+
+Expose the gateway to all adapters:
+
+```yaml
+  gateway:
+    ports:
+       - "8080:8080"
+```
+
+Expose Prometheus only to 127.0.0.1:
+
+```yaml
+  prometheus:
+    ports:
+       - "127.0.0.1:9090:9090"
+```
+
+### Upgrading faasd
+
+To upgrade `faasd` either re-create your VM using Terraform, or simply replace the faasd binary with a newer one.
+
+```bash
+systemctl stop faasd-provider
+systemctl stop faasd
+
+# Replace /usr/local/bin/faasd with the desired release
+
+# Replace /var/lib/faasd/docker-compose.yaml with the matching version for
+# that release.
+# Remember to keep any custom patches you make such as exposing additional 
+# ports, or updating timeout values
+
+systemctl start faasd
+systemctl start faasd-provider
+```
+
+You could also perform this task over SSH, or use a configuration management tool.
+
+> Note: if you are using Caddy or Let's Encrypt for free SSL certificates, that you may hit rate-limits for generating new certificates if you do this too often within a given week.
+
+## What does faasd deploy?
+
+* faasd - itself, and its [faas-provider](https://github.com/openfaas/faas-provider) for containerd - CRUD for functions and services, implements the OpenFaaS REST API
+* [Prometheus](https://github.com/prometheus/prometheus) - for monitoring of services, metrics, scaling and dashboards
+* [OpenFaaS Gateway](https://github.com/openfaas/faas/tree/master/gateway) - the UI portal, CLI, and other OpenFaaS tooling can talk to this.
+* [OpenFaaS queue-worker for NATS](https://github.com/openfaas/nats-queue-worker) - run your invocations in the background without adding any code. See also: [asynchronous invocations](https://docs.openfaas.com/reference/triggers/#async-nats-streaming)
+* [NATS](https://nats.io) for asynchronous processing and queues
+
+You'll also need:
+
+* [CNI](https://github.com/containernetworking/plugins)
+* [containerd](https://github.com/containerd/containerd)
+* [runc](https://github.com/opencontainers/runc)
+
+You can use the standard [faas-cli](https://github.com/openfaas/faas-cli) along with pre-packaged functions from *the Function Store*, or build your own using any OpenFaaS template.
+
 ### Manual / developer instructions
 
 See [here for manual / developer instructions](docs/DEV.md)
@@ -107,9 +180,17 @@ For community functions see `faas-cli store --help`
 
 For templates built by the community see: `faas-cli template store list`, you can also use the `dockerfile` template if you just want to migrate an existing service without the benefits of using a template.
 
-### Workshop
+### Training and courses
 
-[The OpenFaaS workshop](https://github.com/openfaas/workshop/) is a set of 12 self-paced labs and provides a great starting point
+#### LinuxFoundation training course
+
+The founder of faasd and OpenFaaS has written a training course for the LinuxFoundation which also covers how to use OpenFaaS on Kubernetes. Much of the same concepts can be applied to faasd, and the course is free:
+
+* [Introduction to Serverless on Kubernetes](https://www.edx.org/course/introduction-to-serverless-on-kubernetes)
+
+#### Community workshop
+
+[The OpenFaaS workshop](https://github.com/openfaas/workshop/) is a set of 12 self-paced labs and provides a great starting point for learning the features of openfaas. Not all features will be available or usable with faasd.
 
 ### Community support
 
@@ -117,7 +198,7 @@ An active community of almost 3000 users awaits you on Slack. Over 250 of those
 
 * [Join Slack](https://slack.openfaas.io/)
 
-## Backlog
+## Roadmap
 
 ### Supported operations
 
@@ -141,19 +222,26 @@ Other operations are pending development in the provider such as:
 
 * `faas auth` - supported for Basic Authentication, but OAuth2 & OIDC require a patch
 
-## Todo
+### Backlog
 
-Pending:
+* [ ] [Store and retrieve annotations in function spec](https://github.com/openfaas/faasd/pull/86) - in progress
-
+* [ ] Offer live rolling-updates, with zero downtime - requires moving to IDs vs. names for function containers
-* [ ] Add support for using container images in third-party public registries
+* [ ] An installer for faasd and dependencies - runc, containerd
-* [ ] Add support for using container images in private third-party registries
 * [ ] Monitor and restart any of the core components at runtime if the container stops
-* [ ] Bundle/package/automate installation of containerd - [see bootstrap from k3s](https://github.com/rancher/k3s)
 * [ ] Provide ufw rules / example for blocking access to everything but a reverse proxy to the gateway container
 * [ ] Provide [simple Caddyfile example](https://blog.alexellis.io/https-inlets-local-endpoints/) in the README showing how to expose the faasd proxy on port 80/443 with TLS
 
-Done:
+### Known-issues
 
+* [ ] [containerd can't pull image from Github Docker Package Registry](https://github.com/containerd/containerd/issues/3291)
+
+### Completed
+
+* [x] Provide a cloud-init configuration for faasd bootstrap
+* [x] Configure core services from a docker-compose.yaml file
+* [x] Store and fetch logs from the journal
+* [x] Add support for using container images in third-party public registries
+* [x] Add support for using container images in private third-party registries
 * [x] Provide a cloud-config.txt file for automated deployments of `faasd`
 * [x] Inject / manage IPs between core components for service to service communication - i.e. so Prometheus can scrape the OpenFaaS gateway - done via `/etc/hosts` mount
 * [x] Add queue-worker and NATS
@@ -165,4 +253,9 @@ Done:
 * [x] Configure `basic_auth` to protect the OpenFaaS gateway and faasd-provider HTTP API
 * [x] Setup custom working directory for faasd `/var/lib/faasd/`
 * [x] Use CNI to create network namespaces and adapters
+* [x] Optionally expose core services from the docker-compose.yaml file, locally or to all adapters.
 
+WIP:
+
+* [ ] Annotation support (PR ready)
+* [ ] Hard memory limits for functions (PR ready)

cloud-config.txt

@@ -1,5 +1,6 @@
 #cloud-config
 ssh_authorized_keys:
+## Note: Replace with your own public key
   - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8Q/aUYUr3P1XKVucnO9mlWxOjJm+K01lHJR90MkHC9zbfTqlp8P7C3J26zKAuzHXOeF+VFxETRr6YedQKW9zp5oP7sN+F2gr/pO7GV3VmOqHMV7uKfyUQfq7H1aVzLfCcI7FwN2Zekv3yB7kj35pbsMa1Za58aF6oHRctZU6UWgXXbRxP+B04DoVU7jTstQ4GMoOCaqYhgPHyjEAS3DW0kkPW6HzsvJHkxvVcVlZ/wNJa1Ie/yGpzOzWIN0Ol0t2QT/RSWOhfzO1A2P0XbPuZ04NmriBonO9zR7T1fMNmmtTuK7WazKjQT3inmYRAqU6pe8wfX8WIWNV7OowUjUsv alex@alexr.local
 
 package_update: true
@@ -8,16 +9,16 @@ packages:
  - runc
 
 runcmd:
-- curl -sLSf https://github.com/containerd/containerd/releases/download/v1.3.2/containerd-1.3.2.linux-amd64.tar.gz > /tmp/containerd.tar.gz && tar -xvf /tmp/containerd.tar.gz -C /usr/local/bin/ --strip-components=1
+- curl -sLSf https://github.com/containerd/containerd/releases/download/v1.3.5/containerd-1.3.5-linux-amd64.tar.gz > /tmp/containerd.tar.gz && tar -xvf /tmp/containerd.tar.gz -C /usr/local/bin/ --strip-components=1
-- curl -SLfs https://raw.githubusercontent.com/containerd/containerd/v1.3.2/containerd.service | tee /etc/systemd/system/containerd.service
+- curl -SLfs https://raw.githubusercontent.com/containerd/containerd/v1.3.5/containerd.service | tee /etc/systemd/system/containerd.service
 - systemctl daemon-reload && systemctl start containerd
 - systemctl enable containerd
 - /sbin/sysctl -w net.ipv4.conf.all.forwarding=1
 - mkdir -p /opt/cni/bin
 - curl -sSL https://github.com/containernetworking/plugins/releases/download/v0.8.5/cni-plugins-linux-amd64-v0.8.5.tgz | tar -xz -C /opt/cni/bin
 - mkdir -p /go/src/github.com/openfaas/
-- cd /go/src/github.com/openfaas/ && git clone https://github.com/openfaas/faasd && git checkout 0.9.0
+- cd /go/src/github.com/openfaas/ && git clone https://github.com/openfaas/faasd && git checkout 0.9.2
-- curl -fSLs "https://github.com/openfaas/faasd/releases/download/0.9.0/faasd" --output "/usr/local/bin/faasd" && chmod a+x "/usr/local/bin/faasd"
+- curl -fSLs "https://github.com/openfaas/faasd/releases/download/0.9.2/faasd" --output "/usr/local/bin/faasd" && chmod a+x "/usr/local/bin/faasd"
 - cd /go/src/github.com/openfaas/faasd/ && /usr/local/bin/faasd install
 - systemctl status -l containerd --no-pager
 - journalctl -u faasd-provider --no-pager

cmd/up.go

@@ -7,7 +7,6 @@ import (
 	"os"
 	"os/signal"
 	"path"
-	"strings"
 	"sync"
 	"syscall"
 	"time"
@@ -72,20 +71,15 @@ func runUp(cmd *cobra.Command, _ []string) error {
 	log.Printf("Supervisor created in: %s\n", time.Since(start).String())
 
 	start = time.Now()
-
+	if err := supervisor.Start(services); err != nil {
-	err = supervisor.Start(services)
-
-	if err != nil {
 		return err
 	}
-
 	defer supervisor.Close()
 
 	log.Printf("Supervisor init done in: %s\n", time.Since(start).String())
 
 	shutdownTimeout := time.Second * 1
 	timeout := time.Second * 60
-	proxyDoneCh := make(chan bool)
 
 	wg := sync.WaitGroup{}
 	wg.Add(1)
@@ -102,38 +96,38 @@ func runUp(cmd *cobra.Command, _ []string) error {
 			fmt.Println(err)
 		}
 
-		// Close proxy
+		// TODO: close proxies
-		proxyDoneCh <- true
 		time.AfterFunc(shutdownTimeout, func() {
 			wg.Done()
 		})
 	}()
 
-	gatewayURLChan := make(chan string, 1)
+	localResolver := pkg.NewLocalResolver(path.Join(cfg.workingDir, "hosts"))
-	proxyPort := 8080
+	go localResolver.Start()
-	proxy := pkg.NewProxy(proxyPort, timeout)
-	go proxy.Start(gatewayURLChan, proxyDoneCh)
 
-	go func() {
+	proxies := map[uint32]*pkg.Proxy{}
-		time.Sleep(3 * time.Second)
+	for _, svc := range services {
+		for _, port := range svc.Ports {
 
-		fileData, fileErr := ioutil.ReadFile(path.Join(cfg.workingDir, "hosts"))
+			listenPort := port.Port
-		if fileErr != nil {
+			if _, ok := proxies[listenPort]; ok {
-			log.Println(fileErr)
+				return fmt.Errorf("port %d already allocated", listenPort)
-			return
-		}
-
-		host := ""
-		lines := strings.Split(string(fileData), "\n")
-		for _, line := range lines {
-			if strings.Index(line, "gateway") > -1 {
-				host = line[:strings.Index(line, "\t")]
 			}
+
+			hostIP := "0.0.0.0"
+			if len(port.HostIP) > 0 {
+				hostIP = port.HostIP
+			}
+
+			upstream := fmt.Sprintf("%s:%d", svc.Name, port.TargetPort)
+			proxies[listenPort] = pkg.NewProxy(upstream, listenPort, hostIP, timeout, localResolver)
 		}
-		log.Printf("[up] Sending %s to proxy\n", host)
+	}
-		gatewayURLChan <- host + ":8080"
+
-		close(gatewayURLChan)
+	// TODO: track proxies for later cancellation when receiving sigint/term
-	}()
+	for _, v := range proxies {
+		go v.Start()
+	}
 
 	wg.Wait()
 	return nil

docker-compose.yaml

@@ -1,7 +1,7 @@
 version: "3.7"
 services:
   basic-auth-plugin:
-    image: "docker.io/openfaas/basic-auth-plugin:0.18.17${ARCH_SUFFIX}"
+    image: "docker.io/openfaas/basic-auth-plugin:0.18.18${ARCH_SUFFIX}"
     environment:
       - port=8080
       - secret_mount_path=/run/secrets
@@ -26,6 +26,8 @@ services:
       - "8222"
       - "--store=memory"
       - "--cluster_id=faas-cluster"
+    # ports:
+    #    - "127.0.0.1:8222:8222"
 
   prometheus:
     image: docker.io/prom/prometheus:v2.14.0
@@ -35,9 +37,11 @@ services:
         target: /etc/prometheus/prometheus.yml
     cap_add:
       - CAP_NET_RAW
+    ports:
+       - "127.0.0.1:9090:9090"
 
   gateway:
-    image: "docker.io/openfaas/gateway:0.18.17${ARCH_SUFFIX}"
+    image: "docker.io/openfaas/gateway:0.18.18${ARCH_SUFFIX}"
     environment:
       - basic_auth=true
       - functions_provider_url=http://faasd-provider:8081/
@@ -65,6 +69,8 @@ services:
       - basic-auth-plugin
       - nats
       - prometheus
+    ports:
+       - "8080:8080"
 
   queue-worker:
     image: docker.io/openfaas/queue-worker:0.11.2

docs/DEV.md

@@ -14,10 +14,67 @@
 
     For Windows users, install [Git Bash](https://git-scm.com/downloads) along with multipass or vagrant. You can also use WSL1 or WSL2 which provides a Linux environment.
 
-    You will also need [containerd v1.3.2](https://github.com/containerd/containerd) and the [CNI plugins v0.8.5](https://github.com/containernetworking/plugins)
+    You will also need [containerd v1.3.5](https://github.com/containerd/containerd) and the [CNI plugins v0.8.5](https://github.com/containernetworking/plugins)
 
     [faas-cli](https://github.com/openfaas/faas-cli) is optional, but recommended.
 
+If you're using multipass, then allocate sufficient resources:
+
+```sh
+multipass launch \
+  --mem 4G \
+  -c 2 \
+  -n faasd
+
+# Then access its shell
+multipass shell faasd
+```
+
+### Get runc
+
+```sh
+sudo apt update \
+  && sudo apt install -qy \
+    runc \
+    bridge-utils \
+    make
+```
+
+### Get faas-cli (optional)
+
+Having `faas-cli` on your dev machine is useful for testing and debug.
+
+```bash
+curl -sLS https://cli.openfaas.com | sudo sh
+```
+
+#### Install the CNI plugins:
+
+* For PC run `export ARCH=amd64`
+* For RPi/armhf run `export ARCH=arm`
+* For arm64 run `export ARCH=arm64`
+
+Then run:
+
+```sh
+export ARCH=amd64
+export CNI_VERSION=v0.8.5
+
+sudo mkdir -p /opt/cni/bin
+curl -sSL https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-${ARCH}-${CNI_VERSION}.tgz | sudo tar -xz -C /opt/cni/bin
+
+# Make a config folder for CNI definitions
+sudo mkdir -p /etc/cni/net.d
+
+# Make an initial loopback configuration
+sudo sh -c 'cat >/etc/cni/net.d/99-loopback.conf <<-EOF
+{
+    "cniVersion": "0.3.1",
+    "type": "loopback"
+}
+EOF'
+```
+
 ### Get containerd
 
 You have three options - binaries for PC, binaries for armhf, or build from source.
@@ -25,8 +82,8 @@ You have three options - binaries for PC, binaries for armhf, or build from sour
 * Install containerd `x86_64` only
 
 ```sh
-export VER=1.3.2
+export VER=1.3.5
-curl -sLSf https://github.com/containerd/containerd/releases/download/v$VER/containerd-$VER.linux-amd64.tar.gz > /tmp/containerd.tar.gz \
+curl -sSL https://github.com/containerd/containerd/releases/download/v$VER/containerd-$VER-linux-amd64.tar.gz > /tmp/containerd.tar.gz \
   && sudo tar -xvf /tmp/containerd.tar.gz -C /usr/local/bin/ --strip-components=1
 
 containerd -version
@@ -37,7 +94,7 @@ containerd -version
     Building `containerd` on armhf is extremely slow, so I've provided binaries for you.
 
     ```sh
-    curl -sSL https://github.com/alexellis/containerd-armhf/releases/download/v1.3.2/containerd.tgz | sudo tar -xvz --strip-components=2 -C /usr/local/bin/
+    curl -sSL https://github.com/alexellis/containerd-armhf/releases/download/v1.3.5/containerd.tgz | sudo tar -xvz --strip-components=2 -C /usr/local/bin/
     ```
 
 * Or clone / build / install [containerd](https://github.com/containerd/containerd) from source:
@@ -49,7 +106,7 @@ containerd -version
     git clone https://github.com/containerd/containerd
     cd containerd
     git fetch origin --tags
-    git checkout v1.3.2
+    git checkout v1.3.5
 
     make
     sudo make install
@@ -60,7 +117,11 @@ containerd -version
 #### Ensure containerd is running
 
 ```sh
-curl -sLS https://raw.githubusercontent.com/containerd/containerd/master/containerd.service > /tmp/containerd.service
+curl -sLS https://raw.githubusercontent.com/containerd/containerd/v1.3.5/containerd.service > /tmp/containerd.service
+
+# Extend the timeouts for low-performance VMs
+echo "[Manager]" | tee -a /tmp/containerd.service
+echo "DefaultTimeoutStartSec=3m" | tee -a /tmp/containerd.service
 
 sudo cp /tmp/containerd.service /lib/systemd/system/
 sudo systemctl enable containerd
@@ -69,7 +130,7 @@ sudo systemctl daemon-reload
 sudo systemctl restart containerd
 ```
 
-Or run ad-hoc:
+Or run ad-hoc. This step can be useful for exploring why containerd might fail to start.
 
 ```sh
 sudo containerd &
@@ -106,10 +167,10 @@ You may find alternative package names for CentOS and other Linux distributions.
 #### Install Go 1.13 (x86_64)
 
 ```sh
-curl -sSLf https://dl.google.com/go/go1.13.6.linux-amd64.tar.gz > go.tgz
+curl -sSLf https://dl.google.com/go/go1.13.6.linux-amd64.tar.gz > /tmp/go.tgz
 sudo rm -rf /usr/local/go/
 sudo mkdir -p /usr/local/go/
-sudo tar -xvf go.tgz -C /usr/local/go/ --strip-components=1
+sudo tar -xvf /tmp/go.tgz -C /usr/local/go/ --strip-components=1
 
 export GOPATH=$HOME/go/
 export PATH=$PATH:/usr/local/go/bin/
@@ -120,8 +181,8 @@ go version
 You should also add the following to `~/.bash_profile`:
 
 ```sh
-export GOPATH=$HOME/go/
+echo "export GOPATH=\$HOME/go/" | tee -a $HOME/.bash_profile
-export PATH=$PATH:/usr/local/go/bin/
+echo "export PATH=\$PATH:/usr/local/go/bin/" | tee -a $HOME/.bash_profile
 ```
 
 #### Or on Raspberry Pi (armhf)
@@ -138,22 +199,6 @@ export PATH=$PATH:/usr/local/go/bin/
 go version
 ```
 
-#### Install the CNI plugins:
-
-* For PC run `export ARCH=amd64`
-* For RPi/armhf run `export ARCH=arm`
-* For arm64 run `export ARCH=arm64`
-
-Then run:
-
-```sh
-export ARCH=amd64
-export CNI_VERSION=v0.8.5
-
-sudo mkdir -p /opt/cni/bin
-curl -sSL https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-${ARCH}-${CNI_VERSION}.tgz | sudo tar -xz -C /opt/cni/bin
-```
-
 #### Clone faasd and its systemd unit files
 
 ```sh
@@ -168,9 +213,12 @@ git clone https://github.com/openfaas/faasd
 cd $GOPATH/src/github.com/openfaas/faasd
 cd faasd
 make local
+
+# Install the binary
+sudo cp bin/faasd /usr/local/bin
 ```
 
-#### Build and run `faasd` (binaries)
+#### Or, download and run `faasd` (binaries)
 
 ```sh
 # For x86_64
@@ -191,9 +239,9 @@ sudo mv /tmp/faasd /usr/local/bin/
 
 #### Install `faasd`
 
+This step installs faasd as a systemd unit file, creates files in `/var/lib/faasd`, and writes out networking configuration for the CNI bridge networking plugin.
+
 ```sh
-# Install with systemd
-sudo cp bin/faasd /usr/local/bin
 sudo faasd install
 
 2020/02/17 17:38:06 Writing to: "/var/lib/faasd/secrets/basic-auth-password"

docs/MULTIPASS.md

@@ -0,0 +1,141 @@
+# Tutorial - faasd with multipass
+
+## Get up and running with your own faasd installation on your Mac
+
+[multipass from Canonical](https://multipass.run) is like Docker Desktop, but for getting Ubuntu instead of a Docker daemon. It works on MacOS, Linux, and Windows with the same consistent UX. It's not fully open-source, and uses some proprietary add-ons / binaries, but is free to use.
+
+For Linux using Ubuntu, you can install the packages directly, or use `sudo snap install multipass --classic` and follow this tutorial. For Raspberry Pi, [see my tutorial here](https://blog.alexellis.io/faasd-for-lightweight-serverless/).
+
+John McCabe has also tested faasd on Windows with multipass, [see his tweet](https://twitter.com/mccabejohn/status/1221899154672308224).
+
+## Use-case:
+
+Try out [faasd](https://github.com/openfaas/faasd) in a single command using a cloud-config file to get a VM which has:
+
+* port 22 for administration and
+* port 8080 for the OpenFaaS REST API.
+
+![Example](https://pbs.twimg.com/media/EPNQz00W4AEwDxM?format=jpg&name=medium)
+
+The above screenshot is [from my tweet](https://twitter.com/alexellisuk/status/1221408788395298819/), feel free to comment there.
+
+It took me about 2-3 minutes to run through everything after installing multipass.
+
+## Let's start the tutorial
+
+* Get [multipass.run](https://multipass.run)
+
+* Get my cloud-config.txt file
+
+```sh
+curl -sSLO https://raw.githubusercontent.com/openfaas/faasd/master/cloud-config.txt
+```
+
+* Update the SSH key to match your own, edit `cloud-config.txt`:
+
+Replace the 2nd line with the contents of `~/.ssh/id_rsa.pub`:
+
+```
+ssh_authorized_keys:
+  - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8Q/aUYUr3P1XKVucnO9mlWxOjJm+K01lHJR90MkHC9zbfTqlp8P7C3J26zKAuzHXOeF+VFxETRr6YedQKW9zp5oP7sN+F2gr/pO7GV3VmOqHMV7uKfyUQfq7H1aVzLfCcI7FwN2Zekv3yB7kj35pbsMa1Za58aF6oHRctZU6UWgXXbRxP+B04DoVU7jTstQ4GMoOCaqYhgPHyjEAS3DW0kkPW6HzsvJHkxvVcVlZ/wNJa1Ie/yGpzOzWIN0Ol0t2QT/RSWOhfzO1A2P0XbPuZ04NmriBonO9zR7T1fMNmmtTuK7WazKjQT3inmYRAqU6pe8wfX8WIWNV7OowUjUsv alex@alexr.local
+```
+
+* Boot the VM
+
+```sh
+multipass launch --cloud-init cloud-config.txt  --name faasd
+```
+
+* Get the VM's IP and connect with `ssh`
+
+```sh
+multipass info faasd
+Name:           faasd
+State:          Running
+IPv4:           192.168.64.14
+Release:        Ubuntu 18.04.3 LTS
+Image hash:     a720c34066dc (Ubuntu 18.04 LTS)
+Load:           0.79 0.19 0.06
+Disk usage:     1.1G out of 4.7G
+Memory usage:   145.6M out of 985.7M
+```
+
+Set the variable `IP`:
+
+```
+export IP="192.168.64.14"
+```
+
+You can also try to use `jq` to get the IP into a variable:
+
+```sh
+export IP=$(multipass info faasd --format json| jq '.info.faasd.ipv4[0]' | tr -d '\"')
+```
+
+Connect to the IP listed:
+
+```sh
+ssh ubuntu@$IP
+```
+
+Log out once you know it works.
+
+* Let's capture the authentication password into a file for use with `faas-cli`
+
+```
+ssh ubuntu@192.168.64.14 "sudo cat /var/lib/faasd/secrets/basic-auth-password" > basic-auth-password
+```
+
+## Try faasd (OpenFaaS)
+
+* Login from your laptop (the host)
+
+```
+export OPENFAAS_URL=http://$IP:8080
+cat basic-auth-password | faas-cli login -s
+```
+
+* Deploy a function and invoke it
+
+```
+faas-cli store deploy figlet --env write_timeout=1s
+echo "faasd" | faas-cli invoke figlet
+
+faas-cli describe figlet
+
+# Run async
+curl -i -d "faasd-async" $OPENFAAS_URL/async-function/figlet
+
+# Run async with a callback
+
+curl -i -d "faasd-async" -H "X-Callback-Url: http://some-request-bin.com/path" $OPENFAAS_URL/async-function/figlet
+```
+
+You can also checkout the other store functions: `faas-cli store list`
+
+* Try the UI
+
+Head over to the UI from your laptop and remember that your password is in the `basic-auth-password` file. The username is `admin.:
+
+```
+echo http://$IP:8080
+```
+
+* Stop/start the instance
+
+```sh
+multipass stop faasd
+```
+
+* Delete, if you want to:
+
+```
+multipass delete --purge faasd
+```
+
+You now have a faasd appliance on your Mac. You can also use this cloud-init file with public cloud like AWS or DigitalOcean.
+
+* If you want a public IP for your faasd VM, then just head over to [inlets.dev](https://inlets.dev/)
+* Try my more complete walk-through / tutorial with Raspberry Pi, or run the same steps on your multipass VM, including how to develop your own functions and services - https://blog.alexellis.io/faasd-for-lightweight-serverless/
+* You might also like [Building containers without Docker](https://blog.alexellis.io/building-containers-without-docker/)
+* Star/fork [faasd](https://github.com/openfaas/faasd) on GitHub

docs/bootstrap/.gitignore

@@ -0,0 +1,3 @@
+/.terraform/
+/terraform.tfstate
+/terraform.tfstate.backup

docs/bootstrap/cloud-config.tpl

@@ -1,5 +1,6 @@
 #cloud-config
 ssh_authorized_keys:
+## Note: Replace with your own public key
   - ${ssh_key}
 
 package_update: true
@@ -8,8 +9,8 @@ packages:
  - runc
 
 runcmd:
-- curl -sLSf https://github.com/containerd/containerd/releases/download/v1.3.2/containerd-1.3.2.linux-amd64.tar.gz > /tmp/containerd.tar.gz && tar -xvf /tmp/containerd.tar.gz -C /usr/local/bin/ --strip-components=1
+- curl -sLSf https://github.com/containerd/containerd/releases/download/v1.3.5/containerd-1.3.5-linux-amd64.tar.gz > /tmp/containerd.tar.gz && tar -xvf /tmp/containerd.tar.gz -C /usr/local/bin/ --strip-components=1
-- curl -SLfs https://raw.githubusercontent.com/containerd/containerd/v1.3.2/containerd.service | tee /etc/systemd/system/containerd.service
+- curl -SLfs https://raw.githubusercontent.com/containerd/containerd/v1.3.5/containerd.service | tee /etc/systemd/system/containerd.service
 - systemctl daemon-reload && systemctl start containerd
 - /sbin/sysctl -w net.ipv4.conf.all.forwarding=1
 - mkdir -p /opt/cni/bin
@@ -18,12 +19,12 @@ runcmd:
 - mkdir -p /var/lib/faasd/secrets/
 - echo ${gw_password} > /var/lib/faasd/secrets/basic-auth-password
 - echo admin > /var/lib/faasd/secrets/basic-auth-user
-- cd /go/src/github.com/openfaas/ && git clone https://github.com/openfaas/faasd
+- cd /go/src/github.com/openfaas/ && git clone https://github.com/openfaas/faasd && cd faasd && git checkout 0.9.2
-- curl -fSLs "https://github.com/openfaas/faasd/releases/download/0.8.1/faasd" --output "/usr/local/bin/faasd" && chmod a+x "/usr/local/bin/faasd"
+- curl -fSLs "https://github.com/openfaas/faasd/releases/download/0.9.2/faasd" --output "/usr/local/bin/faasd" && chmod a+x "/usr/local/bin/faasd"
 - cd /go/src/github.com/openfaas/faasd/ && /usr/local/bin/faasd install
 - systemctl status -l containerd --no-pager
 - journalctl -u faasd-provider --no-pager
 - systemctl status -l faasd-provider --no-pager
 - systemctl status -l faasd --no-pager
 - curl -sSLf https://cli.openfaas.com | sh
-- sleep 5 && journalctl -u faasd --no-pager
+- sleep 5 && journalctl -u faasd --no-pager

docs/bootstrap/digitalocean-terraform/.gitignore

@@ -0,0 +1,3 @@
+/.terraform/
+/terraform.tfstate
+/terraform.tfstate.backup

docs/bootstrap/digitalocean-terraform/README.md

@@ -10,6 +10,7 @@
    | ------------ | ------------------- | --------------- |
    | `do_token` | Digitalocean API token | None |
    | `do_domain` | Public domain used for the faasd gateway | None |
+   | `do_subdomain` | Public subdomain used for the faasd gateway | `faasd` |
    | `letsencrypt_email` | Email used by when ordering TLS certificate from Letsencrypt | `""` |
    | `do_create_record` | When set to `true`, a new DNS record will be created. This works only if your domain (`do_domain`) is managed by Digitalocean | `false` |
    | `do_region` | Digitalocean region for creating the droplet | `fra1` |

docs/bootstrap/digitalocean-terraform/cloud-config.tpl

@@ -31,8 +31,8 @@ packages:
  - runc
 
 runcmd:
-- curl -sLSf https://github.com/containerd/containerd/releases/download/v1.3.2/containerd-1.3.2.linux-amd64.tar.gz > /tmp/containerd.tar.gz && tar -xvf /tmp/containerd.tar.gz -C /usr/local/bin/ --strip-components=1
+- curl -sLSf https://github.com/containerd/containerd/releases/download/v1.3.5/containerd-1.3.5-linux-amd64.tar.gz > /tmp/containerd.tar.gz && tar -xvf /tmp/containerd.tar.gz -C /usr/local/bin/ --strip-components=1
-- curl -SLfs https://raw.githubusercontent.com/containerd/containerd/v1.3.2/containerd.service | tee /etc/systemd/system/containerd.service
+- curl -SLfs https://raw.githubusercontent.com/containerd/containerd/v1.3.5/containerd.service | tee /etc/systemd/system/containerd.service
 - systemctl daemon-reload && systemctl start containerd
 - /sbin/sysctl -w net.ipv4.conf.all.forwarding=1
 - mkdir -p /opt/cni/bin
@@ -41,8 +41,8 @@ runcmd:
 - mkdir -p /var/lib/faasd/secrets/
 - echo ${gw_password} > /var/lib/faasd/secrets/basic-auth-password
 - echo admin > /var/lib/faasd/secrets/basic-auth-user
-- cd /go/src/github.com/openfaas/ && git clone https://github.com/openfaas/faasd
+- cd /go/src/github.com/openfaas/ && git clone https://github.com/openfaas/faasd && cd faasd && git checkout 0.9.2
-- curl -fSLs "https://github.com/openfaas/faasd/releases/download/0.8.1/faasd" --output "/usr/local/bin/faasd" && chmod a+x "/usr/local/bin/faasd"
+- curl -fSLs "https://github.com/openfaas/faasd/releases/download/0.9.2/faasd" --output "/usr/local/bin/faasd" && chmod a+x "/usr/local/bin/faasd"
 - cd /go/src/github.com/openfaas/faasd/ && /usr/local/bin/faasd install
 - systemctl status -l containerd --no-pager
 - journalctl -u faasd-provider --no-pager
@@ -50,7 +50,7 @@ runcmd:
 - systemctl status -l faasd --no-pager
 - curl -sSLf https://cli.openfaas.com | sh
 - sleep 5 && journalctl -u faasd --no-pager
-- wget https://github.com/caddyserver/caddy/releases/download/v2.0.0-rc.2/caddy_2.0.0-rc.2_linux_amd64.tar.gz -O /tmp/caddy.tar.gz && tar -zxvf /tmp/caddy.tar.gz -C /usr/bin/ caddy
+- wget https://github.com/caddyserver/caddy/releases/download/v2.1.1/caddy_2.1.1_linux_amd64.tar.gz -O /tmp/caddy.tar.gz && tar -zxvf /tmp/caddy.tar.gz -C /usr/bin/ caddy
 - wget https://raw.githubusercontent.com/caddyserver/dist/master/init/caddy.service -O /etc/systemd/system/caddy.service
 - systemctl daemon-reload
 - systemctl enable caddy

docs/bootstrap/digitalocean-terraform/main.tf

@@ -8,6 +8,10 @@ variable "do_token" {
 variable "do_domain" {
   description = "Your public domain"
 }
+variable "do_subdomain" {
+  description = "Your public subdomain"
+  default = "faasd"
+}
 variable "letsencrypt_email" {
   description = "Email used to order a certificate from Letsencrypt"
 }
@@ -43,7 +47,7 @@ data "template_file" "cloud_init" {
     vars = {
       gw_password=random_password.password.result,
       ssh_key=data.local_file.ssh_key.content,
-      faasd_domain_name="faasd.${var.do_domain}"
+      faasd_domain_name="${var.do_subdomain}.${var.do_domain}"
       letsencrypt_email=var.letsencrypt_email
     }
 }
@@ -70,7 +74,7 @@ output "droplet_ip" {
 }
 
 output "gateway_url" {
-  value = "https://faasd.${var.do_domain}/"
+  value = "https://${var.do_subdomain}.${var.do_domain}/"
 }
 
 output "password" {
@@ -78,5 +82,5 @@ output "password" {
 }
 
 output "login_cmd" {
-  value = "faas-cli login -g https://faasd.${var.do_domain}/ -p ${random_password.password.result}"
+  value = "faas-cli login -g https://${var.do_subdomain}.${var.do_domain}/ -p ${random_password.password.result}"
 }

docs/bootstrap/digitalocean-terraform/main.tfvars

@@ -1,3 +1,4 @@
 do_token          = ""
 do_domain         = ""
+do_subdomain      = ""
 letsencrypt_email = ""

pkg/depends_on.go

@@ -1,37 +0,0 @@
-package pkg
-
-import (
-	"log"
-)
-
-func buildInstallOrder(svcs []Service) []string {
-	graph := Graph{nodes: []*Node{}}
-
-	nodeMap := map[string]*Node{}
-	for _, s := range svcs {
-		n := &Node{Name: s.Name}
-		nodeMap[s.Name] = n
-		graph.nodes = append(graph.nodes, n)
-	}
-
-	for _, s := range svcs {
-		for _, d := range s.DependsOn {
-			nodeMap[s.Name].Edges = append(nodeMap[s.Name].Edges, nodeMap[d])
-		}
-	}
-
-	resolved := &Graph{}
-	unresolved := &Graph{}
-	for _, g := range graph.nodes {
-		resolve(g, resolved, unresolved)
-	}
-
-	log.Printf("Start-up order:\n")
-	order := []string{}
-	for _, node := range resolved.nodes {
-		log.Printf("- %s\n", node.Name)
-		order = append(order, node.Name)
-	}
-
-	return order
-}

pkg/depgraph/depgraph.go

@@ -1,4 +1,4 @@
-package pkg
+package depgraph
 
 import "log"
 
@@ -14,6 +14,17 @@ type Graph struct {
 	nodes []*Node
 }
 
+func NewDepgraph() *Graph {
+	return &Graph{
+		nodes: []*Node{},
+	}
+}
+
+// Nodes returns the nodes within the graph
+func (g *Graph) Nodes() []*Node {
+	return g.nodes
+}
+
 // Contains returns true if the target Node is found
 // in its list
 func (g *Graph) Contains(target *Node) bool {
@@ -47,10 +58,31 @@ func (g *Graph) Remove(target *Node) {
 	}
 }
 
-// resolve finds the order of dependencies for a graph
+// Resolve retruns a list of node names in order of their dependencies.
-// of nodes.
+// A use case may be for determining the correct order to install
-// Inspired by algorithm from
+// software packages, or to start services.
+// Based upon the algorithm described by Ferry Boender in the following article
 // https://www.electricmonk.nl/log/2008/08/07/dependency-resolving-algorithm/
+func (g *Graph) Resolve() []string {
+	resolved := &Graph{}
+	unresolved := &Graph{}
+	for _, node := range g.nodes {
+		resolve(node, resolved, unresolved)
+	}
+
+	order := []string{}
+
+	for _, node := range resolved.Nodes() {
+		order = append(order, node.Name)
+	}
+
+	return order
+}
+
+// resolve mutates the resolved graph for a given starting
+// node. The unresolved graph is used to detect a circular graph
+// error and will throw a panic. This can be caught with a resolve
+// in a go routine.
 func resolve(node *Node, resolved, unresolved *Graph) {
 	unresolved.Add(node)
 

pkg/depgraph/depgraph_test.go

@@ -1,4 +1,4 @@
-package pkg
+package depgraph
 
 import "testing"
 

pkg/deployment_order.go

@@ -0,0 +1,41 @@
+package pkg
+
+import (
+	"log"
+
+	"github.com/openfaas/faasd/pkg/depgraph"
+)
+
+func buildDeploymentOrder(svcs []Service) []string {
+
+	graph := buildServiceGraph(svcs)
+
+	order := graph.Resolve()
+
+	log.Printf("Start-up order:\n")
+	for _, node := range order {
+		log.Printf("- %s\n", node)
+	}
+
+	return order
+}
+
+func buildServiceGraph(svcs []Service) *depgraph.Graph {
+	graph := depgraph.NewDepgraph()
+
+	nodeMap := map[string]*depgraph.Node{}
+	for _, s := range svcs {
+		n := &depgraph.Node{Name: s.Name}
+		nodeMap[s.Name] = n
+		graph.Add(n)
+
+	}
+
+	for _, s := range svcs {
+		for _, d := range s.DependsOn {
+			nodeMap[s.Name].Edges = append(nodeMap[s.Name].Edges, nodeMap[d])
+		}
+	}
+
+	return graph
+}

pkg/deployment_order_test.go

@@ -5,7 +5,7 @@ import (
 	"testing"
 )
 
-func Test_buildInstallOrder_ARequiresB(t *testing.T) {
+func Test_buildDeploymentOrder_ARequiresB(t *testing.T) {
 	svcs := []Service{
 		{
 			Name:      "A",
@@ -17,7 +17,7 @@ func Test_buildInstallOrder_ARequiresB(t *testing.T) {
 		},
 	}
 
-	order := buildInstallOrder(svcs)
+	order := buildDeploymentOrder(svcs)
 
 	if len(order) < len(svcs) {
 		t.Fatalf("length of order too short: %d", len(order))
@@ -30,7 +30,7 @@ func Test_buildInstallOrder_ARequiresB(t *testing.T) {
 	}
 }
 
-func Test_buildInstallOrder_ARequiresBAndC(t *testing.T) {
+func Test_buildDeploymentOrder_ARequiresBAndC(t *testing.T) {
 	svcs := []Service{
 		{
 			Name:      "A",
@@ -46,7 +46,7 @@ func Test_buildInstallOrder_ARequiresBAndC(t *testing.T) {
 		},
 	}
 
-	order := buildInstallOrder(svcs)
+	order := buildDeploymentOrder(svcs)
 
 	if len(order) < len(svcs) {
 		t.Fatalf("length of order too short: %d", len(order))
@@ -65,7 +65,7 @@ func Test_buildInstallOrder_ARequiresBAndC(t *testing.T) {
 
 }
 
-func Test_buildInstallOrder_ARequiresBRequiresC(t *testing.T) {
+func Test_buildDeploymentOrder_ARequiresBRequiresC(t *testing.T) {
 	svcs := []Service{
 		{
 			Name:      "A",
@@ -81,7 +81,7 @@ func Test_buildInstallOrder_ARequiresBRequiresC(t *testing.T) {
 		},
 	}
 
-	order := buildInstallOrder(svcs)
+	order := buildDeploymentOrder(svcs)
 
 	if len(order) < len(svcs) {
 		t.Fatalf("length of order too short: %d", len(order))
@@ -104,7 +104,7 @@ func Test_buildInstallOrder_ARequiresBRequiresC(t *testing.T) {
 	}
 }
 
-func Test_buildInstallOrderCircularARequiresBRequiresA(t *testing.T) {
+func Test_buildDeploymentOrderCircularARequiresBRequiresA(t *testing.T) {
 	svcs := []Service{
 		{
 			Name:      "A",
@@ -118,12 +118,12 @@ func Test_buildInstallOrderCircularARequiresBRequiresA(t *testing.T) {
 
 	defer func() { recover() }()
 
-	buildInstallOrder(svcs)
+	buildDeploymentOrder(svcs)
 
 	t.Fatalf("did not panic as expected")
 }
 
-func Test_buildInstallOrderComposeFile(t *testing.T) {
+func Test_buildDeploymentOrderComposeFile(t *testing.T) {
 	// svcs := []Service{}
 	file, err := LoadComposeFileWithArch("../", "docker-compose.yaml", func() (string, string) {
 		return "x86_64", "Linux"
@@ -145,7 +145,7 @@ func Test_buildInstallOrderComposeFile(t *testing.T) {
 		}
 	}
 
-	order := buildInstallOrder(svcs)
+	order := buildDeploymentOrder(svcs)
 
 	if len(order) < len(svcs) {
 		t.Fatalf("length of order too short: %d", len(order))
@@ -167,7 +167,7 @@ func Test_buildInstallOrderComposeFile(t *testing.T) {
 	}
 }
 
-func Test_buildInstallOrderOpenFaaS(t *testing.T) {
+func Test_buildDeploymentOrderOpenFaaS(t *testing.T) {
 	svcs := []Service{
 		{
 			Name:      "queue-worker",
@@ -191,7 +191,7 @@ func Test_buildInstallOrderOpenFaaS(t *testing.T) {
 		},
 	}
 
-	order := buildInstallOrder(svcs)
+	order := buildDeploymentOrder(svcs)
 
 	if len(order) < len(svcs) {
 		t.Fatalf("length of order too short: %d", len(order))

pkg/local_resolver.go

@@ -0,0 +1,104 @@
+package pkg
+
+import (
+	"io/ioutil"
+	"log"
+	"os"
+	"strings"
+	"sync"
+	"time"
+)
+
+// LocalResolver provides hostname to IP look-up for faasd core services
+type LocalResolver struct {
+	Path  string
+	Map   map[string]string
+	Mutex *sync.RWMutex
+}
+
+// NewLocalResolver creates a new resolver for reading from a hosts file
+func NewLocalResolver(path string) Resolver {
+	return &LocalResolver{
+		Path:  path,
+		Mutex: &sync.RWMutex{},
+		Map:   make(map[string]string),
+	}
+}
+
+// Start polling the disk for the hosts file in Path
+func (l *LocalResolver) Start() {
+	var lastStat os.FileInfo
+
+	for {
+		rebuild := false
+		if info, err := os.Stat(l.Path); err == nil {
+			if lastStat == nil {
+				rebuild = true
+			} else {
+				if !lastStat.ModTime().Equal(info.ModTime()) {
+					rebuild = true
+				}
+			}
+			lastStat = info
+		}
+
+		if rebuild {
+			log.Printf("Resolver rebuilding map")
+			l.rebuild()
+		}
+		time.Sleep(time.Second * 3)
+	}
+}
+
+func (l *LocalResolver) rebuild() {
+	l.Mutex.Lock()
+	defer l.Mutex.Unlock()
+
+	fileData, fileErr := ioutil.ReadFile(l.Path)
+	if fileErr != nil {
+		log.Printf("resolver rebuild error: %s", fileErr.Error())
+		return
+	}
+
+	lines := strings.Split(string(fileData), "\n")
+
+	for _, line := range lines {
+		index := strings.Index(line, "\t")
+
+		if len(line) > 0 && index > -1 {
+			ip := line[:index]
+			host := line[index+1:]
+			log.Printf("Resolver: %q=%q", host, ip)
+			l.Map[host] = ip
+		}
+	}
+}
+
+// Get resolves a hostname to an IP, or timesout after the duration has passed
+func (l *LocalResolver) Get(upstream string, got chan<- string, timeout time.Duration) {
+	start := time.Now()
+	for {
+		if val := l.get(upstream); len(val) > 0 {
+			got <- val
+			break
+		}
+
+		if time.Now().After(start.Add(timeout)) {
+			log.Printf("Timed out after %s getting host %q", timeout.String(), upstream)
+			break
+		}
+
+		time.Sleep(time.Millisecond * 250)
+	}
+}
+
+func (l *LocalResolver) get(upstream string) string {
+	l.Mutex.RLock()
+	defer l.Mutex.RUnlock()
+
+	if val, ok := l.Map[upstream]; ok {
+		return val
+	}
+
+	return ""
+}

pkg/provider/handlers/deploy.go

@@ -69,6 +69,7 @@ func deploy(ctx context.Context, req types.FunctionDeployment, client *container
 	if err != nil {
 		return err
 	}
+
 	imgRef := reference.TagNameOnly(r).String()
 
 	snapshotter := ""
@@ -98,6 +99,11 @@ func deploy(ctx context.Context, req types.FunctionDeployment, client *container
 
 	name := req.Service
 
+	labels := map[string]string{}
+	if req.Labels != nil {
+		labels = *req.Labels
+	}
+
 	container, err := client.NewContainer(
 		ctx,
 		name,
@@ -108,7 +114,7 @@ func deploy(ctx context.Context, req types.FunctionDeployment, client *container
 			oci.WithCapabilities([]string{"CAP_NET_RAW"}),
 			oci.WithMounts(mounts),
 			oci.WithEnv(envs)),
-		containerd.WithContainerLabels(*req.Labels),
+		containerd.WithContainerLabels(labels),
 	)
 
 	if err != nil {
@@ -122,7 +128,6 @@ func deploy(ctx context.Context, req types.FunctionDeployment, client *container
 func createTask(ctx context.Context, client *containerd.Client, container containerd.Container, cni gocni.CNI) error {
 
 	name := container.ID()
-	// task, taskErr := container.NewTask(ctx, cio.NewCreator(cio.WithStdio))
 
 	task, taskErr := container.NewTask(ctx, cio.BinaryIO("/usr/local/bin/faasd", nil))
 

pkg/provider/handlers/functions.go

@@ -68,6 +68,7 @@ func GetFunction(client *containerd.Client, name string) (Function, error) {
 			if err != nil {
 				return Function{}, fmt.Errorf("unable to get task status for container: %s %s", name, err)
 			}
+
 			if svc.Status == "running" {
 				replicas = 1
 				f.pid = task.Pid()
@@ -85,7 +86,7 @@ func GetFunction(client *containerd.Client, name string) (Function, error) {
 
 		f.replicas = replicas
 		return f, nil
-
 	}
+
 	return Function{}, fmt.Errorf("unable to find function: %s, error %s", name, err)
 }

pkg/provider/handlers/scale.go

@@ -11,6 +11,7 @@ import (
 	"github.com/containerd/containerd"
 	"github.com/containerd/containerd/namespaces"
 	gocni "github.com/containerd/go-cni"
+
 	"github.com/openfaas/faas-provider/types"
 	faasd "github.com/openfaas/faasd/pkg"
 )
@@ -58,46 +59,71 @@ func MakeReplicaUpdateHandler(client *containerd.Client, cni gocni.CNI) func(w h
 			return
 		}
 
-		taskExists := true
+		var taskExists bool
+		var taskStatus *containerd.Status
+
 		task, taskErr := ctr.Task(ctx, nil)
 		if taskErr != nil {
 			msg := fmt.Sprintf("cannot load task for service %s, error: %s", name, taskErr)
 			log.Printf("[Scale] %s\n", msg)
 			taskExists = false
+		} else {
+			taskExists = true
+			status, statusErr := task.Status(ctx)
+			if statusErr != nil {
+				msg := fmt.Sprintf("cannot load task status for %s, error: %s", name, statusErr)
+				log.Printf("[Scale] %s\n", msg)
+				http.Error(w, msg, http.StatusInternalServerError)
+				return
+			} else {
+				taskStatus = &status
+			}
 		}
 
-		if req.Replicas > 0 {
+		createNewTask := false
-			if taskExists {
+
-				if status, statusErr := task.Status(ctx); statusErr == nil {
+		// Scale to zero
-					if status.Status == containerd.Paused {
+		if req.Replicas == 0 {
-						if resumeErr := task.Resume(ctx); resumeErr != nil {
+			// If a task is running, pause it
-							log.Printf("[Scale] error resuming task %s, error: %s\n", name, resumeErr)
+			if taskExists && taskStatus.Status == containerd.Running {
-							http.Error(w, resumeErr.Error(), http.StatusBadRequest)
+				if pauseErr := task.Pause(ctx); pauseErr != nil {
-						}
+					wrappedPauseErr := fmt.Errorf("error pausing task %s, error: %s", name, pauseErr)
-					}
+					log.Printf("[Scale] %s\n", wrappedPauseErr.Error())
-				}
+					http.Error(w, wrappedPauseErr.Error(), http.StatusNotFound)
-			} else {
-				deployErr := createTask(ctx, client, ctr, cni)
-				if deployErr != nil {
-					log.Printf("[Scale] error deploying %s, error: %s\n", name, deployErr)
-					http.Error(w, deployErr.Error(), http.StatusBadRequest)
 					return
 				}
-				return
-			}
-		} else {
-			if taskExists {
-				if status, statusErr := task.Status(ctx); statusErr == nil {
-					if status.Status == containerd.Running {
-						if pauseErr := task.Pause(ctx); pauseErr != nil {
-							log.Printf("[Scale] error pausing task %s, error: %s\n", name, pauseErr)
-							http.Error(w, pauseErr.Error(), http.StatusBadRequest)
-						}
-					}
-				}
 			}
 		}
 
-	}
+		if taskExists {
+			if taskStatus != nil {
+				if taskStatus.Status == containerd.Paused {
+					if resumeErr := task.Resume(ctx); resumeErr != nil {
+						log.Printf("[Scale] error resuming task %s, error: %s\n", name, resumeErr)
+						http.Error(w, resumeErr.Error(), http.StatusBadRequest)
+						return
+					}
+				} else if taskStatus.Status == containerd.Stopped {
+					// Stopped tasks cannot be restarted, must be removed, and created again
+					if _, delErr := task.Delete(ctx); delErr != nil {
+						log.Printf("[Scale] error deleting stopped task %s, error: %s\n", name, delErr)
+						http.Error(w, delErr.Error(), http.StatusBadRequest)
+						return
+					}
+					createNewTask = true
+				}
+			}
+		} else {
+			createNewTask = true
+		}
 
+		if createNewTask {
+			deployErr := createTask(ctx, client, ctr, cni)
+			if deployErr != nil {
+				log.Printf("[Scale] error deploying %s, error: %s\n", name, deployErr)
+				http.Error(w, deployErr.Error(), http.StatusBadRequest)
+				return
+			}
+		}
+	}
 }

pkg/proxy.go

@@ -1,126 +1,116 @@
 package pkg
 
 import (
-	"context"
 	"fmt"
 	"io"
-	"io/ioutil"
 	"log"
+	"net"
 	"net/http"
+	"strconv"
+	"strings"
 
 	"time"
 )
 
-// NewProxy creates a HTTP proxy to expose the gateway container
+// NewProxy creates a HTTP proxy to expose a host
-// from OpenFaaS to the host
+func NewProxy(upstream string, listenPort uint32, hostIP string, timeout time.Duration, resolver Resolver) *Proxy {
-func NewProxy(port int, timeout time.Duration) *Proxy {
 
 	return &Proxy{
-		Port:    port,
+		Upstream: upstream,
-		Timeout: timeout,
+		Port:     listenPort,
+		HostIP:   hostIP,
+		Timeout:  timeout,
+		Resolver: resolver,
 	}
 }
 
 // Proxy for exposing a private container
 type Proxy struct {
 	Timeout time.Duration
-	Port    int
+
+	// Port on which to listen to traffic
+	Port uint32
+
+	// Upstream is where to send traffic when received
+	Upstream string
+
+	// The IP to use to bind locally
+	HostIP string
+
+	Resolver Resolver
 }
 
 // Start listening and forwarding HTTP to the host
-func (p *Proxy) Start(gatewayChan chan string, done chan bool) error {
+func (p *Proxy) Start() error {
-	tcp := p.Port
 
 	http.DefaultClient.CheckRedirect = func(req *http.Request, via []*http.Request) error {
 		return http.ErrUseLastResponse
 	}
-	ps := proxyState{
+	upstreamHost, upstreamPort, err := getUpstream(p.Upstream, p.Port)
-		Host: "",
+	if err != nil {
+		return err
 	}
 
-	ps.Host = <-gatewayChan
+	log.Printf("Looking up IP for: %q", upstreamHost)
+	got := make(chan string, 1)
 
-	log.Printf("Starting faasd proxy on %d\n", tcp)
+	go p.Resolver.Get(upstreamHost, got, time.Second*5)
 
-	fmt.Printf("Gateway: %s\n", ps.Host)
+	ipAddress := <-got
+	close(got)
 
-	s := &http.Server{
+	upstreamAddr := fmt.Sprintf("%s:%d", ipAddress, upstreamPort)
-		Addr:           fmt.Sprintf(":%d", tcp),
+
-		ReadTimeout:    p.Timeout,
+	localBind := fmt.Sprintf("%s:%d", p.HostIP, p.Port)
-		WriteTimeout:   p.Timeout,
+	log.Printf("Proxy from: %s, to: %s (%s)\n", localBind, p.Upstream, ipAddress)
-		MaxHeaderBytes: 1 << 20, // Max header of 1MB
+
-		Handler:        http.HandlerFunc(makeProxy(&ps)),
+	l, err := net.Listen("tcp", localBind)
+	if err != nil {
+		log.Printf("Error: %s", err.Error())
+		return err
 	}
 
-	go func() {
+	defer l.Close()
-		log.Printf("[proxy] Begin listen on %d\n", p.Port)
+	for {
-		if err := s.ListenAndServe(); err != http.ErrServerClosed {
+		// Wait for a connection.
-			log.Printf("Error ListenAndServe: %v", err)
+		conn, err := l.Accept()
+		if err != nil {
+			acceptErr := fmt.Errorf("Unable to accept on %d, error: %s",
+				p.Port,
+				err.Error())
+			log.Printf("%s", acceptErr.Error())
+			return acceptErr
 		}
-	}()
 
-	log.Println("[proxy] Wait for done")
+		upstream, err := net.Dial("tcp", upstreamAddr)
-	<-done
-	log.Println("[proxy] Done received")
-	if err := s.Shutdown(context.Background()); err != nil {
-		log.Printf("[proxy] Error in Shutdown: %v", err)
-	}
 
-	return nil
+		if err != nil {
-}
+			log.Printf("unable to dial to %s, error: %s", upstreamAddr, err.Error())
+			return err
+		}
 
-// copyHeaders clones the header values from the source into the destination.
+		go pipe(conn, upstream)
-func copyHeaders(destination http.Header, source *http.Header) {
+		go pipe(upstream, conn)
-	for k, v := range *source {
-		vClone := make([]string, len(v))
-		copy(vClone, v)
-		destination[k] = vClone
 	}
 }
 
-type proxyState struct {
+func pipe(from net.Conn, to net.Conn) {
-	Host string
+	defer from.Close()
+	io.Copy(from, to)
 }
 
-func makeProxy(ps *proxyState) func(w http.ResponseWriter, r *http.Request) {
+func getUpstream(val string, defaultPort uint32) (string, uint32, error) {
-	return func(w http.ResponseWriter, r *http.Request) {
+	upstreamHostname := val
+	upstreamPort := defaultPort
 
-		query := ""
+	if in := strings.Index(val, ":"); in > -1 {
-		if len(r.URL.RawQuery) > 0 {
+		upstreamHostname = val[:in]
-			query = "?" + r.URL.RawQuery
+		port, err := strconv.ParseInt(val[in+1:], 10, 32)
+		if err != nil {
+			return "", defaultPort, err
 		}
-
+		upstreamPort = uint32(port)
-		upstream := fmt.Sprintf("http://%s%s%s", ps.Host, r.URL.Path, query)
-		fmt.Printf("[faasd] proxy: %s\n", upstream)
-
-		if r.Body != nil {
-			defer r.Body.Close()
-		}
-
-		wrapper := ioutil.NopCloser(r.Body)
-		upReq, upErr := http.NewRequest(r.Method, upstream, wrapper)
-
-		copyHeaders(upReq.Header, &r.Header)
-
-		if upErr != nil {
-			log.Println(upErr)
-
-			http.Error(w, upErr.Error(), http.StatusInternalServerError)
-			return
-		}
-
-		upRes, upResErr := http.DefaultClient.Do(upReq)
-
-		if upResErr != nil {
-			log.Println(upResErr)
-
-			http.Error(w, upResErr.Error(), http.StatusInternalServerError)
-			return
-		}
-
-		copyHeaders(w.Header(), &upRes.Header)
-
-		w.WriteHeader(upRes.StatusCode)
-		io.Copy(w, upRes.Body)
 	}
+
+	return upstreamHostname, upstreamPort, nil
 }

pkg/proxy_test.go

@@ -16,7 +16,7 @@ func Test_Proxy_ToPrivateServer(t *testing.T) {
 
 	wantBodyText := "OK"
 	wantBody := []byte(wantBodyText)
-	upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+	upstreamSvr := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 
 		if r.Body != nil {
 			defer r.Body.Close()
@@ -27,17 +27,19 @@ func Test_Proxy_ToPrivateServer(t *testing.T) {
 
 	}))
 
-	defer upstream.Close()
+	defer upstreamSvr.Close()
 	port := 8080
-	proxy := NewProxy(port, time.Second*1)
+	u, _ := url.Parse(upstreamSvr.URL)
+	log.Println("Host", u.Host)
+
+	upstreamAddr := u.Host
+	proxy := NewProxy(upstreamAddr, 8080, "127.0.0.1", time.Second*1, &mockResolver{})
 
 	gwChan := make(chan string, 1)
 	doneCh := make(chan bool)
 
-	go proxy.Start(gwChan, doneCh)
+	go proxy.Start()
 
-	u, _ := url.Parse(upstream.URL)
-	log.Println("Host", u.Host)
 	wg := sync.WaitGroup{}
 	wg.Add(1)
 	go func() {
@@ -71,3 +73,14 @@ func Test_Proxy_ToPrivateServer(t *testing.T) {
 		doneCh <- true
 	}()
 }
+
+type mockResolver struct {
+}
+
+func (m *mockResolver) Start() {
+
+}
+
+func (m *mockResolver) Get(upstream string, got chan<- string, timeout time.Duration) {
+	got <- upstream
+}

pkg/resolver.go

@@ -0,0 +1,12 @@
+package pkg
+
+import "time"
+
+// Resolver resolves an upstream IP address for a given upstream host
+type Resolver interface {
+	// Start any polling or connections required to resolve
+	Start()
+
+	// Get an IP address using an asynchronous operation
+	Get(upstream string, got chan<- string, timeout time.Duration)
+}

pkg/supervisor.go

@@ -41,6 +41,13 @@ type Service struct {
 	Caps      []string
 	Args      []string
 	DependsOn []string
+	Ports     []ServicePort
+}
+
+type ServicePort struct {
+	TargetPort uint32
+	Port       uint32
+	HostIP     string
 }
 
 type Mount struct {
@@ -112,7 +119,7 @@ func (s *Supervisor) Start(svcs []Service) error {
 		}
 	}
 
-	order := buildInstallOrder(svcs)
+	order := buildDeploymentOrder(svcs)
 
 	for _, key := range order {
 
@@ -154,7 +161,7 @@ func (s *Supervisor) Start(svcs []Service) error {
 			Options:     []string{"rbind", "ro"},
 		})
 
-		newContainer, containerCreateErr := s.client.NewContainer(
+		newContainer, err := s.client.NewContainer(
 			ctx,
 			svc.Name,
 			containerd.WithImage(image),
@@ -166,14 +173,14 @@ func (s *Supervisor) Start(svcs []Service) error {
 				oci.WithEnv(svc.Env)),
 		)
 
-		if containerCreateErr != nil {
+		if err != nil {
-			log.Printf("Error creating container: %s\n", containerCreateErr)
+			log.Printf("Error creating container: %s\n", err)
-			return containerCreateErr
+			return err
 		}
 
 		log.Printf("Created container: %s\n", newContainer.ID())
 
-		task, err := newContainer.NewTask(ctx, cio.NewCreator(cio.WithStdio))
+		task, err := newContainer.NewTask(ctx, cio.BinaryIO("/usr/local/bin/faasd", nil))
 		if err != nil {
 			log.Printf("Error creating task: %s\n", err)
 			return err
@@ -181,13 +188,14 @@ func (s *Supervisor) Start(svcs []Service) error {
 
 		labels := map[string]string{}
 		network, err := cninetwork.CreateCNINetwork(ctx, s.cni, task, labels)
-
 		if err != nil {
+			log.Printf("Error creating CNI for %s: %s", svc.Name, err)
 			return err
 		}
 
 		ip, err := cninetwork.GetIPAddress(network, task)
 		if err != nil {
+			log.Printf("Error getting IP for %s: %s", svc.Name, err)
 			return err
 		}
 
@@ -297,12 +305,26 @@ func ParseCompose(config *compose.Config) ([]Service, error) {
 			Env:       env,
 			Mounts:    mounts,
 			DependsOn: s.DependsOn,
+			Ports:     convertPorts(s.Ports),
 		}
 	}
 
 	return services, nil
 }
 
+func convertPorts(ports []compose.ServicePortConfig) []ServicePort {
+	servicePorts := []ServicePort{}
+	for _, p := range ports {
+		servicePorts = append(servicePorts, ServicePort{
+			Port:       p.Published,
+			TargetPort: p.Target,
+			HostIP:     p.HostIP,
+		})
+	}
+
+	return servicePorts
+}
+
 // LoadComposeFile is a helper method for loading a docker-compose file
 func LoadComposeFile(wd string, file string) (*compose.Config, error) {
 	return LoadComposeFileWithArch(wd, file, env.GetClientArch)

pkg/testdata/docker-compose.yaml

@@ -26,6 +26,8 @@ services:
       - "8222"
       - "--store=memory"
       - "--cluster_id=faas-cluster"
+    ports:
+       - "127.0.0.1:8222:8222"
 
   prometheus:
     image: docker.io/prom/prometheus:v2.14.0
@@ -35,6 +37,8 @@ services:
         target: /etc/prometheus/prometheus.yml
     cap_add:
       - CAP_NET_RAW
+    ports:
+       - "127.0.0.1:9090:9090"
 
   gateway:
     image: "docker.io/openfaas/gateway:0.18.17${ARCH_SUFFIX}"
@@ -65,6 +69,8 @@ services:
       - basic-auth-plugin
       - nats
       - prometheus
+    ports:
+       - "8080:8080"
 
   queue-worker:
     image: docker.io/openfaas/queue-worker:0.11.2