Compare commits

..

20 Commits
0.8.0 ... 0.8.3

Author SHA1 Message Date
d64edeb648 Update OpenFaaS core components for faasd
basic-auth-plugin: 0.18.10 -> 0.18.17
gateway: 0.18.8 -> 0.18.17
queue-worker: 0.9.0 -> 0.11.2
Signed-off-by: Hsiny <yangxinhust@hotmail.com>
2020-05-30 10:18:39 +01:00
42b9cc6b71 Update suffix approach for dev guide
Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
2020-05-27 12:07:08 +01:00
25c553a87c Reorganise docs/ folder
Signed-off-by: Mehdi Yedes <mehdi.yedes@gmail.com>
2020-05-27 12:00:06 +01:00
8bc39f752e Update README.md
Signed-off-by: Mehdi Yedes <mehdi.yedes@gmail.com>
2020-05-27 12:00:06 +01:00
cbff6fa8f6 Include instructions for bootstrapping faasd on digitalocean
Signed-off-by: Mehdi Yedes <mehdi.yedes@gmail.com>
2020-05-27 12:00:06 +01:00
3e29408518 Update README.md 2020-05-24 11:03:34 +01:00
04f1807d92 Bump instructions to latest patch release
Includes a fix for security in 0.8.2

Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
2020-04-29 16:18:53 +01:00
35e017b526 Remove output fields from the logs test case
**What**
- Fix the test to match the new logs command builder, without the output
  fields flag

Signed-off-by: Lucas Roesler <roesler.lucas@gmail.com>
2020-04-29 14:48:57 +01:00
e54da61283 Use quote for test failure output
Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
2020-04-29 12:23:01 +01:00
84353d0cae Format buildCmd test output
Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
2020-04-29 12:23:01 +01:00
e33a60862d Fix vulnerability in authenticated secrets API
This patch fixes a vulnerability in the secrets API, however
it is important to stress that the user must be authenticated
as the admin user on the REST API before they can attempt this.

Reported by Appsecco via email. @lucasroesler, Appsecco and
myself believe this to be of low severity.

The fix prevents directory traversal characters from being
used in secret names. If a secret name such as:
../../root/.ssh/authorized_keys were to be used, an attacker
could remove the value and write their own.

Tested with unit tests and tests are now made to run
via the CI and a new Makefile target.

Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
2020-04-29 12:23:01 +01:00
7b67ff22e6 Update README.md 2020-03-17 20:04:13 +00:00
19abc9f7b9 Use an unbuffered log msg channel
**What**
- Instead of sleeping to ensure the msg channel is populated, use an
  unbuffered channel. This seems to work just as well in all the manual
  tests cases

Signed-off-by: Lucas Roesler <roesler.lucas@gmail.com>
2020-03-08 15:00:42 +00:00
480f566819 Update to 0.8.1 faasd 2020-03-07 11:19:51 +00:00
cece6cf1ef Improve journalctl version compat
**What**
- Remove the `output-fields` flag because not all journalctl versions
  support it
- Add a short sleep to the start of the log stream to avoid some kind of
  race/buffering condition with the Handler

Signed-off-by: Lucas Roesler <roesler.lucas@gmail.com>
2020-03-07 10:11:09 +00:00
22882e2643 Initial journald log provider attempt
**What**
- journald log provider using exec to journalctl
```
journalctl -t <namespace>:<name>  --output=json --since=<timestamp> <--follow> --output-fields=SYSLOG_IDENTIFIER,MESSAGE,_PID,_SOURCE_REALTIME_TIMESTAMP
```
- This can be tested manually using `faas-cli logs` as normal, e.g.
  `faas-cli logs nodeinfo` should tail the last 5 mins of logs.
- Very basic tests ensuring that the `journalctl` comamand is correctly
  construction and that the json log entrys are parsed correctly.
- Add simple e2e test to grep the function logs

Signed-off-by: Lucas Roesler <roesler.lucas@gmail.com>
2020-03-07 10:11:09 +00:00
667d74aaf7 Skip adding function if GetFunction returns error
When ListFunctions populate it's function map, it should not add
functions that GetFunction returned error.

Signed-off-by: Carlos de Paula <me@carlosedp.com>
2020-03-07 07:25:19 +00:00
9dcdbfb7e3 Update DEV.md 2020-03-05 15:28:16 +00:00
3a9b81200e Promote to 0.8.0 with pull policy of always
Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
2020-03-02 17:23:48 +00:00
734425de25 Update the dev workflow
Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
2020-03-01 20:37:19 +00:00
27 changed files with 947 additions and 54 deletions

View File

@ -10,6 +10,7 @@ addons:
- runc
script:
- make test
- make dist
- make prepare-test
- make test-e2e

8
Gopkg.lock generated
View File

@ -175,8 +175,8 @@
name = "github.com/coreos/go-systemd"
packages = ["journal"]
pruneopts = "UT"
revision = "2d78030078ef61b3cae27f42ad6d0e46db51b339"
version = "v22.0.0"
revision = "d3cd4ed1dbcf5835feba465b180436db54f20228"
version = "v21"
[[projects]]
digest = "1:92ebc9c068ab8e3fff03a58694ee33830964f6febd0130069aadce328802de14"
@ -361,12 +361,13 @@
version = "0.18.10"
[[projects]]
digest = "1:6f21508bd38feec0d440ca862f5adcb4c955713f3eb4e075b9af731e6ef258ba"
digest = "1:7a20be0bdfb2c05a4a7b955cb71645fe2983aa3c0bbae10d6bba3e2dd26ddd0d"
name = "github.com/openfaas/faas-provider"
packages = [
".",
"auth",
"httputil",
"logs",
"proxy",
"types",
]
@ -593,6 +594,7 @@
"github.com/morikuni/aec",
"github.com/opencontainers/runtime-spec/specs-go",
"github.com/openfaas/faas-provider",
"github.com/openfaas/faas-provider/logs",
"github.com/openfaas/faas-provider/proxy",
"github.com/openfaas/faas-provider/types",
"github.com/openfaas/faas/gateway/requests",

View File

@ -11,6 +11,10 @@ all: local
local:
CGO_ENABLED=0 GOOS=linux go build -o bin/faasd
.PHONY: test
test:
CGO_ENABLED=0 GOOS=linux go test -ldflags $(LDFLAGS) ./...
.PHONY: dist
dist:
CGO_ENABLED=0 GOOS=linux go build -ldflags $(LDFLAGS) -a -installsuffix cgo -o bin/faasd
@ -48,3 +52,5 @@ test-e2e:
/usr/local/bin/faas-cli remove figlet
sleep 3
/usr/local/bin/faas-cli list
sleep 1
/usr/local/bin/faas-cli logs figlet --follow=false | grep Forking

View File

@ -1,10 +1,11 @@
# faasd - serverless with containerd and CNI 🐳
# faasd - lightweight OSS serverless 🐳
[![Build Status](https://travis-ci.com/openfaas/faasd.svg?branch=master)](https://travis-ci.com/openfaas/faasd)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![OpenFaaS](https://img.shields.io/badge/openfaas-serverless-blue.svg)](https://www.openfaas.com)
![Downloads](https://img.shields.io/github/downloads/openfaas/faasd/total)
faasd is the same OpenFaaS experience and ecosystem, but without Kubernetes. Functions and microservices can be deployed anywhere with reduced overheads whilst retaining the portability of containers and cloud-native tooling.
faasd is the same OpenFaaS experience and ecosystem, but without Kubernetes. Functions and microservices can be deployed anywhere with reduced overheads whilst retaining the portability of containers and cloud-native tooling such as containerd and CNI.
## About faasd
@ -55,7 +56,9 @@ You can run this tutorial on your Raspberry Pi, or adapt the steps for a regular
Automate everything within < 60 seconds and get a public URL and IP address back. Customise as required, or adapt to your preferred cloud such as AWS EC2.
* [Provision faasd 0.7.5 on DigitalOcean with Terraform 0.12.0](https://gist.github.com/alexellis/fd618bd2f957eb08c44d086ef2fc3906)
* [Provision faasd 0.8.1 on DigitalOcean with Terraform 0.12.0](docs/bootstrap/README.md)
* [Provision faasd on DigitalOcean with built-in TLS support](docs/bootstrap/digitalocean-terraform/README.md)
### A note on private repos / registries
@ -121,7 +124,7 @@ An active community of almost 3000 users awaits you on Slack. Over 250 of those
* `faas login`
* `faas up`
* `faas list`
* `faas describe`
* `faas describe`
* `faas deploy --update=true --replace=false`
* `faas invoke --async`
* `faas invoke`
@ -130,12 +133,12 @@ An active community of almost 3000 users awaits you on Slack. Over 250 of those
* `faas version`
* `faas namespace`
* `faas secret`
* `faas logs`
Scale from and to zero is also supported. On a Dell XPS with a small, pre-pulled image unpausing an existing task took 0.19s and starting a task for a killed function took 0.39s. There may be further optimizations to be gained.
Other operations are pending development in the provider such as:
* `faas logs` - to stream logs on-demand for a known function, for the time being you can find logs via `journalctl -u faasd-provider`
* `faas auth` - supported for Basic Authentication, but OAuth2 & OIDC require a patch
## Todo

View File

@ -17,7 +17,7 @@ runcmd:
- curl -sSL https://github.com/containernetworking/plugins/releases/download/v0.8.5/cni-plugins-linux-amd64-v0.8.5.tgz | tar -xz -C /opt/cni/bin
- mkdir -p /go/src/github.com/openfaas/
- cd /go/src/github.com/openfaas/ && git clone https://github.com/openfaas/faasd
- curl -fSLs "https://github.com/openfaas/faasd/releases/download/0.7.7/faasd" --output "/usr/local/bin/faasd" && chmod a+x "/usr/local/bin/faasd"
- curl -fSLs "https://github.com/openfaas/faasd/releases/download/0.8.2/faasd" --output "/usr/local/bin/faasd" && chmod a+x "/usr/local/bin/faasd"
- cd /go/src/github.com/openfaas/faasd/ && /usr/local/bin/faasd install
- systemctl status -l containerd --no-pager
- journalctl -u faasd-provider --no-pager

View File

@ -11,9 +11,11 @@ import (
"github.com/containerd/containerd"
bootstrap "github.com/openfaas/faas-provider"
"github.com/openfaas/faas-provider/logs"
"github.com/openfaas/faas-provider/proxy"
"github.com/openfaas/faas-provider/types"
"github.com/openfaas/faasd/pkg/cninetwork"
faasdlogs "github.com/openfaas/faasd/pkg/logs"
"github.com/openfaas/faasd/pkg/provider/config"
"github.com/openfaas/faasd/pkg/provider/handlers"
"github.com/spf13/cobra"
@ -93,14 +95,7 @@ func makeProviderCmd() *cobra.Command {
InfoHandler: handlers.MakeInfoHandler(Version, GitCommit),
ListNamespaceHandler: listNamespaces(),
SecretHandler: handlers.MakeSecretHandler(client, userSecretPath),
LogHandler: func(w http.ResponseWriter, r *http.Request) {
if r.Body != nil {
defer r.Body.Close()
}
w.WriteHeader(http.StatusNotImplemented)
w.Write([]byte(`Logs are not implemented for faasd`))
},
LogHandler: logs.NewLogHandlerFunc(faasdlogs.New(), config.ReadTimeout),
}
log.Printf("Listening on TCP port: %d\n", *config.TCPPort)

View File

@ -175,7 +175,7 @@ func makeServiceDefinitions(archSuffix string) []pkg.Service {
return []pkg.Service{
{
Name: "basic-auth-plugin",
Image: "docker.io/openfaas/basic-auth-plugin:0.18.10" + archSuffix,
Image: "docker.io/openfaas/basic-auth-plugin:0.18.17" + archSuffix,
Env: []string{
"port=8080",
"secret_mount_path=" + containerSecretMountDir,
@ -230,7 +230,7 @@ func makeServiceDefinitions(archSuffix string) []pkg.Service {
"secret_mount_path=" + containerSecretMountDir,
"scale_from_zero=true",
},
Image: "docker.io/openfaas/gateway:0.18.8" + archSuffix,
Image: "docker.io/openfaas/gateway:0.18.17" + archSuffix,
Mounts: []pkg.Mount{
{
Src: path.Join(path.Join(wd, "secrets"), "basic-auth-password"),
@ -256,7 +256,7 @@ func makeServiceDefinitions(archSuffix string) []pkg.Service {
"basic_auth=true",
"secret_mount_path=" + containerSecretMountDir,
},
Image: "docker.io/openfaas/queue-worker:0.9.0",
Image: "docker.io/openfaas/queue-worker:0.11.2",
Mounts: []pkg.Mount{
{
Src: path.Join(path.Join(wd, "secrets"), "basic-auth-password"),

View File

@ -174,19 +174,19 @@ make local
```sh
# For x86_64
sudo curl -fSLs "https://github.com/openfaas/faasd/releases/download/0.7.4/faasd" \
-o "/usr/local/bin/faasd" \
&& sudo chmod a+x "/usr/local/bin/faasd"
export SUFFIX=""
# armhf
sudo curl -fSLs "https://github.com/openfaas/faasd/releases/download/0.7.4/faasd-armhf" \
-o "/usr/local/bin/faasd" \
&& sudo chmod a+x "/usr/local/bin/faasd"
export SUFFIX="-armhf"
# arm64
sudo curl -fSLs "https://github.com/openfaas/faasd/releases/download/0.7.4/faasd-arm64" \
-o "/usr/local/bin/faasd" \
&& sudo chmod a+x "/usr/local/bin/faasd"
export SUFFIX="-arm64"
# Then download
curl -fSLs "https://github.com/openfaas/faasd/releases/download/0.8.2/faasd$SUFFIX" \
-o "/tmp/faasd" \
&& chmod +x "/tmp/faasd"
sudo mv /tmp/faasd /usr/local/bin/
```
#### Install `faasd`
@ -225,6 +225,54 @@ To get the CLI for the command above run:
curl -sSLf https://cli.openfaas.com | sudo sh
```
#### Make a change to `faasd`
There are two components you can hack on:
For function CRUD you will work on `faasd provider` which is started from `cmd/provider.go`
For faasd itself, you will work on the code from `faasd up`, which is started from `cmd/up.go`
Before working on either, stop the systemd services:
```
sudo systemctl stop faasd & # up command
sudo systemctl stop faasd-provider # provider command
```
Here is a workflow you can use for each code change:
Enter the directory of the source code, and build a new binary:
```bash
cd $GOPATH/src/github.com/openfaas/faasd
go build
```
Copy that binary to `/usr/local/bin/`
```bash
cp faasd /usr/local/bin/
```
To run `faasd up`, run it from its working directory as root
```bash
sudo -i
cd /var/lib/faasd
faasd up
```
Now to run `faasd provider`, run it from its working directory:
```bash
sudo -i
cd /var/lib/faasd-provider
faasd provider
```
#### At run-time
Look in `hosts` in the current working folder or in `/var/lib/faasd/` to get the IP for the gateway or Prometheus

20
docs/bootstrap/README.md Normal file
View File

@ -0,0 +1,20 @@
# Bootstrap faasd on Digitalocean
1) [Sign up to DigitalOcean](https://www.digitalocean.com/?refcode=2962aa9e56a1&utm_campaign=Referral_Invite&utm_medium=Referral_Program&utm_source=CopyPaste)
2) [Download Terraform](https://www.terraform.io)
3) Clone this gist using the URL from the address bar
4) Run `terraform init`
5) Run `terraform apply -var="do_token=$(cat $HOME/digitalocean-access-token)"`
6) View the output for the login command and gateway URL i.e.
```
gateway_url = http://178.128.39.201:8080/
login_cmd = faas-cli login -g http://178.128.39.201:8080/ -p rvIU49CEcFcHmqxj
password = rvIU49CEcFcHmqxj
```
Note that the user-data may take a couple of minutes to come up since it will be pulling in various components and preparing the machine.
A single host with 1GB of RAM will be deployed for you, to remove at a later date simply use `terraform destroy`.
If required, you can remove the VM via `terraform destroy -var="do_token=$(cat $HOME/digitalocean-access-token)"`

View File

@ -0,0 +1,29 @@
#cloud-config
ssh_authorized_keys:
- ${ssh_key}
package_update: true
packages:
- runc
runcmd:
- curl -sLSf https://github.com/containerd/containerd/releases/download/v1.3.2/containerd-1.3.2.linux-amd64.tar.gz > /tmp/containerd.tar.gz && tar -xvf /tmp/containerd.tar.gz -C /usr/local/bin/ --strip-components=1
- curl -SLfs https://raw.githubusercontent.com/containerd/containerd/v1.3.2/containerd.service | tee /etc/systemd/system/containerd.service
- systemctl daemon-reload && systemctl start containerd
- /sbin/sysctl -w net.ipv4.conf.all.forwarding=1
- mkdir -p /opt/cni/bin
- curl -sSL https://github.com/containernetworking/plugins/releases/download/v0.8.5/cni-plugins-linux-amd64-v0.8.5.tgz | tar -xz -C /opt/cni/bin
- mkdir -p /go/src/github.com/openfaas/
- mkdir -p /var/lib/faasd/secrets/
- echo ${gw_password} > /var/lib/faasd/secrets/basic-auth-password
- echo admin > /var/lib/faasd/secrets/basic-auth-user
- cd /go/src/github.com/openfaas/ && git clone https://github.com/openfaas/faasd
- curl -fSLs "https://github.com/openfaas/faasd/releases/download/0.8.1/faasd" --output "/usr/local/bin/faasd" && chmod a+x "/usr/local/bin/faasd"
- cd /go/src/github.com/openfaas/faasd/ && /usr/local/bin/faasd install
- systemctl status -l containerd --no-pager
- journalctl -u faasd-provider --no-pager
- systemctl status -l faasd-provider --no-pager
- systemctl status -l faasd --no-pager
- curl -sSLf https://cli.openfaas.com | sh
- sleep 5 && journalctl -u faasd --no-pager

View File

@ -0,0 +1,37 @@
# Bootstrap faasd with TLS support on Digitalocean
1) [Sign up to DigitalOcean](https://www.digitalocean.com/?refcode=2962aa9e56a1&utm_campaign=Referral_Invite&utm_medium=Referral_Program&utm_source=CopyPaste)
2) [Download Terraform](https://www.terraform.io)
3) Clone this gist using the URL from the address bar
4) Run `terraform init`
5) Configure terraform variables as needed by updating the `main.tfvars` file:
| Variable | Description | Default |
| ------------ | ------------------- | --------------- |
| `do_token` | Digitalocean API token | None |
| `do_domain` | Public domain used for the faasd gateway | None |
| `letsencrypt_email` | Email used by when ordering TLS certificate from Letsencrypt | `""` |
| `do_create_record` | When set to `true`, a new DNS record will be created. This works only if your domain (`do_domain`) is managed by Digitalocean | `false` |
| `do_region` | Digitalocean region for creating the droplet | `fra1` |
| `ssh_key_file` | Path to public SSH key file |`~/.ssh/id_rsa.pub` |
> Environment variables can also be used to set terraform variables when running the `terraform apply` command using the format `TF_VAR_name`.
6) Run `terraform apply`
1) Add `-var-file=main.tfvars` if you have set the variables in `main.tfvars`.
2) OR [use environment variables](https://www.terraform.io/docs/commands/environment-variables.html#tf_var_name) for setting the terraform variables when running the `apply` command
7) View the output for the login command and gateway URL i.e.
```
droplet_ip = 178.128.39.201
gateway_url = https://faasd.example.com/
login_cmd = faas-cli login -g https://faasd.example.com/ -p rvIU49CEcFcHmqxj
password = rvIU49CEcFcHmqxj
```
8) Use your browser to access the OpenFaaS interface
Note that the user-data may take a couple of minutes to come up since it will be pulling in various components and preparing the machine.
Also take into consideration the DNS propagation time for the new DNS record.
A single host with 1GB of RAM will be deployed for you, to remove at a later date simply use `terraform destroy`.

View File

@ -0,0 +1,57 @@
#cloud-config
ssh_authorized_keys:
- ${ssh_key}
groups:
- caddy
users:
- name: caddy
gecos: Caddy web server
primary_group: caddy
groups: caddy
shell: /usr/sbin/nologin
homedir: /var/lib/caddy
write_files:
- content: |
{
email ${letsencrypt_email}
}
${faasd_domain_name} {
reverse_proxy 127.0.0.1:8080
}
path: /etc/caddy/Caddyfile
package_update: true
packages:
- runc
runcmd:
- curl -sLSf https://github.com/containerd/containerd/releases/download/v1.3.2/containerd-1.3.2.linux-amd64.tar.gz > /tmp/containerd.tar.gz && tar -xvf /tmp/containerd.tar.gz -C /usr/local/bin/ --strip-components=1
- curl -SLfs https://raw.githubusercontent.com/containerd/containerd/v1.3.2/containerd.service | tee /etc/systemd/system/containerd.service
- systemctl daemon-reload && systemctl start containerd
- /sbin/sysctl -w net.ipv4.conf.all.forwarding=1
- mkdir -p /opt/cni/bin
- curl -sSL https://github.com/containernetworking/plugins/releases/download/v0.8.5/cni-plugins-linux-amd64-v0.8.5.tgz | tar -xz -C /opt/cni/bin
- mkdir -p /go/src/github.com/openfaas/
- mkdir -p /var/lib/faasd/secrets/
- echo ${gw_password} > /var/lib/faasd/secrets/basic-auth-password
- echo admin > /var/lib/faasd/secrets/basic-auth-user
- cd /go/src/github.com/openfaas/ && git clone https://github.com/openfaas/faasd
- curl -fSLs "https://github.com/openfaas/faasd/releases/download/0.8.1/faasd" --output "/usr/local/bin/faasd" && chmod a+x "/usr/local/bin/faasd"
- cd /go/src/github.com/openfaas/faasd/ && /usr/local/bin/faasd install
- systemctl status -l containerd --no-pager
- journalctl -u faasd-provider --no-pager
- systemctl status -l faasd-provider --no-pager
- systemctl status -l faasd --no-pager
- curl -sSLf https://cli.openfaas.com | sh
- sleep 5 && journalctl -u faasd --no-pager
- wget https://github.com/caddyserver/caddy/releases/download/v2.0.0-rc.2/caddy_2.0.0-rc.2_linux_amd64.tar.gz -O /tmp/caddy.tar.gz && tar -zxvf /tmp/caddy.tar.gz -C /usr/bin/ caddy
- wget https://raw.githubusercontent.com/caddyserver/dist/master/init/caddy.service -O /etc/systemd/system/caddy.service
- systemctl daemon-reload
- systemctl enable caddy
- systemctl start caddy

View File

@ -0,0 +1,82 @@
terraform {
required_version = ">= 0.12"
}
variable "do_token" {
description = "Digitalocean API token"
}
variable "do_domain" {
description = "Your public domain"
}
variable "letsencrypt_email" {
description = "Email used to order a certificate from Letsencrypt"
}
variable "do_create_record" {
default = false
description = "Whether to create a DNS record on Digitalocean"
}
variable "do_region" {
default = "fra1"
description = "The Digitalocean region where the faasd droplet will be created."
}
variable "ssh_key_file" {
default = "~/.ssh/id_rsa.pub"
description = "Path to the SSH public key file"
}
provider "digitalocean" {
token = var.do_token
}
data "local_file" "ssh_key"{
filename = pathexpand(var.ssh_key_file)
}
resource "random_password" "password" {
length = 16
special = true
override_special = "_-#"
}
data "template_file" "cloud_init" {
template = "${file("cloud-config.tpl")}"
vars = {
gw_password=random_password.password.result,
ssh_key=data.local_file.ssh_key.content,
faasd_domain_name="faasd.${var.do_domain}"
letsencrypt_email=var.letsencrypt_email
}
}
resource "digitalocean_droplet" "faasd" {
region = var.do_region
image = "ubuntu-18-04-x64"
name = "faasd"
size = "s-1vcpu-1gb"
user_data = data.template_file.cloud_init.rendered
}
resource "digitalocean_record" "faasd" {
domain = var.do_domain
type = "A"
name = "faasd"
value = digitalocean_droplet.faasd.ipv4_address
# Only creates record if do_create_record is true
count = var.do_create_record == true ? 1 : 0
}
output "droplet_ip" {
value = digitalocean_droplet.faasd.ipv4_address
}
output "gateway_url" {
value = "https://faasd.${var.do_domain}/"
}
output "password" {
value = random_password.password.result
}
output "login_cmd" {
value = "faas-cli login -g https://faasd.${var.do_domain}/ -p ${random_password.password.result}"
}

View File

@ -0,0 +1,3 @@
do_token = ""
do_domain = ""
letsencrypt_email = ""

56
docs/bootstrap/main.tf Normal file
View File

@ -0,0 +1,56 @@
terraform {
required_version = ">= 0.12"
}
variable "do_token" {}
variable "ssh_key_file" {
default = "~/.ssh/id_rsa.pub"
description = "Path to the SSH public key file"
}
provider "digitalocean" {
token = var.do_token
}
resource "random_password" "password" {
length = 16
special = true
override_special = "_-#"
}
data "local_file" "ssh_key"{
filename = pathexpand(var.ssh_key_file)
}
data "template_file" "cloud_init" {
template = "${file("cloud-config.tpl")}"
vars = {
gw_password=random_password.password.result,
ssh_key=data.local_file.ssh_key.content,
}
}
resource "digitalocean_droplet" "faasd" {
region = "lon1"
image = "ubuntu-18-04-x64"
name = "faasd"
# Plans: https://developers.digitalocean.com/documentation/changelog/api-v2/new-size-slugs-for-droplet-plan-changes/
#size = "512mb"
size = "s-1vcpu-1gb"
user_data = data.template_file.cloud_init.rendered
}
output "password" {
value = random_password.password.result
}
output "gateway_url" {
value = "http://${digitalocean_droplet.faasd.ipv4_address}:8080/"
}
output "login_cmd" {
value = "faas-cli login -g http://${digitalocean_droplet.faasd.ipv4_address}:8080/ -p ${random_password.password.result}"
}

6
pkg/contants.go Normal file
View File

@ -0,0 +1,6 @@
package pkg
const (
// FunctionNamespace is the default containerd namespace functions are created
FunctionNamespace = "openfaas-fn"
)

183
pkg/logs/requestor.go Normal file
View File

@ -0,0 +1,183 @@
package logs
import (
"context"
"encoding/json"
"fmt"
"io"
"log"
"os/exec"
"strconv"
"strings"
"time"
"github.com/openfaas/faas-provider/logs"
faasd "github.com/openfaas/faasd/pkg"
)
type requester struct{}
// New returns a new journalctl log Requester
func New() logs.Requester {
return &requester{}
}
// Query submits a log request to the actual logging system.
func (r *requester) Query(ctx context.Context, req logs.Request) (<-chan logs.Message, error) {
_, err := exec.LookPath("journalctl")
if err != nil {
return nil, fmt.Errorf("can not find journalctl: %w", err)
}
cmd := buildCmd(ctx, req)
stdout, err := cmd.StdoutPipe()
if err != nil {
return nil, fmt.Errorf("failed to create journalctl pipe: %w", err)
}
stderr, err := cmd.StderrPipe()
if err != nil {
return nil, fmt.Errorf("failed to create journalctl err pipe: %w", err)
}
err = cmd.Start()
if err != nil {
return nil, fmt.Errorf("failed to create journalctl: %w", err)
}
// call start and get the stdout prior to streaming so that we can return a meaningful
// error for as long as possible. If the cmd starts correctly, we are highly likely to
// succeed anyway
msgs := make(chan logs.Message)
go streamLogs(ctx, cmd, stdout, msgs)
go logErrOut(stderr)
return msgs, nil
}
// buildCmd reeturns the equivalent of
//
// journalctl -t <namespace>:<name> \
// --output=json \
// --since=<timestamp> \
// <--follow> \
func buildCmd(ctx context.Context, req logs.Request) *exec.Cmd {
// // set the cursor position based on req, default to 5m
since := time.Now().Add(-5 * time.Minute)
if req.Since != nil && req.Since.Before(time.Now()) {
since = *req.Since
}
namespace := req.Namespace
if namespace == "" {
namespace = faasd.FunctionNamespace
}
// find the description of the fields here
// https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html
// the available fields can vary greatly, the selected fields were detemined by
// trial and error with journalctl in an ubuntu VM (via multipass)
args := []string{
"--utc",
"--no-pager",
"--output=json",
"--identifier=" + namespace + ":" + req.Name,
fmt.Sprintf("--since=%s", since.UTC().Format("2006-01-02 15:04:05")),
}
if req.Follow {
args = append(args, "--follow")
}
if req.Tail > 0 {
args = append(args, fmt.Sprintf("--lines=%d", req.Tail))
}
return exec.CommandContext(ctx, "journalctl", args...)
}
// streamLogs copies log entries from the journalctl `cmd`/`out` to `msgs`
// the loop is based on the Decoder example in the docs
// https://golang.org/pkg/encoding/json/#Decoder.Decode
func streamLogs(ctx context.Context, cmd *exec.Cmd, out io.ReadCloser, msgs chan logs.Message) {
log.Println("starting journal stream using ", cmd.String())
// will ensure `out` is closed and all related resources cleaned up
go func() {
err := cmd.Wait()
log.Println("wait result", err)
}()
defer func() {
log.Println("closing journal stream")
close(msgs)
}()
dec := json.NewDecoder(out)
for dec.More() {
if ctx.Err() != nil {
log.Println("log stream context cancelled")
return
}
// the journalctl outputs all the values as a string, so a struct with json
// tags wont help much
entry := map[string]string{}
err := dec.Decode(&entry)
if err != nil {
log.Printf("error decoding journalctl output: %s", err)
return
}
msg, err := parseEntry(entry)
if err != nil {
log.Printf("error parsing journalctl output: %s", err)
return
}
msgs <- msg
}
}
// parseEntry reads the deserialized json from journalctl into a log.Message
//
// The following fields are parsed from the journal
// - MESSAGE
// - _PID
// - SYSLOG_IDENTIFIER
// - __REALTIME_TIMESTAMP
func parseEntry(entry map[string]string) (logs.Message, error) {
logMsg := logs.Message{
Text: entry["MESSAGE"],
Instance: entry["_PID"],
}
identifier := entry["SYSLOG_IDENTIFIER"]
parts := strings.Split(identifier, ":")
if len(parts) != 2 {
return logMsg, fmt.Errorf("invalid SYSLOG_IDENTIFIER")
}
logMsg.Namespace = parts[0]
logMsg.Name = parts[1]
ts, ok := entry["__REALTIME_TIMESTAMP"]
if !ok {
return logMsg, fmt.Errorf("missing required field __REALTIME_TIMESTAMP")
}
ms, err := strconv.ParseInt(ts, 10, 64)
if err != nil {
return logMsg, fmt.Errorf("invalid timestamp: %w", err)
}
logMsg.Timestamp = time.Unix(0, ms*1000).UTC()
return logMsg, nil
}
func logErrOut(out io.ReadCloser) {
defer log.Println("stderr closed")
defer out.Close()
io.Copy(log.Writer(), out)
}

View File

@ -0,0 +1,73 @@
package logs
import (
"context"
"encoding/json"
"fmt"
"strings"
"testing"
"time"
"github.com/openfaas/faas-provider/logs"
)
func Test_parseEntry(t *testing.T) {
rawEntry := `{ "__CURSOR" : "s=71c4550142d14ace8e2959e3540cc15c;i=133c;b=44864010f0d94baba7b6bf8019f82a56;m=2945cd3;t=5a00d4eb59180;x=8ed47f7f9b3d798", "__REALTIME_TIMESTAMP" : "1583353899094400", "__MONOTONIC_TIMESTAMP" : "43277523", "_BOOT_ID" : "44864010f0d94baba7b6bf8019f82a56", "SYSLOG_IDENTIFIER" : "openfaas-fn:nodeinfo", "_PID" : "2254", "MESSAGE" : "2020/03/04 20:31:39 POST / - 200 OK - ContentLength: 83", "_SOURCE_REALTIME_TIMESTAMP" : "1583353899094372" }`
expectedEntry := logs.Message{
Name: "nodeinfo",
Namespace: "openfaas-fn",
Text: "2020/03/04 20:31:39 POST / - 200 OK - ContentLength: 83",
Timestamp: time.Unix(0, 1583353899094400*1000).UTC(),
}
value := map[string]string{}
json.Unmarshal([]byte(rawEntry), &value)
entry, err := parseEntry(value)
if err != nil {
t.Fatalf("unexpected error %s", err)
}
if entry.Name != expectedEntry.Name {
t.Fatalf("want Name: %q, got %q", expectedEntry.Name, entry.Name)
}
if entry.Namespace != expectedEntry.Namespace {
t.Fatalf("want Namespace: %q, got %q", expectedEntry.Namespace, entry.Namespace)
}
if entry.Timestamp != expectedEntry.Timestamp {
t.Fatalf("want Timestamp: %q, got %q", expectedEntry.Timestamp, entry.Timestamp)
}
if entry.Text != expectedEntry.Text {
t.Fatalf("want Text: %q, got %q", expectedEntry.Text, entry.Text)
}
}
func Test_buildCmd(t *testing.T) {
ctx := context.TODO()
now := time.Now()
req := logs.Request{
Name: "loggyfunc",
Namespace: "spacetwo",
Follow: true,
Since: &now,
Tail: 5,
}
expectedArgs := fmt.Sprintf(
"--utc --no-pager --output=json --identifier=spacetwo:loggyfunc --since=%s --follow --lines=5",
now.UTC().Format("2006-01-02 15:04:05"),
)
cmd := buildCmd(ctx, req).String()
wantCmd := "journalctl"
if !strings.Contains(cmd, wantCmd) {
t.Fatalf("cmd want: %q, got: %q", wantCmd, cmd)
}
if !strings.HasSuffix(cmd, expectedArgs) {
t.Fatalf("arg want: %q\ngot: %q", expectedArgs, cmd)
}
}

View File

@ -12,6 +12,8 @@ import (
"github.com/containerd/containerd/namespaces"
gocni "github.com/containerd/go-cni"
"github.com/openfaas/faas/gateway/requests"
faasd "github.com/openfaas/faasd/pkg"
cninetwork "github.com/openfaas/faasd/pkg/cninetwork"
"github.com/openfaas/faasd/pkg/service"
)
@ -49,7 +51,7 @@ func MakeDeleteHandler(client *containerd.Client, cni gocni.CNI) func(w http.Res
return
}
ctx := namespaces.WithNamespace(context.Background(), FunctionNamespace)
ctx := namespaces.WithNamespace(context.Background(), faasd.FunctionNamespace)
// TODO: this needs to still happen if the task is paused
if function.replicas != 0 {

View File

@ -18,6 +18,7 @@ import (
"github.com/docker/distribution/reference"
"github.com/opencontainers/runtime-spec/specs-go"
"github.com/openfaas/faas-provider/types"
faasd "github.com/openfaas/faasd/pkg"
cninetwork "github.com/openfaas/faasd/pkg/cninetwork"
"github.com/openfaas/faasd/pkg/service"
"github.com/pkg/errors"
@ -52,7 +53,7 @@ func MakeDeployHandler(client *containerd.Client, cni gocni.CNI, secretMountPath
}
name := req.Service
ctx := namespaces.WithNamespace(context.Background(), FunctionNamespace)
ctx := namespaces.WithNamespace(context.Background(), faasd.FunctionNamespace)
deployErr := deploy(ctx, req, client, cni, secretMountPath, alwaysPull)
if deployErr != nil {

View File

@ -8,6 +8,8 @@ import (
"github.com/containerd/containerd"
"github.com/containerd/containerd/namespaces"
"github.com/openfaas/faasd/pkg/cninetwork"
faasd "github.com/openfaas/faasd/pkg"
)
type Function struct {
@ -20,31 +22,29 @@ type Function struct {
labels map[string]string
}
const (
// FunctionNamespace is the containerd namespace functions are created
FunctionNamespace = "openfaas-fn"
)
// ListFunctions returns a map of all functions with running tasks on namespace
func ListFunctions(client *containerd.Client) (map[string]Function, error) {
ctx := namespaces.WithNamespace(context.Background(), FunctionNamespace)
ctx := namespaces.WithNamespace(context.Background(), faasd.FunctionNamespace)
functions := make(map[string]Function)
containers, _ := client.Containers(ctx)
for _, k := range containers {
name := k.ID()
functions[name], _ = GetFunction(client, name)
f, err := GetFunction(client, name)
if err != nil {
continue
}
functions[name] = f
}
return functions, nil
}
// GetFunction returns a function that matches name
func GetFunction(client *containerd.Client, name string) (Function, error) {
ctx := namespaces.WithNamespace(context.Background(), FunctionNamespace)
ctx := namespaces.WithNamespace(context.Background(), faasd.FunctionNamespace)
c, err := client.LoadContainer(ctx, name)
if err == nil {
image, _ := c.Image(ctx)
containerName := c.ID()
@ -55,7 +55,7 @@ func GetFunction(client *containerd.Client, name string) (Function, error) {
f := Function{
name: containerName,
namespace: FunctionNamespace,
namespace: faasd.FunctionNamespace,
image: image.Name(),
labels: labels,
}

View File

@ -12,6 +12,7 @@ import (
"github.com/containerd/containerd/namespaces"
gocni "github.com/containerd/go-cni"
"github.com/openfaas/faas-provider/types"
faasd "github.com/openfaas/faasd/pkg"
)
func MakeReplicaUpdateHandler(client *containerd.Client, cni gocni.CNI) func(w http.ResponseWriter, r *http.Request) {
@ -47,7 +48,7 @@ func MakeReplicaUpdateHandler(client *containerd.Client, cni gocni.CNI) func(w h
return
}
ctx := namespaces.WithNamespace(context.Background(), FunctionNamespace)
ctx := namespaces.WithNamespace(context.Background(), faasd.FunctionNamespace)
ctr, ctrErr := client.LoadContainer(ctx, name)
if ctrErr != nil {

View File

@ -2,11 +2,13 @@ package handlers
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
"os"
"path"
"strings"
"github.com/containerd/containerd"
"github.com/openfaas/faas-provider/types"
@ -76,17 +78,6 @@ func createSecret(c *containerd.Client, w http.ResponseWriter, r *http.Request,
}
}
func parseSecret(r *http.Request) (types.Secret, error) {
secret := types.Secret{}
bytesOut, err := ioutil.ReadAll(r.Body)
if err != nil {
return secret, err
}
err = json.Unmarshal(bytesOut, &secret)
return secret, err
}
func deleteSecret(c *containerd.Client, w http.ResponseWriter, r *http.Request, mountPath string) {
secret, err := parseSecret(r)
if err != nil {
@ -103,3 +94,29 @@ func deleteSecret(c *containerd.Client, w http.ResponseWriter, r *http.Request,
return
}
}
func parseSecret(r *http.Request) (types.Secret, error) {
secret := types.Secret{}
bytesOut, err := ioutil.ReadAll(r.Body)
if err != nil {
return secret, err
}
err = json.Unmarshal(bytesOut, &secret)
if err != nil {
return secret, err
}
if isTraversal(secret.Name) {
return secret, fmt.Errorf(traverseErrorSt)
}
return secret, err
}
const traverseErrorSt = "directory traversal found in name"
func isTraversal(name string) bool {
return strings.Contains(name, fmt.Sprintf("%s", string(os.PathSeparator))) ||
strings.Contains(name, "..")
}

View File

@ -0,0 +1,63 @@
package handlers
import (
"bytes"
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"github.com/openfaas/faas-provider/types"
)
func Test_parseSecretValidName(t *testing.T) {
s := types.Secret{Name: "authorized_keys"}
body, _ := json.Marshal(s)
reader := bytes.NewReader(body)
r := httptest.NewRequest(http.MethodPost, "/", reader)
_, err := parseSecret(r)
if err != nil {
t.Fatalf("secret name is valid with no traversal characters")
}
}
func Test_parseSecretValidNameWithDot(t *testing.T) {
s := types.Secret{Name: "authorized.keys"}
body, _ := json.Marshal(s)
reader := bytes.NewReader(body)
r := httptest.NewRequest(http.MethodPost, "/", reader)
_, err := parseSecret(r)
if err != nil {
t.Fatalf("secret name is valid with no traversal characters")
}
}
func Test_parseSecretWithTraversalWithSlash(t *testing.T) {
s := types.Secret{Name: "/root/.ssh/authorized_keys"}
body, _ := json.Marshal(s)
reader := bytes.NewReader(body)
r := httptest.NewRequest(http.MethodPost, "/", reader)
_, err := parseSecret(r)
if err == nil {
t.Fatalf("secret name should fail due to path traversal")
}
}
func Test_parseSecretWithTraversalWithDoubleDot(t *testing.T) {
s := types.Secret{Name: ".."}
body, _ := json.Marshal(s)
reader := bytes.NewReader(body)
r := httptest.NewRequest(http.MethodPost, "/", reader)
_, err := parseSecret(r)
if err == nil {
t.Fatalf("secret name should fail due to path traversal")
}
}

View File

@ -12,6 +12,8 @@ import (
"github.com/containerd/containerd/namespaces"
gocni "github.com/containerd/go-cni"
"github.com/openfaas/faas-provider/types"
faasd "github.com/openfaas/faasd/pkg"
"github.com/openfaas/faasd/pkg/cninetwork"
"github.com/openfaas/faasd/pkg/service"
)
@ -53,7 +55,7 @@ func MakeUpdateHandler(client *containerd.Client, cni gocni.CNI, secretMountPath
http.Error(w, err.Error(), http.StatusBadRequest)
}
ctx := namespaces.WithNamespace(context.Background(), FunctionNamespace)
ctx := namespaces.WithNamespace(context.Background(), faasd.FunctionNamespace)
if function.replicas != 0 {
err = cninetwork.DeleteCNINetwork(ctx, cni, client, name)
if err != nil {

View File

@ -0,0 +1,144 @@
package logs
import (
"context"
"encoding/json"
"log"
"net/http"
"net/url"
"strconv"
"time"
"github.com/openfaas/faas-provider/httputil"
)
// Requester submits queries the logging system.
// This will be passed to the log handler constructor.
type Requester interface {
// Query submits a log request to the actual logging system.
Query(context.Context, Request) (<-chan Message, error)
}
// NewLogHandlerFunc creates an http HandlerFunc from the supplied log Requestor.
func NewLogHandlerFunc(requestor Requester, timeout time.Duration) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
if r.Body != nil {
defer r.Body.Close()
}
cn, ok := w.(http.CloseNotifier)
if !ok {
log.Println("LogHandler: response is not a CloseNotifier, required for streaming response")
http.NotFound(w, r)
return
}
flusher, ok := w.(http.Flusher)
if !ok {
log.Println("LogHandler: response is not a Flusher, required for streaming response")
http.NotFound(w, r)
return
}
logRequest, err := parseRequest(r)
if err != nil {
log.Printf("LogHandler: could not parse request %s", err)
httputil.Errorf(w, http.StatusUnprocessableEntity, "could not parse the log request")
return
}
ctx, cancelQuery := context.WithTimeout(r.Context(), timeout)
defer cancelQuery()
messages, err := requestor.Query(ctx, logRequest)
if err != nil {
// add smarter error handling here
httputil.Errorf(w, http.StatusInternalServerError, "function log request failed")
return
}
// Send the initial headers saying we're gonna stream the response.
w.Header().Set("Connection", "Keep-Alive")
w.Header().Set("Transfer-Encoding", "chunked")
w.Header().Set(http.CanonicalHeaderKey("Content-Type"), "application/x-ndjson")
w.WriteHeader(http.StatusOK)
flusher.Flush()
// ensure that we always try to send the closing chunk, not the inverted order due to how
// the defer stack works. We need two flush statements to ensure that the empty slice is
// sent as its own chunk
defer flusher.Flush()
defer w.Write([]byte{})
defer flusher.Flush()
jsonEncoder := json.NewEncoder(w)
for messages != nil {
select {
case <-cn.CloseNotify():
log.Println("LogHandler: client stopped listening")
return
case msg, ok := <-messages:
if !ok {
log.Println("LogHandler: end of log stream")
messages = nil
return
}
// serialize and write the msg to the http ResponseWriter
err := jsonEncoder.Encode(msg)
if err != nil {
// can't actually write the status header here so we should json serialize an error
// and return that because we have already sent the content type and status code
log.Printf("LogHandler: failed to serialize log message: '%s'\n", msg.String())
log.Println(err.Error())
// write json error message here ?
jsonEncoder.Encode(Message{Text: "failed to serialize log message"})
flusher.Flush()
return
}
flusher.Flush()
}
}
return
}
}
// parseRequest extracts the logRequest from the GET variables or from the POST body
func parseRequest(r *http.Request) (logRequest Request, err error) {
query := r.URL.Query()
logRequest.Name = getValue(query, "name")
logRequest.Namespace = getValue(query, "namespace")
logRequest.Instance = getValue(query, "instance")
tailStr := getValue(query, "tail")
if tailStr != "" {
logRequest.Tail, err = strconv.Atoi(tailStr)
if err != nil {
return logRequest, err
}
}
// ignore error because it will default to false if we can't parse it
logRequest.Follow, _ = strconv.ParseBool(getValue(query, "follow"))
sinceStr := getValue(query, "since")
if sinceStr != "" {
since, err := time.Parse(time.RFC3339, sinceStr)
logRequest.Since = &since
if err != nil {
return logRequest, err
}
}
return logRequest, nil
}
// getValue returns the value for the given key. If the key has more than one value, it returns the
// last value. if the value does not exist, it returns the empty string.
func getValue(queryValues url.Values, name string) string {
values := queryValues[name]
if len(values) == 0 {
return ""
}
return values[len(values)-1]
}

62
vendor/github.com/openfaas/faas-provider/logs/logs.go generated vendored Normal file
View File

@ -0,0 +1,62 @@
// Package logs provides the standard interface and handler for OpenFaaS providers to expose function logs.
//
// The package defines the Requester interface that OpenFaaS providers should implement and then expose using
// the predefined NewLogHandlerFunc. See the example folder for a minimal log provider implementation.
//
// The Requester is where the actual specific logic for connecting to and querying the log system should be implemented.
//
package logs
import (
"fmt"
"time"
)
// Request is the query to return the function logs.
type Request struct {
// Name is the function name and is required
Name string `json:"name"`
// Namespace is the namespace the function is deployed to, how a namespace is defined
// is faas-provider specific
Namespace string `json:"namespace"`
// Instance is the optional container name, that allows you to request logs from a specific function instance
Instance string `json:"instance"`
// Since is the optional datetime value to start the logs from
Since *time.Time `json:"since"`
// Tail sets the maximum number of log messages to return, <=0 means unlimited
Tail int `json:"tail"`
// Follow is allows the user to request a stream of logs until the timeout
Follow bool `json:"follow"`
}
// String implements that Stringer interface and prints the log Request in a consistent way that
// allows you to safely compare if two requests have the same value.
func (r Request) String() string {
return fmt.Sprintf(
"name:%s namespace: %s instance:%s since:%v tail:%d follow:%v",
r.Name, r.Namespace, r.Instance, r.Since, r.Tail, r.Follow,
)
}
// Message is a specific log message from a function container log stream
type Message struct {
// Name is the function name
Name string `json:"name"`
// Namespace is the namespace the function is deployed to, how a namespace is defined
// is faas-provider specific
Namespace string `json:"namespace"`
// instance is the name/id of the specific function instance
Instance string `json:"instance"`
// Timestamp is the timestamp of when the log message was recorded
Timestamp time.Time `json:"timestamp"`
// Text is the raw log message content
Text string `json:"text"`
}
// String implements the Stringer interface and allows for nice and simple string formatting of a log Message.
func (m Message) String() string {
return fmt.Sprintf(
"%s %s (%s %s) %s",
m.Timestamp.String(), m.Name, m.Namespace, m.Instance, m.Text,
)
}