Compare commits

...

29 Commits

Author SHA1 Message Date
8ac45f5379 Avoid providing memory limit if not set explicitly
Signed-off-by: Nitishkumar Singh <nitishkumarsingh71@gmail.com>
2022-01-25 16:17:30 +00:00
3579061423 Bump gateway version 0.21.3
Signed-off-by: Nitishkumar Singh <nitishkumarsingh71@gmail.com>
2022-01-25 16:12:12 +00:00
761d1847bf Update ISSUE_TEMPLATE.md 2022-01-21 09:12:27 +00:00
8003748b73 Review feedback for Labeller
Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
2022-01-19 18:13:05 +00:00
a2ea804d2c Handled list secrets for no secret in namespaces
Signed-off-by: Nitishkumar Singh <nitishkumarsingh71@gmail.com>

Test case included for default and non-default

Signed-off-by: Nitishkumar Singh <nitishkumarsingh71@gmail.com>

Changed Fake Labeller Implementation

Signed-off-by: Nitishkumar Singh <nitishkumarsingh71@gmail.com>
2022-01-19 18:05:56 +00:00
551e6645b7 chore: improve multipass cloud-config and instructions
Improve the Multipass cloud-config.txt by using the install.sh script
instead of stale install instructions. This ensures that the latest
release is installed and reduces the number of install instructions we
need to maintain.

Also, improve the instructions for using multipass by including a
one-line command for setting the correct ssh key _and_ starting the VM.

Finally, improve the markdown formatting by indenting the paragraph
bodies of the list items. This ensures that the content is properly
aligned with the bullet list.

Signed-off-by: Lucas Roesler <roesler.lucas@gmail.com>
2022-01-19 12:44:47 +00:00
77867f17e3 Migrate to Go 1.17
Tested during local development and deployment to multipass
and Ubuntu. Worked as expected.

Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
2021-12-21 13:26:38 +00:00
5aed707354 Create volumes automatically for NATS/Prometheus
Fixes: #223

Tested by checking the logs of Prometheus and NATS was tested
by running async requests with hey then restarting faasd,
which deletes the NATS and queue-worker containers. The work
left over in the queue was restarted as expected.

Pre-created volumes are read from docker-compose.yaml and
only numeric user IDs are supported at this time. Users
can still specify a textual username, if they create the
source directory for a mount before restarting faasd,
it won't try to overwrite the directory.

Add comment for workingDirectoryPermission

Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
2021-12-21 13:26:38 +00:00
8fbdd1a461 Update ROADMAP.md 2021-11-14 08:58:52 +00:00
8dd48b8957 Update README.md 2021-11-14 08:56:56 +00:00
6763ed6d66 Update README.md 2021-11-14 08:55:53 +00:00
acb5d0bd1c Amend patches.md gh command to include 'pr'
Signed-off-by: Richard Gee richard@technologee.co.uk
2021-11-03 19:48:39 +00:00
2c9eb3904e Add guide for testing patches.
Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
2021-11-01 13:00:26 +00:00
b42066d1a1 Fixed bad memory display and refactor test cases in functions_test.go
Signed-off-by: Shikachuu <zcmate@gmail.com>
2021-11-01 10:07:25 +00:00
17188b8de9 Added unit tests for readMemoryLimitFromSpec
Signed-off-by: Shikachuu <zcmate@gmail.com>
2021-11-01 10:07:25 +00:00
0c0088e8b0 Change readMemoryLimitFromSpec, to a more clear implementation, edited error message.
Signed-off-by: Shikachuu <zcmate@gmail.com>
2021-11-01 10:07:25 +00:00
c5f167df21 Change plain number response to Decimal String.
Signed-off-by: Shikachuu <zcmate@gmail.com>
2021-11-01 10:07:25 +00:00
d5fcc7b2ab Fixed nil pointer dereference while parsing memory limit
Signed-off-by: Shikachuu <zcmate@gmail.com>
2021-11-01 10:07:25 +00:00
cbfefb6fa5 Extend the Function type with a memoryLimit field, create a conversion to k8s resource value and return it through the REST API.
Signed-off-by: Shikachuu <zcmate@gmail.com>
2021-11-01 10:07:25 +00:00
ea62c1b12d feat: add support for raw secret values
Load the secret value from the RawValue field, if it is empty, use the
string value. Add unit tests for the creation handler.

Refactor secret parser tests.

Resolves #208

Signed-off-by: Lucas Roesler <roesler.lucas@gmail.com>
2021-10-17 18:04:06 +01:00
8f40618a5c Update README.md 2021-10-17 15:49:25 +01:00
3fe0d8d8d3 Update messages to want/got for unit tests
This is the style used in the openfaas project.

Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
2021-09-16 10:44:36 +01:00
5aa4c69e03 Inline namespace check and create const for label
* Inlines the namespace check for valid faasd namespaces
* Creates a const for the namespace label applied to faasd
namespaces

Tested with go build and go test.

Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
2021-09-16 10:43:21 +01:00
12b5e8ca7f Add check for namespace label openfaas=true
This commit adds the checks that the namespace supplied by the user has
the `openfaas=true` label. Without this check the user can
deploy/update/read functions in any namespace  using the CLI.

The UI is not effected because it calls the listnamesaces endpoint,
which has the check for the label

Signed-off-by: Alistair Hey <alistair@heyal.co.uk>
2021-09-16 10:37:32 +01:00
195e81f595 Fix for #201
Old secrets are now copied, rather than moved, so that any
existing functions do not need to be redeployed by the user.

As a maintenance task, users should remove the older secrets.

Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
2021-09-15 19:49:28 +01:00
06fbca83bf Fix syntax error with error wrapping
Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
2021-09-15 16:17:12 +01:00
e71d2c27c5 Update some errors to wrapped syntax
Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
2021-09-15 15:54:44 +01:00
13f4a487ce Correct error formatting
Errors should not start with an uppercase letter.

Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
2021-09-15 15:52:13 +01:00
13412841aa Rename getMounts to getOSMounts
A more descriptive name

Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
2021-09-15 15:51:43 +01:00
31 changed files with 806 additions and 220 deletions

View File

@ -1,3 +1,7 @@
## Due diligence
Before you for help or support, make sure that you've [consulted the faasd manual "Serverless For Everyone Else"](https://openfaas.gumroad.com/l/serverless-for-everyone-else).
<!--- Provide a general summary of the issue in the Title above -->
## Expected Behaviour

View File

@ -12,7 +12,7 @@ jobs:
GO111MODULE: off
strategy:
matrix:
go-version: [1.16.x]
go-version: [1.17.x]
os: [ubuntu-latest]
runs-on: ${{ matrix.os }}
steps:

View File

@ -9,7 +9,7 @@ jobs:
publish:
strategy:
matrix:
go-version: [ 1.16.x ]
go-version: [ 1.17.x ]
os: [ ubuntu-latest ]
runs-on: ${{ matrix.os }}
steps:

View File

@ -1,5 +1,6 @@
# faasd - a lightweight & portable faas engine
[![Sponsor this](https://img.shields.io/static/v1?label=Sponsor&message=%E2%9D%A4&logo=GitHub&link=https://github.com/sponsors/openfaas)](https://github.com/sponsors/openfaas)
[![Build Status](https://github.com/openfaas/faasd/workflows/build/badge.svg?branch=master)](https://github.com/openfaas/faasd/actions)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![OpenFaaS](https://img.shields.io/badge/openfaas-serverless-blue.svg)](https://www.openfaas.com)
@ -44,7 +45,7 @@ Additional resources:
Most importantly, it's easy to manage so you can set it up and leave it alone to run your functions.
![demo](https://pbs.twimg.com/media/EPNQz00W4AEwDxM?format=jpg&name=small)
[![demo](https://pbs.twimg.com/media/EPNQz00W4AEwDxM?format=jpg&name=medium)](https://www.youtube.com/watch?v=WX1tZoSXy8E)
> Demo of faasd running asynchronous functions
@ -166,4 +167,6 @@ For completed features, WIP and upcoming roadmap see:
See [ROADMAP.md](docs/ROADMAP.md)
Want to build a patch without setting up a complete development environment? See [docs/PATCHES.md](docs/PATCHES.md)
Are you looking to hack on faasd? Follow the [developer instructions](docs/DEV.md) for a manual installation, or use the `hack/install.sh` script and pick up from there.

View File

@ -10,17 +10,7 @@ packages:
- git
runcmd:
- curl -sLSf https://github.com/containerd/containerd/releases/download/v1.5.4/containerd-1.5.4-linux-amd64.tar.gz > /tmp/containerd.tar.gz && tar -xvf /tmp/containerd.tar.gz -C /usr/local/bin/ --strip-components=1
- curl -SLfs https://raw.githubusercontent.com/containerd/containerd/v1.5.4/containerd.service | tee /etc/systemd/system/containerd.service
- systemctl daemon-reload && systemctl start containerd
- systemctl enable containerd
- /sbin/sysctl -w net.ipv4.conf.all.forwarding=1
- mkdir -p /opt/cni/bin
- curl -sSL https://github.com/containernetworking/plugins/releases/download/v0.8.5/cni-plugins-linux-amd64-v0.8.5.tgz | tar -xz -C /opt/cni/bin
- mkdir -p /go/src/github.com/openfaas/
- cd /go/src/github.com/openfaas/ && git clone --depth 1 --branch 0.13.0 https://github.com/openfaas/faasd
- curl -fSLs "https://github.com/openfaas/faasd/releases/download/0.13.0/faasd" --output "/usr/local/bin/faasd" && chmod a+x "/usr/local/bin/faasd"
- cd /go/src/github.com/openfaas/faasd/ && /usr/local/bin/faasd install
- curl -sfL https://raw.githubusercontent.com/openfaas/faasd/master/hack/install.sh | sh -s -
- systemctl status -l containerd --no-pager
- journalctl -u faasd-provider --no-pager
- systemctl status -l faasd-provider --no-pager

View File

@ -109,7 +109,16 @@ func binExists(folder, name string) error {
}
return nil
}
func ensureSecretsDir(folder string) error {
if _, err := os.Stat(folder); err != nil {
err = os.MkdirAll(folder, secretDirPermission)
if err != nil {
return err
}
}
return nil
}
func ensureWorkingDir(folder string) error {
if _, err := os.Stat(folder); err != nil {
err = os.MkdirAll(folder, workingDirectoryPermission)

View File

@ -2,6 +2,7 @@ package cmd
import (
"fmt"
"io"
"io/ioutil"
"log"
"net/http"
@ -21,6 +22,8 @@ import (
"github.com/spf13/cobra"
)
const secretDirPermission = 0755
func makeProviderCmd() *cobra.Command {
var command = &cobra.Command{
Use: "provider",
@ -82,25 +85,25 @@ func makeProviderCmd() *cobra.Command {
invokeResolver := handlers.NewInvokeResolver(client)
userSecretPath := path.Join(wd, "secrets")
err = moveSecretsToDefaultNamespaceSecrets(userSecretPath, faasd.FunctionNamespace)
if err != nil {
baseUserSecretsPath := path.Join(wd, "secrets")
if err := moveSecretsToDefaultNamespaceSecrets(
baseUserSecretsPath,
faasd.DefaultFunctionNamespace); err != nil {
return err
}
bootstrapHandlers := types.FaaSHandlers{
FunctionProxy: proxy.NewHandlerFunc(*config, invokeResolver),
DeleteHandler: handlers.MakeDeleteHandler(client, cni),
DeployHandler: handlers.MakeDeployHandler(client, cni, userSecretPath, alwaysPull),
DeployHandler: handlers.MakeDeployHandler(client, cni, baseUserSecretsPath, alwaysPull),
FunctionReader: handlers.MakeReadHandler(client),
ReplicaReader: handlers.MakeReplicaReaderHandler(client),
ReplicaUpdater: handlers.MakeReplicaUpdateHandler(client, cni),
UpdateHandler: handlers.MakeUpdateHandler(client, cni, userSecretPath, alwaysPull),
UpdateHandler: handlers.MakeUpdateHandler(client, cni, baseUserSecretsPath, alwaysPull),
HealthHandler: func(w http.ResponseWriter, r *http.Request) {},
InfoHandler: handlers.MakeInfoHandler(Version, GitCommit),
ListNamespaceHandler: handlers.MakeNamespacesLister(client),
SecretHandler: handlers.MakeSecretHandler(client, userSecretPath),
SecretHandler: handlers.MakeSecretHandler(client.NamespaceService(), baseUserSecretsPath),
LogHandler: logs.NewLogHandlerFunc(faasdlogs.New(), config.ReadTimeout),
}
@ -116,29 +119,58 @@ func makeProviderCmd() *cobra.Command {
* Mutiple namespace support was added after release 0.13.0
* Function will help users to migrate on multiple namespace support of faasd
*/
func moveSecretsToDefaultNamespaceSecrets(secretPath string, namespace string) error {
newSecretPath := path.Join(secretPath, namespace)
func moveSecretsToDefaultNamespaceSecrets(baseSecretPath string, defaultNamespace string) error {
newSecretPath := path.Join(baseSecretPath, defaultNamespace)
err := ensureWorkingDir(newSecretPath)
err := ensureSecretsDir(newSecretPath)
if err != nil {
return err
}
files, err := ioutil.ReadDir(secretPath)
files, err := ioutil.ReadDir(baseSecretPath)
if err != nil {
return err
}
for _, f := range files {
if !f.IsDir() {
oldPath := path.Join(secretPath, f.Name())
newPath := path.Join(newSecretPath, f.Name())
err = os.Rename(oldPath, newPath)
if err != nil {
return err
// A non-nil error means the file wasn't found in the
// destination path
if _, err := os.Stat(newPath); err != nil {
oldPath := path.Join(baseSecretPath, f.Name())
if err := copyFile(oldPath, newPath); err != nil {
return err
}
log.Printf("[Migration] Copied %s to %s", oldPath, newPath)
}
}
}
return nil
}
func copyFile(src, dst string) error {
inputFile, err := os.Open(src)
if err != nil {
return fmt.Errorf("opening %s failed %w", src, err)
}
defer inputFile.Close()
outputFile, err := os.OpenFile(dst, os.O_CREATE|os.O_WRONLY|os.O_APPEND, secretDirPermission)
if err != nil {
return fmt.Errorf("opening %s failed %w", dst, err)
}
defer outputFile.Close()
// Changed from os.Rename due to issue in #201
if _, err := io.Copy(outputFile, inputFile); err != nil {
return fmt.Errorf("writing into %s failed %w", outputFile.Name(), err)
}
return nil
}

View File

@ -20,28 +20,43 @@ services:
nats:
image: docker.io/library/nats-streaming:0.22.0
# nobody
user: "65534"
command:
- "/nats-streaming-server"
- "-m"
- "8222"
- "--store=memory"
- "--store=file"
- "--dir=/nats"
- "--cluster_id=faas-cluster"
volumes:
# Data directory
- type: bind
source: ./nats
target: /nats
# ports:
# - "127.0.0.1:8222:8222"
prometheus:
image: docker.io/prom/prometheus:v2.14.0
# nobody
user: "65534"
volumes:
# Config directory
- type: bind
source: ./prometheus.yml
target: /etc/prometheus/prometheus.yml
# Data directory
- type: bind
source: ./prometheus
target: /prometheus
cap_add:
- CAP_NET_RAW
ports:
- "127.0.0.1:9090:9090"
gateway:
image: ghcr.io/openfaas/gateway:0.21.0
image: ghcr.io/openfaas/gateway:0.21.3
environment:
- basic_auth=true
- functions_provider_url=http://faasd-provider:8081/

View File

@ -1,7 +1,11 @@
## Instructions for hacking on faasd itself
## Instructions for building and testing faasd locally
> Note: if you're just wanting to try out faasd, then it's likely that you're on the wrong page. This is a detailed set of instructions for those wanting to contribute or customise faasd. Feel free to go back to the homepage and pick a tutorial instead.
Do you want to help the community test a pull request?
See these instructions instead: [Testing patches](/docs/PATCHES.md)
### Pre-reqs
> It's recommended that you do not install Docker on the same host as faasd, since 1) they may both use different versions of containerd and 2) docker's networking rules can disrupt faasd's networking. When using faasd - make your faasd server a faasd server, and build container image on your laptop or in a CI pipeline.

View File

@ -27,111 +27,112 @@ It took me about 2-3 minutes to run through everything after installing multipas
* Get my cloud-config.txt file
```sh
curl -sSLO https://raw.githubusercontent.com/openfaas/faasd/master/cloud-config.txt
```
```sh
curl -sSLO https://raw.githubusercontent.com/openfaas/faasd/master/cloud-config.txt
```
* Update the SSH key to match your own, edit `cloud-config.txt`:
* Boot the VM
Replace the 2nd line with the contents of `~/.ssh/id_rsa.pub`:
The `cloud-config.txt` contains an ssh key to allow your local machine to access the VM. However, this must be updated with your local ssh key.
This command will update the key with your local public key value and start the VM.
```
ssh_authorized_keys:
- ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8Q/aUYUr3P1XKVucnO9mlWxOjJm+K01lHJR90MkHC9zbfTqlp8P7C3J26zKAuzHXOeF+VFxETRr6YedQKW9zp5oP7sN+F2gr/pO7GV3VmOqHMV7uKfyUQfq7H1aVzLfCcI7FwN2Zekv3yB7kj35pbsMa1Za58aF6oHRctZU6UWgXXbRxP+B04DoVU7jTstQ4GMoOCaqYhgPHyjEAS3DW0kkPW6HzsvJHkxvVcVlZ/wNJa1Ie/yGpzOzWIN0Ol0t2QT/RSWOhfzO1A2P0XbPuZ04NmriBonO9zR7T1fMNmmtTuK7WazKjQT3inmYRAqU6pe8wfX8WIWNV7OowUjUsv alex@alexr.local
```
```sh
sed "s/ssh-rsa.*/$(cat $HOME/.ssh/id_*.pub)/" cloud-config.txt | multipass launch --name faasd --cloud-init -
```
* Boot the VM
This can also be done manually, just replace the 2nd line of the `cloud-config.txt` with the coPntents of your public ssh key, usually either `~/.ssh/id_rsa.pub` or `~/.ssh/id_ed25519.pub`
```sh
multipass launch --cloud-init cloud-config.txt --name faasd
```
```
ssh_authorized_keys:
- ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8Q/aUYUr3P1XKVucnO9mlWxOjJm+K01lHJR90MkHC9zbfTqlp8P7C3J26zKAuzHXOeF+VFxETRr6YedQKW9zp5oP7sN+F2gr/pO7GV3VmOqHMV7uKfyUQfq7H1aVzLfCcI7FwN2Zekv3yB7kj35pbsMa1Za58aF6oHRctZU6UWgXXbRxP+B04DoVU7jTstQ4GMoOCaqYhgPHyjEAS3DW0kkPW6HzsvJHkxvVcVlZ/wNJa1Ie/yGpzOzWIN0Ol0t2QT/RSWOhfzO1A2P0XbPuZ04NmriBonO9zR7T1fMNmmtTuK7WazKjQT3inmYRAqU6pe8wfX8WIWNV7OowUjUsv alex@alexr.local
```
* Get the VM's IP and connect with `ssh`
```sh
multipass info faasd
Name: faasd
State: Running
IPv4: 192.168.64.14
Release: Ubuntu 18.04.3 LTS
Image hash: a720c34066dc (Ubuntu 18.04 LTS)
Load: 0.79 0.19 0.06
Disk usage: 1.1G out of 4.7G
Memory usage: 145.6M out of 985.7M
```
```sh
multipass info faasd
Name: faasd
State: Running
IPv4: 192.168.64.14
Release: Ubuntu 18.04.3 LTS
Image hash: a720c34066dc (Ubuntu 18.04 LTS)
Load: 0.79 0.19 0.06
Disk usage: 1.1G out of 4.7G
Memory usage: 145.6M out of 985.7M
```
Set the variable `IP`:
Set the variable `IP`:
```
export IP="192.168.64.14"
```
```
export IP="192.168.64.14"
```
You can also try to use `jq` to get the IP into a variable:
You can also try to use `jq` to get the IP into a variable:
```sh
export IP=$(multipass info faasd --format json| jq -r '.info.faasd.ipv4[0]')
```
```sh
export IP=$(multipass info faasd --format json| jq -r '.info.faasd.ipv4[0]')
```
Connect to the IP listed:
Connect to the IP listed:
```sh
ssh ubuntu@$IP
```
```sh
ssh ubuntu@$IP
```
Log out once you know it works.
Log out once you know it works.
* Let's capture the authentication password into a file for use with `faas-cli`
```
ssh ubuntu@$IP "sudo cat /var/lib/faasd/secrets/basic-auth-password" > basic-auth-password
```
```
ssh ubuntu@$IP "sudo cat /var/lib/faasd/secrets/basic-auth-password" > basic-auth-password
```
## Try faasd (OpenFaaS)
* Login from your laptop (the host)
```
export OPENFAAS_URL=http://$IP:8080
cat basic-auth-password | faas-cli login -s
```
```
export OPENFAAS_URL=http://$IP:8080
cat basic-auth-password | faas-cli login -s
```
* Deploy a function and invoke it
```
faas-cli store deploy figlet --env write_timeout=1s
echo "faasd" | faas-cli invoke figlet
```
faas-cli store deploy figlet --env write_timeout=1s
echo "faasd" | faas-cli invoke figlet
faas-cli describe figlet
faas-cli describe figlet
# Run async
curl -i -d "faasd-async" $OPENFAAS_URL/async-function/figlet
# Run async
curl -i -d "faasd-async" $OPENFAAS_URL/async-function/figlet
# Run async with a callback
# Run async with a callback
curl -i -d "faasd-async" -H "X-Callback-Url: http://some-request-bin.com/path" $OPENFAAS_URL/async-function/figlet
```
curl -i -d "faasd-async" -H "X-Callback-Url: http://some-request-bin.com/path" $OPENFAAS_URL/async-function/figlet
```
You can also checkout the other store functions: `faas-cli store list`
You can also checkout the other store functions: `faas-cli store list`
* Try the UI
Head over to the UI from your laptop and remember that your password is in the `basic-auth-password` file. The username is `admin`:
Head over to the UI from your laptop and remember that your password is in the `basic-auth-password` file. The username is `admin`:
```
echo http://$IP:8080
```
```
echo http://$IP:8080
```
* Stop/start the instance
```sh
multipass stop faasd
```
```sh
multipass stop faasd
```
* Delete, if you want to:
```
multipass delete --purge faasd
```
```
multipass delete --purge faasd
```
You now have a faasd appliance on your Mac. You can also use this cloud-init file with public cloud like AWS or DigitalOcean.

88
docs/PATCHES.md Normal file
View File

@ -0,0 +1,88 @@
## Instructions for testing a patch for faasd
### Launch a virtual machine
You can use any kind of Linux virtual machine, Ubuntu 20.04 is recommended.
Launch a cloud VM or use [Multipass](https://multipass.run), which is free to use an can be run locally. A Raspberry Pi 3 or 4 could also be used, but will need you to run `make dist` to cross compile a valid binary.
### Copy over your SSH key
Your SSH key will be used, so that you can copy a new faasd binary over to the host.
```bash
multipass launch \
--mem 4G \
-c 2 \
-n faasd
# Then access its shell
multipass shell faasd
# Edit .ssh/authorized_keys
# Add .ssh/id_rsa.pub from your host and save the file
```
### Install faasd on the VM
You start off with the upstream version of faasd on the host, then add the new version over the top later on.
```bash
cd /tmp/
git clone https://github.com/openfaas/faasd --depth=1
cd faasd/hack
./install.sh
# Run the login command given to you at the end of the script
```
Get the multipass IP address:
```bash
export IP=$(multipass info faasd --format json| jq -r '.info.faasd.ipv4[0]')
```
### Build a new faasd binary with the patch
Check out faasd on your local computer
```bash
git clone https://github.com/openfaas/faasd
cd faasd
gh pr checkout #PR_NUMBER_HERE
GOOS=linux go build
# You can also run "make dist" which is slower, but includes
# a version and binaries for other platforms such as the Raspberry Pi
```
### Copy it over to the VM
Now build a new faasd binary and copy it to the VM:
```bash
scp faasd ubuntu@$IP:~/
```
Now deploy the new version on the VM:
```bash
killall -9 faasd-linux; killall -9 faasd-linux ; mv ./faasd-linux /usr/local/bin/faasd
```
### Check it worked and test that patch
Now run a command with `faas-cli` such as:
* `faas-cli list`
* `faas-cli version`
See the testing instructions on the PR and run through those steps.
Post your results on GitHub to assist the creator of the pull request.
You can see how to get the logs for various components using the [eBook Serverless For Everyone Else](https://gumroad.com/l/serverless-for-everyone-else), or by consulting the [DEV.md](/docs/DEV.md) guide.

View File

@ -75,10 +75,10 @@ sudo systemctl restart faasd
Should have:
* [ ] Resolve core services from functions by populating/sharing `/etc/hosts` between `faasd` and `faasd-provider`
* [ ] Docs or examples on how to use the various connectors and connector-sdk
* [ ] Offer a recommendation or implement a strategy for faasd replication/HA
* [ ] Monitor and restart any of the core components at runtime if the container stops
* [ ] Asynchronous deletion instead of synchronous
* [ ] Asynchronous function deletion instead of synchronous
* [ ] Asynchronous function start-up instead of synchronous
Nice to Have:
@ -89,6 +89,8 @@ Nice to Have:
### Completed
* [x] Docs or examples on how to use the various event connectors (Yes in the eBook)
* [x] Resolve core services from functions by populating/sharing `/etc/hosts` between `faasd` and `faasd-provider`
* [x] Provide a cloud-init configuration for faasd bootstrap
* [x] Configure core services from a docker-compose.yaml file
* [x] Store and fetch logs from the journal

View File

@ -1,8 +1,11 @@
package pkg
const (
// FunctionNamespace is the default containerd namespace functions are created
FunctionNamespace = "openfaas-fn"
// DefaultFunctionNamespace is the default containerd namespace functions are created
DefaultFunctionNamespace = "openfaas-fn"
// NamespaceLabel indicates that a namespace is managed by faasd
NamespaceLabel = "openfaas"
// FaasdNamespace is the containerd namespace services are created
FaasdNamespace = "openfaas"

View File

@ -71,7 +71,7 @@ func buildCmd(ctx context.Context, req logs.Request) *exec.Cmd {
namespace := req.Namespace
if namespace == "" {
namespace = faasd.FunctionNamespace
namespace = faasd.DefaultFunctionNamespace
}
// find the description of the fields here

View File

@ -42,6 +42,18 @@ func MakeDeleteHandler(client *containerd.Client, cni gocni.CNI) func(w http.Res
lookupNamespace := getRequestNamespace(readNamespaceFromQuery(r))
// Check if namespace exists, and it has the openfaas label
valid, err := validNamespace(client.NamespaceService(), lookupNamespace)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
if !valid {
http.Error(w, "namespace not valid", http.StatusBadRequest)
return
}
name := req.FunctionName
function, err := GetFunction(client, name, lookupNamespace)

View File

@ -52,10 +52,25 @@ func MakeDeployHandler(client *containerd.Client, cni gocni.CNI, secretMountPath
}
namespace := getRequestNamespace(req.Namespace)
// Check if namespace exists, and it has the openfaas label
valid, err := validNamespace(client.NamespaceService(), namespace)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
if !valid {
http.Error(w, "namespace not valid", http.StatusBadRequest)
return
}
namespaceSecretMountPath := getNamespaceSecretMountPath(secretMountPath, namespace)
err = validateSecrets(namespaceSecretMountPath, req.Secrets)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
name := req.Service
@ -111,7 +126,7 @@ func deploy(ctx context.Context, req types.FunctionDeployment, client *container
}
envs := prepareEnv(req.EnvProcess, req.EnvVars)
mounts := getMounts()
mounts := getOSMounts()
for _, secret := range req.Secrets {
mounts = append(mounts, specs.Mount{
@ -126,7 +141,7 @@ func deploy(ctx context.Context, req types.FunctionDeployment, client *container
labels, err := buildLabels(&req)
if err != nil {
return fmt.Errorf("Unable to apply labels to conatiner: %s, error: %s", name, err)
return fmt.Errorf("unable to apply labels to container: %s, error: %w", name, err)
}
var memory *specs.LinuxMemory
@ -157,7 +172,7 @@ func deploy(ctx context.Context, req types.FunctionDeployment, client *container
)
if err != nil {
return fmt.Errorf("unable to create container: %s, error: %s", name, err)
return fmt.Errorf("unable to create container: %s, error: %w", name, err)
}
return createTask(ctx, client, container, cni)
@ -195,7 +210,7 @@ func createTask(ctx context.Context, client *containerd.Client, container contai
task, taskErr := container.NewTask(ctx, cio.BinaryIO("/usr/local/bin/faasd", nil))
if taskErr != nil {
return fmt.Errorf("unable to start task: %s, error: %s", name, taskErr)
return fmt.Errorf("unable to start task: %s, error: %w", name, taskErr)
}
log.Printf("Container ID: %s\tTask ID %s:\tTask PID: %d\t\n", name, task.ID(), task.Pid())
@ -247,7 +262,9 @@ func prepareEnv(envProcess string, reqEnvVars map[string]string) []string {
return envs
}
func getMounts() []specs.Mount {
// getOSMounts provides a mount for os-specific files such
// as the hosts file and resolv.conf
func getOSMounts() []specs.Mount {
// Prior to hosts_dir env-var, this value was set to
// os.Getwd()
hostsDir := "/var/lib/faasd"

View File

@ -53,7 +53,7 @@ func Test_BuildLabels_WithAnnotations(t *testing.T) {
}
if !reflect.DeepEqual(val, tc.result) {
t.Errorf("Got: %s, expected %s", val, tc.result)
t.Errorf("Want: %s, got: %s", val, tc.result)
}
})
}

View File

@ -2,6 +2,7 @@ package handlers
import (
"context"
"errors"
"fmt"
"log"
"strings"
@ -11,6 +12,7 @@ import (
"github.com/containerd/containerd"
"github.com/containerd/containerd/namespaces"
"github.com/openfaas/faasd/pkg"
faasd "github.com/openfaas/faasd/pkg"
"github.com/openfaas/faasd/pkg/cninetwork"
)
@ -27,11 +29,23 @@ type Function struct {
secrets []string
envVars map[string]string
envProcess string
memoryLimit int64
createdAt time.Time
}
// ListFunctions returns a map of all functions with running tasks on namespace
func ListFunctions(client *containerd.Client, namespace string) (map[string]*Function, error) {
// Check if namespace exists, and it has the openfaas label
valid, err := validNamespace(client.NamespaceService(), namespace)
if err != nil {
return nil, err
}
if !valid {
return nil, errors.New("namespace not valid")
}
ctx := namespaces.WithNamespace(context.Background(), namespace)
functions := make(map[string]*Function)
@ -60,7 +74,7 @@ func GetFunction(client *containerd.Client, name string, namespace string) (Func
c, err := client.LoadContainer(ctx, name)
if err != nil {
return Function{}, fmt.Errorf("unable to find function: %s, error %s", name, err)
return Function{}, fmt.Errorf("unable to find function: %s, error %w", name, err)
}
image, err := c.Image(ctx)
@ -72,19 +86,19 @@ func GetFunction(client *containerd.Client, name string, namespace string) (Func
allLabels, labelErr := c.Labels(ctx)
if labelErr != nil {
log.Printf("cannot list container %s labels: %s", containerName, labelErr.Error())
log.Printf("cannot list container %s labels: %s", containerName, labelErr)
}
labels, annotations := buildLabelsAndAnnotations(allLabels)
spec, err := c.Spec(ctx)
if err != nil {
return Function{}, fmt.Errorf("unable to load function spec for reading secrets: %s, error %s", name, err)
return Function{}, fmt.Errorf("unable to load function %s error: %w", name, err)
}
info, err := c.Info(ctx)
if err != nil {
return Function{}, fmt.Errorf("can't load info for: %s, error %s", name, err)
return Function{}, fmt.Errorf("can't load info for: %s, error %w", name, err)
}
envVars, envProcess := readEnvFromProcessEnv(spec.Process.Env)
@ -99,6 +113,7 @@ func GetFunction(client *containerd.Client, name string, namespace string) (Func
fn.envVars = envVars
fn.envProcess = envProcess
fn.createdAt = info.CreatedAt
fn.memoryLimit = readMemoryLimitFromSpec(spec)
replicas := 0
task, err := c.Task(ctx, nil)
@ -106,7 +121,7 @@ func GetFunction(client *containerd.Client, name string, namespace string) (Func
// Task for container exists
svc, err := task.Status(ctx)
if err != nil {
return Function{}, fmt.Errorf("unable to get task status for container: %s %s", name, err)
return Function{}, fmt.Errorf("unable to get task status for container: %s %w", name, err)
}
if svc.Status == "running" {
@ -187,7 +202,7 @@ func ListNamespaces(client *containerd.Client) []string {
namespaces, err := store.List(context.Background())
if err != nil {
log.Printf("Error listing namespaces: %s", err.Error())
set = append(set, faasd.FunctionNamespace)
set = append(set, faasd.DefaultFunctionNamespace)
return set
}
@ -198,12 +213,12 @@ func ListNamespaces(client *containerd.Client) []string {
continue
}
if _, found := labels["openfaas"]; found {
if _, found := labels[pkg.NamespaceLabel]; found {
set = append(set, namespace)
}
if !findNamespace(faasd.FunctionNamespace, set) {
set = append(set, faasd.FunctionNamespace)
if !findNamespace(faasd.DefaultFunctionNamespace, set) {
set = append(set, faasd.DefaultFunctionNamespace)
}
}
@ -218,3 +233,10 @@ func findNamespace(target string, items []string) bool {
}
return false
}
func readMemoryLimitFromSpec(spec *specs.Spec) int64 {
if spec.Linux == nil || spec.Linux.Resources == nil || spec.Linux.Resources.Memory == nil || spec.Linux.Resources.Memory.Limit == nil {
return 0
}
return *spec.Linux.Resources.Memory.Limit
}

View File

@ -32,54 +32,54 @@ func Test_BuildLabelsAndAnnotationsFromServiceSpec_Annotations(t *testing.T) {
}
func Test_SplitMountToSecrets(t *testing.T) {
type test struct {
Name string
Input []specs.Mount
Expected []string
type testCase struct {
Name string
Input []specs.Mount
Want []string
}
tests := []test{
{Name: "No matching openfaas secrets", Input: []specs.Mount{{Destination: "/foo/"}}, Expected: []string{}},
{Name: "Nil mounts", Input: nil, Expected: []string{}},
{Name: "No Mounts", Input: []specs.Mount{{Destination: "/foo/"}}, Expected: []string{}},
{Name: "One Mounts IS secret", Input: []specs.Mount{{Destination: "/var/openfaas/secrets/secret1"}}, Expected: []string{"secret1"}},
{Name: "Multiple Mounts 1 secret", Input: []specs.Mount{{Destination: "/var/openfaas/secrets/secret1"}, {Destination: "/some/other/path"}}, Expected: []string{"secret1"}},
{Name: "Multiple Mounts all secrets", Input: []specs.Mount{{Destination: "/var/openfaas/secrets/secret1"}, {Destination: "/var/openfaas/secrets/secret2"}}, Expected: []string{"secret1", "secret2"}},
tests := []testCase{
{Name: "No matching openfaas secrets", Input: []specs.Mount{{Destination: "/foo/"}}, Want: []string{}},
{Name: "Nil mounts", Input: nil, Want: []string{}},
{Name: "No Mounts", Input: []specs.Mount{{Destination: "/foo/"}}, Want: []string{}},
{Name: "One Mounts IS secret", Input: []specs.Mount{{Destination: "/var/openfaas/secrets/secret1"}}, Want: []string{"secret1"}},
{Name: "Multiple Mounts 1 secret", Input: []specs.Mount{{Destination: "/var/openfaas/secrets/secret1"}, {Destination: "/some/other/path"}}, Want: []string{"secret1"}},
{Name: "Multiple Mounts all secrets", Input: []specs.Mount{{Destination: "/var/openfaas/secrets/secret1"}, {Destination: "/var/openfaas/secrets/secret2"}}, Want: []string{"secret1", "secret2"}},
}
for _, tc := range tests {
t.Run(tc.Name, func(t *testing.T) {
got := readSecretsFromMounts(tc.Input)
if !reflect.DeepEqual(got, tc.Expected) {
t.Fatalf("expected %s, got %s", tc.Expected, got)
if !reflect.DeepEqual(got, tc.Want) {
t.Fatalf("Want %s, got %s", tc.Want, got)
}
})
}
}
func Test_ProcessEnvToEnvVars(t *testing.T) {
type test struct {
type testCase struct {
Name string
Input []string
Expected map[string]string
Want map[string]string
fprocess string
}
tests := []test{
{Name: "No matching EnvVars", Input: []string{"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "fprocess=python index.py"}, Expected: make(map[string]string), fprocess: "python index.py"},
{Name: "No EnvVars", Input: []string{}, Expected: make(map[string]string), fprocess: ""},
{Name: "One EnvVar", Input: []string{"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "fprocess=python index.py", "env=this"}, Expected: map[string]string{"env": "this"}, fprocess: "python index.py"},
{Name: "Multiple EnvVars", Input: []string{"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "this=that", "env=var", "fprocess=python index.py"}, Expected: map[string]string{"this": "that", "env": "var"}, fprocess: "python index.py"},
{Name: "Nil EnvVars", Input: nil, Expected: make(map[string]string)},
tests := []testCase{
{Name: "No matching EnvVars", Input: []string{"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "fprocess=python index.py"}, Want: make(map[string]string), fprocess: "python index.py"},
{Name: "No EnvVars", Input: []string{}, Want: make(map[string]string), fprocess: ""},
{Name: "One EnvVar", Input: []string{"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "fprocess=python index.py", "env=this"}, Want: map[string]string{"env": "this"}, fprocess: "python index.py"},
{Name: "Multiple EnvVars", Input: []string{"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "this=that", "env=var", "fprocess=python index.py"}, Want: map[string]string{"this": "that", "env": "var"}, fprocess: "python index.py"},
{Name: "Nil EnvVars", Input: nil, Want: make(map[string]string)},
}
for _, tc := range tests {
t.Run(tc.Name, func(t *testing.T) {
got, fprocess := readEnvFromProcessEnv(tc.Input)
if !reflect.DeepEqual(got, tc.Expected) {
t.Fatalf("expected: %s, got: %s", tc.Expected, got)
if !reflect.DeepEqual(got, tc.Want) {
t.Fatalf("Want: %s, got: %s", tc.Want, got)
}
if fprocess != tc.fprocess {
t.Fatalf("expected fprocess: %s, got: %s", tc.fprocess, got)
t.Fatalf("Want fprocess: %s, got: %s", tc.fprocess, got)
}
})
@ -87,22 +87,46 @@ func Test_ProcessEnvToEnvVars(t *testing.T) {
}
func Test_findNamespace(t *testing.T) {
type test struct {
type testCase struct {
Name string
foundNamespaces []string
namespace string
Expected bool
Want bool
}
tests := []test{
{Name: "Namespace Found", namespace: "fn", foundNamespaces: []string{"fn", "openfaas-fn"}, Expected: true},
{Name: "namespace Not Found", namespace: "fn", foundNamespaces: []string{"openfaas-fn"}, Expected: false},
tests := []testCase{
{Name: "Namespace Found", namespace: "fn", foundNamespaces: []string{"fn", "openfaas-fn"}, Want: true},
{Name: "namespace Not Found", namespace: "fn", foundNamespaces: []string{"openfaas-fn"}, Want: false},
}
for _, tc := range tests {
t.Run(tc.Name, func(t *testing.T) {
got := findNamespace(tc.namespace, tc.foundNamespaces)
if got != tc.Expected {
t.Fatalf("expected %t, got %t", tc.Expected, got)
if got != tc.Want {
t.Fatalf("Want %t, got %t", tc.Want, got)
}
})
}
}
func Test_readMemoryLimitFromSpec(t *testing.T) {
type testCase struct {
Name string
Spec *specs.Spec
Want int64
}
testLimit := int64(64)
tests := []testCase{
{Name: "specs.Linux not found", Spec: &specs.Spec{Linux: nil}, Want: int64(0)},
{Name: "specs.LinuxResource not found", Spec: &specs.Spec{Linux: &specs.Linux{Resources: nil}}, Want: int64(0)},
{Name: "specs.LinuxMemory not found", Spec: &specs.Spec{Linux: &specs.Linux{Resources: &specs.LinuxResources{Memory: nil}}}, Want: int64(0)},
{Name: "specs.LinuxMemory.Limit not found", Spec: &specs.Spec{Linux: &specs.Linux{Resources: &specs.LinuxResources{Memory: &specs.LinuxMemory{Limit: nil}}}}, Want: int64(0)},
{Name: "Memory limit set as Want", Spec: &specs.Spec{Linux: &specs.Linux{Resources: &specs.LinuxResources{Memory: &specs.LinuxMemory{Limit: &testLimit}}}}, Want: int64(64)},
}
for _, tc := range tests {
t.Run(tc.Name, func(t *testing.T) {
got := readMemoryLimitFromSpec(tc.Spec)
if got != tc.Want {
t.Fatalf("Want %d, got %d", tc.Want, got)
}
})
}

View File

@ -24,7 +24,7 @@ func (i *InvokeResolver) Resolve(functionName string) (url.URL, error) {
actualFunctionName := functionName
log.Printf("Resolve: %q\n", actualFunctionName)
namespace := getNamespace(functionName, faasd.FunctionNamespace)
namespace := getNamespace(functionName, faasd.DefaultFunctionNamespace)
if strings.Contains(functionName, ".") {
actualFunctionName = strings.TrimSuffix(functionName, "."+namespace)

View File

@ -5,6 +5,8 @@ import (
"log"
"net/http"
"k8s.io/apimachinery/pkg/api/resource"
"github.com/containerd/containerd"
"github.com/openfaas/faas-provider/types"
)
@ -14,6 +16,17 @@ func MakeReadHandler(client *containerd.Client) func(w http.ResponseWriter, r *h
return func(w http.ResponseWriter, r *http.Request) {
lookupNamespace := getRequestNamespace(readNamespaceFromQuery(r))
// Check if namespace exists, and it has the openfaas label
valid, err := validNamespace(client.NamespaceService(), lookupNamespace)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
if !valid {
http.Error(w, "namespace not valid", http.StatusBadRequest)
return
}
res := []types.FunctionStatus{}
fns, err := ListFunctions(client, lookupNamespace)
@ -26,7 +39,8 @@ func MakeReadHandler(client *containerd.Client) func(w http.ResponseWriter, r *h
for _, fn := range fns {
annotations := &fn.annotations
labels := &fn.labels
res = append(res, types.FunctionStatus{
memory := resource.NewQuantity(fn.memoryLimit, resource.BinarySI)
status := types.FunctionStatus{
Name: fn.name,
Image: fn.image,
Replicas: uint64(fn.replicas),
@ -37,7 +51,16 @@ func MakeReadHandler(client *containerd.Client) func(w http.ResponseWriter, r *h
EnvVars: fn.envVars,
EnvProcess: fn.envProcess,
CreatedAt: fn.createdAt,
})
}
// Do not remove below memory check for 0
// Memory limit should not be included in status until set explicitly
limit := &types.FunctionResources{Memory: memory.String()}
if limit.Memory != "0" {
status.Limits = limit
}
res = append(res, status)
}
body, _ := json.Marshal(res)

View File

@ -16,6 +16,18 @@ func MakeReplicaReaderHandler(client *containerd.Client) func(w http.ResponseWri
functionName := vars["name"]
lookupNamespace := getRequestNamespace(readNamespaceFromQuery(r))
// Check if namespace exists, and it has the openfaas label
valid, err := validNamespace(client.NamespaceService(), lookupNamespace)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
if !valid {
http.Error(w, "namespace not valid", http.StatusBadRequest)
return
}
if f, err := GetFunction(client, functionName, lookupNamespace); err == nil {
found := types.FunctionStatus{
Name: functionName,

View File

@ -41,6 +41,18 @@ func MakeReplicaUpdateHandler(client *containerd.Client, cni gocni.CNI) func(w h
namespace := getRequestNamespace(readNamespaceFromQuery(r))
// Check if namespace exists, and it has the openfaas label
valid, err := validNamespace(client.NamespaceService(), namespace)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
if !valid {
http.Error(w, "namespace not valid", http.StatusBadRequest)
return
}
name := req.ServiceName
if _, err := GetFunction(client, name, namespace); err != nil {

View File

@ -10,14 +10,14 @@ import (
"path"
"strings"
"github.com/containerd/containerd"
"github.com/openfaas/faas-provider/types"
provider "github.com/openfaas/faasd/pkg/provider"
)
const secretFilePermission = 0644
const secretDirPermission = 0755
func MakeSecretHandler(c *containerd.Client, mountPath string) func(w http.ResponseWriter, r *http.Request) {
func MakeSecretHandler(store provider.Labeller, mountPath string) func(w http.ResponseWriter, r *http.Request) {
err := os.MkdirAll(mountPath, secretFilePermission)
if err != nil {
@ -31,13 +31,13 @@ func MakeSecretHandler(c *containerd.Client, mountPath string) func(w http.Respo
switch r.Method {
case http.MethodGet:
listSecrets(c, w, r, mountPath)
listSecrets(store, w, r, mountPath)
case http.MethodPost:
createSecret(c, w, r, mountPath)
createSecret(w, r, mountPath)
case http.MethodPut:
createSecret(c, w, r, mountPath)
createSecret(w, r, mountPath)
case http.MethodDelete:
deleteSecret(c, w, r, mountPath)
deleteSecret(w, r, mountPath)
default:
w.WriteHeader(http.StatusBadRequest)
return
@ -46,13 +46,32 @@ func MakeSecretHandler(c *containerd.Client, mountPath string) func(w http.Respo
}
}
func listSecrets(c *containerd.Client, w http.ResponseWriter, r *http.Request, mountPath string) {
func listSecrets(store provider.Labeller, w http.ResponseWriter, r *http.Request, mountPath string) {
lookupNamespace := getRequestNamespace(readNamespaceFromQuery(r))
// Check if namespace exists, and it has the openfaas label
valid, err := validNamespace(store, lookupNamespace)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
if !valid {
http.Error(w, "namespace not valid", http.StatusBadRequest)
return
}
mountPath = getNamespaceSecretMountPath(mountPath, lookupNamespace)
files, err := ioutil.ReadDir(mountPath)
files, err := os.ReadDir(mountPath)
if os.IsNotExist(err) {
bytesOut, _ := json.Marshal([]types.Secret{})
w.Write(bytesOut)
return
}
if err != nil {
fmt.Printf("Error Occured: %s \n", err)
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
@ -66,7 +85,7 @@ func listSecrets(c *containerd.Client, w http.ResponseWriter, r *http.Request, m
w.Write(bytesOut)
}
func createSecret(c *containerd.Client, w http.ResponseWriter, r *http.Request, mountPath string) {
func createSecret(w http.ResponseWriter, r *http.Request, mountPath string) {
secret, err := parseSecret(r)
if err != nil {
log.Printf("[secret] error %s", err.Error())
@ -74,6 +93,14 @@ func createSecret(c *containerd.Client, w http.ResponseWriter, r *http.Request,
return
}
err = validateSecret(secret)
if err != nil {
log.Printf("[secret] error %s", err.Error())
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
log.Printf("[secret] is valid: %q", secret.Name)
namespace := getRequestNamespace(secret.Namespace)
mountPath = getNamespaceSecretMountPath(mountPath, namespace)
@ -84,7 +111,12 @@ func createSecret(c *containerd.Client, w http.ResponseWriter, r *http.Request,
return
}
err = ioutil.WriteFile(path.Join(mountPath, secret.Name), []byte(secret.Value), secretFilePermission)
data := secret.RawValue
if len(data) == 0 {
data = []byte(secret.Value)
}
err = ioutil.WriteFile(path.Join(mountPath, secret.Name), data, secretFilePermission)
if err != nil {
log.Printf("[secret] error %s", err.Error())
@ -93,7 +125,7 @@ func createSecret(c *containerd.Client, w http.ResponseWriter, r *http.Request,
}
}
func deleteSecret(c *containerd.Client, w http.ResponseWriter, r *http.Request, mountPath string) {
func deleteSecret(w http.ResponseWriter, r *http.Request, mountPath string) {
secret, err := parseSecret(r)
if err != nil {
log.Printf("[secret] error %s", err.Error())
@ -125,10 +157,6 @@ func parseSecret(r *http.Request) (types.Secret, error) {
return secret, err
}
if isTraversal(secret.Name) {
return secret, fmt.Errorf(traverseErrorSt)
}
return secret, err
}
@ -138,3 +166,13 @@ func isTraversal(name string) bool {
return strings.Contains(name, fmt.Sprintf("%s", string(os.PathSeparator))) ||
strings.Contains(name, "..")
}
func validateSecret(secret types.Secret) error {
if strings.TrimSpace(secret.Name) == "" {
return fmt.Errorf("non-empty name is required")
}
if isTraversal(secret.Name) {
return fmt.Errorf(traverseErrorSt)
}
return nil
}

View File

@ -1,63 +1,252 @@
package handlers
import (
"bytes"
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
"github.com/openfaas/faas-provider/types"
"github.com/openfaas/faasd/pkg"
provider "github.com/openfaas/faasd/pkg/provider"
)
func Test_parseSecretValidName(t *testing.T) {
func Test_parseSecret(t *testing.T) {
cases := []struct {
name string
payload string
expError string
expSecret types.Secret
}{
{
name: "no error when name is valid without extention and with no traversal",
payload: `{"name": "authorized_keys", "value": "foo"}`,
expSecret: types.Secret{Name: "authorized_keys", Value: "foo"},
},
{
name: "no error when name is valid and parses RawValue correctly",
payload: `{"name": "authorized_keys", "rawValue": "YmFy"}`,
expSecret: types.Secret{Name: "authorized_keys", RawValue: []byte("bar")},
},
{
name: "no error when name is valid with dot and with no traversal",
payload: `{"name": "authorized.keys", "value": "foo"}`,
expSecret: types.Secret{Name: "authorized.keys", Value: "foo"},
},
}
s := types.Secret{Name: "authorized_keys"}
body, _ := json.Marshal(s)
reader := bytes.NewReader(body)
r := httptest.NewRequest(http.MethodPost, "/", reader)
_, err := parseSecret(r)
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
reader := strings.NewReader(tc.payload)
r := httptest.NewRequest(http.MethodPost, "/", reader)
secret, err := parseSecret(r)
if err != nil && tc.expError == "" {
t.Fatalf("unexpected error: %s", err)
return
}
if tc.expError != "" {
if err == nil {
t.Fatalf("expected error: %s, got nil", tc.expError)
}
if err.Error() != tc.expError {
t.Fatalf("expected error: %s, got: %s", tc.expError, err)
}
return
}
if !reflect.DeepEqual(secret, tc.expSecret) {
t.Fatalf("expected secret: %+v, got: %+v", tc.expSecret, secret)
}
})
}
}
func TestSecretCreation(t *testing.T) {
mountPath, err := os.MkdirTemp("", "test_secret_creation")
if err != nil {
t.Fatalf("secret name is valid with no traversal characters")
t.Fatalf("unexpected error while creating temp directory: %s", err)
}
defer os.RemoveAll(mountPath)
handler := MakeSecretHandler(nil, mountPath)
cases := []struct {
name string
verb string
payload string
status int
secretPath string
secret string
err string
}{
{
name: "returns error when the name contains a traversal",
verb: http.MethodPost,
payload: `{"name": "/root/.ssh/authorized_keys", "value": "foo"}`,
status: http.StatusBadRequest,
err: "directory traversal found in name\n",
},
{
name: "returns error when the name contains a traversal",
verb: http.MethodPost,
payload: `{"name": "..", "value": "foo"}`,
status: http.StatusBadRequest,
err: "directory traversal found in name\n",
},
{
name: "empty request returns a validation error",
verb: http.MethodPost,
payload: `{}`,
status: http.StatusBadRequest,
err: "non-empty name is required\n",
},
{
name: "can create secret from string",
verb: http.MethodPost,
payload: `{"name": "foo", "value": "bar"}`,
status: http.StatusOK,
secretPath: "/openfaas-fn/foo",
secret: "bar",
},
{
name: "can create secret from raw value",
verb: http.MethodPost,
payload: `{"name": "foo", "rawValue": "YmFy"}`,
status: http.StatusOK,
secretPath: "/openfaas-fn/foo",
secret: "bar",
},
{
name: "can create secret in non-default namespace from raw value",
verb: http.MethodPost,
payload: `{"name": "pity", "rawValue": "dGhlIGZvbw==", "namespace": "a-team"}`,
status: http.StatusOK,
secretPath: "/a-team/pity",
secret: "the foo",
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
req := httptest.NewRequest(tc.verb, "http://example.com/foo", strings.NewReader(tc.payload))
w := httptest.NewRecorder()
handler(w, req)
resp := w.Result()
if resp.StatusCode != tc.status {
t.Logf("response body: %s", w.Body.String())
t.Fatalf("expected status: %d, got: %d", tc.status, resp.StatusCode)
}
if resp.StatusCode != http.StatusOK && w.Body.String() != tc.err {
t.Fatalf("expected error message: %q, got %q", tc.err, w.Body.String())
}
if tc.secretPath != "" {
data, err := os.ReadFile(filepath.Join(mountPath, tc.secretPath))
if err != nil {
t.Fatalf("can not read the secret from disk: %s", err)
}
if string(data) != tc.secret {
t.Fatalf("expected secret value: %s, got %s", tc.secret, string(data))
}
}
})
}
}
func Test_parseSecretValidNameWithDot(t *testing.T) {
s := types.Secret{Name: "authorized.keys"}
body, _ := json.Marshal(s)
reader := bytes.NewReader(body)
r := httptest.NewRequest(http.MethodPost, "/", reader)
_, err := parseSecret(r)
func TestListSecrets(t *testing.T) {
mountPath, err := os.MkdirTemp("", "test_secret_creation")
if err != nil {
t.Fatalf("secret name is valid with no traversal characters")
}
}
func Test_parseSecretWithTraversalWithSlash(t *testing.T) {
s := types.Secret{Name: "/root/.ssh/authorized_keys"}
body, _ := json.Marshal(s)
reader := bytes.NewReader(body)
r := httptest.NewRequest(http.MethodPost, "/", reader)
_, err := parseSecret(r)
if err == nil {
t.Fatalf("secret name should fail due to path traversal")
}
}
func Test_parseSecretWithTraversalWithDoubleDot(t *testing.T) {
s := types.Secret{Name: ".."}
body, _ := json.Marshal(s)
reader := bytes.NewReader(body)
r := httptest.NewRequest(http.MethodPost, "/", reader)
_, err := parseSecret(r)
if err == nil {
t.Fatalf("secret name should fail due to path traversal")
t.Fatalf("unexpected error while creating temp directory: %s", err)
}
defer os.RemoveAll(mountPath)
cases := []struct {
name string
verb string
namespace string
labels map[string]string
status int
secretPath string
secret string
err string
expected []types.Secret
}{
{
name: "Get empty secret list for default namespace having no secret",
verb: http.MethodGet,
status: http.StatusOK,
secretPath: "/test-fn/foo",
secret: "bar",
expected: make([]types.Secret, 0),
},
{
name: "Get empty secret list for non-default namespace having no secret",
verb: http.MethodGet,
status: http.StatusOK,
secretPath: "/test-fn/foo",
secret: "bar",
expected: make([]types.Secret, 0),
namespace: "other-ns",
labels: map[string]string{
pkg.NamespaceLabel: "true",
},
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
labelStore := provider.NewFakeLabeller(tc.labels)
handler := MakeSecretHandler(labelStore, mountPath)
path := "http://example.com/foo"
if len(tc.namespace) > 0 {
path = path + fmt.Sprintf("?namespace=%s", tc.namespace)
}
req := httptest.NewRequest(tc.verb, path, nil)
w := httptest.NewRecorder()
handler(w, req)
resp := w.Result()
if resp.StatusCode != tc.status {
t.Fatalf("want status: %d, but got: %d", tc.status, resp.StatusCode)
}
if resp.StatusCode != http.StatusOK && w.Body.String() != tc.err {
t.Fatalf("want error message: %q, but got %q", tc.err, w.Body.String())
}
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
t.Fatalf("can't read response of list %v", err)
}
var res []types.Secret
err = json.Unmarshal(body, &res)
if err != nil {
t.Fatalf("unable to unmarshal %q, error: %v", string(body), err)
}
if !reflect.DeepEqual(res, tc.expected) {
t.Fatalf("want response: %v, but got: %v", tc.expected, res)
}
})
}
}

View File

@ -41,6 +41,19 @@ func MakeUpdateHandler(client *containerd.Client, cni gocni.CNI, secretMountPath
}
name := req.Service
namespace := getRequestNamespace(req.Namespace)
// Check if namespace exists, and it has the openfaas label
valid, err := validNamespace(client.NamespaceService(), namespace)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
if !valid {
http.Error(w, "namespace not valid", http.StatusBadRequest)
return
}
namespaceSecretMountPath := getNamespaceSecretMountPath(secretMountPath, namespace)
function, err := GetFunction(client, name, namespace)

View File

@ -1,10 +1,13 @@
package handlers
import (
"context"
"net/http"
"path"
"github.com/openfaas/faasd/pkg"
faasd "github.com/openfaas/faasd/pkg"
provider "github.com/openfaas/faasd/pkg/provider"
)
func getRequestNamespace(namespace string) string {
@ -12,7 +15,7 @@ func getRequestNamespace(namespace string) string {
if len(namespace) > 0 {
return namespace
}
return faasd.FunctionNamespace
return faasd.DefaultFunctionNamespace
}
func readNamespaceFromQuery(r *http.Request) string {
@ -23,3 +26,22 @@ func readNamespaceFromQuery(r *http.Request) string {
func getNamespaceSecretMountPath(userSecretPath string, namespace string) string {
return path.Join(userSecretPath, namespace)
}
// validNamespace indicates whether the namespace is eligable to be
// used for OpenFaaS functions.
func validNamespace(store provider.Labeller, namespace string) (bool, error) {
if namespace == faasd.DefaultFunctionNamespace {
return true, nil
}
labels, err := store.Labels(context.Background(), namespace)
if err != nil {
return false, err
}
if value, found := labels[pkg.NamespaceLabel]; found && value == "true" {
return true, nil
}
return false, nil
}

View File

@ -15,7 +15,7 @@ func Test_getRequestNamespace(t *testing.T) {
requestNamespace string
expectedNamespace string
}{
{name: "RequestNamespace is not provided", requestNamespace: "", expectedNamespace: faasd.FunctionNamespace},
{name: "RequestNamespace is not provided", requestNamespace: "", expectedNamespace: faasd.DefaultFunctionNamespace},
{name: "RequestNamespace is provided", requestNamespace: "user-namespace", expectedNamespace: "user-namespace"},
}
@ -23,7 +23,7 @@ func Test_getRequestNamespace(t *testing.T) {
t.Run(tc.name, func(t *testing.T) {
actualNamespace := getRequestNamespace(tc.requestNamespace)
if actualNamespace != tc.expectedNamespace {
t.Errorf("Got: %s, expected %s", actualNamespace, tc.expectedNamespace)
t.Errorf("Want: %s, got: %s", actualNamespace, tc.expectedNamespace)
}
})
}
@ -36,7 +36,7 @@ func Test_getNamespaceSecretMountPath(t *testing.T) {
requestNamespace string
expectedSecretPath string
}{
{name: "Default Namespace is provided", requestNamespace: faasd.FunctionNamespace, expectedSecretPath: "/var/openfaas/secrets/" + faasd.FunctionNamespace},
{name: "Default Namespace is provided", requestNamespace: faasd.DefaultFunctionNamespace, expectedSecretPath: "/var/openfaas/secrets/" + faasd.DefaultFunctionNamespace},
{name: "User Namespace is provided", requestNamespace: "user-namespace", expectedSecretPath: "/var/openfaas/secrets/user-namespace"},
}
@ -44,7 +44,7 @@ func Test_getNamespaceSecretMountPath(t *testing.T) {
t.Run(tc.name, func(t *testing.T) {
actualNamespace := getNamespaceSecretMountPath(userSecretPath, tc.requestNamespace)
if actualNamespace != tc.expectedSecretPath {
t.Errorf("Got: %s, expected %s", actualNamespace, tc.expectedSecretPath)
t.Errorf("Want: %s, got: %s", actualNamespace, tc.expectedSecretPath)
}
})
}
@ -68,7 +68,7 @@ func Test_readNamespaceFromQuery(t *testing.T) {
actualNamespace := readNamespaceFromQuery(r)
if actualNamespace != tc.expectedNamespace {
t.Errorf("Got: %s, expected %s", actualNamespace, tc.expectedNamespace)
t.Errorf("Want: %s, got: %s", actualNamespace, tc.expectedNamespace)
}
})
}

25
pkg/provider/labeller.go Normal file
View File

@ -0,0 +1,25 @@
package provider
import "context"
// Labeller can return labels for a namespace from containerd.
type Labeller interface {
Labels(ctx context.Context, namespace string) (map[string]string, error)
}
//
// FakeLabeller can be used to fake labels applied on namespace to mark
// them valid/invalid for openfaas functions
type FakeLabeller struct {
labels map[string]string
}
func NewFakeLabeller(labels map[string]string) Labeller {
return &FakeLabeller{
labels: labels,
}
}
func (s *FakeLabeller) Labels(ctx context.Context, namespace string) (map[string]string, error) {
return s.labels, nil
}

View File

@ -33,7 +33,7 @@ func Remove(ctx context.Context, client *containerd.Client, name string) error {
if errdefs.IsNotFound(err) {
taskFound = false
} else {
return fmt.Errorf("unable to get task %s: ", err)
return fmt.Errorf("unable to get task %w: ", err)
}
}
@ -47,12 +47,12 @@ func Remove(ctx context.Context, client *containerd.Client, name string) error {
log.Printf("Need to kill task: %s\n", name)
if err = killTask(ctx, t); err != nil {
return fmt.Errorf("error killing task %s, %s, %s", container.ID(), name, err)
return fmt.Errorf("error killing task %s, %s, %w", container.ID(), name, err)
}
}
if err := container.Delete(ctx, containerd.WithSnapshotCleanup); err != nil {
return fmt.Errorf("error deleting container %s, %s, %s", container.ID(), name, err)
return fmt.Errorf("error deleting container %s, %s, %w", container.ID(), name, err)
}
} else {
@ -79,9 +79,10 @@ func killTask(ctx context.Context, task containerd.Task) error {
if task != nil {
wait, err := task.Wait(ctx)
if err != nil {
err = fmt.Errorf("error waiting on task: %s", err)
log.Printf("error waiting on task: %s", err)
return
}
if err := task.Kill(ctx, unix.SIGTERM, containerd.WithKillAll); err != nil {
log.Printf("error killing container task: %s", err)
}

View File

@ -8,6 +8,8 @@ import (
"os"
"path"
"sort"
"strconv"
"strings"
"github.com/alexellis/k3sup/pkg/env"
"github.com/compose-spec/compose-go/loader"
@ -26,7 +28,8 @@ import (
)
const (
workingDirectoryPermission = 0644
// workingDirectoryPermission user read/write/execute, group and others: read-only
workingDirectoryPermission = 0744
)
type Service struct {
@ -145,6 +148,28 @@ func (s *Supervisor) Start(svcs []Service) error {
Type: "bind",
Options: []string{"rbind", "rw"},
})
// Only create directories, not files.
// Some files don't have a suffix, such as secrets.
if len(path.Ext(mnt.Src)) == 0 &&
!strings.HasPrefix(mnt.Src, "/var/lib/faasd/secrets/") {
// src is already prefixed with wd from an earlier step
src := mnt.Src
fmt.Printf("Creating local directory: %s\n", src)
if err := os.MkdirAll(src, workingDirectoryPermission); err != nil {
if !errors.Is(os.ErrExist, err) {
fmt.Printf("Unable to create: %s, %s\n", src, err)
}
}
if len(svc.User) > 0 {
uid, err := strconv.Atoi(svc.User)
if err == nil {
if err := os.Chown(src, uid, -1); err != nil {
fmt.Printf("Unable to chown: %s to %d, error: %s\n", src, uid, err)
}
}
}
}
}
}