During some exploratory testing, I ran into an issue where
the gateway would attempt to scale a deployment from zero
replicas to min, despite there already being min replicas.
Why?
The scaling logic was looking for Available replicas when
it should have looked for Desired replicas. So when a
deployment had zero ready replicas due to readiness checks
failing, the gateway was attempting to scale from zero
to min.
This logic has been corrected and separated from the
a holding pattern where the gateway waits for a ready
replica.
Tested with KinD and an edited function which had a
readiness probe, which was failing and no ready
replicas. As desired, the gateway did not scale to min.
However, when setting desired replicas to zero, the
gateway did scale up as expected.
This change also modifies all print statements for
"seconds" and makes them use 4 decimal places instead of
the default which was a longer, more verbose string for
the logs.
Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
Introduces a single-flight call to a function's health
endpoint to verify that it is registered with an Istio
sidecar (Envoy) before letting the invocation through.
Results are cached for 5 seconds, before a probe is
required again.
Tested without Istio, with probe_functions environment
variable set to true, I saw a probe execute in the logs.
Fixes: #1721 for Istio users.
Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alex@openfaas.com>
The histogram for gateway_functions_seconds excluded the status
code that gives important information for setting up SLOs.
Fixes: #1725
Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alex@openfaas.com>
* Add service target metric
* Add service min replicas metric
* Add scale type metric
These combined allow new auto-scaling modes and parameters
for OpenFaaS Pro customers.
Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
Polls every 100 ms instead of every 50ms for use with a new
faas-netes PR:
https://github.com/openfaas/faas-netes/pull/726
The call is much faster now, so this request should me made
at a lower frequency.
Also adds error handling for URL building in the external
service query code.
Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
This field doesn't appear to be used and is supplied via the
querystring which faas-netes (the only ns-enabled provider
already consumes)
Related to PR:
https://github.com/openfaas/faas-netes/pull/671
Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
Enables publishing to various topics according to annotations
on the functions. The function cache is moved up one level so
that it can be shared between the scale from zero code and the
queue proxy.
Unit tests added for new internal methods.
Tested e2e with arkade and the newest queue-worker and RC
gateway image with two queues and an annotation on one of the
functions of com.openfaas.queue. It worked as expected including
with multiple namespace support.
Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
Allows alerts to trigger functions to scale when they
also have an optional namespace set.
Tested e2e with Kubernetes 1.15 and a non-default namespace.
Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
The function deployment and status structs have been moved away
into the faas-provider package.
Tested with a build, running tests, and CI.
Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
- max_conns / idle / per host are now read from env-vars and have
defaults set to 1024 for both values
- logging / metrics are collected in the client transaction
rather than via defer (this may impact throughput)
- function cache moved to use RWMutex to try to improve latency
around locking when updating cache
- logging message added to show latency in running GetReplicas
because this was observed to increase in a linear fashion under
high concurrency
- changes tested against 3-node bare-metal 1.13 K8s cluster
with kubeadm
Signed-off-by: Alex Ellis (VMware) <alexellis2@gmail.com>
- extracting this package means it can be used in other components
such as the asynchronous nats-queue-worker which may need to
invoke functions which are scaled down to zero replicas.
Ref: https://github.com/openfaas/nats-queue-worker/issues/32
Tested on Docker Swarm for scaling up, already scaled and not
found error.
Signed-off-by: Alex Ellis (VMware) <alexellis2@gmail.com>
- updates comments and adds where missing
- updates locks so that unlock is done via defer instead of
at the end of the statement
- extracts timeout variable in two places
- remove makeClient() unused method from metrics package
No-harm changes tested via go build.
Signed-off-by: Alex Ellis (VMware) <alexellis2@gmail.com>
Within MakeScalingHandler() there is a call to GetReplicas() which was not returning an error when a non-200 http response was received from /system/function/. The call would also return a populated struct, so the perception was that a function existed an had been scaled to zero. This meant that the function would be added to the function cache and the code would continue into SetReplicas() where an attempt would be made to scale up a non-existent function.
This change amends GetReplicas() so that it will return an error if the gateway returns anything other than a 200 reponse code from the /system/function/ endpoint. This causes MakeScalingHandler() to return earlier with an error indicating that the function could not be found. The cache.Set call is also moved to after the error check so that the cache is only updated to include existent functions.
During investigations as to the cause of #876 tests were added to function_cache to check that Get() is behaving as intended when function exists and when not. Tests are also added to plugin/external to test that GetReplicas() and SetReplicas() are following their intended modes of operation when 200 and non-200 responses are received from the gateway.
Signed-off-by: Richard Gee <richard@technologee.co.uk>
Updated gateway to accept 202 as valid response code for
/system/scale-function/<function> along with 200.
Fixes: #faas-netes/245
Signed-off-by: Vivek Singh <vivekkmr45@yahoo.in>
Previous PR from Simon or Ken broke build due to missing gofmt
in the PR. This PR applies it to resolve the build issue.
Signed-off-by: Alex Ellis (VMware) <alexellis2@gmail.com>
Based on the received feedback I updated the documentation of the function. Also replaced variable temp by an more declaritive variable.
Signed-off-by: Simon Pelczer <templum.dev@gmail.com>
Further I created some unit test which should cover all relevant scenarios for the created function.
Signed-off-by: Simon Pelczer <templum.dev@gmail.com>
Introduced an new label to set the scaling factor that is used to calculate th proportions, setting it to 0 also allows to disable scaling.
Updated the tests to reflect the changes and added a new test which shows that setting the scaling factor to 0 indeed does disable scaling.
Ensured that the scaling factor is always between [0 and 100].
Signed-off-by: Simon Pelczer <templum.dev@gmail.com>
Note, not all `alexellis/github` references should be changed, there are
a number of repos which are not part of the openfaas org, this commit
excludes those.
Signed-off-by: John McCabe <john@johnmccabe.net>
