Changes to canary tutorial (#664)

* Changes to canary tutorial

* Updated mkdocs

* image link

* conformance testing is yet to come

Author: sriumcp

.github/workflows/knative-e2e-tests.yaml

@@ -62,6 +62,53 @@ jobs:
         export ITER8=$(pwd)
         source $ITER8/samples/knative/quickstart/check.sh
 
+  canary:
+    name: canary experiment
+    runs-on: ubuntu-latest
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v2
+    
+    - uses: engineerd/[email protected]
+    - name: platform setup
+      run: |
+        export ITER8=$(pwd)
+        $ITER8/samples/knative/quickstart/platformsetup.sh contour
+
+    - name: create Knative service and readiness check
+      run: |
+        export ITER8=$(pwd)
+        kubectl apply -f $ITER8/samples/knative/quickstart/baseline.yaml
+        kubectl apply -f $ITER8/samples/knative/quickstart/experimentalservice.yaml
+        kubectl wait --for=condition=Ready ksvc/sample-app --timeout=240s
+
+    - name: create Fortio job and create experiment
+      run: |
+        export ITER8=$(pwd)
+        sed -i "s/6000s/120s/g" $ITER8/samples/knative/quickstart/fortio.yaml
+        URL_VALUE=$(kubectl get ksvc sample-app -o json | jq .status.address.url)
+        sed "s+URL_VALUE+${URL_VALUE}+g" $ITER8/samples/knative/quickstart/fortio.yaml | kubectl apply -f -
+        
+        # Wait for Fortio to be ready
+        pod_name=$(kubectl get pods --selector=job-name=fortio -o jsonpath='{.items[*].metadata.name}')
+        kubectl wait --for=condition=Ready pods/"$pod_name" --timeout=240s
+
+        # Define metrics
+        kubectl apply -f $ITER8/samples/knative/quickstart/metrics.yaml
+
+        # Starting the experiment
+        kubectl apply -f $ITER8/samples/knative/canary/experiment.yaml
+
+    - name: Sleep until end of experiment
+      run: |
+        sleep 150.0
+        kubectl get experiment canary-exp -o yaml
+    
+    - name: Check if experiment is complete and successful
+      run: |
+        export ITER8=$(pwd)
+        source $ITER8/samples/knative/canary/check.sh
+
   canary-progressive-helm:
     name: progressive canary experiment using helm
     runs-on: ubuntu-latest

mkdocs/docs/getting-started/quick-start.md

@@ -20,6 +20,8 @@ template: main.html
     1. The [kubectl CLI](https://kubernetes.io/docs/tasks/tools/install-kubectl/).
     2. [Go 1.13+](https://golang.org/doc/install).
 
+    > **Note:** Please choose the same K8s stack (for example, Istio, KFServing, or Knative) consistently throughout this tutorial. If you wish to switch K8s stacks between tutorials, start from a clean K8s cluster (step 1 below), so that your cluster is correctly setup.
+
 ## 1. Create Kubernetes cluster
 
 Create a local cluster using Kind or Minikube as follows, or use a managed Kubernetes cluster. Ensure that the cluster has sufficient resources, for example, 8 CPUs and 12GB of memory.
@@ -52,8 +54,6 @@ export ITER8=$(pwd)
 ## 3. Install K8s stack and Iter8
 Choose the K8s stack over which you are performing the A/B testing experiment.
 
-> **Note:** Please choose the same K8s stack consistently throughout the tutorial. If you finished running the tutorial for a specific stack, and wish to try it on another stack, start from Step 1 (i.e., a clean K8s cluster), so that your cluster is correctly setup.
-
 === "Istio"
     Setup Istio, Iter8, a mock New Relic service, and Prometheus add-on within your cluster.
 
@@ -1183,7 +1183,7 @@ When the experiment completes, you will see the experiment stage change from `Ru
 ???+ info "Understanding what happened"
     1. You created two versions of your app/ML model.
     2. You generated requests for your app/ML model versions. At the start of the experiment, 100% of the requests are sent to the baseline and 0% to the candidate.
-    3. You created an Iter8 experiment with A/B testing pattern and progressive deployment pattern. In each iteration, Iter8 observed the latency and error-rate metrics collected by Prometheus, and the user-engagement metric from New Relic, verified that the candidate satisfied all objectives, verified that the candidate improved over the baseline in terms of user-engagement, identified candidate as the winner, progressively shifted traffic from the baseline to the candidate, and promoted the candidate.
+    3. You created an Iter8 experiment with A/B testing pattern and progressive deployment pattern. In each iteration, Iter8 observed the latency and error-rate metrics collected by Prometheus, and the user-engagement metric from New Relic; Iter8 verified that the candidate satisfied all objectives, verified that the candidate improved over the baseline in terms of user-engagement, identified candidate as the winner, progressively shifted traffic from the baseline to the candidate, and promoted the candidate.
 
 ## 9. Cleanup
 === "Istio"

mkdocs/docs/images/canary-exp.png

@@ -0,0 +1,7 @@
+---
+template: main.html
+---
+
+# A/B Testing
+
+A/B testing pattern is documented as part of [quick start](../../../../getting-started/quick-start/).