DistOS 2021F Experience 2 - Revision history

Housedhorse: Clarify question 3

2021-11-29T17:08:48Z

Clarify question 3

← Older revision		Revision as of 17:08, 29 November 2021
Line 132:		Line 132:
	<li><p>Printerfacts is now a CRUD app that supports creating, reading, updating, and deleting facts from the Cassandra database. In particular, we now support the following endpoints:</p>		<li><p>Printerfacts is now a CRUD app that supports creating, reading, updating, and deleting facts from the Cassandra database. In particular, we now support the following endpoints:</p>
	<ul>		<ul>
	<li><code>~~GET~~</code> ~~/fact~~: Get a random fact from the database as a JSON object</li>		<li>GET <code>10.96.0.201/fact</code>: Get a random fact from the database as a JSON object</li>
	<li~~><code~~>GET</code> /fact/keys: Get a list of all fact keys in the database as a JSON object</li>		<li>GET <code>10.96.0.201/fact/keys</code>: Get a list of all fact keys in the database as a JSON object</li>
	<li~~><code~~>GET</code> /fact/<key>: Get the fact with the key <key> in the database as a JSON object</li>		<li>GET <code>10.96.0.201/fact/<key></code>: Get the fact with the key <key> in the database as a JSON object</li>
	<li><code>~~POST~~</code> ~~/fact~~: Create a new fact where the request body is a JSON object of the form <code>{ "fact": "Fact Here", "kind": "Kind of fact (e.g. Cat fact)" }</code></li>		<li>POST <code>10.96.0.201/fact</code>: Create a new fact where the request body is a JSON object of the form <code>{ "fact": "Fact Here", "kind": "Kind of fact (e.g. Cat fact)" }</code></li>
	<li~~><code~~>PUT</code> /fact/<key>: Modify the fact with key <key> in the database where the request body is a JSON object of the form <code>{ "fact": "Fact Here", "kind": "Kind of fact (e.g. Cat fact)" }</code></li>		<li>PUT <code>10.96.0.201/fact/<key></code>: Modify the fact with key <key> in the database where the request body is a JSON object of the form <code>{ "fact": "Fact Here", "kind": "Kind of fact (e.g. Cat fact)" }</code></li>
	<li~~><code~~>DELETE</code> /fact/<key>: Delete the fact with key <key> from the database</li></ul>		<li>DELETE <code>10.96.0.201/fact/<key></code>: Delete the fact with key <key> from the database</li></ul>

	<p>Try out each of the printerfacts endpoints. Note that you can specify the HTTP request type using <code>curl -X <type></code> (for example, <code>curl -X POST</code> to send a POST request). You can send a JSON body as a payload by using <code>curl -H 'Content-type: application/json' -d '{"key": "value"}'</code> where you replace the <code>{"key": "value"}</code> part with your JSON object.</p>		<p>Try out each of the printerfacts endpoints. Note that you can specify the HTTP request type using <code>curl -X <type></code> (for example, <code>curl -X POST</code> to send a POST request). You can send a JSON body as a payload by using <code>curl -H 'Content-type: application/json' -d '{"key": "value"}'</code> where you replace the <code>{"key": "value"}</code> part with your JSON object.</p>

Housedhorse: Add optional part to question 3

2021-11-27T16:42:59Z

Add optional part to question 3

← Older revision		Revision as of 16:42, 27 November 2021
Line 139:		Line 139:
	<li><code>DELETE</code> /fact/<key>: Delete the fact with key <key> from the database</li></ul>		<li><code>DELETE</code> /fact/<key>: Delete the fact with key <key> from the database</li></ul>

	<p>Try out each of the printerfacts endpoints. Note that you can specify the HTTP request type using <code>curl -X <type></code> (for example, <code>curl -X POST</code> to send a POST request). You can send a JSON body as a payload by using <code>curl -H 'Content-type: application/json' -d '{"key": "value"}'</code> where you replace the <code>{"key": "value"}</code> part with your JSON object.</p></li>		<p>Try out each of the printerfacts endpoints. Note that you can specify the HTTP request type using <code>curl -X <type></code> (for example, <code>curl -X POST</code> to send a POST request). You can send a JSON body as a payload by using <code>curl -H 'Content-type: application/json' -d '{"key": "value"}'</code> where you replace the <code>{"key": "value"}</code> part with your JSON object.</p>

			<p>Optional: Try restarting the Cassandra pods using <code>kubectl rollout restart statefulset cassandra</code>. While the pods are restarting, watch the output of <code>kubectl get pods --watch</code> and try making requests to the printerfacts service at the same time. Do you notice any downtime?
			</p></li>
	<li><p>Let’s use Cilium and Hubble to observe dataflows between our Cassandra database and the printerfacts service. To do this, run the following Hubble command: <code>hubble observe --label component=server --label component=printerfacts -f</code>. Make a few requests to the printerfacts service, exercising various API endpoints. Try and make sense of some of the network traffic you observe.</p></li>		<li><p>Let’s use Cilium and Hubble to observe dataflows between our Cassandra database and the printerfacts service. To do this, run the following Hubble command: <code>hubble observe --label component=server --label component=printerfacts -f</code>. Make a few requests to the printerfacts service, exercising various API endpoints. Try and make sense of some of the network traffic you observe.</p></li>
	<li><p>In an effort to secure your cluster, you decide to employ a Cilium network policy that locks down the printerfacts service. First, download the example security policy by running <code>wget https://homeostasis.scs.carleton.ca/~will/printerfacts-policy.yml</code>, then run <code>kubectl apply -f policy.yml</code> to apply it. The example policy allows GET requests to <code>/</code>, <code>/fact</code>, <code>/fact/keys</code>, and <code>/fact/<key></code>. Examine the policy file and make sure you understand how it works.</p>		<li><p>In an effort to secure your cluster, you decide to employ a Cilium network policy that locks down the printerfacts service. First, download the example security policy by running <code>wget https://homeostasis.scs.carleton.ca/~will/printerfacts-policy.yml</code>, then run <code>kubectl apply -f policy.yml</code> to apply it. The example policy allows GET requests to <code>/</code>, <code>/fact</code>, <code>/fact/keys</code>, and <code>/fact/<key></code>. Examine the policy file and make sure you understand how it works.</p>

Housedhorse: /* Part 1: Multi-Node Kubernetes, Cilium, and Cassandra (Easy) */

2021-11-27T01:14:54Z

Part 1: Multi-Node Kubernetes, Cilium, and Cassandra (Easy)

← Older revision		Revision as of 01:14, 27 November 2021
Line 119:		Line 119:

	<ol style="list-style-type: decimal;">		<ol style="list-style-type: decimal;">
	<li><p>Thanks the Cilium CNI, your Kubernetes ~~clusters~~ has been outfitted with new eBPF superpowers. In particular, Cilium installs a series of eBPF programs into your (VM’s) kernel which can be used to monitor traffic between containers, pods, and nodes, as well as enforce L4–L7 security policy. To test out your new superpowers, run <code>cilium hubble port-forward&</code> followed by <code>hubble observe</code>. What is all that output? Can you make sense of any of it?</p>		<li><p>Thanks the Cilium CNI, your Kubernetes cluster has been outfitted with new eBPF superpowers. In particular, Cilium installs a series of eBPF programs into your (VM’s) kernel which can be used to monitor traffic between containers, pods, and nodes, as well as enforce L4–L7 security policy. To test out your new superpowers, run <code>cilium hubble port-forward&</code> followed by <code>hubble observe</code>. What is all that output? Can you make sense of any of it?</p>
	<ul>		<ul>
	<li>Hint 1: Try checking <code>hubble --help</code> and <code>hubble observe --help</code></li>		<li>Hint 1: Try checking <code>hubble --help</code> and <code>hubble observe --help</code></li>

Housedhorse: Touch experience 2

2021-11-27T01:08:30Z

Touch experience 2

New page

= Introduction =

In this experience, you will be playing with a multi-node Kubernetes cluster simulated using <code>minikube</code>, running the [https://docs.cilium.io/en/v1.10/ Cilium CNI], a replicated [https://cassandra.apache.org/doc/latest/ Apache Cassandra] database and an updated version of our beloved “printerfacts” service that consumes facts stored in the Cassandra database. As usual, you may wish to consult the relevant documentation if you get stuck. Links to documentation will be provided along with hints later on in this document.

With the exception of Part 2, completing this experience shouldn’t take more than a couple of hours. Feel free to collaborate with other students if you get stuck. However, '''you must acknowledge any collaboration'''. Additionally, '''copying and pasting or simply “changing up” each other’s answers will be treated as an academic integrity violation'''.

== Submissions ==

Your experience report should be submitted as a PDF file on Brightspace, written in '''paragraph form'''. Code snippets or screenshots are allowed to augment your prose but are not required. We will post a submission link on Brightspace within the next few days.

The official due date for the experience report is December 10th (the last day of class), but we will continue to accept submissions up until the date of the final exam if you need extra time.

== Receiving Your Grade ==

This experience is broken up into three parts:

# A series of easy tasks designed to get you familiar Cilium, Cassandra, and a multi-node K8s cluster.
# A harder challenge that involves programming a client to interact with the Cassandra database and writing a custom Cilium security policy to lock down your cluster.
# An opportunity to reflect on this experience and make connections to overall themes in the course.

Students are expected to complete Part 1 and Part 3 to get a grade of at most a B+. '''Part 2 is optional''', but must be completed to receive a grade of A- or higher. Marks will be deducted for insufficient explanations or answers that demonstrate a lack of effort. By this logic, you should have a fairly clear idea of what grade you will receive when you submit this experience, based on how much effort you put in.

== Setting Up Your Environment ==

For this experience, a new VM image has been provided for you. It has a larger disk size, more RAM, and a batteries-included installation of the [https://github.com/cilium/cilium-cli Cilium] and [https://github.com/cilium/hubble Hubble] CLI. First, '''delete your old instance''' using the OpenStack web console, then create a new one. You can follow the [https://homeostasis.scs.carleton.ca/wiki/index.php/DistOS_2021F:_Using_Openstack same instructions as before], except replace the <code>COMP4000-studentvm-v1</code> image with <code>COMP4000-studentvm-v2</code>. When selecting your flavour, make sure you pick the flavour with '''16GiB of disk space and 8GB of RAM'''.

Once your VM is set up, SSH into it using your preferred SSH client. You should probably be working with at least 2 terminals for the rest of this experience. After SSHing into your instance, run the following command to spin up a multi-node k8s cluster: <code>minikube start -n3 --cni=cilium</code>. This might take a few minutes to run to completion.

Verify your cluster has started correctly using <code>minikube status</code>. You should see something like the following:

<pre>minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

minikube-m02
type: Worker
host: Running
kubelet: Running

minikube-m03
type: Worker
host: Running
kubelet: Running</pre>
=== Cilium and Hubble ===

Once your cluster is running, you can configure the cluster to use Cilium and Hubble. Due to some minikube and Cilium quirks, we need to uninstall the default version of Cilium installed by minikube and reinstall it using the Cilium CLI. You can do so with the following commands:

<pre>cilium uninstall && cilium install && cilium hubble enable</pre>
This may take a few minutes to run to completion.

You can run the command <code>cilium status</code> to check the status of your Cilium installation. Once everything has installed correctly, you should see something like the following when running <code>cilium status</code>:

<pre>student@alpine:~$ cilium status
/¯¯\
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
/¯¯\__/¯¯\ Hubble: OK
\__/¯¯\__/ ClusterMesh: disabled
\__/

Deployment cilium-operator Desired: 1, Ready: 1/1, Available: 1/1
Deployment hubble-relay Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet cilium Desired: 3, Ready: 3/3, Available: 3/3
Containers: cilium Running: 3
cilium-operator Running: 1
hubble-relay Running: 1
Cluster Pods: 2/2 managed by Cilium
Image versions cilium quay.io/cilium/cilium:v1.10.5: 3
cilium-operator quay.io/cilium/operator-generic:v1.10.5: 1
hubble-relay quay.io/cilium/hubble-relay:v1.10.5: 1</pre>
=== Cassandra ===

We will be using the <code>helm</code> package manger to install Cassandra in our k8s cluster. To do so, run the following commands:

<pre>wget https://homeostasis.scs.carleton.ca/~will/cassandra.yml
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install cassandra bitnami/cassandra -f cassandra.yml</pre>
You can verify the status of your Cassandra installation using <code>kubectl get pods --watch</code>. Wait until both <code>cassandra-0</code> and <code>cassandra-1</code> show <code>1/1</code> under <code>READY</code>. This might take a few minutes. The output should look something like this:

<pre>student@alpine:~$ kubectl get pods
NAME READY STATUS RESTARTS AGE
cassandra-0 1/1 Running 0 4m2s
cassandra-1 1/1 Running 0 2m8s</pre>
=== Printerfacts ===

The printerfacts service from last experience has received an upgrade to work with our new Cassandra database. First, run the migrations (provided as a batch job) to initialize the Cassandra database and populate it with some facts about printers:

<pre>wget https://homeostasis.scs.carleton.ca/~will/migrations.yml
kubectl apply -f migrations.yml</pre>
After applying the migrations, you’re ready to deploy the printerfacts service:

<pre>wget https://homeostasis.scs.carleton.ca/~will/deploy.yml
kubectl apply -f deploy.yml</pre>
Verify that printerfacts has been correctly deployed with <code>kubectl get pods</code>. You should see something like this:

<pre>student@alpine:~$ kubectl get pods
NAME READY STATUS RESTARTS AGE
cassandra-0 1/1 Running 0 31m
cassandra-1 1/1 Running 0 30m
server-85f9f4465b-9vhn9 1/1 Running 0 38m
server-85f9f4465b-b7s2n 1/1 Running 0 38m
server-85f9f4465b-bvgqs 1/1 Running 0 38m
server-85f9f4465b-bwf42 1/1 Running 0 38m
server-85f9f4465b-gn5sm 1/1 Running 0 38m</pre>
Finally, run <code>minikube tunnel</code> in a separate terminal to set up port forwarding for our <code>LoadBalancer</code> API objects. This will enable you to interact with printerfacts from your VM, rather than needing to spin up a client pod like last time.

Verify that it worked correctly by running <code>curl 10.96.0.201</code> (this is the IP you will use to talk to printerfacts). Note that depending on your specific configuration, you may need to SSH into minikube using <code>minikube ssh</code> before you can access the load balancer service. If your curl command hangs forever, this may be the issue.

= Tasks =

== Part 1: Multi-Node Kubernetes, Cilium, and Cassandra (Easy) ==

Follow the instructions for each of the following numbered tasks. Make an effort to answer the accompanying questions, but more importantly please note down all of your observations and describe what you did for each task. You should also feel free to write down whatever questions you may have about a given task.

To achieve the best possible grade in this section, you must demonstrate that you have made an effort to understand the results of each task. (Note that an effort does not strictly mean a full understanding; it is okay to have questions!)

<ol style="list-style-type: decimal;">
<li>Thanks the Cilium CNI, your Kubernetes clusters has been outfitted with new eBPF superpowers. In particular, Cilium installs a series of eBPF programs into your (VM’s) kernel which can be used to monitor traffic between containers, pods, and nodes, as well as enforce L4–L7 security policy. To test out your new superpowers, run <code>cilium hubble port-forward&</code> followed by <code>hubble observe</code>. What is all that output? Can you make sense of any of it?
<ul>
<li>Hint 1: Try checking <code>hubble --help</code> and <code>hubble observe --help</code></li>
<li>Hint 2: [https://docs.cilium.io/en/v1.10/gettingstarted/hubble/ Check out the Hubble docs]</li></ul>
</li>
<li>Unlike last time, our cluster consists of three nodes rather than just one. Try running <code>curl 10.96.0.201</code> a few times and notice that the output now includes a node name in addition to a pod name. Do you notice any patterns in the output? Compare what you see with the output from running <code>kubectl get pods -o wide</code>. Try to come up with an explanation for the distribution of printerfacts pods over the nodes in your cluster.
<ul>
<li>Hint 1: Have a look at the <code>affinity</code> section of <code>deploy.yml</code>.</li>
<li>Hint 2: Have a look at the [https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity relevant documentation].</li>
<li>Hint 3: If you still can’t get it, an educated guess will suffice.</li></ul>
</li>
<li>Printerfacts is now a CRUD app that supports creating, reading, updating, and deleting facts from the Cassandra database. In particular, we now support the following endpoints:
<ul>
<li><code>GET</code> /fact: Get a random fact from the database as a JSON object</li>
<li><code>GET</code> /fact/keys: Get a list of all fact keys in the database as a JSON object</li>
<li><code>GET</code> /fact/<key>: Get the fact with the key <key> in the database as a JSON object</li>
<li><code>POST</code> /fact: Create a new fact where the request body is a JSON object of the form <code>{ "fact": "Fact Here", "kind": "Kind of fact (e.g. Cat fact)" }</code></li>
<li><code>PUT</code> /fact/<key>: Modify the fact with key <key> in the database where the request body is a JSON object of the form <code>{ "fact": "Fact Here", "kind": "Kind of fact (e.g. Cat fact)" }</code></li>
<li><code>DELETE</code> /fact/<key>: Delete the fact with key <key> from the database</li></ul>

Try out each of the printerfacts endpoints. Note that you can specify the HTTP request type using <code>curl -X <type></code> (for example, <code>curl -X POST</code> to send a POST request). You can send a JSON body as a payload by using <code>curl -H 'Content-type: application/json' -d '{"key": "value"}'</code> where you replace the <code>{"key": "value"}</code> part with your JSON object.</li>
<li>Let’s use Cilium and Hubble to observe dataflows between our Cassandra database and the printerfacts service. To do this, run the following Hubble command: <code>hubble observe --label component=server --label component=printerfacts -f</code>. Make a few requests to the printerfacts service, exercising various API endpoints. Try and make sense of some of the network traffic you observe.</li>
<li>In an effort to secure your cluster, you decide to employ a Cilium network policy that locks down the printerfacts service. First, download the example security policy by running <code>wget https://homeostasis.scs.carleton.ca/~will/printerfacts-policy.yml</code>, then run <code>kubectl apply -f policy.yml</code> to apply it. The example policy allows GET requests to <code>/</code>, <code>/fact</code>, <code>/fact/keys</code>, and <code>/fact/<key></code>. Examine the policy file and make sure you understand how it works.
Try out your policy by making a few valid requests followed by some invalid ones. While your policy is applied, try using Hubble to observe the HTTP traffic by running <code>hubble observe --protocol http</code>. Optionally, try extending the policy to make some other valid routes work (e.g. PUT, POST, and DELETE requests).
Hint: You may wish to consult the [https://docs.cilium.io/en/v1.10/gettingstarted/http/ Cilium docs]</li>
<li>Now that you’re familiar with printerfacts, it’s time to simulate a node failure. Since both printerfacts and Cassandra are replicated across multiple nodes in our cluster, taking one of them down should not impact either service.
Let’s first try a planned node disruption. This kind of disruption might occur when a cluster administrator wants to take a node down for maintenance, for example to perform a kernel update. Start by draining node <code>minikube-m02</code> with the command <code>kubectl drain minikube-m02 --ignore-daemonsets</code>. Now try interacting with the printerfacts service as before. Make note of any unusual behaviour and try to come up with a best-guess explanation.
Now you can bring your node back up using <code>kubectl uncordon minikube-m02</code>. Once your node has been uncordoned, it should be ready for scheduling again. Try scaling up the printerfacts deployment to get some pods running on the node again.
Finally, let’s simulate a total node failure. To do this, we will kill the underlying kubelet for our node, which is running on your system as a Docker container. Use the <code>docker ps</code> command to find the container ID that corresponds to your node, then kill it using <code>docker kill</code>. Repeat the same experiments from before, documenting any unusual behaviour you observe.</li></ol>

== Part 2: Interacting with Cassandra (Hard) ==

Note that this part of the experience is only required if you wish to achieve a grade of A- or higher. You can also choose to skip one of the two questions here, but doing so will likely impact your grade.

<ol style="list-style-type: decimal;">
<li>Write a Cilium security policy that allows only SELECT and INSERT queries to the <code>pfacts.facts</code> table in Cassandra. All other queries should be denied. Apply your policy and demonstrate how it works using a few examples queries. You may wish to consult [https://docs.cilium.io/en/v1.10/gettingstarted/cassandra/#how-to-secure-a-cassandra-database the Cilium policy docs].
Hint: You can use this [https://homeostasis.scs.carleton.ca/~will/cassandra-policy.yml policy template] as a starting point.</li>
<li>Write and deploy your own containerized application as a replicated deployment to interact with the Cassandra database in some interesting way. You can choose to extend the printerfacts schema or come up with your own schema, depending on your preference. Be sure to explain how your application deals with Cassandra’s consistency model in a replicated cluster. In order to achieve points for this question, you '''must''' make some meaningful modifications to the database. Just consuming the existing data in a new way is not sufficient.
Hint 1: Cassandra allows the client to choose their consistency level when making queries. Hint 2: Cassandra queries are made in CQL, a NoSQL query language that is similar to but not totally compatible with SQL. Hint 3: You can spawn a <code>cqlsh</code> session to issue test CQL queries like <code>kubectl exec -it cassandra-0 -- cqlsh</code></li></ol>

== Part 3: Reflection ==

Summarize your experience with multi-node Kubernetes, Cilium, and Cassandra in a few paragraphs (both the good and the bad). What concepts do you see reflected here from the research papers we have read thus far? After having some hands on experience with a distributed system technology, have any of your opinions or initial assumptions changed? Feel free to list any other thoughts you have.

= Acknowledgements =

The idea for the printerfacts API comes from Christine Dodrill’s [https://christine.website/blog/dev-printerfact-2021-04-17 wonderful blog post].