COMP3000 Operating Systems F23: Tutorial 9 - Revision history

Abdou at 02:11, 25 October 2023

2023-10-25T02:11:57Z

Abdou: /* General Instructions (same for all tutorials) */

2023-10-25T02:09:06Z

General Instructions (same for all tutorials)

← Older revision		Revision as of 02:09, 25 October 2023
Line 1:		Line 1:
	By the end of this tutorial, you will have a basic familiarity with the Docker platform and be able to run, operate, and customize containers, and create simple containers of your own. You will see how kernel building blocks like namespaces make containers possible.		By the end of this tutorial, you will have a basic familiarity with the Docker platform and be able to run, operate, and customize containers, and create simple containers of your own. You will see how kernel building blocks like namespaces make containers possible.

	~~==General Instructions (same for all tutorials)==~~

	~~<div class="mw-collapsible mw-collapsed">~~
	~~Click on Expand to view.~~
	~~<div class="mw-collapsible-content">~~
	Tutorials are graded based on participation and effort ('''so no need to struggle to have the “correct” answers — what matters is the attempts and sufficient thinking reflected in your answers'''), but you should still turn in your work. Submit your answers on Brightspace as a single text file named "<username>-comp3000-t<n>.txt" (where username is your MyCarletonOne username and n is the tutorial number). The first four lines of this file should be "COMP 3000 Tutorial <n>", your name, student number, and the date of submission. Your submitted answers must at least respond to all the items in the '''Tasks/Questions''' section.

	The deadline is usually four days after the tutorial date (see the actual deadline on the submission entry). Note that the submission entry is enforced by the system, so you may fail to get the effort marks even if it is one minute past the deadline.

	You should also check in with your assigned TA (in person or by responding to a poll if online). Your TA will be your first point of contact when you have questions or encounter any issues during the tutorial session.

	~~You get 1.5 marks for submitting answers that shows your effort and 0.5 for checking in, making this tutorial worth 2 points total.~~

	~~Read through the instructions before starting your work to get an overall picture. When source files are needed, you can download them by clicking on the hyperlink.~~
	~~</div>~~
	~~</div>~~

	==Background==		==Background==

Abdou: Created page with "By the end of this tutorial, you will have a basic familiarity with the Docker platform and be able to run, operate, and customize containers, and create simple containers of your own. You will see how kernel building blocks like namespaces make containers possible. ==General Instructions (same for all tutorials)==
Click on Expand to view.
Tutorials are graded based on participation and effo..."

2023-10-25T02:09:00Z

Created page with "By the end of this tutorial, you will have a basic familiarity with the Docker platform and be able to run, operate, and customize containers, and create simple containers of your own. You will see how kernel building blocks like namespaces make containers possible. ==General Instructions (same for all tutorials)== <div class="mw-collapsible mw-collapsed"> Click on Expand to view. <div class="mw-collapsible-content"> Tutorials are graded based on participation and effo..."

New page

By the end of this tutorial, you will have a basic familiarity with the Docker platform and be able to run, operate, and customize containers, and create simple containers of your own. You will see how kernel building blocks like namespaces make containers possible.

==General Instructions (same for all tutorials)==

<div class="mw-collapsible mw-collapsed">
Click on Expand to view.
<div class="mw-collapsible-content">
Tutorials are graded based on participation and effort ('''so no need to struggle to have the “correct” answers — what matters is the attempts and sufficient thinking reflected in your answers'''), but you should still turn in your work. Submit your answers on Brightspace as a single text file named "<username>-comp3000-t<n>.txt" (where username is your MyCarletonOne username and n is the tutorial number). The first four lines of this file should be "COMP 3000 Tutorial <n>", your name, student number, and the date of submission. Your submitted answers must at least respond to all the items in the '''Tasks/Questions''' section.

The deadline is usually four days after the tutorial date (see the actual deadline on the submission entry). Note that the submission entry is enforced by the system, so you may fail to get the effort marks even if it is one minute past the deadline.

You should also check in with your assigned TA (in person or by responding to a poll if online). Your TA will be your first point of contact when you have questions or encounter any issues during the tutorial session.

You get 1.5 marks for submitting answers that shows your effort and 0.5 for checking in, making this tutorial worth 2 points total.

Read through the instructions before starting your work to get an overall picture. When source files are needed, you can download them by clicking on the hyperlink.
</div>
</div>

==Background==

===Docker containers===
Docker is one of the OS-level virtualization technologies that allows the creation of isolated virtual execution environments, called containers, sharing the same OS kernel and certain runtime services/libraries with other containers. The workload running inside a container will feel as if it had its own everything, OS kernel, root file system, etc. This is possible thanks to the way Docker containers are designed and implemented. The Linux kernel does not directly support the construction of containers (the userspace libraries do) but its various building blocks play an important role, among which we will cover namespaces in this tutorial.

===Docker images===
A Docker image is a file containing a root file system as well as configuration parameters. It is very similar to other disk image formats in this sense. One of the main differences is that it employs a layered file system to maximize storage efficiency and allow flexible reuse of existing files.

===Dockerfile===
A Dockerfile is a text file and serves as instructions to build a Docker image. Unlike storing and distributing binary data directly in images, using Dockerfiles can save space and rebuild the Docker image by always downloading the latest version of software.

===Docker Registry===
The Registry is a server-side application containing and managing repositories of Docker images. We use Docker Hub (among other choices by different providers) to pull images from in this tutorial.

===Namespaces===
Namespaces allow for multiple “spaces” where identical names can coexist. For example, the same process ID (PID) may refer to completely different processes. This is like how we can distinguish Tom (PID) in Ottawa (namespace) from Tom in Toronto (namespace). Namespaces are enforced by the OS kernel. They are the fundamental building block of containers.

A few namespaces we will cover here: Mount (<tt>mnt</tt>), Network (<tt>net</tt>), User ID (<tt>uid</tt>) and Process ID (<tt>pid</tt>).

You can use the command <tt>lsns</tt> to see the namespaces the current process is in. A namespace ID is actually an '''inode'''.

Also, the command <tt>nsenter</tt> allows you to run programs with namespaces of another process. Use the <tt>-t</tt> option to specify the target PID. For instance, <tt>sudo nsenter -t 2772 -m ls /</tt> will list the root file system of the mount namespace of process 2772.

==Tasks/Questions==
The purpose of the following questions and tasks is to help you understand how containers work, which is an important artifact on top of and based on modern operating systems. The focus is on the OS building blocks – how it is made possible by the OS. Docker is just a typical and popular example to facilitate understanding.

'''You need to <tt>sudo</tt> for all the <tt>docker</tt> commands below'''.

Install Docker with <tt>sudo apt install docker.io</tt> if it’s not already installed.

# Use <tt>docker images</tt> and <tt>docker ps</tt> (similar to the purpose of <tt>ps</tt>) to see what images you have and whether any containers are running. Run your first container with <tt>docker run hello-world</tt>. Now, check the images and <tt>docker ps</tt> again, what do you see and <u>why</u>?
# Look at the hello-world image above and you may wonder how the tiny-sized image forms a full container. Find out where the files are located (hint: using the command <tt>docker inspect</tt> and looking for paths starting with <tt>/var/lib/docker</tt>). What is the size of the found file? Run the file directly. <u>What do you think</u> is the way the tiny-sized image forms the full container?
# <tt>docker ps -a</tt> will show something different. Next, let’s try an image of a bit more complexity that will remain running. Try the centos (a Linux distribution) image: <tt>docker run -d centos sleep infinity</tt> (<tt>man docker-run</tt> for the meaning of the options).
# Start a new shell: <tt>docker exec -it</tt> <CONTAINER ID> <tt>/bin/bash</tt> (you can find out the container ID with <tt>docker ps -a</tt>. Note that you can specify just the first few digits of the ID as long as it’s distinctive). Explore the file system therein.
#; Next, we will examine the namespaces.
# Compare the mount namespaces (different views of the file system structure) between the centos container and the host, using <tt>ls -l /proc/self/ns/mnt</tt>. Are they the same and if not, what are they (format: <tt>mnt:[????]</tt>)? In particular, take a look at the root mount point (“<tt>/</tt>”) inside and outside the container (<tt>mount | grep "on / "</tt>). You can always use two sessions (one in the container and one in the host) or <tt>tmux</tt> as you feel convenient.
# With the <tt>lsns</tt> command, you can get an overview of all namespaces. Comparing the centos container and the host, <u>which namespaces are the same (shared) and which are different</u>?
# Inside the container, use the command <tt>id</tt> to see what user you’re logged in as (uid, gid and group). Then <tt>ps -eo uid,gid,pid,command | grep sleep</tt> inside the container and outside the container respectively. Are their outputs the same?
# Using the pid of “sleep” you have got outside the container above, find its pid namespace (<tt>sudo ls -l /proc/<pid>/ns/pid</tt>, format <tt>pid:[????]</tt>). Is it consistent with the <tt>lsns</tt> output inside the container?
# From step 6, you have the individual namespaces used by the “sleep” container (no need to go inside the container again). Let’s check the network namespace (<tt>sudo nsenter -t </tt><pid><tt> -n ip addr</tt>).
#: Note that uid remapping is not enabled by default. So the user (uid) inside the container will be the same outside (which does not consider security).
# Now, make your own Docker container. Read the current [https://raw.githubusercontent.com/docker-library/hello-world/7ecae6978055d2fb6960e2a29d24a2af612e2716/amd64/hello-world/Dockerfile Dockerfile of hello-world]. Change it in a way that it is no longer from scratch but ''from <tt>hello-world </tt>''. Here we can avoid defining <tt>csimpleshell</tt> as the default command as it works with a terminal for interaction (then it will inherit <tt>CMD</tt> from <tt>hello-world</tt>). Build a new image called <tt>csimpleshell</tt> by doing the following:
#: <tt>mkdir anyname</tt>
#: copy your <tt>csimipleshell</tt> and the Dockerfile of <tt>hello-world</tt> into this directory
#: <tt>cd anyname</tt>
#: make necessary changes to the Dockerfile
#: <tt>docker build -t csimpleshell .</tt>
# Run the new csimpleshell container (e.g., <tt>docker run -it csimpleshell /csimpleshell</tt>). Depending on <tt>how you compiled csimpleshell</tt>, check whether the container runs or if it does whether csimpleshell works as before. Try compiling csimpleshell statically (if it was dynamic) and repeat the steps.
# Edit the Dockerfile again and make it from “centos” instead of “hello-world”. Remove any reference to hello. Rebuild and run it. How is csimpleshell now? <u>Why</u> is there such a difference?

← Older revision		Revision as of 02:11, 25 October 2023
Line 18:		Line 18:
	Namespaces allow for multiple “spaces” where identical names can coexist. For example, the same process ID (PID) may refer to completely different processes. This is like how we can distinguish Tom (PID) in Ottawa (namespace) from Tom in Toronto (namespace). Namespaces are enforced by the OS kernel. They are the fundamental building block of containers.		Namespaces allow for multiple “spaces” where identical names can coexist. For example, the same process ID (PID) may refer to completely different processes. This is like how we can distinguish Tom (PID) in Ottawa (namespace) from Tom in Toronto (namespace). Namespaces are enforced by the OS kernel. They are the fundamental building block of containers.

	A few namespaces we will cover here: Mount (<tt>mnt</tt>), Network (<tt>net</tt>), User ID (<tt>uid</tt>) and Process ID (<tt>pid</tt>).		A few namespaces we will cover here: Mount (<code>mnt</code>), Network (<code>net</code>), User ID (<code>uid</code>) and Process ID (<code>pid</code>).

	You can use the command <tt>lsns</tt> to see the namespaces the current process is in. A namespace ID is actually an '''inode'''.		You can use the command <code>lsns</code> to see the namespaces the current process is in. A namespace ID is actually an '''inode'''.

	Also, the command <tt>nsenter</tt> allows you to run programs with namespaces of another process. Use the <tt>-t</tt> option to specify the target PID. For instance, <tt>sudo nsenter -t 2772 -m ls /</tt> will list the root file system of the mount namespace of process 2772.		Also, the command <code>nsenter</code> allows you to run programs with namespaces of another process. Use the <code>-t</code> option to specify the target PID. For instance, <code>sudo nsenter -t 2772 -m ls /</code> will list the root file system of the mount namespace of process 2772.

	==Tasks/Questions==		==Tasks/Questions==
	The purpose of the following questions and tasks is to help you understand how containers work, which is an important artifact on top of and based on modern operating systems. The focus is on the OS building blocks – how it is made possible by the OS. Docker is just a typical and popular example to facilitate understanding.		The purpose of the following questions and tasks is to help you understand how containers work, which is an important artifact on top of and based on modern operating systems. The focus is on the OS building blocks – how it is made possible by the OS. Docker is just a typical and popular example to facilitate understanding.

	'''You need to <tt>sudo</tt> for all the <tt>docker</tt> commands below'''.		'''You need to <code>sudo</code> for all the <code>docker</code> commands below'''.

	Install Docker with <tt>sudo apt install docker.io</tt> if it’s not already installed.		Install Docker with <code>sudo apt install docker.io</code> if it’s not already installed.

	# Use <tt>docker images</tt> and <tt>docker ps</tt> (similar to the purpose of <tt>ps</tt>) to see what images you have and whether any containers are running. Run your first container with <tt>docker run hello-world</tt>. Now, check the images and <tt>docker ps</tt> again, what do you see and <u>why</u>?		# Use <code>docker images</code> and <code>docker ps</code> (similar to the purpose of <code>ps</code>) to see what images you have and whether any containers are running. Run your first container with <code>docker run hello-world</code>. Now, check the images and <code>docker ps</code> again, what do you see and <u>why</u>?
	# Look at the hello-world image above and you may wonder how the tiny-sized image forms a full container. Find out where the files are located (hint: using the command <tt>docker inspect</tt> and looking for paths starting with <tt>/var/lib/docker</tt>). What is the size of the found file? Run the file directly. <u>What do you think</u> is the way the tiny-sized image forms the full container?		# Look at the hello-world image above and you may wonder how the tiny-sized image forms a full container. Find out where the files are located (hint: using the command <code>docker inspect</code> and looking for paths starting with <code>/var/lib/docker</code>). What is the size of the found file? Run the file directly. <u>What do you think</u> is the way the tiny-sized image forms the full container?
	# <tt>docker ps -a</tt> will show something different. Next, let’s try an image of a bit more complexity that will remain running. Try the centos (a Linux distribution) image: <tt>docker run -d centos sleep infinity</tt> (<tt>man docker-run</tt> for the meaning of the options).		# <code>docker ps -a</code> will show something different. Next, let’s try an image of a bit more complexity that will remain running. Try the centos (a Linux distribution) image: <code>docker run -d centos sleep infinity</code> (<code>man docker-run</code> for the meaning of the options).
	# Start a new shell: <tt>docker exec -it</tt> <CONTAINER ID> <tt>/bin/bash</tt> (you can find out the container ID with <tt>docker ps -a</tt>. Note that you can specify just the first few digits of the ID as long as it’s distinctive). Explore the file system therein.		# Start a new shell: <code>docker exec -it</code> <CONTAINER ID> <code>/bin/bash</code> (you can find out the container ID with <code>docker ps -a</code>. Note that you can specify just the first few digits of the ID as long as it’s distinctive). Explore the file system therein.
	#; Next, we will examine the namespaces.		#; Next, we will examine the namespaces.
	# Compare the mount namespaces (different views of the file system structure) between the centos container and the host, using <tt>ls -l /proc/self/ns/mnt</tt>. Are they the same and if not, what are they (format: <tt>mnt:[????]</tt>)? In particular, take a look at the root mount point (“<tt>/</tt>”) inside and outside the container (<tt>mount \| grep "on / "</tt>). You can always use two sessions (one in the container and one in the host) or <tt>tmux</tt> as you feel convenient.		# Compare the mount namespaces (different views of the file system structure) between the centos container and the host, using <code>ls -l /proc/self/ns/mnt</code>. Are they the same and if not, what are they (format: <code>mnt:[????]</code>)? In particular, take a look at the root mount point (“<code>/</code>”) inside and outside the container (<code>mount \| grep "on / "</code>). You can always use two sessions (one in the container and one in the host) or <code>tmux</code> as you feel convenient.
	# With the <tt>lsns</tt> command, you can get an overview of all namespaces. Comparing the centos container and the host, <u>which namespaces are the same (shared) and which are different</u>?		# With the <code>lsns</code> command, you can get an overview of all namespaces. Comparing the centos container and the host, <u>which namespaces are the same (shared) and which are different</u>?
	# Inside the container, use the command <tt>id</tt> to see what user you’re logged in as (uid, gid and group). Then <tt>ps -eo uid,gid,pid,command \| grep sleep</tt> inside the container and outside the container respectively. Are their outputs the same?		# Inside the container, use the command <code>id</code> to see what user you’re logged in as (uid, gid and group). Then <code>ps -eo uid,gid,pid,command \| grep sleep</code> inside the container and outside the container respectively. Are their outputs the same?
	# Using the pid of “sleep” you have got outside the container above, find its pid namespace (<tt>sudo ls -l /proc/<pid>/ns/pid</tt>, format <tt>pid:[????]</tt>). Is it consistent with the <tt>lsns</tt> output inside the container?		# Using the pid of “sleep” you have got outside the container above, find its pid namespace (<code>sudo ls -l /proc/<pid>/ns/pid</code>, format <code>pid:[????]</code>). Is it consistent with the <code>lsns</code> output inside the container?
	# From step 6, you have the individual namespaces used by the “sleep” container (no need to go inside the container again). Let’s check the network namespace (<tt>sudo nsenter -t </tt><pid><tt> -n ip addr</tt>).		# From step 6, you have the individual namespaces used by the “sleep” container (no need to go inside the container again). Let’s check the network namespace (<code>sudo nsenter -t </code><pid><code> -n ip addr</code>).
	#: Note that uid remapping is not enabled by default. So the user (uid) inside the container will be the same outside (which does not consider security).		#: Note that uid remapping is not enabled by default. So the user (uid) inside the container will be the same outside (which does not consider security).
	# Now, make your own Docker container. Read the current [https://raw.githubusercontent.com/docker-library/hello-world/7ecae6978055d2fb6960e2a29d24a2af612e2716/amd64/hello-world/Dockerfile Dockerfile of hello-world]. Change it in a way that it is no longer from scratch but ''from <tt>hello-world </tt>''. Here we can avoid defining <tt>csimpleshell</tt> as the default command as it works with a terminal for interaction (then it will inherit <tt>CMD</tt> from <tt>hello-world</tt>). Build a new image called <tt>csimpleshell</tt> by doing the following:		# Now, make your own Docker container. Read the current [https://raw.githubusercontent.com/docker-library/hello-world/7ecae6978055d2fb6960e2a29d24a2af612e2716/amd64/hello-world/Dockerfile Dockerfile of hello-world]. Change it in a way that it is no longer from scratch but ''from <code>hello-world </code>''. Here we can avoid defining <code>csimpleshell</code> as the default command as it works with a terminal for interaction (then it will inherit <code>CMD</code> from <code>hello-world</code>). Build a new image called <code>csimpleshell</code> by doing the following:
	#: <tt>mkdir anyname</tt>		#: <code>mkdir anyname</code>
	#: copy your <tt>csimipleshell</tt> and the Dockerfile of <tt>hello-world</tt> into this directory		#: copy your <code>csimipleshell</code> and the Dockerfile of <code>hello-world</code> into this directory
	#: <tt>cd anyname</tt>		#: <code>cd anyname</code>
	#: make necessary changes to the Dockerfile		#: make necessary changes to the Dockerfile
	#: <tt>docker build -t csimpleshell .</tt>		#: <code>docker build -t csimpleshell .</code>
	# Run the new csimpleshell container (e.g., <tt>docker run -it csimpleshell /csimpleshell</tt>). Depending on <tt>how you compiled csimpleshell</tt>, check whether the container runs or if it does whether csimpleshell works as before. Try compiling csimpleshell statically (if it was dynamic) and repeat the steps.		# Run the new csimpleshell container (e.g., <code>docker run -it csimpleshell /csimpleshell</code>). Depending on <code>how you compiled csimpleshell</code>, check whether the container runs or if it does whether csimpleshell works as before. Try compiling csimpleshell statically (if it was dynamic) and repeat the steps.
	# Edit the Dockerfile again and make it from “centos” instead of “hello-world”. Remove any reference to hello. Rebuild and run it. How is csimpleshell now? <u>Why</u> is there such a difference?		# Edit the Dockerfile again and make it from “centos” instead of “hello-world”. Remove any reference to hello. Rebuild and run it. How is csimpleshell now? <u>Why</u> is there such a difference?