Operating Systems 2022F: Tutorial 2 - Revision history

Soma: /* C: Examining the runtime memory map */

2022-09-24T20:23:40Z

C: Examining the runtime memory map

← Older revision		Revision as of 20:23, 24 September 2022
Line 87:		Line 87:

	Compile and run [https://homeostasis.scs.carleton.ca/~soma/os-2019f/code/3000memview.c 3000memview.c], then consider the following questions.		Compile and run [https://homeostasis.scs.carleton.ca/~soma/os-2019f/code/3000memview.c 3000memview.c], then consider the following questions.
	# Why are the addresses inconsistent between runs? What happens if you run "setarch ~~x86-64~~ -R 3000memview" (~~where~~ 3000memview is ~~the name of the compiled program~~)?		# Why are the addresses inconsistent between runs? What happens if you run the program with the command "setarch -R ./3000memview"? (Note this changes how 3000memview is run.)
	# Roughly where does the stack seem to be? The heap? Code? Global variables?		# Roughly where does the stack seem to be? The heap? Code? Global variables?
	# Observe how the heap grows (i.e. the value of sbrk changes) in response to malloc calls. Would you expect the heap to ever run into the stack? Why or why not?		# Observe how the heap grows (i.e. the value of sbrk changes) in response to malloc calls. Would you expect the heap to ever run into the stack? Why or why not?

Soma: /* C: Examining the runtime memory map */

2022-09-24T20:20:37Z

C: Examining the runtime memory map

← Older revision		Revision as of 20:20, 24 September 2022
Line 87:		Line 87:

	Compile and run [https://homeostasis.scs.carleton.ca/~soma/os-2019f/code/3000memview.c 3000memview.c], then consider the following questions.		Compile and run [https://homeostasis.scs.carleton.ca/~soma/os-2019f/code/3000memview.c 3000memview.c], then consider the following questions.
	# Why are the addresses inconsistent between runs? What happens if ~~then~~ run `setarch x86-64 -R 3000memview`?		# Why are the addresses inconsistent between runs? What happens if you run "setarch x86-64 -R 3000memview" (where 3000memview is the name of the compiled program)?
	# Roughly where does the stack seem to be? The heap? Code? Global variables?		# Roughly where does the stack seem to be? The heap? Code? Global variables?
	# Observe how the heap grows (i.e. the value of sbrk changes) in response to malloc calls. Would you expect the heap to ever run into the stack? Why or why not?		# Observe how the heap grows (i.e. the value of sbrk changes) in response to malloc calls. Would you expect the heap to ever run into the stack? Why or why not?

Soma: /* C: Examining the runtime memory map */

2022-09-24T20:19:30Z

C: Examining the runtime memory map

← Older revision		Revision as of 20:19, 24 September 2022
Line 87:		Line 87:

	Compile and run [https://homeostasis.scs.carleton.ca/~soma/os-2019f/code/3000memview.c 3000memview.c], then consider the following questions.		Compile and run [https://homeostasis.scs.carleton.ca/~soma/os-2019f/code/3000memview.c 3000memview.c], then consider the following questions.
	# Why are the addresses inconsistent between runs?		# Why are the addresses inconsistent between runs? What happens if then run `setarch x86-64 -R 3000memview`?
	# Roughly where does the stack seem to be? The heap? Code? Global variables?		# Roughly where does the stack seem to be? The heap? Code? Global variables?
	# Observe how the heap grows (i.e. the value of sbrk changes) in response to malloc calls. Would you expect the heap to ever run into the stack? Why or why not?		# Observe how the heap grows (i.e. the value of sbrk changes) in response to malloc calls. Would you expect the heap to ever run into the stack? Why or why not?

Soma: /* B: Comparing C and assembly */

2022-09-17T19:35:06Z

B: Comparing C and assembly

← Older revision		Revision as of 19:35, 17 September 2022
Line 65:		Line 65:

	Resources on x86-64 assembly language:		Resources on x86-64 assembly language:
	* [https://cs61.seas.harvard.edu/site/2018/Asm1/ Eddie Kohler's assembly language basics (2018)]		* [https://cs61.seas.harvard.edu/site/2018/Asm1/ Eddie Kohler's assembly language basics (2018)] (also see other notes from this class)
	* [https://en.wikibooks.org/wiki/X86_Assembly/GAS_Syntax AT&T/GNU Assembler syntax]		* [https://en.wikibooks.org/wiki/X86_Assembly/GAS_Syntax AT&T/GNU Assembler syntax]
	* [https://en.wikipedia.org/wiki/X86_calling_conventions#System_V_AMD64_ABI the Wikipedia article on calling conventions]		* [https://en.wikipedia.org/wiki/X86_calling_conventions#System_V_AMD64_ABI the Wikipedia article on calling conventions]

Soma: /* B: Comparing C and assembly */

2022-09-17T19:34:15Z

B: Comparing C and assembly

← Older revision		Revision as of 19:34, 17 September 2022
Line 59:		Line 59:
	Do the following with hello.c and syscall-hello.c, as before.		Do the following with hello.c and syscall-hello.c, as before.

	A few tips on ~~reading~~ assembly ~~code. See [https~~://en.wikibooks.org/wiki/X86_Assembly/GAS_Syntax AT&T/GNU Assembler syntax] for more information on syntax and [https://en.wikipedia.org/wiki/X86_calling_conventions#System_V_AMD64_ABI the Wikipedia article on calling conventions] for how functions are called.		A few tips on x86-64 assembly language:
	* The last letter of many instructions refers to the size of the operand. For example, callq means call a function using a "quad" value (64 bits).		* The last letter of many instructions refers to the size of the operand. For example, callq means call a function using a "quad" value (64 bits).
	* A dollar sign preceding a value means that it is a literal value, a percent sign means it is a register.		* A dollar sign preceding a value means that it is a literal value, a percent sign means it is a register.
	* If a register is in parentheses, then it is being used as a "pointer" (it contains an address, so the CPU goes to that address and interacts with the memory there). If there is a number before the parentheses, it is an offset to the register's value.		* If a register is in parentheses, then it is being used as a "pointer" (it contains an address, so the CPU goes to that address and interacts with the memory there). If there is a number before the parentheses, it is an offset to the register's value.

			Resources on x86-64 assembly language:
			* [https://cs61.seas.harvard.edu/site/2018/Asm1/ Eddie Kohler's assembly language basics (2018)]
			* [https://en.wikibooks.org/wiki/X86_Assembly/GAS_Syntax AT&T/GNU Assembler syntax]
			* [https://en.wikipedia.org/wiki/X86_calling_conventions#System_V_AMD64_ABI the Wikipedia article on calling conventions]


	# Using the <tt>nm</tt> command, see what symbols are defined in prog-static and prog-dyn. Which defines more symbols?		# Using the <tt>nm</tt> command, see what symbols are defined in prog-static and prog-dyn. Which defines more symbols?

Soma: /* A: Function calls, library calls, and system calls */

2022-09-16T13:56:22Z

A: Function calls, library calls, and system calls

← Older revision		Revision as of 13:56, 16 September 2022
Line 50:		Line 50:
	# Run <tt>ldd</tt> on the static and dynamic versions of the program. How does the output compare? Why?		# Run <tt>ldd</tt> on the static and dynamic versions of the program. How does the output compare? Why?
	# See what system calls prog-static produces by running <tt>strace -o syscalls-static.log ./prog-static</tt>. Do the same for <tt>prog-dyn</tt>. Which version generates more system calls? '''Note: system calls are saved in the log file syscalls-static.log. Feel free to save them in a different file.'''		# See what system calls prog-static produces by running <tt>strace -o syscalls-static.log ./prog-static</tt>. Do the same for <tt>prog-dyn</tt>. Which version generates more system calls? '''Note: system calls are saved in the log file syscalls-static.log. Feel free to save them in a different file.'''
	# See what library calls prog-static produces by running <tt>ltrace -x '*' -o library-static.log ./prog-static</tt>. Do the same for <tt>prog-dyn</tt>. Which version generates more library calls?		# See what library calls prog-static produces by running <tt>ltrace -x '*' -o library-static.log ./prog-static</tt>. Do the same for <tt>prog-dyn</tt>. Which version generates more library calls? (Note: you will have to run <tt>sudo apt install ltrace</tt> to enable the command.)
	# Compile the program dynamically but with "lazy" linking: <tt>gcc -O2 -z lazy prog.c -o prog-dynlazy</tt>. Run <tt>ltrace -o library-lazy.log ./prog-dynlazy</tt>. How does the output of this compare to that of the previous ltrace?		# Compile the program dynamically but with "lazy" linking: <tt>gcc -O2 -z lazy prog.c -o prog-dynlazy</tt>. Run <tt>ltrace -o library-lazy.log ./prog-dynlazy</tt>. How does the output of this compare to that of the previous ltrace?
	# Use the command <tt>ls -l</tt> to see the metadata associated with prog.c and prog-dyn, and prog-static. Who owns these files? What group are they in? Do you notice any pattern with the permissions (rwx) associated with each file?		# Use the command <tt>ls -l</tt> to see the metadata associated with prog.c and prog-dyn, and prog-static. Who owns these files? What group are they in? Do you notice any pattern with the permissions (rwx) associated with each file?

Soma: Created page with "In this tutorial we're going to look at how processes work at a low level: how they make system calls & library calls, how C and assembly compare, and and how memory is laid out. If you haven't already, please set up and use a VM on openstack for this work. ==Background== A process on UNIX-like systems are separated from each other: they run in their own address space - pointers can only refer to code and data in that program, not in other programs. Because programs..."

2022-09-12T00:54:03Z

Created page with "In this tutorial we're going to look at how processes work at a low level: how they make system calls & library calls, how C and assembly compare, and and how memory is laid out. If you haven't already, please set up and use a VM on openstack for this work. ==Background== A process on UNIX-like systems are separated from each other: they run in their own address space - pointers can only refer to code and data in that program, not in other programs. Because programs..."

New page

In this tutorial we're going to look at how processes work at a low level: how they make system calls & library calls, how C and assembly compare, and and how memory is laid out. If you haven't already, please set up and use a VM on openstack for this work.

==Background==

A process on UNIX-like systems are separated from each other: they run in their own address space - pointers can only refer to code and data in that program, not in other programs. Because programs are separated, they can't access anything external to them without help.

Here we're going to talk about libraries and system calls, two ways programs gain access to additional functionality. Libraries are for external code that is loaded into a process, while system calls allow for code in other processes or the OS kernel to be accessed.

===Assembly Language===

When we write code in C, it has to be compiled to machine code before it can be run. This compilation step doesn't happen all at once. Compilation has distinct phases:

* Compile C code into assembly code (.s files).
* Assemble assembly code into machine code placed in object code files (.o files).
* Link object code files together to create a runnable binary.

If you run <tt>gcc -v</tt>, you'll see these steps all happen in a very verbose fashion.

===Static & Dynamic Libraries===

Most applications are not self contained; they rely on lots of external code. In compiled languages such as C, external code can be brought into the process through '''linking'''. There are two basic types of linking, static and dynamic linking:
* With '''static linking''', code is brought in at compile time (specifically, in the link stage) and added to the executable. The code is now the same as other application code. (Static libraries are just collections of .o files.)
* With '''dynamic linking''', a reference to the library code is added to the binary. The actual library code has to later be loaded when the program is executed. This loading will happen before <tt>main()</tt> is called.
Static and dynamic libraries are stored in /lib and /usr/lib, traditionally.

The dynamic libraries associated with a program binary can be found using the <tt>ldd</tt> command. You can use <tt>ltrace</tt> to see calls to functions that are dynamically linked. If a program is statically linked <tt>ldd</tt> will have nothing to report and will generally produce an error.

Note that code in static and dynamic libraries runs inside of the process loading the code; thus, library code has the same privileges as other application code. It can do everything regular application code can do (it can access all of your code and data), but it can do no more than your code can do (it has the same restrictions on accessing system resources).

===System Calls===

A process on its own has limited access to the system. It cannot directly access any external devices or data sources (e.g., files, keyboard, the screen, networks) on its own. To access these external resources, to allocate memory, or otherwise change its runtime environment, it must make '''system calls'''. Note that system calls run code outside of a process and thus cannot be called like regular function calls. The standard C library provides function wrappers for most commonly-used system calls so they can be accessed like regular C functions. Under the hood, however, these functions make use of special compiler directives in order to generate the machine code necessary to invoke system calls.

You can see the system calls produced by a process using the <tt>strace</tt> command.

In general, the code you call through a system call has more privileges than regular application code. This is because a system call is a request to the operating system kernel to do something on behalf of the process, and the kernel has full privileges to the system. (Indeed, it is the part of the system that implements the process abstraction.) We're going to talk a lot about system calls this semester, this is just your introduction to the concept.

==Tasks==

===A: Function calls, library calls, and system calls===

For [http://homeostasis.scs.carleton.ca/~soma/os-2017f/code/tut1/hello.c hello.c] and [http://homeostasis.scs.carleton.ca/~soma/os-2017f/code/tut1/syscall-hello.c syscall-hello.c] do the following (<b>substituting the appropriate source file for prog.c</b>). For example, for hello.c, you would replace all instances of "prog" in a command with "hello".

To download programs to your VM, use the wget command, e.g.
<pre>
wget https://homeostasis.scs.carleton.ca/~soma/os-2017f/code/tut1/hello.c
</pre>
# Compile the program prog.c using <tt>gcc -O2 prog.c -o prog-dyn</tt> and run prog-dyn. What does it do?
# Statically compile and optimize prog.c by running <tt>gcc -O2 -static prog.c -o prog-static</tt>. How does the size compare with <tt>prog</tt>?
# Run <tt>ldd</tt> on the static and dynamic versions of the program. How does the output compare? Why?
# See what system calls prog-static produces by running <tt>strace -o syscalls-static.log ./prog-static</tt>. Do the same for <tt>prog-dyn</tt>. Which version generates more system calls? '''Note: system calls are saved in the log file syscalls-static.log. Feel free to save them in a different file.'''
# See what library calls prog-static produces by running <tt>ltrace -x '*' -o library-static.log ./prog-static</tt>. Do the same for <tt>prog-dyn</tt>. Which version generates more library calls?
# Compile the program dynamically but with "lazy" linking: <tt>gcc -O2 -z lazy prog.c -o prog-dynlazy</tt>. Run <tt>ltrace -o library-lazy.log ./prog-dynlazy</tt>. How does the output of this compare to that of the previous ltrace?
# Use the command <tt>ls -l</tt> to see the metadata associated with prog.c and prog-dyn, and prog-static. Who owns these files? What group are they in? Do you notice any pattern with the permissions (rwx) associated with each file?
# (optional) Look up the documentation for each of the system calls made by the static versions of the programs. You may need to append a 2 or 3 to the manpage invocation, e.g. "man 2 write" gets you the write system call documentation.

===B: Comparing C and assembly===

Do the following with hello.c and syscall-hello.c, as before.

A few tips on reading assembly code. See [https://en.wikibooks.org/wiki/X86_Assembly/GAS_Syntax AT&T/GNU Assembler syntax] for more information on syntax and [https://en.wikipedia.org/wiki/X86_calling_conventions#System_V_AMD64_ABI the Wikipedia article on calling conventions] for how functions are called.
* The last letter of many instructions refers to the size of the operand. For example, callq means call a function using a "quad" value (64 bits).
* A dollar sign preceding a value means that it is a literal value, a percent sign means it is a register.
* If a register is in parentheses, then it is being used as a "pointer" (it contains an address, so the CPU goes to that address and interacts with the memory there). If there is a number before the parentheses, it is an offset to the register's value.

# Using the <tt>nm</tt> command, see what symbols are defined in prog-static and prog-dyn. Which defines more symbols?
# Run the command <tt>gcc -c -O2 prog.c</tt> to produce an object file. What file was produced? What symbols does it define?
# Look at the assembly code of the program by running <tt>gcc -S -O2 prog.c</tt>. What file was produced? Identify the following in the assembly code (if present):
#* A function call (call)
#* A return from a function (ret)
#* Registers being saved onto the stack (push)
#* Registers being retrieved from the stack (pop)
#* Subtraction (sub)
#* A system call (syscall)
# Disassemble the object file using <tt>objdump -d</tt>. How does this disassembly compare with the output from gcc -S?
# Examine the headers of object file, dynamically linked executable, and the statically linked executable using <tt>objdump -h</tt>
# Examine the contents of object file, dynamically linked executable, and the statically linked executable using <tt>objdump -s</tt>
# Re-run all of the previous gcc commands adding the "-v" flag. What is all of that output?

===C: Examining the runtime memory map===

Compile and run [https://homeostasis.scs.carleton.ca/~soma/os-2019f/code/3000memview.c 3000memview.c], then consider the following questions.
# Why are the addresses inconsistent between runs?
# Roughly where does the stack seem to be? The heap? Code? Global variables?
# Observe how the heap grows (i.e. the value of sbrk changes) in response to malloc calls. Would you expect the heap to ever run into the stack? Why or why not?
# Change each malloc call to allocate more than 128K. What happens to the values of sbrk? Why? (Hint: use strace)
# Add more code and data to the program, and add more printf's to see where things are. Are things where you expect them to be?

==Code==

===[https://homeostasis.scs.carleton.ca/~soma/os-2017f/code/tut1/hello.c hello.c]===
<syntaxhighlight lang="c" line>
#include <stdio.h>

int main(int argc, char *argv[]) {

printf("Hello world!\n");

return 0;
}
</syntaxhighlight>

===[http://homeostasis.scs.carleton.ca/~soma/os-2017f/code/tut1/syscall-hello.c syscall-hello.c]===
<syntaxhighlight lang="c" line>
#include <unistd.h>
#include <sys/syscall.h>

char *buf = "Hello world!\n";

int main(int argc, char *argv) {
size_t result;

/* "man 2 write" to see arguments to write syscall */
result = syscall(SYS_write, 1, buf, 13);

return (int) result;
}
</syntaxhighlight>

===[https://homeostasis.scs.carleton.ca/~soma/os-2019f/code/3000memview.c 3000memview.c]===
<syntaxhighlight lang="c" line>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

char *gmsg = "Global Message";

const int buffer_size = 100;

int main(int argc, char *argv[], char *envp[])
{
char *lmsg = "Local Message";
char *buf[buffer_size];
int i;

printf("Memory report\n");
printf("argv: %lx\n", (unsigned long) argv);
printf("argv[0]: %lx\n", (unsigned long) argv[0]);
printf("envp: %lx\n", (unsigned long) envp);
printf("envp[0]: %lx\n", (unsigned long) envp[0]);

printf("lmsg: %lx\n", (unsigned long) lmsg);
printf("&lmsg: %lx\n", (unsigned long) &lmsg);
printf("gmsg: %lx\n", (unsigned long) gmsg);
printf("&gmsg: %lx\n", (unsigned long) &gmsg);

printf("main: %lx\n", (unsigned long) &main);

printf("sbrk(0): %lx\n", (unsigned long) sbrk(0));
printf("&buf: %lx\n", (unsigned long) &buf);

for (i = 0; i<buffer_size; i++) {
buf[i] = (char *) malloc(4096);
}

printf("buf[0]: %lx\n", (unsigned long) buf[0]);
printf("sbrk(0): %lx\n", (unsigned long) sbrk(0));

return 0;
}
</syntaxhighlight>