Operating Systems 2020W Lecture 9

Video

The video from the lecture given on February 5, 2020 is now available.

Topics

standard file descriptors (stdin, stdout, stderr) vs command line arguments vs environment variables
open file descriptors, /proc
files, inodes
directories
block devices, character devices
superblocks, inode blocks, data blocks
filesystems
vfs
mount
fsck
sshfs

Readings

At this point, we've covered most of these chapters:

 http://pages.cs.wisc.edu/~remzi/OSTEP/cpu-intro.pdf
 http://pages.cs.wisc.edu/~remzi/OSTEP/cpu-api.pdf
 http://pages.cs.wisc.edu/~remzi/OSTEP/cpu-mechanisms.pdf  (partially)
 http://pages.cs.wisc.edu/~remzi/OSTEP/vm-intro.pdf

For files and filesystems, see these chapters

 http://pages.cs.wisc.edu/~remzi/OSTEP/file-intro.pdf
 http://pages.cs.wisc.edu/~remzi/OSTEP/file-implementation.pdf (at a high level)
 http://pages.cs.wisc.edu/~remzi/OSTEP/file-journaling.pdf (at a high level)

The textbook goes into much more detail that I will on some topics, but it is worth looking through that detail if material covered in lecture seems too abstract.

It is worth your time, however, looking through the textbook from the beginning and skimming sections that we haven't covered.

Notes

Lecture 9
---------

information overload
* I'm covering a lot of material in this course, more than I expect you to retain.  I expect there to be areas that are not completely clear at the end of the semester.  That's just the nature of modern operating systems.
* So here are some tips:
  - use the textbook, it gives a different presentation
  - take the tutorials seriously, that is where the learning should happen
  - ask questions
  - learn the tools
  
standard file descriptors (stdin, stdout, stderr) vs command line arguments vs environment variables
 - remember:
   - standard in is file descriptor 0
   - standard out is fd 1
   - standard error is fd 2

What is a process, really?
 - address space + execution context
 - RAM plus CPU registers

But if this is what a process is, how can it interact with the outside world?

When a program is first run by execve, its inputs are as follows:
 - command line (supplied by execve)
 - environment variables (supplied by execve)
 - data, code in the executable file (supplied by execve)
 - file descriptors, i.e., files opened by whomever did execve
    - standard in, out, err are just open file descriptors
      that are expected to be there by default, by convention

What is a file descriptor, really?
 - it is a value passed to the kernel so the kernel knows where we want to
   read from or write to (using a read or write system call)
 - they are "created" by open system calls and disposed of using close
 - if you don't want to do reads and writes, you could have a custom system
   call to do the thing you want
 - but most I/O on UNIX is done using reads and writes

0, 1, and 2 are the standard file descriptors, but we can of course open more
than three files - so file descriptors are allocated as needed

File descriptors are references to open files
 - you can think of them as indices to arrays of structs
   *that are in the kernel* (you can't access them directly from userspace)

What is a file?

To understand a file, we have to discuss filesystems.


When you look at files and you do an ls -l, you'll see a few different
characters in the first column
 * -: regular file
 * d: directory  <--- use getdents, not read, to get data
 * l: symbolic link  <--- use lstat to get info, not fstat
 * c: character device  
 * b: block device 
 * p: named pipe (fifo) <--- way of doing | in a different way
      (will discuss later)

These are all "files", so you'll open them and deal with them using file descriptors.  But they are very different things.

Character devices and block devices represent devices
 - real hardware or virtual hardware
 - ways to interact with "the real world"

character devices are for byte-by-byte I/O, block devices are for
block-by-block I/O

(A block is a fixed unit of storage, can vary, but is typically some small power of two, e.g. 512, 1024, 4096)

Mass storage (hard disks, SSDs) is represented using block devices
everything else is a character device

pseudo-tty's are character devices

Note that block and character devices don't have a "size"
In that column in ls, you instead see two numbers separated by commas
  - first is the major number
  - second is the minor number

These two numbers tell the kernel how to deal with this particular device
file.  (separate namespace for block and character devices)

RAM is not a device, but...
 - on some UNIX flavors you can access RAM through a device but it isn't
   generally done nowadays
 - RAM is accessed in the form of processes that do memory allocations

But then what's a filesystem?
 - a set of "files", potentially of all types
 - what happens when accessing each file can vary completely
   depending on the type of filesystem

To try and clarify
 - device files each have their own special code in the kernel to handle each
   (these are drivers)
 - regular files are implemented through filesystems, where common code
   implements open, read, write, etc for all files in the filesystem
   - different filesystems can have very different implementations

You've seen many filesystems already in this class
 - ext4 (the files you regularly use)
 - proc (for /proc)
 - udev (for /dev)

If you look in /proc/filesystems, you get a list of filesystems your current system supports.

  - ones that say "nodev" are virtual filesystems that don't result in
  storing data to any disk, they generally are ways to access kernel
  data structures or other things

Let's say you attach a USB stick formatted with hfs+ (an older disk format used by MacOS)
 - you can use the same USB storage drivers as for any USB stick
 - but the kernel also needs a filesystem driver that understands hfs+
    - otherwise you can read the raw block-level data but won't
      be able to see it as files


Think of a block device as an array, where each element of the array
is a 4096 byte buffer.
- filesystems are ways of turning this array into a set of files and
  directories
- as you can guess, there are *lots* of ways of doing this, with different tradeoffs
- any OS can in principle read any filesystem with the right data
  - for social and economic rather than technical reasons, Linux is much
    better at reading MacOS and Windows filesystems than the reverse

Roughly speaking, you can use some sets of blocks to store directories, and
the rest for data
  - so the directory blocks would contain filenames and lists of data blocks
    containing the data in the files

But UNIX doesn't do things this way.  It adds a level of indirection.

So you have three kinds of blocks
 - directory blocks
 - data blocks
 - *inode* blocks

 - inodes contain all the metadata of files *except* their name
    - size of file, owner, permissions, timestamps, and pointers
      to the data blocks

 - directories then just list filenames and inode numbers

 - (with symbolic links, the inode contains a filename)

 - inodes enable hard links

 - windows shortcuts are similar to symbolic links (except symbolic links on
   UNIX/Linux work almost everywhere a regular file would)

"mounting" a filesystem means associating an empty directory with a filesystem
 - on boot, the directory / is associated with the "root" filesystem
    (filesystem that is at the base of the filesystem hierarchy, nothing
     to do with the root user)
 - other filesystems are then mounted on directories in this root filesystem

it used to be df would only output a few things
 - we're using a lot more mounts and filesystems than we used to!