Operating Systems 2017F Lecture 18: Difference between revisions
Line 72: | Line 72: | ||
So, on modern systems we update metadata (and sometimes data) by writing sequentially to disk...and then later writing randomly <br> | So, on modern systems we update metadata (and sometimes data) by writing sequentially to disk...and then later writing randomly <br> | ||
* Sequential writes go to the journal <br> | * Sequential writes go to the journal <br> | ||
<br> | |||
On fsck on a journaled filesystem | |||
* just check the journal for pending operations (replaying the journal) <br> | |||
There exists filesytems for optimizing writes that are pure journal<br> | |||
* log-based filesystem <br> | |||
<br> | |||
Logs and journal inherently create multiple copies of data and metadata that are hard to track. This makes deletion nearly impossible (at least to garauntee) <br> | |||
<br> | |||
Only way to garauntee...encrypt everything <br> | |||
* if every file has its own key, you can delete the key and this "delete" the data <br> |
Revision as of 19:02, 16 November 2017
Additional Notes
Lec 18
- More on filesystems
- How can you recover a fs and how do you delete a file?
A filesystem is a:
- Persistent data structure
- Stored in fixed size blocks (at least 512 bytes in size)
- Maps hierarchical filenames to file contents
- Has metadata about files somehow
What's in a filesystem
- data blocks
- metadata blocks, you need someway to find the blocks
How do you organize metadata?
First identify basic characteristics of the filesystem
You need a "superblock" which is a "summary" block that tells you about everything else
Normally the superblock is the first block of the filesystem
In the superblock
- Type of filesystem
- What filesystem magic number is there
- file command to know file type
- What filesystem magic number is there
- Size of the filesystem
- How the filesystem is organized
- Where can I find the rest of the metadata
He opened a .jpg as a binary file to show us the magic number in a file, first several bytes identify type of file. Kernel does not care about file extension. Userspace programs may care about the extension.
POSIX is a standard for UNIX
For POSIX filesystems
- File metadata is stored in INODES
- most have pre-reserved inodes
Usenet is a worldwide distribute discussion system that is deprecated now because it could not handle the spam people uploaded into it, lol. Format for usenet was every message stored in individual file. You will have lots of files!
So we have:
- superblock
- inode blocks
- data blocks
- data blocks for directories
- data blocks for files
- data blocks for directories
How do you recover from damage?
- filesystems never "reboot", must remain correct over the course of years
- Errors will happen: bitrot, accidental corruption, computer failiure/memory corruption/hard reboot
To make filesystems fast, data and meta-data is cached in RAM
- Bad things happen if this data hasn't been writen to disk and you reboot
- Even worse things happen if your RAM is bad and corrupts the data
- FSCK is like scandisk in Windows 98
Also bad...what if you lose the superblock?
- You could lose EVERYTHING
- node trunc dd command blew away first bytes of the file system so you could not mount it because you corrupted the superblock. However, fsck fixed this because we have backup superblocks :D
Old scandisk/fsck was slow because we they had to scan all filesystem metadata
- Not to recover data, but to fix metadata
- lost+found might have some files that you might recover. It is a part of the filesystem for fsck to use.
Nowadays fsck is very fast and we rarely lose data due to losing power
- What this means is we must be writing to disk all the time
- But isn't writing slow? All writes aren't slow particularly on conventional hard disks.
On Magnetic Hard Disks (not SSD's)
- sequential oeprations are fast
- random access is slow
- we have to move the read/write head
- we have to move the read/write head
So, on modern systems we update metadata (and sometimes data) by writing sequentially to disk...and then later writing randomly
- Sequential writes go to the journal
On fsck on a journaled filesystem
- just check the journal for pending operations (replaying the journal)
There exists filesytems for optimizing writes that are pure journal
- log-based filesystem
Logs and journal inherently create multiple copies of data and metadata that are hard to track. This makes deletion nearly impossible (at least to garauntee)
Only way to garauntee...encrypt everything
- if every file has its own key, you can delete the key and this "delete" the data