Okay, so it is.
Let's go ahead and get started then.
Okay, good morning. let's get started.
So I have a lot of stuff I want to talk about today.
We're five permissions for tomorrow. Okay.
Okay, so.
Okay, so I got several questions here that came on during through this stuff below this is why I really want to talk about I will try to get through this stuff, quickly.
So, first off,
some odd man page, explains the bits
for two one, they're here, the explanations on that now tomorrow.
I can talk about more if it's unclear but, yeah, that's pretty straightforward.
Restoring from access let me show you that. So, let's just do it. Okay, so see desktop.
So I've got files here.
So if I run FCS backup.
Yes.
So, it's a win win so this is all making my backup.
Jason thanks within the chat there that's a good little summary there.
Okay.
Yeah.
That's one thing if you're doing the backup.
Try to cancel out. If you notice you're doing a backup and you've got something like this large.
Make sure you Ashley.
Okay.
Is our command to return to modify our normal default there's no such thing as normal default.
You just have to set it to what you want.
So this, I brought it back up. So the question is, what command and I actually run well and bash bash aliases. This is what I ran.
So this is the command here. So, this basically is the command with adding in a you know my summer is my username at CES. So this is exactly what was there so the question is now, how do I reverse it.
So to reverse it. It's just a sign of makeup.
I don't make a restore directory so you override everything. And I just changed this I don't need the exclude.
Because I don't need to exclude anything.
This,
too,
so the leading slashes in the site up leading the trailing slashes and commands to our sink matter because they tell you how to how it handles directories, you can see the man page for the exact thing but do you use it, a slash at the end and basically
means put everything to overwrite that direction so i'm saying is the contents of the comp 3000 vm dash backup on my access account, will we put in restore.
So I did that and now if I looked at restore all my files are there. Now if I wanted to, I could have just run.
I could have done slash homes I screwed it, and that would have over it and everything in my home directory, not sure I'm just going to do that right now.
Notice what it did is, restore is no longer there, but everything else is exactly as it was because I debt so
don't use slash rather than restore because slash would be would overwrite.
You don't want to do this.
That would be bad because that you know what is Ella slashes that. So, it would delete, if you read it as root it would delete everything except and replace just the home directory, you would be bad.
It would be bad. What if our sinks, if arcing skips files that are synced only copies of files that need to be copied. So if it skips a file, it's already there.
Restore is backing up in the opposite direction exactly does this backup include home, other no this backup, only the backup only includes it. So,
the backup only includes home student, it doesn't backup your whole VM, it just backs up the files that were in your home directory. So if you want a backup other or backup the whole VM there but if you backup the whole VM you will make this really big
thing.
So,
so does our sync up a copy files with newer, more reason time scans or modification, with the way the commands are, if you have a new file of the source, and an older file the destination, but so if you have an old file the source and the destination
will overwrite the new file. It's basically in favor of the source is copied to the destination. So, by default, if you don't add in these delete force, it just updates. The destination with the current source, but it doesn't delete anything a destination. That's not the source but if you have to delete that will that will
that will that will eliminate files and force me to move entire sub directories at the destination if they're not in the source so dangerous, which is one reason I did not shoot.
You should be, you know, play with our sake oh by the way if you're playing with arcing of wondering what the heck is going to do use dash end.
Because that will simulate it.
And
like touch.
Okay so who's here. So this command should race food, because I don't have food and my backup
and but food is still here right now. But if I get to the cause of this is just a simulation of the RC, so.
Okay, so.
Okay, So here we go.
Yes, simulator yes it's a dry run. Ok.
Ok.
I can blow me my I can try doing a blog away my VM at the end of class but I need to be on potentially there. So that's our sake so now one of the things we taught SSH agent.
So
she wants to come in, okay when she come in.
We might have a cat visit to come on, you know, I think she's looking at me saying she wants to be fed Roshi.
Are we doing here.
Are we good.
Okay.
Okay, back to where we were.
she ran away.
She doesn't want to say hi.
So SSH agent.
SSH agent or something equivalent is basically running on most major desktop operating systems. So what that means is, So if I add in. So just.
So first of all, this is the agent is what allows me to just log in without it asking for a password because I already unlocked, my local key is protected by a pass phrase.
But now, if I wanted to keep using that I can use the dash capital A, and so now what I should be able to do is,
it does that.
Now, the way this is working, is actually through an environment variable.
So there's a number of SSH agents, I state related things. So if I SSH over here.
You'll notice that here it's telling me.
Oh wow, there's some interesting things here but the base of the authentication sockets here is basically giving information on how to basically where things are so if I now SSH here.
This is all telling you know.
So this you know client is coming in from the VM so it knows this IP address, and the connection
is interesting where it's all like, I'm not sure what this SSH connection.
Thing is,
so you don't actually notice you should not have to normally run as if you're running SSH agent manually, you're probably doing things wrong, because you should be able to on any of your desktops,
just do this and it should list the keys you currently have, I have multiple keys. So it's two it shows them a
check on your desktop and if you're just you sh.
public private key pair. You then add it to your agent, and then you start SSH around as long as everyone has the appropriate public key, and they're authorized keys file you never have to type it a password.
So SSH agent only runs.
We still need to connect with the VPN, I knew.
Okay.
So,
play with sh, sh agent, if you can if you want it's a very useful tool, I can address that later but I've got bigger fish to fry right now.
So let's talk about.
Okay I got the past phrase is not stored in the environment variable. It takes the past phrase, and encrypts the, the private key and then stores it in the memory of the agent.
So if you kill the agent you get rid of the key and that way when you reboot there's no copy of the key around was just in memory.
So anyway, but let's let's go on to other things here so where do we end stuff so supervisor versus user mode.
So that's what we are ending things with last time that the CPU has.
at least two to
three, six.
So.
So what is a ring I mean this is just a terminology of the rings, I mean user mode, it means for particulars, one and two aren't used.
One and two.
Bring wanting to look up multiplex.
Now, the other rings are there they're just not used.
So,
when you switch modes. there's all these registers as all this cash there's all this stuff is all the state that has to be dealt with. So once you switch modes.
It all has to get chucked out, or, or put to the side or side so that's inherently an expensive thing.
Okay so access. So, is access the only thing I'm so
mode.
You things. You can't
discuss okay so. So, what is the job of the colonel what does it really mean.
So, what, why do we have this surprise most of the, the role of the colonel.
So, what the girls doing is that abstracting allocate resources for the programs. Now, we talked about these abstractions what are we talking about what some of the things like
prices get turned into files. So that's one thing we've already seen there.
other is.
So when I say one CPU many CPUs i'm saying is we want to be able to share CPU, the CPU between prophecies, I mean if we go and we look at
me, I've got 326 prophecies.
So on this machine, I have, I have a.
I think I have four cores which, which are the eight threads. So, I can't devote a run it, you know, each process does not get its own core.
So, we have to if you want to run a program.
can multiple cores use different modes yes different, different cores can be in different modes.
Okay so this process of sharing
process scheduling is a generally a big concern in operating systems, is how do you schedule processes between
does know JS work like the CPU I mean,
kinda yeah that I can't quite parse that question right now that's all.
We can talk about that. I'll ask one thread.
kind of, well we'll get to that and we'll get to that in a second. So,
what they are lots to talk about scheduling.
We'll do that later.
So, I will talk about scheduling schedule will be very important for concurrency tutorial which will be next week's tutorial, we will so we'll get to that soon.
But I wanted to go back when we talked about extracting things.
So this is the big thing I want to talk about now and the other big abstraction.
CPU, RAM you basically get they get turned into character or block devices devices get turned into files, your CPU gets turned too many CPUs basically.
we're getting to the character device thing in a, in a second here.
But the other thing I want to talk about here is physical memory to virtual memory. So.
So, if you remember, we looked at the memory locations in a process we tend to see the same values the same range of values may exactly the same values you set arch between different runs and you could have different prophecies and they all have roughly
the same addresses. So,
So process.
Your process access memory using virtual addresses, so.
so.
So, different address.
Not a name spaces and address space.
So, but the addresses are kind of name. So you could say so, a process a process be they're both running if process, he has addressed 2000 process be has addressed 2000, they're completely separate things.
But.
dress.
So when you have ram in your computer and you have a dim car now, built on the CPU package or whatever we have this abstraction called ram where we have these numbers called addresses which allows two things that I say I'm using this virtual addresses
that's what process have.
But we have to have physical memory too because we have to say there is as much RAM. I've got eight gigs I've got 16 gigs of RAM in my computer, which memory location, am I putting stuff in I can access it on a per byte basis.
That's why I use it address. So.
so I'll explain why it's called a page table in a little bit. But, logically every process has its own set of memory, but the asset, go into physical memory.
So we have this abstraction call the page table on every memory access.
So, we're going to talk about the logically, it's a key value pair, but it's a little.
It's a little different. Can you explain why it's fine to choose our products at the same virtual address to 2000. Well, because they're so huge, there every process has its own merit, but we'll talk about the data structure in a little bit but when every
memory access the CPUs got to look this up.
So,
time.
the cache of virtual the physical mappings is a TLB remember a cache is essentially a small amount of memory, which for which we can get higher speed things up by accessing it in some way.
So, this cache is faster to access than the full page table. And so that's what's actually used and it turns out it is very fast, so it barely slows down the CPU and does every memory access.
Can you know can look it up and, and we.
And if the cash is big enough we have to look up things the page table, relatively and frequently and so we don't slow things down.
So, so what is. So when we talk about ym.
So in terms of terminology, it's just a fixed amount of storage for AK that number should sound familiar.
If we're talking about block devices and discs we know we talked a fixed size blocks and RAM we talked about pages.
I'm getting to that here.
So, the size of a page.
Okay so so random.
Is this is the to be a data structure. No, it's a piece of hardware.
It's an actual piece of hardware, it's a which implements an associative map, but it's dedicated hardware and is among the fastest ram on your fastest storage on your CPU because it has to be very fast.
Because otherwise, all your memory axes can be slow.
Okay.
Do you want to learn about.
if you want to.
So, If you want to learn about
how computers work, it's I mean really, this is the low level stuff it's.
So, look that up. I can't explain everything about how CPU works I don't have time for it, But it's actually the.
What's cool about this book is it doesn't act it's actually, it doesn't actually have assembly so much, the types of things at a higher level.
But what I was saying is, so your CPU, your mapping these.
So.
so the mapping of virtual to physical addresses is not every dress is not this address goes to this address this address goes to this address. It's done this page goes to this page.
Okay.
So
when we talk about virtual memory.
Oh, in the last tutorial, we're gonna do some show some kernel stuff of how you access page table so we will talk more about this right now I just need to kind of give the, the picture of this.
and the reason this picture matters, is because if you understand that this virtual to physical mapping is happening, it explains a lot of things, because one of the cool things.
So why do we do this well one thing is.
So, I can just divide memory up needs for AK chunks and I can say okay this process gets these pieces and this one get those and they're just all scattered all over the place and I don't care about the scattering because to a process.
They've all been assembled into a nice regular range of memory because, you know what, if I want to have an array of 100 K, and I get Oh, you can actually have an array of 100 k that's I don't have a range of 100 k that's free.
I have 4k over here I 4k over here I 4k We are 4k here. If you had to manage memory that way you probably go a little nuts.
But here you don't have to think about it because virtual memory assembles all those pieces together and you just see a nice range of continuous memory.
So that helps with, I mean that's a problem called fragmentation. We will talk about memory prior to it but basically minimises.
It minimizes external external fragmentation isn't an issue, you get a little bit of an issue with internal fragmentation because you do only allocate memory and 4k chunks.
What if I only need one bite, sorry you get the rest.
But nowadays that's not so much with big deal, but.
yeah, exactly.
Exactly. This is why, because there are all sorts of different the physical addresses. Exactly.
virtual memory gives us lots of things.
So, that link takes advantage of it.
So we were told there is no limit to virtual memory, um, there is a limit but it's 64 bit address space is I mean you're not going to run into it.
Yeah, lots of this so so
so now I get to talk about a map.
So remember we've been seeing lots of calls to end map.
Specifically, what's getting I can get rid of this.
Let's go to have tutorials seven. Let's go to tutorial six.
So tutorial six. If you notice, right now it's lecture tutorial six.
Yeah, so,
Charles six as an end map.
So, when I do when I do let's say let's do an ls catchphrase us.
So I just done this race here, we all these things. There's all these calls here let's go to top so
there's a bunch of maps here.
Let's start at the beginning here knows LD so cash LD so preload.
You saw that now this stuff being access we had to have that working to make dynamic looking working in our confined environment. But then you'll notice things like these opens IPOs of the cash, and then we have these maps.
Here, but these maps are doing are setting up virtual address space mappings. And you can see now what file descriptor they're doing
so well. So here here's the one that so we're opening Lipsey here.
And then we doing a read check it period. And then we do an M map of file script to three so I guess it wanting to access some bites initially, but then it maps the whole thing into address space, I'm not sure why it doesn't just map out.
Probably because it wants to set different permissions for different parts of memory.
So, this part, deny right here, this part make it executable.
Okay so, so what we're doing here is, every.
We have permissions basically on any range of memory and this is part of how you get segue faults and how we make sure things work properly. His code is loaded in as read an executable data areas are normally Mark read right.
You can't execute code them unless you set the permissions specially because a lot of security attacks can make use of data areas that are executable.
But also it allows us to control which parts of memory are being used. So think about for.
This is virtual memories what allows us to have fork and for fork to actually be fast because you have two processes now where you had one process. These two process was to have a complete copy of memory, but I don't have to copy all that memory.
I can just copy the, the address mappings, which is, which is much less space, and then logically I have two copies and they just mark both of them as read only.
So, if what any process rights to memory. I just got all well that 4k chunk, I better duplicate it.
And now they each have their own distinct copy so logically they still have separate address spaces, but I'm sharing all the memory that I can.
So.
So, when you look at these, These things, and look at the maps.
Look at the flags that are given so this one's protection read this is a private, also.
So we thought multi threaded prophecies, it just means that it, it's almost like most of our process.
Only during the process and a thread really is, are you sharing your memory mappings.
So affiliate do we use virtual memory only for address space, I, what else would it be used for.
I'm not.
Are you seeing something specific.
So when malice call a bunch of em maps, what files are being mapped into the virtual address space, no files are being addressed into the virtual address space there are anonymous ones.
Maybe. So let me or I said that. So, So these are all recency the ones with.
Okay, So now let's.
So.
So like for example this one here.
This map. He's allocating memory, because it says, read right private fixed anonymous. So this is an anonymous map the you knows the file descriptor it gives us minus one, because there is no file, because there's no corresponding file.
mp3 is being reused I mean before it was.
up.
Okay, I'll talk threads, of course, when we run a command in the terminal on the channel for things like these for pupil cases the virtual address space mappings.
Yes.
Forth what you're doing is you're logically duplicating a process, but it doesn't, you don't actually have to duplicate my program because of you can share the virtual memory mappings.
But then you do an exact ve, you actually exactly you really mean throw out the memory map, create a new one, and load that program into it and set initialize it.
What's the direction allocate memory using a map and Matlock now walk is a C function like C library function that's portable between operating systems, a map is a, is a Unix specific operating system call is exactly slower than fork in general, very
much so yes fork is actually quite fast exactly is inherently slow because it has to load a program into memory and initialize it and everything is that he has to a lot of work for because they can rely on virtual memory and all this sharing can just
be like this.
So yeah, so it's it's very lazy so.
So when we run.
When we do a PSA UX.
What's going on, so.
So we get these things we have these numbers here with PSA UX.
So when you see here and you see what's taking up the space here, we have this virtual resident and shared. Now we can explain what these mean virtual is the size of the virtual address space for that process that's how much memory is allocated resident
is how much of it has been loaded into round.
And then here, share, says how much of this is actually shared with another process. So it's not exclusive to another one. So you could say as well this is taking up.
This is a five gig process, oh my god that is that is a piggy virtual address space, but in fact it's only using 646 megabytes of RAM, a physical Ram.
God that's a big difference.
I'm not sure exactly why I wanted me so the virtual space this fake. This is all the RAM is and of the 657 646 megabytes of RAM. Only 100 and 170 of that is shared with other processes.
You know my ex or, you know, hundred megabytes you are shared with it.
My, my sound thing you know, top even it's got a little bit sure but you see it's much smaller, but this is all talking about these mappings.
Okay so, So when you do truncate that's dealing with files.
Not memory similar but very different one in particular, Ram is.
We think about things on disk you talk about things that are persistent things that will stick around. After you reboot. When you're talking about process memory you reboot it all goes away it's volatile
shared memory included in the resident.
I'm not sure.
I think that.
i don't know if
I think so. I think the shared is what's, because it's on disk it's not going to talk about shared so yeah so so shared is how much of the resident memory is actually shared with another process.
So
here's another thing to think about is
so important question to realize when you realize all these mechanisms, how does the colonel know when it doesn't have any more RAM, like how does it make sure of processes Oh, I want to have a gigabyte of RAM.
Can I have it in the kernel, has no doesn't have it available because, so we can kicked into memory so can we set out.
So I actually here's the thing.
as.
Just, just.
Yes.
Yeah.
So.
so when people talk about a swap file or a page file, or things this is true on Windows, Mac OS Linux, it could be an older Linux traditionally used a separate partition for it but now most modern Linux systems tend to use a page file like Windows does.
So if you, if you're racing run out of RAM you need somewhere to put it me if it's one thing if you have a copy of on disk, if you don't have a copy you can write it to disk.
That's what the page file is for.
Then.
so.
yeah.
Yeah, so this is so this is all.
So, let's go make sure it doesn't take too much on can Linux break our computer with less Ram.
Yeah, so,
so.
So, if you once you get to a point where the allocations.
Linux is very optimistic, it's requested they say you want memory or give it and so what will happen is rather than the system to stopping it will tend to get slower and slower, until
until there's you know until basically you go like, yeah, someone will do something and kill off the bad process or otherwise you're just gonna be stuck.
Can there be a list of priority prophecies, yes it's.
It won't kill PhD one because there is. This is a simplification of what actually happens.
Yeah, something's like it's not going to die.
So why is the curl initially check the RAM. The reason why is it doesn't know.
So, think of it this way, when we out how things are so we have a process that uses one gig of virtual dress was like two gigs, two gigs. Grace
Farley Meg of.
So let's so we have a process that use two gigabytes of virtual address space, 500 megabytes of RAM, you know, so we're, you know, and this is on a system with eight gigs of RAM total with one gig of ram free.
So, system
fork.
No problem.
work.
So, the process has a two gig address space. It was only 500 megabytes. It was a memory. Everything's fine. It does a fourth, we're still basically in the same situation.
But now the child starts going whoo let's do stuff. And as it does stuff and now starts us It doesn't even allocate more memory it's just starts using the memory it already has its address space, Maybe you know quite low to kind things we loaded from
disk or Ryan to memory or whatever, all of a sudden now, it could end up using that one gig of ram real you know that one gig real free and it's still in the kernel.
So.
You can impose limits.
You can do this thing called see groups, which will allow you to specify resource groups on things and you can limit how much memory different prophecies get and you can make things all nicer but by default the Linux kernel is just going like, hey, it'll
probably work out, and by being optimistic, I can make things crazy fast because again there's all this sharing of memory it's really non deterministic.
You know, you don't really know exactly what me much of the time will be perfectly fine but then some process changes behavior then all of a sudden you're screwed.
And there's no way for the colonel to predict that and in fact there are classically efforts to try and predict it but none of them are really general.
So if you have a workload, where you need to have.
You need to have guarantees that certain things will interfere with others.
It's on you to do that like for desktop. It doesn't matter because you're there you know some programs as misbehavior you can just kill it.
For an embedded system or maybe a server something you're going to maybe have to have some smarter mechanisms.
Well, it's not it's not so much cheaper and easier make the physical memory large I mean well sometimes it's easier it's just more of that that is
more efficient Oh, one thing I should mention
phones.
Servers nowadays.
So if you look on iOS or Android or something you might notice that, like on Android. A lot of times, background proxies are killed all the time you when they're theoretically supposed to keep running on iOS you switch away from an app and then you switch
back you've noticed, like it's completely reloaded like you're playing a game you switch away from it now the game has to reload and it takes a second.
This is because these operating systems have to be very aggressive and how they manage and basically they're always kind of going like, you aren't meeting that kick you out, neither care, they won't be this, they aren't set to be so optimistic saying,
oh, we'll just let it run, because that also would consume more battery power, and also they have this like don't have swap.
And so they have to always make sure they keep some ram free.
So you can just quit Let's swap. So.
So we talked about pages program paging to disk or something we're just talking about moving the pages of the program from physical memory to disk.
This was from a line swap on phones.
Turns out flash memory.
In general, if you had swap swap can involve writing the same things over and over again to disk.
If you're currently doing them flash it would really potentially wear it out. Also, phones you're wanting to save power. So you want to minimize the amount of extra work you minimize extra work you do so it's just like photo essays are all about processes.
for food.
So if you if you played with Android, iOS and development there you'll notice that processes there any of your programs are always set to always be killed.
You're always specifying state in such a way that when it gets killed it can automatically restore it give you your state and so the user won't realize that the process was destroyed and recreated and the whole reason this whole thing happens is because
RAM is so tight, and they don't want to just write on a desktop we'll just oh so we'll go to dispel this kind of slow down and stuff. Well, that sort of swapping can consume huge amount of CPU resources which would kill your battery and everything so
it's more like a screw it, and also as a flash. No, we're not doing that.
stinking stores yes yeah.
Yeah, so swap is less power efficient a constantly killing random process Yes, definitely.
In general, but again, it all depends on the kind of workload
target for execute So what was it so as well.
Same basic kernels.
Okay,
so it doesn't swap the pages but just the state of the process.
The state of the process of all.
Just as well. It's, as opposed to it is the state this is what when you swap it is the state of the process is just posed to everything. Well, typically won't save all that state will be just kept in memory.
I'm
getting to it.
I don't like to just ask polls asked poses when I have a question for you all. So, we give you if I haven't asked a poll we give you credit for showing up so you should actually probably not encouraged one
phone. Yes.
doesn't swap the pages, so it doesn't want the pages.
Patrick you got a comment, yeah well no I'm see
guys, you're instructing here.
Actually, I do want to ask something here I've got a couple of minutes here is.
Let's do this.
I am kind of curious I, I learned this from, come on here.
We'll stop the share here, and I will
pull.
Let's see.
Okay, here we go.
Let's do this.
I'm now I have now made it to a poll for you All for you to ask.
And I want
three eight or four midterm Wow.
Yeah.
Yeah, two weeks for the underclass I don't. Yeah, that's kind of late.
So I've asked you about tutorials five through seven, and just kind of wondering where you all are. I just kind of wondering where you are I know some of you are a bit behind.
Please do try to catch up. If you are up.
When tutorial six okay.
I didn't say, oh that was stupid.
I didn't, I messed up my pole I meant to ask assignment three separately.
Yeah, there's no thought for a second.
Yeah I messed up the pole. Anyway,
let me end this poll, we asked another one poll.
Okay. Oh,
There we go.
Okay, I'm now launching another tutorial, either poll right now because with the fixed one cuz I really am curious how you guys are doing with this.
Go. And then I'll end up, so she is a former uncover at 78 and T nine, actually it's going to cover.
Tonight, it will cover so much t nine, but but T nine will really kind of reinforce other concepts
tutorial sevens do with assignments for
tutorial seven assignment three at the same time.
They're kind of different.
Yeah assignment Simon three is just for tutorials five and six.
Exactly.
Yeah tutorials go tonight yes tutorial nine will kind of be like the assignment we made last night with a lien on the 277 and eight.