Soma-notes - User contributions [en]

Talk:Untrusted Distributed Storage

2008-11-03T22:23:09Z

Joshua Tessier:

==Group 1==
Pond: 
1) Why use Java? 
2) How was the inner-ring chosen? 
3) How big was the prototype able to scale? 

Farsite: 
1) What is convergence cryptography? 
2) What are the advantages/disadvantages of not locking the directory name of an open file handle? 
3) What assumptions did they make about concurrency in the system and how did they plan to handle it? 

Retro: 
1) Why did this never move beyond a research project? 
2) (in lessons learned) networking turned out to be the limiting factor over disk space. 
Why had they assumed that networking wouldn't be an issue? 
3) What planned goals did they achieve? 

==Group 2==
OceanStore

# What was the purpose of introspection in terms of nomadic data?
# How does the less-reliable-but-faster probabalistic lookup work?
# What is a Bloom filter and how is it used in OceanStore?

FarSite

# Farsite was desgined to look like NTFS. How do Farsite's semantics differ from NTFS?
# How is the content lease system similar to lease systems in distributed systems we've already seen, and which is most similar?
# What is the scope of Farsite? Could it work as a world wide filesystem like OceanStore.

Retro

# How did the lease system change between planning and implementation?
# What was the programming model used in the implementation of Farsite?
# What was the biggest disadvantage to the implementation?

==Group 3 - Farsite==

OceanStore

1) What is convergence cryptography?

Ans. The file key is used to encrypt the hashes rather than to encrypt the file blocks directly.

2) What are the advantages/disadvantages of not locking the directory name of an open file handle?

Ans. Advantages - The results of directory rename operations are not propagated synchronously
to all descendent directory groups during the rename operation, because this would unacceptably
retard the rename operation, particularly for directories near the root of the namespace tree.

Disadvantages- because they used lazy propagation, other users wouldn’t see the name immediately, also
more then one user can change the name at the same time.

3) What assumptions did they make about concurrency in the system and how did they plan to handle it?

Ans. The authors assume that no files are both read by many users and also frequently updated by at least one user.
How did they handle it? There are four classes of leases in Farsite: content leases, name leases,mode leases, and
access leases.

Retro

1)Whats is the main target environment for farsite?

Ans. The target was governments and universities environments

2) What are the 3 different type of certificates? And what are their purposes?

Ans. Namespace certificates – associated the root of a file system namespace with a set of machines that manage the root metadata.

User certificates - associates a user with his personal public key, so that the user identity can be validated for access control.

Machine certificates - associates a machine with its own public key, which is used for establishing the validity of the machine as a physically unique resource

3) What is convergence encryption?

Ans. The file key is used to encrypt the hashes rather than to encrypt the file blocks directly.

Ponds

1) Farsite was designed to look like NTFS. How do Farsite’s semantics differ from NTFS?

2) How is the content lease system similar to lease systems in distributed systems we’ve already seen and which is most similar.

3) What is the scope of Farsite? Could it work as a World Wide file system like OceanStore.

Ans. Farsites main scope was that of a univertisy, governement or large company. Analysis points to a scale of approximately 10^5 machines whereas OceanStore is around 10^10

==Group 4 - Farsite Retrospective==
Some questions were received by paper so they are put here as well as "FROM <GROUP>"

TO OceanStore
# What is their business model?
# What is introspective and what are its many applications?¬
# What are the advantages of using a “Version Control System” over a typical file system model?

TO Pond
# What is Tapestry and how does it work?
# How did they meet the expectations of the original OceanStore paper and vision? (i.e. how did they solve design problems)
# What is the difference between the primary and secondary replicas?

TO FARSITE
# What was the target environment for FARSITE?
# What were the three different types of certificates and what were their purposes?
# What is convergent encryption?

FROM Pond

# How did the lease system change between planning and implementation?
# What are the programming models used in the implementation?
# What was the biggest disadvantage to their implementation?

From FARSITE(?)

# What are the trends in technology that justify FARSITE?
# Were the team members familiar with windows SIS driver?
# What's WebDAV?

Talk:Distributed Shared Memory

2008-09-24T01:52:58Z

Joshua Tessier: /* Difference between DSM and NUMA? */

Here is the page where we will be discussing the DSM readings.

== IVY ==
[[User:Soma|Anil]]: What were the key characteristics of IVY? What exactly did Kai Li build?

[[user:Alireza|Alireza]] : IVY was a software based DSM system that's been developed to allow users share their local memories in a distributed manner. IVY was designed to be used in loosely coupled environments. It had five main modules including memory allocation, process management, initialization, remote operation and memory mapping. The main advantage of IVY was gaining performance in parallel applications comparing.

[[user:Alireza|Alireza]] :(Question) Name some of the applications that you would think benefit from using IVY environment? Distributed Database system is the one that is mentioned in the dissertation. Thinks of something different.

[[user:Azalia|Azalia]]:(Answer) Some of the current sample applications can be (CRM) Customer Relationship Management or (ERP) Enterprise Resource Planning applications that serve multiple users across an organization.Another example, imagine a billing system that has to calculate the telephone bills of thousands of customers can benefit from this environment by calculating the bill of multiple customers at the same time on different machines distributed across the network. Even though, each customer bill calculation can be done separately, using a shared memory space for reading input values like cost per minute, or cost per text message can be very useful. In addition, since each customer bill is a separate object the write operation is done in different pages of the shared memory and even using multiple writer algorithm, in this case, does not introduce any concurrency issue.

== Current DSM systems? ==

[[User:soma|Anil]]: What is a current production system that uses distributed shared memory? What about the underlying problem makes DSM a good technological fit?

[[user:Azalia|Azalia]]:(Answer) What is a current production system that uses distributed shared memory?
Any application with complex independent steps that can be parallelized would be suitable for DSM environment. Some of the current sample applications can be (CRM) Customer Relationship Management or (ERP) Enterprise Resource Planning applications that serve multiple users across an organization.
What about the underlying problem makes DSM a good technological fit?
Apart from DSM there are alternative methods for using in distributed environment (e.g. RPC and message passing), they have some inadequacies that DSM has been introduced to address them. For instance, message passing and RPC, have difficulties in sending complex data structures and pointers over the network due to different memory address spaces. The distributed shared memory can be a solution of this problem since all the processors share the same memory address space. In addition, if we consider current RPC technologies like Web Services, we'll realize that for each task we have to pack and send a lot of XML data around. With DSM we can share a memory space and prevent overloading network by sending XML messages.

== Difference between DSM and NUMA? ==

[[User:Soma|Anil]] What are the differences between DSM and NUMA? Under what circumstances are each appropriate?

[[User:Alirez|Alireza]]: NUMA follows SMP paradigm where there is common memory bus for accessing shared memories. In addition, one of the most important aspects of the NUMA is that it provides different access time for the processers based on their locations. For instance local processors can have faster access to local memories. In addition, NUMA access to memory is hardware based.

[[User: Joshua Tessier|Joshua Tessier]]: Correct me if I'm wrong but NUMA is basically a type of DSM. In a NUMA system, each processor has access to a common memory, however this common memory is distributed across each of the processors. For example, if there are 8 processors, the total memory is divided into 8 sections. As stated above, the processors have different access times to memory stores. Meanwhile, DSM is just dynamic shared memory; not a specific type like NUMA.

== DSM Implementations? ==
[[User:Azalia|Azalia]]:(Question) What are the different types of DSM Implementations?

[[User:Ywahyudi|Yohan]]:(Answer) There are 3 different types of DSM implementation. The first one is Software-level implementation which can be achieved in user-level, run-time library routine, the OS, or the programming language, for example IVY, Mermaid, Munin, etc. The second one is Hardware-level implementation which ensures that automatic replication of shared data in local memories and processors caches, transparently for software-layer, for example Memnet, Dash, SCI KSR1, etc. Since software is used in hardware support to optimize memory reference, and hardware is used in software solution such as virtual memory management, then the third one is Hybrid Level Implementation which is a combination of both implementation. Several examples of such implementation are Plus, Galactica Net, Alewife, etc.

[[User:Joshua Tessier|Joshua Tessier]]:(Question) Does the hybrid solution hold much relevance today? From what I got in the paper, it came to light due to some limitations of the hardware/OS layers at the time. Today, we have a ton of different tools at our disposal and these limitations are no longer present. How would such a solution be divided today?

Talk:Distributed Shared Memory

2008-09-24T01:36:07Z

Joshua Tessier: /* DSM Implementations? */

Here is the page where we will be discussing the DSM readings.

== IVY ==
[[User:Soma|Anil]]: What were the key characteristics of IVY? What exactly did Kai Li build?

[[user:Alireza|Alireza]] : IVY was a software based DSM system that's been developed to allow users share their local memories in a distributed manner. IVY was designed to be used in loosely coupled environments. It had five main modules including memory allocation, process management, initialization, remote operation and memory mapping. The main advantage of IVY was gaining performance in parallel applications comparing.

[[user:Alireza|Alireza]] :(Question) Name some of the applications that you would think benefit from using IVY environment? Distributed Database system is the one that is mentioned in the dissertation. Thinks of something different.

[[user:Azalia|Azalia]]:(Answer) Some of the current sample applications can be (CRM) Customer Relationship Management or (ERP) Enterprise Resource Planning applications that serve multiple users across an organization.Another example, imagine a billing system that has to calculate the telephone bills of thousands of customers can benefit from this environment by calculating the bill of multiple customers at the same time on different machines distributed across the network. Even though, each customer bill calculation can be done separately, using a shared memory space for reading input values like cost per minute, or cost per text message can be very useful. In addition, since each customer bill is a separate object the write operation is done in different pages of the shared memory and even using multiple writer algorithm, in this case, does not introduce any concurrency issue.

== Current DSM systems? ==

[[User:soma|Anil]]: What is a current production system that uses distributed shared memory? What about the underlying problem makes DSM a good technological fit?

[[user:Azalia|Azalia]]:(Answer) What is a current production system that uses distributed shared memory?
Any application with complex independent steps that can be parallelized would be suitable for DSM environment. Some of the current sample applications can be (CRM) Customer Relationship Management or (ERP) Enterprise Resource Planning applications that serve multiple users across an organization.
What about the underlying problem makes DSM a good technological fit?
Apart from DSM there are alternative methods for using in distributed environment (e.g. RPC and message passing), they have some inadequacies that DSM has been introduced to address them. For instance, message passing and RPC, have difficulties in sending complex data structures and pointers over the network due to different memory address spaces. The distributed shared memory can be a solution of this problem since all the processors share the same memory address space. In addition, if we consider current RPC technologies like Web Services, we'll realize that for each task we have to pack and send a lot of XML data around. With DSM we can share a memory space and prevent overloading network by sending XML messages.

== Difference between DSM and NUMA? ==

[[User:Soma|Anil]] What are the differences between DSM and NUMA? Under what circumstances are each appropriate?

[[User:Alirez|Alireza]]: NUMA follows SMP paradigm where there is common memory bus for accessing shared memories. In addition, one of the most important aspects of the NUMA is that it provides different access time for the processers based on their locations. For instance local processors can have faster access to local memories. In addition, NUMA access to memory is hardware based.

== DSM Implementations? ==
[[User:Azalia|Azalia]]:(Question) What are the different types of DSM Implementations?

[[User:Ywahyudi|Yohan]]:(Answer) There are 3 different types of DSM implementation. The first one is Software-level implementation which can be achieved in user-level, run-time library routine, the OS, or the programming language, for example IVY, Mermaid, Munin, etc. The second one is Hardware-level implementation which ensures that automatic replication of shared data in local memories and processors caches, transparently for software-layer, for example Memnet, Dash, SCI KSR1, etc. Since software is used in hardware support to optimize memory reference, and hardware is used in software solution such as virtual memory management, then the third one is Hybrid Level Implementation which is a combination of both implementation. Several examples of such implementation are Plus, Galactica Net, Alewife, etc.

[[User:Joshua Tessier|Joshua Tessier]]:(Question) Does the hybrid solution hold much relevance today? From what I got in the paper, it came to light due to some limitations of the hardware/OS layers at the time. Today, we have a ton of different tools at our disposal and these limitations are no longer present. How would such a solution be divided today?

Talk:Distributed Shared Memory

2008-09-24T01:35:33Z

Joshua Tessier: /* DSM Implementations? */

Here is the page where we will be discussing the DSM readings.

== IVY ==
[[User:Soma|Anil]]: What were the key characteristics of IVY? What exactly did Kai Li build?

[[user:Alireza|Alireza]] : IVY was a software based DSM system that's been developed to allow users share their local memories in a distributed manner. IVY was designed to be used in loosely coupled environments. It had five main modules including memory allocation, process management, initialization, remote operation and memory mapping. The main advantage of IVY was gaining performance in parallel applications comparing.

[[user:Alireza|Alireza]] :(Question) Name some of the applications that you would think benefit from using IVY environment? Distributed Database system is the one that is mentioned in the dissertation. Thinks of something different.

[[user:Azalia|Azalia]]:(Answer) Some of the current sample applications can be (CRM) Customer Relationship Management or (ERP) Enterprise Resource Planning applications that serve multiple users across an organization.Another example, imagine a billing system that has to calculate the telephone bills of thousands of customers can benefit from this environment by calculating the bill of multiple customers at the same time on different machines distributed across the network. Even though, each customer bill calculation can be done separately, using a shared memory space for reading input values like cost per minute, or cost per text message can be very useful. In addition, since each customer bill is a separate object the write operation is done in different pages of the shared memory and even using multiple writer algorithm, in this case, does not introduce any concurrency issue.

== Current DSM systems? ==

[[User:soma|Anil]]: What is a current production system that uses distributed shared memory? What about the underlying problem makes DSM a good technological fit?

[[user:Azalia|Azalia]]:(Answer) What is a current production system that uses distributed shared memory?
Any application with complex independent steps that can be parallelized would be suitable for DSM environment. Some of the current sample applications can be (CRM) Customer Relationship Management or (ERP) Enterprise Resource Planning applications that serve multiple users across an organization.
What about the underlying problem makes DSM a good technological fit?
Apart from DSM there are alternative methods for using in distributed environment (e.g. RPC and message passing), they have some inadequacies that DSM has been introduced to address them. For instance, message passing and RPC, have difficulties in sending complex data structures and pointers over the network due to different memory address spaces. The distributed shared memory can be a solution of this problem since all the processors share the same memory address space. In addition, if we consider current RPC technologies like Web Services, we'll realize that for each task we have to pack and send a lot of XML data around. With DSM we can share a memory space and prevent overloading network by sending XML messages.

== Difference between DSM and NUMA? ==

[[User:Soma|Anil]] What are the differences between DSM and NUMA? Under what circumstances are each appropriate?

[[User:Alirez|Alireza]]: NUMA follows SMP paradigm where there is common memory bus for accessing shared memories. In addition, one of the most important aspects of the NUMA is that it provides different access time for the processers based on their locations. For instance local processors can have faster access to local memories. In addition, NUMA access to memory is hardware based.

== DSM Implementations? ==
[[User:Azalia|Azalia]]:(Question) What are the different types of DSM Implementations?

[[User:Ywahyudi|Yohan]]:(Answer) There are 3 different types of DSM implementation. The first one is Software-level implementation which can be achieved in user-level, run-time library routine, the OS, or the programming language, for example IVY, Mermaid, Munin, etc. The second one is Hardware-level implementation which ensures that automatic replication of shared data in local memories and processors caches, transparently for software-layer, for example Memnet, Dash, SCI KSR1, etc. Since software is used in hardware support to optimize memory reference, and hardware is used in software solution such as virtual memory management, then the third one is Hybrid Level Implementation which is a combination of both implementation. Several examples of such implementation are Plus, Galactica Net, Alewife, etc.

[[User:Joshua Tessier]]:(Question) Does the hybrid solution hold much relevance today? From what I got in the paper, it came to light due to some limitations of the hardware/OS layers at the time. Today, we have a ton of different tools at our disposal and these limitations are no longer present. How would such a solution be divided today?

Distributed Shared Memory

2008-09-24T01:32:09Z

Joshua Tessier: /* Questions */

==Readings==

There will be no formal class this week. Instead, please read the following:

* [http://homeostasis.scs.carleton.ca/~soma/distos/fall2008/kai-li-shared.pdf Kai Li, ''Shared virtual memory on loosely coupled multiprocessors'' (1986)]. This thesis is the first major work on distributed shared memory. You don't need to follow all of the technical details in this thesis; instead, try to understand Li's framing of the DSM problem and the thread of his argument.

* [http://homeostasis.scs.carleton.ca/~soma/distos/2008-02-11/protic-overview.pdf Jelica Protic et al., "Distributed Shared Memory: Concepts and Systems" (1996)] This paper reviews later work in the area of distributed shared memory. Read it to understand what issues dominated later research into DSM.

==Questions==

* What is the appeal of DSM?
* How is DSM related to virtual memory?
* How do concurrency issues affect the design of DSM systems?
* What sort of programs will benefit from DSM? What do not?
* What were the major problems addressed in DSM research following Kai Li's dissertation?
* Did these advances change the scope of environments and problems appropriate for DSM?
* Why aren't DSM systems commonly in use today?

==Online Discussion==

There is no lecture this week; however, there is a reading response and discussion!

* Please submit your response by Tuesday night (Sept. 23rd). You may submit by WebCT under "Response 2". Or, if you do not have access to WebCT, you may submit your assignment via email at soma at ccsl.carleton.ca, subject line "Response 2".

* We will have an online discussion of the readings through Friday, 5 PM, via the discussion for this wiki page. Everyone should plan to make at least one edit here. Ideally, you should all post at least one question and one response. Your group presentation grade will be based on your participation in this discussion. Note that you can sign posts by using four tilde's. You don't need to use your full name; however, if you use an alias please email me privately with your wiki username.

* Note: I have sent a test email to everyone via Carleton Connect. If you did not get an email, please let me know. Thanks.

Have fun this week, see you online and then Sept. 29th in class!

--[[User:Soma|Anil]]

Distributed Shared Memory

2008-09-22T20:01:44Z

Joshua Tessier: /* Questions */

==Readings==

There will be no formal class this week. Instead, please read the following:

* [http://homeostasis.scs.carleton.ca/~soma/distos/fall2008/kai-li-shared.pdf Kai Li, ''Shared virtual memory on loosely coupled multiprocessors'' (1986)]. This thesis is the first major work on distributed shared memory. You don't need to follow all of the technical details in this thesis; instead, try to understand Li's framing of the DSM problem and the thread of his argument.

* [http://homeostasis.scs.carleton.ca/~soma/distos/2008-02-11/protic-overview.pdf Jelica Protic et al., "Distributed Shared Memory: Concepts and Systems" (1996)] This paper reviews later work in the area of distributed shared memory. Read it to understand what issues dominated later research into DSM.

==Questions==

* What is the appeal of DSM?
* How is DSM related to virtual memory?
* How do concurrency issues affect the design of DSM systems?
* What sort of programs will benefit from DSM? What do not?
* What were the major problems addressed in DSM research following Kai Li's dissertation?
* Did these advances change the scope of environments and problems appropriate for DSM?
* Why aren't DSM systems commonly in use today?

* It doesn't seem to be discussed, but how do we secure sensitive information that is being stored on a remote machine? It seems sort of pointless if you're going to end up in situations where you're stuck with having potentially insecure pieces of code run on some dude's machine...

==Online Discussion==

There is no lecture this week; however, there is a reading response and discussion!

* Please submit your response by Tuesday night (Sept. 23rd). You may submit by WebCT under "Response 2". Or, if you do not have access to WebCT, you may submit your assignment via email at soma at ccsl.carleton.ca, subject line "Response 2".

* We will have an online discussion of the readings through Friday, 5 PM, via the discussion for this wiki page. Everyone should plan to make at least one edit here. Ideally, you should all post at least one question and one response. Your group presentation grade will be based on your participation in this discussion. Note that you can sign posts by using four tilde's. You don't need to use your full name; however, if you use an alias please email me privately with your wiki username.

* Note: I have sent a test email to everyone via Carleton Connect. If you did not get an email, please let me know. Thanks.

Have fun this week, see you online and then Sept. 29th in class!

--[[User:Soma|Anil]]