Soma-notes - User contributions [en]

DistOS-2011W Reputation

2011-04-09T16:54:30Z

Wahmed2: /* Issues & Solutions */

==Members==
* Waheed Ahmed
* Trevor Gelowsky
** MSN: Gelowt@gmail.com
** E-Mail: tgelowsk@sce.carleton.ca
* Michael Du Plessis
* Nicolas Lessard (nick.lessard @t gmail.com / nlessard @t carleton.connect.ca)

==Our presentation==
Our current presentation can be viewed at the following link: https://docs.google.com/present/edit?id=0ASS7kj9hfc1aZGRiMjMzOHJfNGhnNzhuamRr&hl=en&authkey=CMHi3KAD

==Our Paper==

Our final paper can be found here: [[Distributed OS: Winter 2011 Reputation Systems Paper]]

==The problem==
* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
* Where do you store the data?
* Where is the data queried from?
* What defines good/bad reputation?
* Who provides the good/bad reputation?
* Who do we trust for this information?
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
* What entities are able to contribute to reputations?
* How do we access reputation about entities?
* Who is authorized to access particular reputations? How much to reveal? (Information flow)

==What technologies currently exist?==
* Digital signatures
** Certificates signed by trusted organizations

* Black hole- email, spam,
* Google - search reputation
* Credit bureaus
* Yellow pages
* Better business bureau
* CRC - criminal records

== What technologies don't currently exist?==

==Guaranteeing Authenticity/Public Key Infrastructure==

In our paper we must explain why PKI/Authentication fits into reputation. Why must it be handled by both Attribution and Reputation systems?

===Problem Domain===

This portion of a reputation system answers the core question of how reputation information being exchanged is guaranteed to be authentic.

*How do we ensure the information exchanged between peers authentic, and not tampered with?

*How to we attribute information exchanged?

*How do we do this in a highly decentralized, distributed system?

*How can we make sure the information is timely?

===Introduction===
In past few years, Internet has provided platform for a global market place and both business and private users realizes that the revolutionary communications opportunities provided by it will give way to large spectrum of business and private applications.Today online users face multitude of problems and issues like vulnerability to viruses , worms , exposure to sniffers, spoofing their private sessions not only this but also they are also subjected to invasion of privacy with multitude of spy ware available for monitoring how they behave. Today over the internet different kind of activities take place ranging from access to information to entertainment, financial services, product services and even socializing. The frequent usage of internet as an important business tool led to a major increase in deliberate abuse and criminal activities. All the organization operating electronically and trading expose their own information and IT systems to a wide range of security threats. The most common protocols like IP/TCP/UDP are the main targets of potential hackers. Its all because of IPs on which attacks are possible and don't have proper authentication mechanism for any incoming data over internet.

In order to build secure chain of trust Public-Key Infrastructure is used for internet based communication. It consists of various things like security policy , Certificate authority , registration authority , certificate distribution system PKI enabled applications.

===PKI===
With development of modern e-commerce based businesses which has minimal customer face-to-face interactions is demanding more security and integrity. The online web based stores where huge amount of transactions take place needs to ensure customers that there information is confidential and processed through a secure channel. This is where implementation of PKI steps in to provide mechanisms to ensure trusted relationships are established and maintained. The specific security functions in which a PKI can provide foundation are confidentiality, integrity, non-repudiation,and authentication.

PKI provides mean of guaranteeing the authenticity by issuing digital signature. In order to ensure that electronic document is authentic , which means knowing the person who created the document and it hasn't been modified. They are commonly used for software distribution, financial transaction or to detect forgery or tampering. Further to ensure the authentication digital signature relies on certain type encryption. Digital certificate is mode of encryption on large scale for example secure web server. Digital certificate are validated by PKI by verifying the authenticity of certificate , the validity of certificate and that the certificate is trustworthy. They are issued by an authority referred as Certification Authority. The certificate authority act as the middleman that both computer trust. This helps in avoiding man in the middle attack , certificate authority confirms that computer is in fact who they say they are and the provides the public key of each computer to the other. The digital certificate contains the public key of the entity identified in certificate , it matches a public to a particular individual, and that certificate's authenticity is guaranteed by the issuer, thus digital certificate provides solution to the problem of how to find user's public key and know that its valid.These problems are solved by a user obtaining another user's public key from the digital certificate. The user knows it is valid because a trusted certification authority has issued the certificate. For their own authentication digital certificate rely on public key cryptography. The certification authority signs the certificate with its own private as digital certificate is issued. To validate the authenticity of a digital certificate, a user can obtain that certification authority's public key and use it against the certificate to determine if it was signed by the certification authority.

===Issues Faced by DoD using PKI===
I found out there are many different implementations of PKI , and they all focuses on their own issues and solutions. For example PKI used in DoD have following issues

*Lack of PKI-enabled eCommerce applications and lack of interoperability among PKI applications

*DoD is developing a single high assurance PKI

*Very High Cost Impact to the EC/EB community.

*The PKI community lacks metrics for mapping of trust models between the DoD :”high assurance” C2 and EC/EB domains

*Education of everyone (policy maker through user) to a common level of understanding is a huge challenge.

*While the purpose of using PKI in EC/EB is to provide additional trust to allow the Internet to serve as a vehicle for legally binding transactions , problems still exist with the methodologies associated with establishing a long-term burden of proof. Specifically, there are no widely adopted industry standards for maintenance of electronic signatures or for authenticated timestamps for record maintenance that have stood the test of time. These processes are untried and the case law has not yet been established to convince users that there are no issues with enforcement of these new processes. An additional barrier to EC/EB within this space is the current DoD Certificate policy in which DoD accepts

===Common Issues With PKI Implementation===

*Commercial Off-The-Shelf (COTS) versus Customised applications : The choice between COTS or customised products is usually one of cost versus usability. In case of usability the thing to be focused should be error messages. If PKI is built int o applications (transparent to users) than its fine if not than user will require to have some understanding of the use of keys, certificates, Certificate Revocation Lists (CRLs) and directories/certificate repositories so that they can make informed decisions.

*Token Logistics (smart card): The point where keys and certificates are linked to their owner is a very critical point in a PKI. If a fraudulent certificate is issued by a registration officer and the certificate holder uses the certificate to commit a crime or prank, trust in the whole PKI hierarchy may be lost. The physical security requirements are high, and the registration officer, whether a person or a smartcard bureau, must be subject to strict security polices and practices. As it was problem with DoD mentioned in section above.

*Network issues - Traffic : There is no doubt that the implementation of PKI will add to the network load, although just how much depends on the system architecture. Potential additional traffic that should be considered includes: Certificate issuance, Email usage, CRLs , and Directory Replication

*Network issues - Encryption : Many organisations implement anti-virus software and content inspection on servers at the perimeter of their networks. Some have security policies that rejects or quarantines encrypted traffic. To provide user-to-user confidentiality, messages will traverse networks with their payload hidden from inspection by virus and content checking.

*Email address in certificate :In order to use certificates for S/MIME signed/encrypted email, the users’ email address must be in the certificate. Most people change their email addresses more frequently than the certificate. Unless a solution is built which allows users to keep the same email address over a long period, certificates would have to be re-issued every time a user changes email address. S/MIME v.3 stipulates that the receiving application must check the From: or Sender: field in the mail header and compare it to an email address in the sender’s certificate. If the check does not match, the mail application should perform another explicit check to ensure that the person who signed the message is indeed the person who sent it. As usual, the ‘devil is in the detail’ when it comes to implementation.

*Certificate Validity Checking:CRLs have been the conventional method of providing certificate validity checking. CRLs do not scale very well as discussed earlier, but are usually kept for backward compatibility, archiving/historical verification and for use in off-line mode. The other issue with CRLs is that they are generally issued at certain intervals of 6, 12 or 24 hours, causing a time lag from the time a certificate is revoked until it appears on the published CRL. This may present a security risk, as a certificate may verify correctly after it has been reported as compromised and revoked; (however some would argue that the time from actual compromise until the discovery and reporting of it would in most cases be a more significant lag). The Online Certificate Status Protocol (OCSP) (RFC2560) allows a client to query an OCSP responder for the current status of a certificate. This saves searching through a large CRL and can save bandwidth if the CRL would normally be downloaded - although it may increase network traffic. Most OCSP responders are based on CRLs and thus do not solve the problem of time lag as outlined above.

*Availability and storage of reliable user information : For an identity certificate scheme, names in certificates need to be unique, meaningful - and correct. Few large user communities have all their member details in a central and accurate database or directory, and the exercise of consolidating, checking and updating all user data can turn into a massive and expensive exercise.

*Archiving/historic verification : Digital signatures need to be verifiable even after the keys used to sign have expired. Likewise, we need to be able to verify that the certificate was valid at the time the datawas signed. This means we would need to archive: the signed file,the public key certificate of the signer, the CRL that was valid at the time of signing, a reliable timestamp to prove the accuracy of the time of signing and, the hardware environment that can run the software that was used at the time

===What are the solution to these problems?===

*Identity Based Encryption:enables senders to encrypt messages for recipients w/o requiring that a recipients key first be established, certified, and published.

*Certificate-based encryption:it incorporates IBE methods, but uses double encryption so that its CA cant decrypt on behalf of the user.

*Certificateless Public Key Cryptography:it incorporates IBE methods, using partial private keys so PKG cant decrypt on behalf of the user.

*Distributed Computation:There exists methods that distribute cryptographic operations so that the cooperative contribution of a number of entities is required in order to perform an operation such as a signature or decryption. It helps in tighter protection at servers vs clients, but implies that the users mist fully trust servers to apply keys appropriately.

*Alternative Validation Strategies:Hash tree:it offers compact protected representations of the status of large number of certiticates.Highly valued if PKI is operated large scale more benifical than Certificate Revovation List. CRL reflect ststus information at fixed intervals.

==Dissemination==

===The Problem Domain===

===Random Ramblings on Reputation Management and Distribution===

Publish/Subscribe?

This system has unique distribution requirements as compared to most distributed systems in general. In this system, we cannot assume that there will be a universally agreed-upon definition of good, or bad. Similarly, the system must be self-policing. It would be up to each and every group of autonomous systems to decide which updates to accept and reject. Updates themselves also should not cause the network to DDoS itself. Lastly, it would be impossible for every system to know what the reputation for a given system is. Therefore the system must disseminate information in some way that is query-able and localizes reputation information where required.

To this end, we need a way of spreading information that while reliable, does not depend on one universally agreed-upon set of reputations.

For example, on an internet-scale operating system it would be entirely reasonable for one group of systems to not want to accept updates, or want to avoid communication with a given series of systems.

Any solution would assume that the problems of attribution are solved.

===Current Examples of Reputation Dissemination===

The first protocol that immediately comes to mind in this situation is a gossip-based protocol. These protocols are designed to operate in highly decentralized, large-scale systems.

Here's a nice overview:
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4537308 "Reputation management in distributed systems"

Examples are as follows:
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4228013 "Gossip-based Reputation Aggregation for Unstructured Peer-to-Peer Networks"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=5569965 "Improving Accuracy and Coverage in an Internet-Deployed Reputation Mechanism"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4459326 "GossipTrust for Fast Reputation Aggregation in Peer-to-Peer Networks"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4777496 "Adaptive trust management in P2P networks using gossip protocol"

Another possibility is using "Reputation chains"
* http://dx.doi.org.proxy.library.carleton.ca/10.1109/TKDE.2009.45 "P2P Reputation Management Using Distributed Identities and Decentralized Recommendation Chains"

==Maintaining History==

===Problem domain===

* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
*** Do we maintain records based on a fixed set of imposed rules? Or do we build rules as the system emerges and reputations are formed?
* Where do you store the data?
** Distributed storage systems. Reputation in real-life is stored in interactions that an entity has with others. Reputation is not stored centrally. Reputation is most often a shared view of an entity by the masses, but sometimes an entities reputation can be disjoint among the masses: many different entities having differing views of reputation for the same entity.
* Where is the data queried from?
** (should I mention this?)
* What defines good/bad reputation?
** (should I mention this?)
* Who provides the good/bad reputation?
** Impose/Emerge problem: reputation for an interaction can be calculated immediately or can be a function of time.
* Who do we trust for this information?
** Trusting the masses is generally a good way of ensuring trustworthiness. Imposed rules will not always fit every situation well - could potentially set bad reputation to a "good" entity.
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
** Do we maintain an ever-growing set of history items for interactions between entities? Do we look focus on the bad reputations?
* What entities are able to contribute to reputations?
* How do we access reputation about entities?
* Who is authorized to access particular reputations? How much to reveal? (Information flow)
* What assumptions will we make?
* Privacy issues? What will we reveal? Will centralized systems have a know-all mentality?
** Fine grained information will never be revealed (privacy concerns and user rights)
* Which history should I maintain? What to take as important, what to disregard?
* Immutable data structure. Who could add data? Who could remove data? Authority

===Reputation systems===
* record, aggregate, distribute information about an entity's behaviour in distributed applications

* reputation might be based on the entity's past ability to adhere to a license agreement (mutual contract between issuer and licensee)

===History-based access control systems===
* make decision based on an entity's past security-sensitive actions

===Examples of reputation systems (trust-informing technologies)===
* eBay - Feedback forum (positive, neutral, negative)

===Do reputation systems have some validity?===

Resnick et al. argue that reputation systems
foster an incentive for principals to well-behave because of “the expectation of
reciprocity or retaliation in future interactions

Abstractions are used to model the aggregated information of each entity. These abstractions may not encompass the full details of transactions and provide context to specific issues relating to feedback. In turn we can end up with ambiguous values.

So we need a system that provides sufficient information in order to verify the precise properties of a past behaviour.

* Krukow, K. A Logical Framework for Reputation Systems and History-based Access Control. School of Electronics and Computer Science University of Southampton, UK. (March 3, 2011) [http://www.brics.dk/~krukow/research/publications/online_papers/concrete-jcs.pdf]

====Abstract====
Reputation systems are meta systems that record, aggregate and distribute information about principals’ behaviour in distributed applications. Similarly, history-based access control systems make decisions based
on programs’ past security-sensitive actions. While the applications are
distinct, the two types of systems are fundamentally making decisions
based on information about the past behaviour of an entity.
A logical policy-centric framework for such behaviour-based decisionmaking is presented. In the framework, principals specify policies which
state precise requirements on the past behaviour of other principals that
must be fulﬁlled in order for interaction to take place. The framework consists of a formal model of behaviour, based on event structures; a declarative logical language for specifying properties of past behaviour; and
eﬃcient dynamic algorithms for checking whether a particular behaviour
satisﬁes a property from the language. It is shown how the framework can
be extended in several ways, most notably to encompass parameterized
events and quantiﬁcation over parameters. In an extended application, it
is illustrated how the framework can be applied for dynamic history-based
access control for safe execution of unknown and untrusted programs.

* Khosrow-Pour, M. Emerging trends and challenges in information technology management (March 7, 2011) [http://books.google.ca/books?id=ybzS-yylJfAC&lpg=PA822&ots=V7hn_RzqXA&dq=maintaining%20history%20in%20reputation%20systems&pg=PA822#v=onepage&q=maintaining%20history%20in%20reputation%20systems&f=false]

====Abstract====

* Bolton, G. et al. How Effective are Electronic Reputation Mechanisms? (March 10, 2011) [http://ccs.mit.edu/dell/reputation/BKOMSsub.pdf]

====Abstract====

Electronic reputation or “feedback” mechanisms aim to mitigate the moral hazard problems
associated with exchange among strangers by providing the type of information available in
more traditional close-knit groups, where members are frequently involved in one another’s
dealings. In this paper, we compare trading in a market with electronic feedback (as
implemented by many Internet markets) to a market without, as well as to a market in which the
same people interact with one another repeatedly (partners market). We find that, while the
feedback mechanism induces quite a substantial improvement in transaction efficiency, it also
exhibits a kind of public goods problem in that, unlike the partners market, the benefits of trust
and trustworthy behavior go to the whole community and are not completely internalized. We
discuss the implications of this perspective for improving these systems.

This portion of a reputation system answers the core question of how reputation is generated from the information exchanged between systems and how/where it is stored.

==Problem domain:==

This portion of a reputation system answers the core question of how reputation is generated from the information exchanged between systems and how/where it is stored.

Problem domain:

• Emerge vs. Impose reputation on the system?
• Probably both, how do we account for both systems?
• Do we maintain records based on a fixed set of imposed rules? Or do we build rules as the system emerges and reputations are formed?
• Where do you store the data?
• Distributed storage systems. Reputation in real-life is stored in interactions that an entity has with others. Reputation is not stored centrally. Reputation is most often a shared view of an entity by the masses, but sometimes an entities reputation can be disjoint among the masses: many different entities having differing views of reputation for the same entity.
• Who do we trust for this information?
• Trusting the masses is generally a good way of ensuring trustworthiness. Imposed rules will not always fit every situation well - could potentially set bad reputation to a "good" entity.
• Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
• Do we maintain an ever-growing set of history items for interactions between entities? Do we look focus on the bad reputations

Existing systems
• Peer-based systems (emerge)
• eBay - positive/negative rating system
• Youtube - like/dislike/spam comment system
• Policy-based systems (impose)
• Java - policy based security
• Android - policy based security
These two systems are on opposite ends of the Emerge-Impose spectrum.
EigenTrust system
The EigenTrust system utilizes a numerical scale for reputation storage.
Advantages:
• Numerical values are easy to compare.
• Little required storage space.
Disadvantages:
• Information is lost in the abstraction process.
o No concrete data
• Ambiguity
Storing concrete data
Hence, with the given information from the reputation system, we cannot generate an accurate profile of the entity.
We need a system that represents reputation in a concrete form
“If principal p gains access to resource r at time t, then the past behavior of p up until time t satisfies requirement ψr.”
Advantages
• A sufficient amount of information is available to come up with a profile of an entity
Disadvantages
• Storage space
Shmatikov and Talcott
Histories are sets of time-stamped events. Reputation is based on ability to adhere to licenses.
Licenses might permit certain actions OR require certain actions from being performed.
Advantages:
• Store data in concrete form
Disadvantages:
• No notion of sessions (logically connected set of events)
Representation of reputation
If we consider reputation information to encompass the events and actions of an entity, then we can model reputation as a set of events.
An interesting new problem is how to re-evaluate policies efficiently when new information becomes available.
This problem is known as “dynamic model-checking”.
Dynamic model-checking
We want a way of summarizing past reputations.
A solution here is to use: Havelund and Rosu, based on the technique of dynamic programming, used for runtime verification
• Given some information on an entity, how do we convert/abstract this to reputation?
• Is all the information necessary to maintain?
• Now that we have the reputation information, what can we do with it?
o What can we compare it with?
Implementation
Desired functionality of the system:
new()
- Append new reputation information
update()
- Update and summarize past behavior
- This is a reduce function
check()
- Analyze whether the given reputation satisfies the criteria of the policy

==Querying Reputation==

=== Problems ===

* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
***If you want to know someone's reputation, you either need to start asking around for it, imposing yourself. Or you need the data to be sent around, so you already have access to it; emergent.
* Where do you store the data?
***You need to know who has the data to ask them for it, or to go get it yourself.
* Where is the data queried from?
***First you need to know who's storing it. then you need to know if you're allowed to ask that node directly, do you ask a intermediary keeper of data. Will you even need to Query-- that is, do you already have all you need to know on hand? you need not get the latest updates on a node if every other node who's ever talked to it got DDOSed. (or do you?)
* What defines good/bad reputation?
***Should I make my own definition for bad reputation, and query if someone engaged in activities I consider bad, or should their be a global agreed upon reputation?
* Who provides the good/bad reputation?
***Who should I ask for information from?
* Who do we trust for this information?
***Whoever you trust, presumably their opinion on a given node is more important then a node you trust less.
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
***topically, would you bother asking for 10 year old reputation data on a node, if it's been a model citizen for the last 9?
* What entities are able to contribute to reputations?
***Should I ask everyone I trust for an opinion on a given node, or just certain keepers of trust data?
* How do we access reputation about entities?
***You query someone in the know who you trust and are allowed to query.
***you could say, ask everyone you know and trust, and ask them to ask people they know and trust, (and so on...if they're willing) until you find a node with the information you need.
***in a more centralized system you need to ask some kind of keeper of information for the information you want, and that keeper may or may not provide you with the reputation info you want.
* Who is authorized to access particular reputations? How much to reveal? (Information flow)
***The ability to control this would depend on how centralized a system you have. In a truly distributed system where every node has an opinion on any other node they've talked to you'll be able to find somebody who can tell you about the CIA node, but in a more centralized system the keepers of information might be less...willing to give Joe 6 cores information on who Iran is DDOSing.

===Maybe References===
http://www.kirkarts.com/wiki/images/1/13/Resnick_eBay.pdf - ''Trust Among Strangers in Internet Transactions:
Empirical Analysis of eBay’s Reputation System'' (maybe not too relevant)

http://portal.acm.org/citation.cfm?id=544741.544809 - ''An Evidential Model of Distributed Reputation Management''

http://portal.acm.org/citation.cfm?id=775152.775242&type=series%EF%BF%BD%C3%9C -- ''The EigenTrust Algorithm for Reputation Management in
P2P Networks''

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.4.2297&rep=rep1&type=pdf -- ''A Robust Reputation System for Mobile Ad-hoc
Networks''

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.125.8729&rep=rep1&type=pdf -- ''EigenRep: Reputation Management in P2P Networks''

http://www.chennaisunday.com/ieee%202010/Reputation%20Estimation%20and%20Query%20in%20Peer-to-Peer%20Networks.pdf -- ''Reputation Estimation and Query in Peer-to-Peer Networks''

Here is another paper that might be interesting for you. -- Lester
http://dcg.ethz.ch/publications/netecon06.pdf

==Possible implementations==
==Implementation Requirements==
==Conclusion==

==References==
* Joel Weise : "Public Key Infrastructure Overview " http://www.sun.com/blueprints/0801/publickey.pdf Accessed 2nd March 2011

* Security Glossary : http://www.cafesoft.com/support/security-glossary.html Accessed on 2nd March 2011

* Mattila, Anssi; and Mattila, Minna "What is the Effect of Product Attributes on Public-Key Infrastructure adoption? " http://internetjournals.net/journals/tir/2006/January/Paper%2003.pdf Accessed on 2nd March 2011

*Electronic Commerece Conference , PKI Sub-Group , Issue Paper : http://www.defense.gov/dodreform/ecwg/pki.pdf date accessed 5th March 2011

*SANS Institute InfoSec Reading Room, Common issues in PKI implementations - climbing the Slope of Enlightenment : http://www.sans.org/reading_room/whitepapers/authentication/common-issues-pki-implementations-climbing-slope-enlightenment_1198 date accessed 15th March 2011

*"Understanding Digital Certificate" by Microsoft : http://technet.microsoft.com/en-us/library/bb123848%28EXCHG.65%29.aspx date accessed 3rd April 2011

*"How digital certificate works" by IBM: http://publib.boulder.ibm.com/infocenter/wmqv6/v6r0/index.jsp?topic=/com.ibm.mq.csqzas.doc/sy10580_.htm date accessed 3rd April 2011

DistOS-2011W Reputation

2011-04-09T16:53:56Z

Wahmed2: /* References */

==Members==
* Waheed Ahmed
* Trevor Gelowsky
** MSN: Gelowt@gmail.com
** E-Mail: tgelowsk@sce.carleton.ca
* Michael Du Plessis
* Nicolas Lessard (nick.lessard @t gmail.com / nlessard @t carleton.connect.ca)

==Our presentation==
Our current presentation can be viewed at the following link: https://docs.google.com/present/edit?id=0ASS7kj9hfc1aZGRiMjMzOHJfNGhnNzhuamRr&hl=en&authkey=CMHi3KAD

==Our Paper==

Our final paper can be found here: [[Distributed OS: Winter 2011 Reputation Systems Paper]]

==The problem==
* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
* Where do you store the data?
* Where is the data queried from?
* What defines good/bad reputation?
* Who provides the good/bad reputation?
* Who do we trust for this information?
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
* What entities are able to contribute to reputations?
* How do we access reputation about entities?
* Who is authorized to access particular reputations? How much to reveal? (Information flow)

==What technologies currently exist?==
* Digital signatures
** Certificates signed by trusted organizations

* Black hole- email, spam,
* Google - search reputation
* Credit bureaus
* Yellow pages
* Better business bureau
* CRC - criminal records

== What technologies don't currently exist?==

==Guaranteeing Authenticity/Public Key Infrastructure==

In our paper we must explain why PKI/Authentication fits into reputation. Why must it be handled by both Attribution and Reputation systems?

===Problem Domain===

This portion of a reputation system answers the core question of how reputation information being exchanged is guaranteed to be authentic.

*How do we ensure the information exchanged between peers authentic, and not tampered with?

*How to we attribute information exchanged?

*How do we do this in a highly decentralized, distributed system?

*How can we make sure the information is timely?

===Introduction===
In past few years, Internet has provided platform for a global market place and both business and private users realizes that the revolutionary communications opportunities provided by it will give way to large spectrum of business and private applications.Today online users face multitude of problems and issues like vulnerability to viruses , worms , exposure to sniffers, spoofing their private sessions not only this but also they are also subjected to invasion of privacy with multitude of spy ware available for monitoring how they behave. Today over the internet different kind of activities take place ranging from access to information to entertainment, financial services, product services and even socializing. The frequent usage of internet as an important business tool led to a major increase in deliberate abuse and criminal activities. All the organization operating electronically and trading expose their own information and IT systems to a wide range of security threats. The most common protocols like IP/TCP/UDP are the main targets of potential hackers. Its all because of IPs on which attacks are possible and don't have proper authentication mechanism for any incoming data over internet.

In order to build secure chain of trust Public-Key Infrastructure is used for internet based communication. It consists of various things like security policy , Certificate authority , registration authority , certificate distribution system PKI enabled applications.

===PKI===
With development of modern e-commerce based businesses which has minimal customer face-to-face interactions is demanding more security and integrity. The online web based stores where huge amount of transactions take place needs to ensure customers that there information is confidential and processed through a secure channel. This is where implementation of PKI steps in to provide mechanisms to ensure trusted relationships are established and maintained. The specific security functions in which a PKI can provide foundation are confidentiality, integrity, non-repudiation,and authentication.

PKI provides mean of guaranteeing the authenticity by issuing digital signature. In order to ensure that electronic document is authentic , which means knowing the person who created the document and it hasn't been modified. They are commonly used for software distribution, financial transaction or to detect forgery or tampering. Further to ensure the authentication digital signature relies on certain type encryption. Digital certificate is mode of encryption on large scale for example secure web server. Digital certificate are validated by PKI by verifying the authenticity of certificate , the validity of certificate and that the certificate is trustworthy. They are issued by an authority referred as Certification Authority. The certificate authority act as the middleman that both computer trust. This helps in avoiding man in the middle attack , certificate authority confirms that computer is in fact who they say they are and the provides the public key of each computer to the other. The digital certificate contains the public key of the entity identified in certificate , it matches a public to a particular individual, and that certificate's authenticity is guaranteed by the issuer, thus digital certificate provides solution to the problem of how to find user's public key and know that its valid.These problems are solved by a user obtaining another user's public key from the digital certificate. The user knows it is valid because a trusted certification authority has issued the certificate. For their own authentication digital certificate rely on public key cryptography. The certification authority signs the certificate with its own private as digital certificate is issued. To validate the authenticity of a digital certificate, a user can obtain that certification authority's public key and use it against the certificate to determine if it was signed by the certification authority.

===Issues & Solutions===
I found out there are many different implementations of PKI , and they all focuses on their own issues and solutions. For example PKI used in DoD have following issues

*Lack of PKI-enabled eCommerce applications and lack of interoperability among PKI applications

*DoD is developing a single high assurance PKI

*Very High Cost Impact to the EC/EB community.

*The PKI community lacks metrics for mapping of trust models between the DoD :”high assurance” C2 and EC/EB domains

*Education of everyone (policy maker through user) to a common level of understanding is a huge challenge.

*While the purpose of using PKI in EC/EB is to provide additional trust to allow the Internet to serve as a vehicle for legally binding transactions , problems still exist with the methodologies associated with establishing a long-term burden of proof. Specifically, there are no widely adopted industry standards for maintenance of electronic signatures or for authenticated timestamps for record maintenance that have stood the test of time. These processes are untried and the case law has not yet been established to convince users that there are no issues with enforcement of these new processes. An additional barrier to EC/EB within this space is the current DoD Certificate policy in which DoD accepts

===Common Issues With PKI Implementation===

*Commercial Off-The-Shelf (COTS) versus Customised applications : The choice between COTS or customised products is usually one of cost versus usability. In case of usability the thing to be focused should be error messages. If PKI is built int o applications (transparent to users) than its fine if not than user will require to have some understanding of the use of keys, certificates, Certificate Revocation Lists (CRLs) and directories/certificate repositories so that they can make informed decisions.

*Token Logistics (smart card): The point where keys and certificates are linked to their owner is a very critical point in a PKI. If a fraudulent certificate is issued by a registration officer and the certificate holder uses the certificate to commit a crime or prank, trust in the whole PKI hierarchy may be lost. The physical security requirements are high, and the registration officer, whether a person or a smartcard bureau, must be subject to strict security polices and practices. As it was problem with DoD mentioned in section above.

*Network issues - Traffic : There is no doubt that the implementation of PKI will add to the network load, although just how much depends on the system architecture. Potential additional traffic that should be considered includes: Certificate issuance, Email usage, CRLs , and Directory Replication

*Network issues - Encryption : Many organisations implement anti-virus software and content inspection on servers at the perimeter of their networks. Some have security policies that rejects or quarantines encrypted traffic. To provide user-to-user confidentiality, messages will traverse networks with their payload hidden from inspection by virus and content checking.

*Email address in certificate :In order to use certificates for S/MIME signed/encrypted email, the users’ email address must be in the certificate. Most people change their email addresses more frequently than the certificate. Unless a solution is built which allows users to keep the same email address over a long period, certificates would have to be re-issued every time a user changes email address. S/MIME v.3 stipulates that the receiving application must check the From: or Sender: field in the mail header and compare it to an email address in the sender’s certificate. If the check does not match, the mail application should perform another explicit check to ensure that the person who signed the message is indeed the person who sent it. As usual, the ‘devil is in the detail’ when it comes to implementation.

*Certificate Validity Checking:CRLs have been the conventional method of providing certificate validity checking. CRLs do not scale very well as discussed earlier, but are usually kept for backward compatibility, archiving/historical verification and for use in off-line mode. The other issue with CRLs is that they are generally issued at certain intervals of 6, 12 or 24 hours, causing a time lag from the time a certificate is revoked until it appears on the published CRL. This may present a security risk, as a certificate may verify correctly after it has been reported as compromised and revoked; (however some would argue that the time from actual compromise until the discovery and reporting of it would in most cases be a more significant lag). The Online Certificate Status Protocol (OCSP) (RFC2560) allows a client to query an OCSP responder for the current status of a certificate. This saves searching through a large CRL and can save bandwidth if the CRL would normally be downloaded - although it may increase network traffic. Most OCSP responders are based on CRLs and thus do not solve the problem of time lag as outlined above.

*Availability and storage of reliable user information : For an identity certificate scheme, names in certificates need to be unique, meaningful - and correct. Few large user communities have all their member details in a central and accurate database or directory, and the exercise of consolidating, checking and updating all user data can turn into a massive and expensive exercise.

*Archiving/historic verification : Digital signatures need to be verifiable even after the keys used to sign have expired. Likewise, we need to be able to verify that the certificate was valid at the time the datawas signed. This means we would need to archive: the signed file,the public key certificate of the signer, the CRL that was valid at the time of signing, a reliable timestamp to prove the accuracy of the time of signing and, the hardware environment that can run the software that was used at the time

===What are the solution to these problems?===

*Identity Based Encryption:enables senders to encrypt messages for recipients w/o requiring that a recipients key first be established, certified, and published.

*Certificate-based encryption:it incorporates IBE methods, but uses double encryption so that its CA cant decrypt on behalf of the user.

*Certificateless Public Key Cryptography:it incorporates IBE methods, using partial private keys so PKG cant decrypt on behalf of the user.

*Distributed Computation:There exists methods that distribute cryptographic operations so that the cooperative contribution of a number of entities is required in order to perform an operation such as a signature or decryption. It helps in tighter protection at servers vs clients, but implies that the users mist fully trust servers to apply keys appropriately.

*Alternative Validation Strategies:Hash tree:it offers compact protected representations of the status of large number of certiticates.Highly valued if PKI is operated large scale more benifical than Certificate Revovation List. CRL reflect ststus information at fixed intervals.

==Dissemination==

===The Problem Domain===

===Random Ramblings on Reputation Management and Distribution===

Publish/Subscribe?

This system has unique distribution requirements as compared to most distributed systems in general. In this system, we cannot assume that there will be a universally agreed-upon definition of good, or bad. Similarly, the system must be self-policing. It would be up to each and every group of autonomous systems to decide which updates to accept and reject. Updates themselves also should not cause the network to DDoS itself. Lastly, it would be impossible for every system to know what the reputation for a given system is. Therefore the system must disseminate information in some way that is query-able and localizes reputation information where required.

To this end, we need a way of spreading information that while reliable, does not depend on one universally agreed-upon set of reputations.

For example, on an internet-scale operating system it would be entirely reasonable for one group of systems to not want to accept updates, or want to avoid communication with a given series of systems.

Any solution would assume that the problems of attribution are solved.

===Current Examples of Reputation Dissemination===

The first protocol that immediately comes to mind in this situation is a gossip-based protocol. These protocols are designed to operate in highly decentralized, large-scale systems.

Here's a nice overview:
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4537308 "Reputation management in distributed systems"

Examples are as follows:
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4228013 "Gossip-based Reputation Aggregation for Unstructured Peer-to-Peer Networks"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=5569965 "Improving Accuracy and Coverage in an Internet-Deployed Reputation Mechanism"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4459326 "GossipTrust for Fast Reputation Aggregation in Peer-to-Peer Networks"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4777496 "Adaptive trust management in P2P networks using gossip protocol"

Another possibility is using "Reputation chains"
* http://dx.doi.org.proxy.library.carleton.ca/10.1109/TKDE.2009.45 "P2P Reputation Management Using Distributed Identities and Decentralized Recommendation Chains"

==Maintaining History==

===Problem domain===

* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
*** Do we maintain records based on a fixed set of imposed rules? Or do we build rules as the system emerges and reputations are formed?
* Where do you store the data?
** Distributed storage systems. Reputation in real-life is stored in interactions that an entity has with others. Reputation is not stored centrally. Reputation is most often a shared view of an entity by the masses, but sometimes an entities reputation can be disjoint among the masses: many different entities having differing views of reputation for the same entity.
* Where is the data queried from?
** (should I mention this?)
* What defines good/bad reputation?
** (should I mention this?)
* Who provides the good/bad reputation?
** Impose/Emerge problem: reputation for an interaction can be calculated immediately or can be a function of time.
* Who do we trust for this information?
** Trusting the masses is generally a good way of ensuring trustworthiness. Imposed rules will not always fit every situation well - could potentially set bad reputation to a "good" entity.
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
** Do we maintain an ever-growing set of history items for interactions between entities? Do we look focus on the bad reputations?
* What entities are able to contribute to reputations?
* How do we access reputation about entities?
* Who is authorized to access particular reputations? How much to reveal? (Information flow)
* What assumptions will we make?
* Privacy issues? What will we reveal? Will centralized systems have a know-all mentality?
** Fine grained information will never be revealed (privacy concerns and user rights)
* Which history should I maintain? What to take as important, what to disregard?
* Immutable data structure. Who could add data? Who could remove data? Authority

===Reputation systems===
* record, aggregate, distribute information about an entity's behaviour in distributed applications

* reputation might be based on the entity's past ability to adhere to a license agreement (mutual contract between issuer and licensee)

===History-based access control systems===
* make decision based on an entity's past security-sensitive actions

===Examples of reputation systems (trust-informing technologies)===
* eBay - Feedback forum (positive, neutral, negative)

===Do reputation systems have some validity?===

Resnick et al. argue that reputation systems
foster an incentive for principals to well-behave because of “the expectation of
reciprocity or retaliation in future interactions

Abstractions are used to model the aggregated information of each entity. These abstractions may not encompass the full details of transactions and provide context to specific issues relating to feedback. In turn we can end up with ambiguous values.

So we need a system that provides sufficient information in order to verify the precise properties of a past behaviour.

* Krukow, K. A Logical Framework for Reputation Systems and History-based Access Control. School of Electronics and Computer Science University of Southampton, UK. (March 3, 2011) [http://www.brics.dk/~krukow/research/publications/online_papers/concrete-jcs.pdf]

====Abstract====
Reputation systems are meta systems that record, aggregate and distribute information about principals’ behaviour in distributed applications. Similarly, history-based access control systems make decisions based
on programs’ past security-sensitive actions. While the applications are
distinct, the two types of systems are fundamentally making decisions
based on information about the past behaviour of an entity.
A logical policy-centric framework for such behaviour-based decisionmaking is presented. In the framework, principals specify policies which
state precise requirements on the past behaviour of other principals that
must be fulﬁlled in order for interaction to take place. The framework consists of a formal model of behaviour, based on event structures; a declarative logical language for specifying properties of past behaviour; and
eﬃcient dynamic algorithms for checking whether a particular behaviour
satisﬁes a property from the language. It is shown how the framework can
be extended in several ways, most notably to encompass parameterized
events and quantiﬁcation over parameters. In an extended application, it
is illustrated how the framework can be applied for dynamic history-based
access control for safe execution of unknown and untrusted programs.

* Khosrow-Pour, M. Emerging trends and challenges in information technology management (March 7, 2011) [http://books.google.ca/books?id=ybzS-yylJfAC&lpg=PA822&ots=V7hn_RzqXA&dq=maintaining%20history%20in%20reputation%20systems&pg=PA822#v=onepage&q=maintaining%20history%20in%20reputation%20systems&f=false]

====Abstract====

* Bolton, G. et al. How Effective are Electronic Reputation Mechanisms? (March 10, 2011) [http://ccs.mit.edu/dell/reputation/BKOMSsub.pdf]

====Abstract====

Electronic reputation or “feedback” mechanisms aim to mitigate the moral hazard problems
associated with exchange among strangers by providing the type of information available in
more traditional close-knit groups, where members are frequently involved in one another’s
dealings. In this paper, we compare trading in a market with electronic feedback (as
implemented by many Internet markets) to a market without, as well as to a market in which the
same people interact with one another repeatedly (partners market). We find that, while the
feedback mechanism induces quite a substantial improvement in transaction efficiency, it also
exhibits a kind of public goods problem in that, unlike the partners market, the benefits of trust
and trustworthy behavior go to the whole community and are not completely internalized. We
discuss the implications of this perspective for improving these systems.

This portion of a reputation system answers the core question of how reputation is generated from the information exchanged between systems and how/where it is stored.

==Problem domain:==

This portion of a reputation system answers the core question of how reputation is generated from the information exchanged between systems and how/where it is stored.

Problem domain:

• Emerge vs. Impose reputation on the system?
• Probably both, how do we account for both systems?
• Do we maintain records based on a fixed set of imposed rules? Or do we build rules as the system emerges and reputations are formed?
• Where do you store the data?
• Distributed storage systems. Reputation in real-life is stored in interactions that an entity has with others. Reputation is not stored centrally. Reputation is most often a shared view of an entity by the masses, but sometimes an entities reputation can be disjoint among the masses: many different entities having differing views of reputation for the same entity.
• Who do we trust for this information?
• Trusting the masses is generally a good way of ensuring trustworthiness. Imposed rules will not always fit every situation well - could potentially set bad reputation to a "good" entity.
• Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
• Do we maintain an ever-growing set of history items for interactions between entities? Do we look focus on the bad reputations

Existing systems
• Peer-based systems (emerge)
• eBay - positive/negative rating system
• Youtube - like/dislike/spam comment system
• Policy-based systems (impose)
• Java - policy based security
• Android - policy based security
These two systems are on opposite ends of the Emerge-Impose spectrum.
EigenTrust system
The EigenTrust system utilizes a numerical scale for reputation storage.
Advantages:
• Numerical values are easy to compare.
• Little required storage space.
Disadvantages:
• Information is lost in the abstraction process.
o No concrete data
• Ambiguity
Storing concrete data
Hence, with the given information from the reputation system, we cannot generate an accurate profile of the entity.
We need a system that represents reputation in a concrete form
“If principal p gains access to resource r at time t, then the past behavior of p up until time t satisfies requirement ψr.”
Advantages
• A sufficient amount of information is available to come up with a profile of an entity
Disadvantages
• Storage space
Shmatikov and Talcott
Histories are sets of time-stamped events. Reputation is based on ability to adhere to licenses.
Licenses might permit certain actions OR require certain actions from being performed.
Advantages:
• Store data in concrete form
Disadvantages:
• No notion of sessions (logically connected set of events)
Representation of reputation
If we consider reputation information to encompass the events and actions of an entity, then we can model reputation as a set of events.
An interesting new problem is how to re-evaluate policies efficiently when new information becomes available.
This problem is known as “dynamic model-checking”.
Dynamic model-checking
We want a way of summarizing past reputations.
A solution here is to use: Havelund and Rosu, based on the technique of dynamic programming, used for runtime verification
• Given some information on an entity, how do we convert/abstract this to reputation?
• Is all the information necessary to maintain?
• Now that we have the reputation information, what can we do with it?
o What can we compare it with?
Implementation
Desired functionality of the system:
new()
- Append new reputation information
update()
- Update and summarize past behavior
- This is a reduce function
check()
- Analyze whether the given reputation satisfies the criteria of the policy

==Querying Reputation==

=== Problems ===

* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
***If you want to know someone's reputation, you either need to start asking around for it, imposing yourself. Or you need the data to be sent around, so you already have access to it; emergent.
* Where do you store the data?
***You need to know who has the data to ask them for it, or to go get it yourself.
* Where is the data queried from?
***First you need to know who's storing it. then you need to know if you're allowed to ask that node directly, do you ask a intermediary keeper of data. Will you even need to Query-- that is, do you already have all you need to know on hand? you need not get the latest updates on a node if every other node who's ever talked to it got DDOSed. (or do you?)
* What defines good/bad reputation?
***Should I make my own definition for bad reputation, and query if someone engaged in activities I consider bad, or should their be a global agreed upon reputation?
* Who provides the good/bad reputation?
***Who should I ask for information from?
* Who do we trust for this information?
***Whoever you trust, presumably their opinion on a given node is more important then a node you trust less.
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
***topically, would you bother asking for 10 year old reputation data on a node, if it's been a model citizen for the last 9?
* What entities are able to contribute to reputations?
***Should I ask everyone I trust for an opinion on a given node, or just certain keepers of trust data?
* How do we access reputation about entities?
***You query someone in the know who you trust and are allowed to query.
***you could say, ask everyone you know and trust, and ask them to ask people they know and trust, (and so on...if they're willing) until you find a node with the information you need.
***in a more centralized system you need to ask some kind of keeper of information for the information you want, and that keeper may or may not provide you with the reputation info you want.
* Who is authorized to access particular reputations? How much to reveal? (Information flow)
***The ability to control this would depend on how centralized a system you have. In a truly distributed system where every node has an opinion on any other node they've talked to you'll be able to find somebody who can tell you about the CIA node, but in a more centralized system the keepers of information might be less...willing to give Joe 6 cores information on who Iran is DDOSing.

===Maybe References===
http://www.kirkarts.com/wiki/images/1/13/Resnick_eBay.pdf - ''Trust Among Strangers in Internet Transactions:
Empirical Analysis of eBay’s Reputation System'' (maybe not too relevant)

http://portal.acm.org/citation.cfm?id=544741.544809 - ''An Evidential Model of Distributed Reputation Management''

http://portal.acm.org/citation.cfm?id=775152.775242&type=series%EF%BF%BD%C3%9C -- ''The EigenTrust Algorithm for Reputation Management in
P2P Networks''

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.4.2297&rep=rep1&type=pdf -- ''A Robust Reputation System for Mobile Ad-hoc
Networks''

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.125.8729&rep=rep1&type=pdf -- ''EigenRep: Reputation Management in P2P Networks''

http://www.chennaisunday.com/ieee%202010/Reputation%20Estimation%20and%20Query%20in%20Peer-to-Peer%20Networks.pdf -- ''Reputation Estimation and Query in Peer-to-Peer Networks''

Here is another paper that might be interesting for you. -- Lester
http://dcg.ethz.ch/publications/netecon06.pdf

==Possible implementations==
==Implementation Requirements==
==Conclusion==

==References==
* Joel Weise : "Public Key Infrastructure Overview " http://www.sun.com/blueprints/0801/publickey.pdf Accessed 2nd March 2011

* Security Glossary : http://www.cafesoft.com/support/security-glossary.html Accessed on 2nd March 2011

* Mattila, Anssi; and Mattila, Minna "What is the Effect of Product Attributes on Public-Key Infrastructure adoption? " http://internetjournals.net/journals/tir/2006/January/Paper%2003.pdf Accessed on 2nd March 2011

*Electronic Commerece Conference , PKI Sub-Group , Issue Paper : http://www.defense.gov/dodreform/ecwg/pki.pdf date accessed 5th March 2011

*SANS Institute InfoSec Reading Room, Common issues in PKI implementations - climbing the Slope of Enlightenment : http://www.sans.org/reading_room/whitepapers/authentication/common-issues-pki-implementations-climbing-slope-enlightenment_1198 date accessed 15th March 2011

*"Understanding Digital Certificate" by Microsoft : http://technet.microsoft.com/en-us/library/bb123848%28EXCHG.65%29.aspx date accessed 3rd April 2011

*"How digital certificate works" by IBM: http://publib.boulder.ibm.com/infocenter/wmqv6/v6r0/index.jsp?topic=/com.ibm.mq.csqzas.doc/sy10580_.htm date accessed 3rd April 2011

DistOS-2011W Reputation

2011-04-09T16:50:58Z

Wahmed2: /* Uses and Need of PKI */

==Members==
* Waheed Ahmed
* Trevor Gelowsky
** MSN: Gelowt@gmail.com
** E-Mail: tgelowsk@sce.carleton.ca
* Michael Du Plessis
* Nicolas Lessard (nick.lessard @t gmail.com / nlessard @t carleton.connect.ca)

==Our presentation==
Our current presentation can be viewed at the following link: https://docs.google.com/present/edit?id=0ASS7kj9hfc1aZGRiMjMzOHJfNGhnNzhuamRr&hl=en&authkey=CMHi3KAD

==Our Paper==

Our final paper can be found here: [[Distributed OS: Winter 2011 Reputation Systems Paper]]

==The problem==
* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
* Where do you store the data?
* Where is the data queried from?
* What defines good/bad reputation?
* Who provides the good/bad reputation?
* Who do we trust for this information?
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
* What entities are able to contribute to reputations?
* How do we access reputation about entities?
* Who is authorized to access particular reputations? How much to reveal? (Information flow)

==What technologies currently exist?==
* Digital signatures
** Certificates signed by trusted organizations

* Black hole- email, spam,
* Google - search reputation
* Credit bureaus
* Yellow pages
* Better business bureau
* CRC - criminal records

== What technologies don't currently exist?==

==Guaranteeing Authenticity/Public Key Infrastructure==

In our paper we must explain why PKI/Authentication fits into reputation. Why must it be handled by both Attribution and Reputation systems?

===Problem Domain===

This portion of a reputation system answers the core question of how reputation information being exchanged is guaranteed to be authentic.

*How do we ensure the information exchanged between peers authentic, and not tampered with?

*How to we attribute information exchanged?

*How do we do this in a highly decentralized, distributed system?

*How can we make sure the information is timely?

===Introduction===
In past few years, Internet has provided platform for a global market place and both business and private users realizes that the revolutionary communications opportunities provided by it will give way to large spectrum of business and private applications.Today online users face multitude of problems and issues like vulnerability to viruses , worms , exposure to sniffers, spoofing their private sessions not only this but also they are also subjected to invasion of privacy with multitude of spy ware available for monitoring how they behave. Today over the internet different kind of activities take place ranging from access to information to entertainment, financial services, product services and even socializing. The frequent usage of internet as an important business tool led to a major increase in deliberate abuse and criminal activities. All the organization operating electronically and trading expose their own information and IT systems to a wide range of security threats. The most common protocols like IP/TCP/UDP are the main targets of potential hackers. Its all because of IPs on which attacks are possible and don't have proper authentication mechanism for any incoming data over internet.

In order to build secure chain of trust Public-Key Infrastructure is used for internet based communication. It consists of various things like security policy , Certificate authority , registration authority , certificate distribution system PKI enabled applications.

===PKI===
With development of modern e-commerce based businesses which has minimal customer face-to-face interactions is demanding more security and integrity. The online web based stores where huge amount of transactions take place needs to ensure customers that there information is confidential and processed through a secure channel. This is where implementation of PKI steps in to provide mechanisms to ensure trusted relationships are established and maintained. The specific security functions in which a PKI can provide foundation are confidentiality, integrity, non-repudiation,and authentication.

PKI provides mean of guaranteeing the authenticity by issuing digital signature. In order to ensure that electronic document is authentic , which means knowing the person who created the document and it hasn't been modified. They are commonly used for software distribution, financial transaction or to detect forgery or tampering. Further to ensure the authentication digital signature relies on certain type encryption. Digital certificate is mode of encryption on large scale for example secure web server. Digital certificate are validated by PKI by verifying the authenticity of certificate , the validity of certificate and that the certificate is trustworthy. They are issued by an authority referred as Certification Authority. The certificate authority act as the middleman that both computer trust. This helps in avoiding man in the middle attack , certificate authority confirms that computer is in fact who they say they are and the provides the public key of each computer to the other. The digital certificate contains the public key of the entity identified in certificate , it matches a public to a particular individual, and that certificate's authenticity is guaranteed by the issuer, thus digital certificate provides solution to the problem of how to find user's public key and know that its valid.These problems are solved by a user obtaining another user's public key from the digital certificate. The user knows it is valid because a trusted certification authority has issued the certificate. For their own authentication digital certificate rely on public key cryptography. The certification authority signs the certificate with its own private as digital certificate is issued. To validate the authenticity of a digital certificate, a user can obtain that certification authority's public key and use it against the certificate to determine if it was signed by the certification authority.

===Issues & Solutions===
I found out there are many different implementations of PKI , and they all focuses on their own issues and solutions. For example PKI used in DoD have following issues

*Lack of PKI-enabled eCommerce applications and lack of interoperability among PKI applications

*DoD is developing a single high assurance PKI

*Very High Cost Impact to the EC/EB community.

*The PKI community lacks metrics for mapping of trust models between the DoD :”high assurance” C2 and EC/EB domains

*Education of everyone (policy maker through user) to a common level of understanding is a huge challenge.

*While the purpose of using PKI in EC/EB is to provide additional trust to allow the Internet to serve as a vehicle for legally binding transactions , problems still exist with the methodologies associated with establishing a long-term burden of proof. Specifically, there are no widely adopted industry standards for maintenance of electronic signatures or for authenticated timestamps for record maintenance that have stood the test of time. These processes are untried and the case law has not yet been established to convince users that there are no issues with enforcement of these new processes. An additional barrier to EC/EB within this space is the current DoD Certificate policy in which DoD accepts

===Common Issues With PKI Implementation===

*Commercial Off-The-Shelf (COTS) versus Customised applications : The choice between COTS or customised products is usually one of cost versus usability. In case of usability the thing to be focused should be error messages. If PKI is built int o applications (transparent to users) than its fine if not than user will require to have some understanding of the use of keys, certificates, Certificate Revocation Lists (CRLs) and directories/certificate repositories so that they can make informed decisions.

*Token Logistics (smart card): The point where keys and certificates are linked to their owner is a very critical point in a PKI. If a fraudulent certificate is issued by a registration officer and the certificate holder uses the certificate to commit a crime or prank, trust in the whole PKI hierarchy may be lost. The physical security requirements are high, and the registration officer, whether a person or a smartcard bureau, must be subject to strict security polices and practices. As it was problem with DoD mentioned in section above.

*Network issues - Traffic : There is no doubt that the implementation of PKI will add to the network load, although just how much depends on the system architecture. Potential additional traffic that should be considered includes: Certificate issuance, Email usage, CRLs , and Directory Replication

*Network issues - Encryption : Many organisations implement anti-virus software and content inspection on servers at the perimeter of their networks. Some have security policies that rejects or quarantines encrypted traffic. To provide user-to-user confidentiality, messages will traverse networks with their payload hidden from inspection by virus and content checking.

*Email address in certificate :In order to use certificates for S/MIME signed/encrypted email, the users’ email address must be in the certificate. Most people change their email addresses more frequently than the certificate. Unless a solution is built which allows users to keep the same email address over a long period, certificates would have to be re-issued every time a user changes email address. S/MIME v.3 stipulates that the receiving application must check the From: or Sender: field in the mail header and compare it to an email address in the sender’s certificate. If the check does not match, the mail application should perform another explicit check to ensure that the person who signed the message is indeed the person who sent it. As usual, the ‘devil is in the detail’ when it comes to implementation.

*Certificate Validity Checking:CRLs have been the conventional method of providing certificate validity checking. CRLs do not scale very well as discussed earlier, but are usually kept for backward compatibility, archiving/historical verification and for use in off-line mode. The other issue with CRLs is that they are generally issued at certain intervals of 6, 12 or 24 hours, causing a time lag from the time a certificate is revoked until it appears on the published CRL. This may present a security risk, as a certificate may verify correctly after it has been reported as compromised and revoked; (however some would argue that the time from actual compromise until the discovery and reporting of it would in most cases be a more significant lag). The Online Certificate Status Protocol (OCSP) (RFC2560) allows a client to query an OCSP responder for the current status of a certificate. This saves searching through a large CRL and can save bandwidth if the CRL would normally be downloaded - although it may increase network traffic. Most OCSP responders are based on CRLs and thus do not solve the problem of time lag as outlined above.

*Availability and storage of reliable user information : For an identity certificate scheme, names in certificates need to be unique, meaningful - and correct. Few large user communities have all their member details in a central and accurate database or directory, and the exercise of consolidating, checking and updating all user data can turn into a massive and expensive exercise.

*Archiving/historic verification : Digital signatures need to be verifiable even after the keys used to sign have expired. Likewise, we need to be able to verify that the certificate was valid at the time the datawas signed. This means we would need to archive: the signed file,the public key certificate of the signer, the CRL that was valid at the time of signing, a reliable timestamp to prove the accuracy of the time of signing and, the hardware environment that can run the software that was used at the time

===What are the solution to these problems?===

*Identity Based Encryption:enables senders to encrypt messages for recipients w/o requiring that a recipients key first be established, certified, and published.

*Certificate-based encryption:it incorporates IBE methods, but uses double encryption so that its CA cant decrypt on behalf of the user.

*Certificateless Public Key Cryptography:it incorporates IBE methods, using partial private keys so PKG cant decrypt on behalf of the user.

*Distributed Computation:There exists methods that distribute cryptographic operations so that the cooperative contribution of a number of entities is required in order to perform an operation such as a signature or decryption. It helps in tighter protection at servers vs clients, but implies that the users mist fully trust servers to apply keys appropriately.

*Alternative Validation Strategies:Hash tree:it offers compact protected representations of the status of large number of certiticates.Highly valued if PKI is operated large scale more benifical than Certificate Revovation List. CRL reflect ststus information at fixed intervals.

==Dissemination==

===The Problem Domain===

===Random Ramblings on Reputation Management and Distribution===

Publish/Subscribe?

This system has unique distribution requirements as compared to most distributed systems in general. In this system, we cannot assume that there will be a universally agreed-upon definition of good, or bad. Similarly, the system must be self-policing. It would be up to each and every group of autonomous systems to decide which updates to accept and reject. Updates themselves also should not cause the network to DDoS itself. Lastly, it would be impossible for every system to know what the reputation for a given system is. Therefore the system must disseminate information in some way that is query-able and localizes reputation information where required.

To this end, we need a way of spreading information that while reliable, does not depend on one universally agreed-upon set of reputations.

For example, on an internet-scale operating system it would be entirely reasonable for one group of systems to not want to accept updates, or want to avoid communication with a given series of systems.

Any solution would assume that the problems of attribution are solved.

===Current Examples of Reputation Dissemination===

The first protocol that immediately comes to mind in this situation is a gossip-based protocol. These protocols are designed to operate in highly decentralized, large-scale systems.

Here's a nice overview:
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4537308 "Reputation management in distributed systems"

Examples are as follows:
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4228013 "Gossip-based Reputation Aggregation for Unstructured Peer-to-Peer Networks"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=5569965 "Improving Accuracy and Coverage in an Internet-Deployed Reputation Mechanism"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4459326 "GossipTrust for Fast Reputation Aggregation in Peer-to-Peer Networks"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4777496 "Adaptive trust management in P2P networks using gossip protocol"

Another possibility is using "Reputation chains"
* http://dx.doi.org.proxy.library.carleton.ca/10.1109/TKDE.2009.45 "P2P Reputation Management Using Distributed Identities and Decentralized Recommendation Chains"

==Maintaining History==

===Problem domain===

* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
*** Do we maintain records based on a fixed set of imposed rules? Or do we build rules as the system emerges and reputations are formed?
* Where do you store the data?
** Distributed storage systems. Reputation in real-life is stored in interactions that an entity has with others. Reputation is not stored centrally. Reputation is most often a shared view of an entity by the masses, but sometimes an entities reputation can be disjoint among the masses: many different entities having differing views of reputation for the same entity.
* Where is the data queried from?
** (should I mention this?)
* What defines good/bad reputation?
** (should I mention this?)
* Who provides the good/bad reputation?
** Impose/Emerge problem: reputation for an interaction can be calculated immediately or can be a function of time.
* Who do we trust for this information?
** Trusting the masses is generally a good way of ensuring trustworthiness. Imposed rules will not always fit every situation well - could potentially set bad reputation to a "good" entity.
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
** Do we maintain an ever-growing set of history items for interactions between entities? Do we look focus on the bad reputations?
* What entities are able to contribute to reputations?
* How do we access reputation about entities?
* Who is authorized to access particular reputations? How much to reveal? (Information flow)
* What assumptions will we make?
* Privacy issues? What will we reveal? Will centralized systems have a know-all mentality?
** Fine grained information will never be revealed (privacy concerns and user rights)
* Which history should I maintain? What to take as important, what to disregard?
* Immutable data structure. Who could add data? Who could remove data? Authority

===Reputation systems===
* record, aggregate, distribute information about an entity's behaviour in distributed applications

* reputation might be based on the entity's past ability to adhere to a license agreement (mutual contract between issuer and licensee)

===History-based access control systems===
* make decision based on an entity's past security-sensitive actions

===Examples of reputation systems (trust-informing technologies)===
* eBay - Feedback forum (positive, neutral, negative)

===Do reputation systems have some validity?===

Resnick et al. argue that reputation systems
foster an incentive for principals to well-behave because of “the expectation of
reciprocity or retaliation in future interactions

Abstractions are used to model the aggregated information of each entity. These abstractions may not encompass the full details of transactions and provide context to specific issues relating to feedback. In turn we can end up with ambiguous values.

So we need a system that provides sufficient information in order to verify the precise properties of a past behaviour.

* Krukow, K. A Logical Framework for Reputation Systems and History-based Access Control. School of Electronics and Computer Science University of Southampton, UK. (March 3, 2011) [http://www.brics.dk/~krukow/research/publications/online_papers/concrete-jcs.pdf]

====Abstract====
Reputation systems are meta systems that record, aggregate and distribute information about principals’ behaviour in distributed applications. Similarly, history-based access control systems make decisions based
on programs’ past security-sensitive actions. While the applications are
distinct, the two types of systems are fundamentally making decisions
based on information about the past behaviour of an entity.
A logical policy-centric framework for such behaviour-based decisionmaking is presented. In the framework, principals specify policies which
state precise requirements on the past behaviour of other principals that
must be fulﬁlled in order for interaction to take place. The framework consists of a formal model of behaviour, based on event structures; a declarative logical language for specifying properties of past behaviour; and
eﬃcient dynamic algorithms for checking whether a particular behaviour
satisﬁes a property from the language. It is shown how the framework can
be extended in several ways, most notably to encompass parameterized
events and quantiﬁcation over parameters. In an extended application, it
is illustrated how the framework can be applied for dynamic history-based
access control for safe execution of unknown and untrusted programs.

* Khosrow-Pour, M. Emerging trends and challenges in information technology management (March 7, 2011) [http://books.google.ca/books?id=ybzS-yylJfAC&lpg=PA822&ots=V7hn_RzqXA&dq=maintaining%20history%20in%20reputation%20systems&pg=PA822#v=onepage&q=maintaining%20history%20in%20reputation%20systems&f=false]

====Abstract====

* Bolton, G. et al. How Effective are Electronic Reputation Mechanisms? (March 10, 2011) [http://ccs.mit.edu/dell/reputation/BKOMSsub.pdf]

====Abstract====

Electronic reputation or “feedback” mechanisms aim to mitigate the moral hazard problems
associated with exchange among strangers by providing the type of information available in
more traditional close-knit groups, where members are frequently involved in one another’s
dealings. In this paper, we compare trading in a market with electronic feedback (as
implemented by many Internet markets) to a market without, as well as to a market in which the
same people interact with one another repeatedly (partners market). We find that, while the
feedback mechanism induces quite a substantial improvement in transaction efficiency, it also
exhibits a kind of public goods problem in that, unlike the partners market, the benefits of trust
and trustworthy behavior go to the whole community and are not completely internalized. We
discuss the implications of this perspective for improving these systems.

This portion of a reputation system answers the core question of how reputation is generated from the information exchanged between systems and how/where it is stored.

==Problem domain:==

This portion of a reputation system answers the core question of how reputation is generated from the information exchanged between systems and how/where it is stored.

Problem domain:

• Emerge vs. Impose reputation on the system?
• Probably both, how do we account for both systems?
• Do we maintain records based on a fixed set of imposed rules? Or do we build rules as the system emerges and reputations are formed?
• Where do you store the data?
• Distributed storage systems. Reputation in real-life is stored in interactions that an entity has with others. Reputation is not stored centrally. Reputation is most often a shared view of an entity by the masses, but sometimes an entities reputation can be disjoint among the masses: many different entities having differing views of reputation for the same entity.
• Who do we trust for this information?
• Trusting the masses is generally a good way of ensuring trustworthiness. Imposed rules will not always fit every situation well - could potentially set bad reputation to a "good" entity.
• Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
• Do we maintain an ever-growing set of history items for interactions between entities? Do we look focus on the bad reputations

Existing systems
• Peer-based systems (emerge)
• eBay - positive/negative rating system
• Youtube - like/dislike/spam comment system
• Policy-based systems (impose)
• Java - policy based security
• Android - policy based security
These two systems are on opposite ends of the Emerge-Impose spectrum.
EigenTrust system
The EigenTrust system utilizes a numerical scale for reputation storage.
Advantages:
• Numerical values are easy to compare.
• Little required storage space.
Disadvantages:
• Information is lost in the abstraction process.
o No concrete data
• Ambiguity
Storing concrete data
Hence, with the given information from the reputation system, we cannot generate an accurate profile of the entity.
We need a system that represents reputation in a concrete form
“If principal p gains access to resource r at time t, then the past behavior of p up until time t satisfies requirement ψr.”
Advantages
• A sufficient amount of information is available to come up with a profile of an entity
Disadvantages
• Storage space
Shmatikov and Talcott
Histories are sets of time-stamped events. Reputation is based on ability to adhere to licenses.
Licenses might permit certain actions OR require certain actions from being performed.
Advantages:
• Store data in concrete form
Disadvantages:
• No notion of sessions (logically connected set of events)
Representation of reputation
If we consider reputation information to encompass the events and actions of an entity, then we can model reputation as a set of events.
An interesting new problem is how to re-evaluate policies efficiently when new information becomes available.
This problem is known as “dynamic model-checking”.
Dynamic model-checking
We want a way of summarizing past reputations.
A solution here is to use: Havelund and Rosu, based on the technique of dynamic programming, used for runtime verification
• Given some information on an entity, how do we convert/abstract this to reputation?
• Is all the information necessary to maintain?
• Now that we have the reputation information, what can we do with it?
o What can we compare it with?
Implementation
Desired functionality of the system:
new()
- Append new reputation information
update()
- Update and summarize past behavior
- This is a reduce function
check()
- Analyze whether the given reputation satisfies the criteria of the policy

==Querying Reputation==

=== Problems ===

* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
***If you want to know someone's reputation, you either need to start asking around for it, imposing yourself. Or you need the data to be sent around, so you already have access to it; emergent.
* Where do you store the data?
***You need to know who has the data to ask them for it, or to go get it yourself.
* Where is the data queried from?
***First you need to know who's storing it. then you need to know if you're allowed to ask that node directly, do you ask a intermediary keeper of data. Will you even need to Query-- that is, do you already have all you need to know on hand? you need not get the latest updates on a node if every other node who's ever talked to it got DDOSed. (or do you?)
* What defines good/bad reputation?
***Should I make my own definition for bad reputation, and query if someone engaged in activities I consider bad, or should their be a global agreed upon reputation?
* Who provides the good/bad reputation?
***Who should I ask for information from?
* Who do we trust for this information?
***Whoever you trust, presumably their opinion on a given node is more important then a node you trust less.
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
***topically, would you bother asking for 10 year old reputation data on a node, if it's been a model citizen for the last 9?
* What entities are able to contribute to reputations?
***Should I ask everyone I trust for an opinion on a given node, or just certain keepers of trust data?
* How do we access reputation about entities?
***You query someone in the know who you trust and are allowed to query.
***you could say, ask everyone you know and trust, and ask them to ask people they know and trust, (and so on...if they're willing) until you find a node with the information you need.
***in a more centralized system you need to ask some kind of keeper of information for the information you want, and that keeper may or may not provide you with the reputation info you want.
* Who is authorized to access particular reputations? How much to reveal? (Information flow)
***The ability to control this would depend on how centralized a system you have. In a truly distributed system where every node has an opinion on any other node they've talked to you'll be able to find somebody who can tell you about the CIA node, but in a more centralized system the keepers of information might be less...willing to give Joe 6 cores information on who Iran is DDOSing.

===Maybe References===
http://www.kirkarts.com/wiki/images/1/13/Resnick_eBay.pdf - ''Trust Among Strangers in Internet Transactions:
Empirical Analysis of eBay’s Reputation System'' (maybe not too relevant)

http://portal.acm.org/citation.cfm?id=544741.544809 - ''An Evidential Model of Distributed Reputation Management''

http://portal.acm.org/citation.cfm?id=775152.775242&type=series%EF%BF%BD%C3%9C -- ''The EigenTrust Algorithm for Reputation Management in
P2P Networks''

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.4.2297&rep=rep1&type=pdf -- ''A Robust Reputation System for Mobile Ad-hoc
Networks''

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.125.8729&rep=rep1&type=pdf -- ''EigenRep: Reputation Management in P2P Networks''

http://www.chennaisunday.com/ieee%202010/Reputation%20Estimation%20and%20Query%20in%20Peer-to-Peer%20Networks.pdf -- ''Reputation Estimation and Query in Peer-to-Peer Networks''

Here is another paper that might be interesting for you. -- Lester
http://dcg.ethz.ch/publications/netecon06.pdf

==Possible implementations==
==Implementation Requirements==
==Conclusion==

==References==
* Joel Weise : "Public Key Infrastructure Overview " http://www.sun.com/blueprints/0801/publickey.pdf Accessed 2nd March 2011

* Security Glossary : http://www.cafesoft.com/support/security-glossary.html Accessed on 2nd March 2011

* Mattila, Anssi; and Mattila, Minna "What is the Effect of Product Attributes on Public-Key Infrastructure adoption? " http://internetjournals.net/journals/tir/2006/January/Paper%2003.pdf Accessed on 2nd March 2011

*Electronic Commerece Conference , PKI Sub-Group , Issue Paper : http://www.defense.gov/dodreform/ecwg/pki.pdf date accessed 5th March 2011

*SANS Institute InfoSec Reading Room, Common issues in PKI implementations - climbing the Slope of Enlightenment : http://www.sans.org/reading_room/whitepapers/authentication/common-issues-pki-implementations-climbing-slope-enlightenment_1198 date accessed 15th March 2011

DistOS-2011W Reputation

2011-04-09T16:18:25Z

Wahmed2: /* Uses and Need of PKI */

==Members==
* Waheed Ahmed
* Trevor Gelowsky
** MSN: Gelowt@gmail.com
** E-Mail: tgelowsk@sce.carleton.ca
* Michael Du Plessis
* Nicolas Lessard (nick.lessard @t gmail.com / nlessard @t carleton.connect.ca)

==Our presentation==
Our current presentation can be viewed at the following link: https://docs.google.com/present/edit?id=0ASS7kj9hfc1aZGRiMjMzOHJfNGhnNzhuamRr&hl=en&authkey=CMHi3KAD

==Our Paper==

Our final paper can be found here: [[Distributed OS: Winter 2011 Reputation Systems Paper]]

==The problem==
* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
* Where do you store the data?
* Where is the data queried from?
* What defines good/bad reputation?
* Who provides the good/bad reputation?
* Who do we trust for this information?
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
* What entities are able to contribute to reputations?
* How do we access reputation about entities?
* Who is authorized to access particular reputations? How much to reveal? (Information flow)

==What technologies currently exist?==
* Digital signatures
** Certificates signed by trusted organizations

* Black hole- email, spam,
* Google - search reputation
* Credit bureaus
* Yellow pages
* Better business bureau
* CRC - criminal records

== What technologies don't currently exist?==

==Guaranteeing Authenticity/Public Key Infrastructure==

In our paper we must explain why PKI/Authentication fits into reputation. Why must it be handled by both Attribution and Reputation systems?

===Problem Domain===

This portion of a reputation system answers the core question of how reputation information being exchanged is guaranteed to be authentic.

*How do we ensure the information exchanged between peers authentic, and not tampered with?

*How to we attribute information exchanged?

*How do we do this in a highly decentralized, distributed system?

*How can we make sure the information is timely?

===Introduction===
In past few years, Internet has provided platform for a global market place and both business and private users realizes that the revolutionary communications opportunities provided by it will give way to large spectrum of business and private applications.Today online users face multitude of problems and issues like vulnerability to viruses , worms , exposure to sniffers, spoofing their private sessions not only this but also they are also subjected to invasion of privacy with multitude of spy ware available for monitoring how they behave. Today over the internet different kind of activities take place ranging from access to information to entertainment, financial services, product services and even socializing. The frequent usage of internet as an important business tool led to a major increase in deliberate abuse and criminal activities. All the organization operating electronically and trading expose their own information and IT systems to a wide range of security threats. The most common protocols like IP/TCP/UDP are the main targets of potential hackers. Its all because of IPs on which attacks are possible and don't have proper authentication mechanism for any incoming data over internet.

In order to build secure chain of trust Public-Key Infrastructure is used for internet based communication. It consists of various things like security policy , Certificate authority , registration authority , certificate distribution system PKI enabled applications.

===Uses and Need of PKI===
With development of modern e-commerce based businesses which has minimal customer face-to-face interactions is demanding more security and integrity. The online web based stores where huge amount of transactions take place needs to ensure customers that there information is confidential and processed through a secure channel. This is where implementation of PKI steps in to provide mechanisms to ensure trusted relationships are established and maintained. The specific security functions in which a PKI can provide foundation are confidentiality, integrity, non-repudiation,and authentication.

PKI provides mean of guaranteeing the authenticity by issuing digital signature.

===Issues & Solutions===
I found out there are many different implementations of PKI , and they all focuses on their own issues and solutions. For example PKI used in DoD have following issues

*Lack of PKI-enabled eCommerce applications and lack of interoperability among PKI applications

*DoD is developing a single high assurance PKI

*Very High Cost Impact to the EC/EB community.

*The PKI community lacks metrics for mapping of trust models between the DoD :”high assurance” C2 and EC/EB domains

*Education of everyone (policy maker through user) to a common level of understanding is a huge challenge.

*While the purpose of using PKI in EC/EB is to provide additional trust to allow the Internet to serve as a vehicle for legally binding transactions , problems still exist with the methodologies associated with establishing a long-term burden of proof. Specifically, there are no widely adopted industry standards for maintenance of electronic signatures or for authenticated timestamps for record maintenance that have stood the test of time. These processes are untried and the case law has not yet been established to convince users that there are no issues with enforcement of these new processes. An additional barrier to EC/EB within this space is the current DoD Certificate policy in which DoD accepts

===Common Issues With PKI Implementation===

*Commercial Off-The-Shelf (COTS) versus Customised applications : The choice between COTS or customised products is usually one of cost versus usability. In case of usability the thing to be focused should be error messages. If PKI is built int o applications (transparent to users) than its fine if not than user will require to have some understanding of the use of keys, certificates, Certificate Revocation Lists (CRLs) and directories/certificate repositories so that they can make informed decisions.

*Token Logistics (smart card): The point where keys and certificates are linked to their owner is a very critical point in a PKI. If a fraudulent certificate is issued by a registration officer and the certificate holder uses the certificate to commit a crime or prank, trust in the whole PKI hierarchy may be lost. The physical security requirements are high, and the registration officer, whether a person or a smartcard bureau, must be subject to strict security polices and practices. As it was problem with DoD mentioned in section above.

*Network issues - Traffic : There is no doubt that the implementation of PKI will add to the network load, although just how much depends on the system architecture. Potential additional traffic that should be considered includes: Certificate issuance, Email usage, CRLs , and Directory Replication

*Network issues - Encryption : Many organisations implement anti-virus software and content inspection on servers at the perimeter of their networks. Some have security policies that rejects or quarantines encrypted traffic. To provide user-to-user confidentiality, messages will traverse networks with their payload hidden from inspection by virus and content checking.

*Email address in certificate :In order to use certificates for S/MIME signed/encrypted email, the users’ email address must be in the certificate. Most people change their email addresses more frequently than the certificate. Unless a solution is built which allows users to keep the same email address over a long period, certificates would have to be re-issued every time a user changes email address. S/MIME v.3 stipulates that the receiving application must check the From: or Sender: field in the mail header and compare it to an email address in the sender’s certificate. If the check does not match, the mail application should perform another explicit check to ensure that the person who signed the message is indeed the person who sent it. As usual, the ‘devil is in the detail’ when it comes to implementation.

*Certificate Validity Checking:CRLs have been the conventional method of providing certificate validity checking. CRLs do not scale very well as discussed earlier, but are usually kept for backward compatibility, archiving/historical verification and for use in off-line mode. The other issue with CRLs is that they are generally issued at certain intervals of 6, 12 or 24 hours, causing a time lag from the time a certificate is revoked until it appears on the published CRL. This may present a security risk, as a certificate may verify correctly after it has been reported as compromised and revoked; (however some would argue that the time from actual compromise until the discovery and reporting of it would in most cases be a more significant lag). The Online Certificate Status Protocol (OCSP) (RFC2560) allows a client to query an OCSP responder for the current status of a certificate. This saves searching through a large CRL and can save bandwidth if the CRL would normally be downloaded - although it may increase network traffic. Most OCSP responders are based on CRLs and thus do not solve the problem of time lag as outlined above.

*Availability and storage of reliable user information : For an identity certificate scheme, names in certificates need to be unique, meaningful - and correct. Few large user communities have all their member details in a central and accurate database or directory, and the exercise of consolidating, checking and updating all user data can turn into a massive and expensive exercise.

*Archiving/historic verification : Digital signatures need to be verifiable even after the keys used to sign have expired. Likewise, we need to be able to verify that the certificate was valid at the time the datawas signed. This means we would need to archive: the signed file,the public key certificate of the signer, the CRL that was valid at the time of signing, a reliable timestamp to prove the accuracy of the time of signing and, the hardware environment that can run the software that was used at the time

===What are the solution to these problems?===

*Identity Based Encryption:enables senders to encrypt messages for recipients w/o requiring that a recipients key first be established, certified, and published.

*Certificate-based encryption:it incorporates IBE methods, but uses double encryption so that its CA cant decrypt on behalf of the user.

*Certificateless Public Key Cryptography:it incorporates IBE methods, using partial private keys so PKG cant decrypt on behalf of the user.

*Distributed Computation:There exists methods that distribute cryptographic operations so that the cooperative contribution of a number of entities is required in order to perform an operation such as a signature or decryption. It helps in tighter protection at servers vs clients, but implies that the users mist fully trust servers to apply keys appropriately.

*Alternative Validation Strategies:Hash tree:it offers compact protected representations of the status of large number of certiticates.Highly valued if PKI is operated large scale more benifical than Certificate Revovation List. CRL reflect ststus information at fixed intervals.

==Dissemination==

===The Problem Domain===

===Random Ramblings on Reputation Management and Distribution===

Publish/Subscribe?

This system has unique distribution requirements as compared to most distributed systems in general. In this system, we cannot assume that there will be a universally agreed-upon definition of good, or bad. Similarly, the system must be self-policing. It would be up to each and every group of autonomous systems to decide which updates to accept and reject. Updates themselves also should not cause the network to DDoS itself. Lastly, it would be impossible for every system to know what the reputation for a given system is. Therefore the system must disseminate information in some way that is query-able and localizes reputation information where required.

To this end, we need a way of spreading information that while reliable, does not depend on one universally agreed-upon set of reputations.

For example, on an internet-scale operating system it would be entirely reasonable for one group of systems to not want to accept updates, or want to avoid communication with a given series of systems.

Any solution would assume that the problems of attribution are solved.

===Current Examples of Reputation Dissemination===

The first protocol that immediately comes to mind in this situation is a gossip-based protocol. These protocols are designed to operate in highly decentralized, large-scale systems.

Here's a nice overview:
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4537308 "Reputation management in distributed systems"

Examples are as follows:
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4228013 "Gossip-based Reputation Aggregation for Unstructured Peer-to-Peer Networks"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=5569965 "Improving Accuracy and Coverage in an Internet-Deployed Reputation Mechanism"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4459326 "GossipTrust for Fast Reputation Aggregation in Peer-to-Peer Networks"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4777496 "Adaptive trust management in P2P networks using gossip protocol"

Another possibility is using "Reputation chains"
* http://dx.doi.org.proxy.library.carleton.ca/10.1109/TKDE.2009.45 "P2P Reputation Management Using Distributed Identities and Decentralized Recommendation Chains"

==Maintaining History==

===Problem domain===

* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
*** Do we maintain records based on a fixed set of imposed rules? Or do we build rules as the system emerges and reputations are formed?
* Where do you store the data?
** Distributed storage systems. Reputation in real-life is stored in interactions that an entity has with others. Reputation is not stored centrally. Reputation is most often a shared view of an entity by the masses, but sometimes an entities reputation can be disjoint among the masses: many different entities having differing views of reputation for the same entity.
* Where is the data queried from?
** (should I mention this?)
* What defines good/bad reputation?
** (should I mention this?)
* Who provides the good/bad reputation?
** Impose/Emerge problem: reputation for an interaction can be calculated immediately or can be a function of time.
* Who do we trust for this information?
** Trusting the masses is generally a good way of ensuring trustworthiness. Imposed rules will not always fit every situation well - could potentially set bad reputation to a "good" entity.
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
** Do we maintain an ever-growing set of history items for interactions between entities? Do we look focus on the bad reputations?
* What entities are able to contribute to reputations?
* How do we access reputation about entities?
* Who is authorized to access particular reputations? How much to reveal? (Information flow)
* What assumptions will we make?
* Privacy issues? What will we reveal? Will centralized systems have a know-all mentality?
** Fine grained information will never be revealed (privacy concerns and user rights)
* Which history should I maintain? What to take as important, what to disregard?
* Immutable data structure. Who could add data? Who could remove data? Authority

===Reputation systems===
* record, aggregate, distribute information about an entity's behaviour in distributed applications

* reputation might be based on the entity's past ability to adhere to a license agreement (mutual contract between issuer and licensee)

===History-based access control systems===
* make decision based on an entity's past security-sensitive actions

===Examples of reputation systems (trust-informing technologies)===
* eBay - Feedback forum (positive, neutral, negative)

===Do reputation systems have some validity?===

Resnick et al. argue that reputation systems
foster an incentive for principals to well-behave because of “the expectation of
reciprocity or retaliation in future interactions

Abstractions are used to model the aggregated information of each entity. These abstractions may not encompass the full details of transactions and provide context to specific issues relating to feedback. In turn we can end up with ambiguous values.

So we need a system that provides sufficient information in order to verify the precise properties of a past behaviour.

* Krukow, K. A Logical Framework for Reputation Systems and History-based Access Control. School of Electronics and Computer Science University of Southampton, UK. (March 3, 2011) [http://www.brics.dk/~krukow/research/publications/online_papers/concrete-jcs.pdf]

====Abstract====
Reputation systems are meta systems that record, aggregate and distribute information about principals’ behaviour in distributed applications. Similarly, history-based access control systems make decisions based
on programs’ past security-sensitive actions. While the applications are
distinct, the two types of systems are fundamentally making decisions
based on information about the past behaviour of an entity.
A logical policy-centric framework for such behaviour-based decisionmaking is presented. In the framework, principals specify policies which
state precise requirements on the past behaviour of other principals that
must be fulﬁlled in order for interaction to take place. The framework consists of a formal model of behaviour, based on event structures; a declarative logical language for specifying properties of past behaviour; and
eﬃcient dynamic algorithms for checking whether a particular behaviour
satisﬁes a property from the language. It is shown how the framework can
be extended in several ways, most notably to encompass parameterized
events and quantiﬁcation over parameters. In an extended application, it
is illustrated how the framework can be applied for dynamic history-based
access control for safe execution of unknown and untrusted programs.

* Khosrow-Pour, M. Emerging trends and challenges in information technology management (March 7, 2011) [http://books.google.ca/books?id=ybzS-yylJfAC&lpg=PA822&ots=V7hn_RzqXA&dq=maintaining%20history%20in%20reputation%20systems&pg=PA822#v=onepage&q=maintaining%20history%20in%20reputation%20systems&f=false]

====Abstract====

* Bolton, G. et al. How Effective are Electronic Reputation Mechanisms? (March 10, 2011) [http://ccs.mit.edu/dell/reputation/BKOMSsub.pdf]

====Abstract====

Electronic reputation or “feedback” mechanisms aim to mitigate the moral hazard problems
associated with exchange among strangers by providing the type of information available in
more traditional close-knit groups, where members are frequently involved in one another’s
dealings. In this paper, we compare trading in a market with electronic feedback (as
implemented by many Internet markets) to a market without, as well as to a market in which the
same people interact with one another repeatedly (partners market). We find that, while the
feedback mechanism induces quite a substantial improvement in transaction efficiency, it also
exhibits a kind of public goods problem in that, unlike the partners market, the benefits of trust
and trustworthy behavior go to the whole community and are not completely internalized. We
discuss the implications of this perspective for improving these systems.

This portion of a reputation system answers the core question of how reputation is generated from the information exchanged between systems and how/where it is stored.

==Problem domain:==

This portion of a reputation system answers the core question of how reputation is generated from the information exchanged between systems and how/where it is stored.

Problem domain:

• Emerge vs. Impose reputation on the system?
• Probably both, how do we account for both systems?
• Do we maintain records based on a fixed set of imposed rules? Or do we build rules as the system emerges and reputations are formed?
• Where do you store the data?
• Distributed storage systems. Reputation in real-life is stored in interactions that an entity has with others. Reputation is not stored centrally. Reputation is most often a shared view of an entity by the masses, but sometimes an entities reputation can be disjoint among the masses: many different entities having differing views of reputation for the same entity.
• Who do we trust for this information?
• Trusting the masses is generally a good way of ensuring trustworthiness. Imposed rules will not always fit every situation well - could potentially set bad reputation to a "good" entity.
• Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
• Do we maintain an ever-growing set of history items for interactions between entities? Do we look focus on the bad reputations

Existing systems
• Peer-based systems (emerge)
• eBay - positive/negative rating system
• Youtube - like/dislike/spam comment system
• Policy-based systems (impose)
• Java - policy based security
• Android - policy based security
These two systems are on opposite ends of the Emerge-Impose spectrum.
EigenTrust system
The EigenTrust system utilizes a numerical scale for reputation storage.
Advantages:
• Numerical values are easy to compare.
• Little required storage space.
Disadvantages:
• Information is lost in the abstraction process.
o No concrete data
• Ambiguity
Storing concrete data
Hence, with the given information from the reputation system, we cannot generate an accurate profile of the entity.
We need a system that represents reputation in a concrete form
“If principal p gains access to resource r at time t, then the past behavior of p up until time t satisfies requirement ψr.”
Advantages
• A sufficient amount of information is available to come up with a profile of an entity
Disadvantages
• Storage space
Shmatikov and Talcott
Histories are sets of time-stamped events. Reputation is based on ability to adhere to licenses.
Licenses might permit certain actions OR require certain actions from being performed.
Advantages:
• Store data in concrete form
Disadvantages:
• No notion of sessions (logically connected set of events)
Representation of reputation
If we consider reputation information to encompass the events and actions of an entity, then we can model reputation as a set of events.
An interesting new problem is how to re-evaluate policies efficiently when new information becomes available.
This problem is known as “dynamic model-checking”.
Dynamic model-checking
We want a way of summarizing past reputations.
A solution here is to use: Havelund and Rosu, based on the technique of dynamic programming, used for runtime verification
• Given some information on an entity, how do we convert/abstract this to reputation?
• Is all the information necessary to maintain?
• Now that we have the reputation information, what can we do with it?
o What can we compare it with?
Implementation
Desired functionality of the system:
new()
- Append new reputation information
update()
- Update and summarize past behavior
- This is a reduce function
check()
- Analyze whether the given reputation satisfies the criteria of the policy

==Querying Reputation==

=== Problems ===

* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
***If you want to know someone's reputation, you either need to start asking around for it, imposing yourself. Or you need the data to be sent around, so you already have access to it; emergent.
* Where do you store the data?
***You need to know who has the data to ask them for it, or to go get it yourself.
* Where is the data queried from?
***First you need to know who's storing it. then you need to know if you're allowed to ask that node directly, do you ask a intermediary keeper of data. Will you even need to Query-- that is, do you already have all you need to know on hand? you need not get the latest updates on a node if every other node who's ever talked to it got DDOSed. (or do you?)
* What defines good/bad reputation?
***Should I make my own definition for bad reputation, and query if someone engaged in activities I consider bad, or should their be a global agreed upon reputation?
* Who provides the good/bad reputation?
***Who should I ask for information from?
* Who do we trust for this information?
***Whoever you trust, presumably their opinion on a given node is more important then a node you trust less.
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
***topically, would you bother asking for 10 year old reputation data on a node, if it's been a model citizen for the last 9?
* What entities are able to contribute to reputations?
***Should I ask everyone I trust for an opinion on a given node, or just certain keepers of trust data?
* How do we access reputation about entities?
***You query someone in the know who you trust and are allowed to query.
***you could say, ask everyone you know and trust, and ask them to ask people they know and trust, (and so on...if they're willing) until you find a node with the information you need.
***in a more centralized system you need to ask some kind of keeper of information for the information you want, and that keeper may or may not provide you with the reputation info you want.
* Who is authorized to access particular reputations? How much to reveal? (Information flow)
***The ability to control this would depend on how centralized a system you have. In a truly distributed system where every node has an opinion on any other node they've talked to you'll be able to find somebody who can tell you about the CIA node, but in a more centralized system the keepers of information might be less...willing to give Joe 6 cores information on who Iran is DDOSing.

===Maybe References===
http://www.kirkarts.com/wiki/images/1/13/Resnick_eBay.pdf - ''Trust Among Strangers in Internet Transactions:
Empirical Analysis of eBay’s Reputation System'' (maybe not too relevant)

http://portal.acm.org/citation.cfm?id=544741.544809 - ''An Evidential Model of Distributed Reputation Management''

http://portal.acm.org/citation.cfm?id=775152.775242&type=series%EF%BF%BD%C3%9C -- ''The EigenTrust Algorithm for Reputation Management in
P2P Networks''

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.4.2297&rep=rep1&type=pdf -- ''A Robust Reputation System for Mobile Ad-hoc
Networks''

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.125.8729&rep=rep1&type=pdf -- ''EigenRep: Reputation Management in P2P Networks''

http://www.chennaisunday.com/ieee%202010/Reputation%20Estimation%20and%20Query%20in%20Peer-to-Peer%20Networks.pdf -- ''Reputation Estimation and Query in Peer-to-Peer Networks''

Here is another paper that might be interesting for you. -- Lester
http://dcg.ethz.ch/publications/netecon06.pdf

==Possible implementations==
==Implementation Requirements==
==Conclusion==

==References==
* Joel Weise : "Public Key Infrastructure Overview " http://www.sun.com/blueprints/0801/publickey.pdf Accessed 2nd March 2011

* Security Glossary : http://www.cafesoft.com/support/security-glossary.html Accessed on 2nd March 2011

* Mattila, Anssi; and Mattila, Minna "What is the Effect of Product Attributes on Public-Key Infrastructure adoption? " http://internetjournals.net/journals/tir/2006/January/Paper%2003.pdf Accessed on 2nd March 2011

*Electronic Commerece Conference , PKI Sub-Group , Issue Paper : http://www.defense.gov/dodreform/ecwg/pki.pdf date accessed 5th March 2011

*SANS Institute InfoSec Reading Room, Common issues in PKI implementations - climbing the Slope of Enlightenment : http://www.sans.org/reading_room/whitepapers/authentication/common-issues-pki-implementations-climbing-slope-enlightenment_1198 date accessed 15th March 2011

DistOS-2011W Reputation

2011-04-08T23:32:03Z

Wahmed2: /* Uses and Need */

==Members==
* Waheed Ahmed
* Trevor Gelowsky
** MSN: Gelowt@gmail.com
** E-Mail: tgelowsk@sce.carleton.ca
* Michael Du Plessis
* Nicolas Lessard (nick.lessard @t gmail.com / nlessard @t carleton.connect.ca)

==Our presentation==
Our current presentation can be viewed at the following link: https://docs.google.com/present/edit?id=0ASS7kj9hfc1aZGRiMjMzOHJfNGhnNzhuamRr&hl=en&authkey=CMHi3KAD

==Our Paper==

Our final paper can be found here: [[Distributed OS: Winter 2011 Reputation Systems Paper]]

==The problem==
* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
* Where do you store the data?
* Where is the data queried from?
* What defines good/bad reputation?
* Who provides the good/bad reputation?
* Who do we trust for this information?
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
* What entities are able to contribute to reputations?
* How do we access reputation about entities?
* Who is authorized to access particular reputations? How much to reveal? (Information flow)

==What technologies currently exist?==
* Digital signatures
** Certificates signed by trusted organizations

* Black hole- email, spam,
* Google - search reputation
* Credit bureaus
* Yellow pages
* Better business bureau
* CRC - criminal records

== What technologies don't currently exist?==

==Guaranteeing Authenticity/Public Key Infrastructure==

In our paper we must explain why PKI/Authentication fits into reputation. Why must it be handled by both Attribution and Reputation systems?

===Problem Domain===

This portion of a reputation system answers the core question of how reputation information being exchanged is guaranteed to be authentic.

*How do we ensure the information exchanged between peers authentic, and not tampered with?

*How to we attribute information exchanged?

*How do we do this in a highly decentralized, distributed system?

*How can we make sure the information is timely?

===Introduction===
In past few years, Internet has provided platform for a global market place and both business and private users realizes that the revolutionary communications opportunities provided by it will give way to large spectrum of business and private applications.Today online users face multitude of problems and issues like vulnerability to viruses , worms , exposure to sniffers, spoofing their private sessions not only this but also they are also subjected to invasion of privacy with multitude of spy ware available for monitoring how they behave. Today over the internet different kind of activities take place ranging from access to information to entertainment, financial services, product services and even socializing. The frequent usage of internet as an important business tool led to a major increase in deliberate abuse and criminal activities. All the organization operating electronically and trading expose their own information and IT systems to a wide range of security threats. The most common protocols like IP/TCP/UDP are the main targets of potential hackers. Its all because of IPs on which attacks are possible and don't have proper authentication mechanism for any incoming data over internet.

In order to build secure chain of trust Public-Key Infrastructure is used for internet based communication. It consists of various things like security policy , Certificate authority , registration authority , certificate distribution system PKI enabled applications.

===Uses and Need of PKI===
With development of modern e-commerce based businesses which has minimal customer face-to-face interactions is demanding more security and integrity. The online web based stores where huge amount of transactions take place needs to ensure customers that there information is confidential and processed through a secure channel. This is where implementation of PKI steps in to provide mechanisms to ensure trusted relationships are established and maintained. The specific security functions in which a PKI can provide foundation are confidentiality, integrity, non-repudiation,and authentication.

===Issues & Solutions===
I found out there are many different implementations of PKI , and they all focuses on their own issues and solutions. For example PKI used in DoD have following issues

*Lack of PKI-enabled eCommerce applications and lack of interoperability among PKI applications

*DoD is developing a single high assurance PKI

*Very High Cost Impact to the EC/EB community.

*The PKI community lacks metrics for mapping of trust models between the DoD :”high assurance” C2 and EC/EB domains

*Education of everyone (policy maker through user) to a common level of understanding is a huge challenge.

*While the purpose of using PKI in EC/EB is to provide additional trust to allow the Internet to serve as a vehicle for legally binding transactions , problems still exist with the methodologies associated with establishing a long-term burden of proof. Specifically, there are no widely adopted industry standards for maintenance of electronic signatures or for authenticated timestamps for record maintenance that have stood the test of time. These processes are untried and the case law has not yet been established to convince users that there are no issues with enforcement of these new processes. An additional barrier to EC/EB within this space is the current DoD Certificate policy in which DoD accepts

===Common Issues With PKI Implementation===

*Commercial Off-The-Shelf (COTS) versus Customised applications : The choice between COTS or customised products is usually one of cost versus usability. In case of usability the thing to be focused should be error messages. If PKI is built int o applications (transparent to users) than its fine if not than user will require to have some understanding of the use of keys, certificates, Certificate Revocation Lists (CRLs) and directories/certificate repositories so that they can make informed decisions.

*Token Logistics (smart card): The point where keys and certificates are linked to their owner is a very critical point in a PKI. If a fraudulent certificate is issued by a registration officer and the certificate holder uses the certificate to commit a crime or prank, trust in the whole PKI hierarchy may be lost. The physical security requirements are high, and the registration officer, whether a person or a smartcard bureau, must be subject to strict security polices and practices. As it was problem with DoD mentioned in section above.

*Network issues - Traffic : There is no doubt that the implementation of PKI will add to the network load, although just how much depends on the system architecture. Potential additional traffic that should be considered includes: Certificate issuance, Email usage, CRLs , and Directory Replication

*Network issues - Encryption : Many organisations implement anti-virus software and content inspection on servers at the perimeter of their networks. Some have security policies that rejects or quarantines encrypted traffic. To provide user-to-user confidentiality, messages will traverse networks with their payload hidden from inspection by virus and content checking.

*Email address in certificate :In order to use certificates for S/MIME signed/encrypted email, the users’ email address must be in the certificate. Most people change their email addresses more frequently than the certificate. Unless a solution is built which allows users to keep the same email address over a long period, certificates would have to be re-issued every time a user changes email address. S/MIME v.3 stipulates that the receiving application must check the From: or Sender: field in the mail header and compare it to an email address in the sender’s certificate. If the check does not match, the mail application should perform another explicit check to ensure that the person who signed the message is indeed the person who sent it. As usual, the ‘devil is in the detail’ when it comes to implementation.

*Certificate Validity Checking:CRLs have been the conventional method of providing certificate validity checking. CRLs do not scale very well as discussed earlier, but are usually kept for backward compatibility, archiving/historical verification and for use in off-line mode. The other issue with CRLs is that they are generally issued at certain intervals of 6, 12 or 24 hours, causing a time lag from the time a certificate is revoked until it appears on the published CRL. This may present a security risk, as a certificate may verify correctly after it has been reported as compromised and revoked; (however some would argue that the time from actual compromise until the discovery and reporting of it would in most cases be a more significant lag). The Online Certificate Status Protocol (OCSP) (RFC2560) allows a client to query an OCSP responder for the current status of a certificate. This saves searching through a large CRL and can save bandwidth if the CRL would normally be downloaded - although it may increase network traffic. Most OCSP responders are based on CRLs and thus do not solve the problem of time lag as outlined above.

*Availability and storage of reliable user information : For an identity certificate scheme, names in certificates need to be unique, meaningful - and correct. Few large user communities have all their member details in a central and accurate database or directory, and the exercise of consolidating, checking and updating all user data can turn into a massive and expensive exercise.

*Archiving/historic verification : Digital signatures need to be verifiable even after the keys used to sign have expired. Likewise, we need to be able to verify that the certificate was valid at the time the datawas signed. This means we would need to archive: the signed file,the public key certificate of the signer, the CRL that was valid at the time of signing, a reliable timestamp to prove the accuracy of the time of signing and, the hardware environment that can run the software that was used at the time

===What are the solution to these problems?===

*Identity Based Encryption:enables senders to encrypt messages for recipients w/o requiring that a recipients key first be established, certified, and published.

*Certificate-based encryption:it incorporates IBE methods, but uses double encryption so that its CA cant decrypt on behalf of the user.

*Certificateless Public Key Cryptography:it incorporates IBE methods, using partial private keys so PKG cant decrypt on behalf of the user.

*Distributed Computation:There exists methods that distribute cryptographic operations so that the cooperative contribution of a number of entities is required in order to perform an operation such as a signature or decryption. It helps in tighter protection at servers vs clients, but implies that the users mist fully trust servers to apply keys appropriately.

*Alternative Validation Strategies:Hash tree:it offers compact protected representations of the status of large number of certiticates.Highly valued if PKI is operated large scale more benifical than Certificate Revovation List. CRL reflect ststus information at fixed intervals.

==Dissemination==

===The Problem Domain===

===Random Ramblings on Reputation Management and Distribution===

Publish/Subscribe?

This system has unique distribution requirements as compared to most distributed systems in general. In this system, we cannot assume that there will be a universally agreed-upon definition of good, or bad. Similarly, the system must be self-policing. It would be up to each and every group of autonomous systems to decide which updates to accept and reject. Updates themselves also should not cause the network to DDoS itself. Lastly, it would be impossible for every system to know what the reputation for a given system is. Therefore the system must disseminate information in some way that is query-able and localizes reputation information where required.

To this end, we need a way of spreading information that while reliable, does not depend on one universally agreed-upon set of reputations.

For example, on an internet-scale operating system it would be entirely reasonable for one group of systems to not want to accept updates, or want to avoid communication with a given series of systems.

Any solution would assume that the problems of attribution are solved.

===Current Examples of Reputation Dissemination===

The first protocol that immediately comes to mind in this situation is a gossip-based protocol. These protocols are designed to operate in highly decentralized, large-scale systems.

Here's a nice overview:
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4537308 "Reputation management in distributed systems"

Examples are as follows:
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4228013 "Gossip-based Reputation Aggregation for Unstructured Peer-to-Peer Networks"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=5569965 "Improving Accuracy and Coverage in an Internet-Deployed Reputation Mechanism"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4459326 "GossipTrust for Fast Reputation Aggregation in Peer-to-Peer Networks"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4777496 "Adaptive trust management in P2P networks using gossip protocol"

Another possibility is using "Reputation chains"
* http://dx.doi.org.proxy.library.carleton.ca/10.1109/TKDE.2009.45 "P2P Reputation Management Using Distributed Identities and Decentralized Recommendation Chains"

==Maintaining History==

===Problem domain===

* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
*** Do we maintain records based on a fixed set of imposed rules? Or do we build rules as the system emerges and reputations are formed?
* Where do you store the data?
** Distributed storage systems. Reputation in real-life is stored in interactions that an entity has with others. Reputation is not stored centrally. Reputation is most often a shared view of an entity by the masses, but sometimes an entities reputation can be disjoint among the masses: many different entities having differing views of reputation for the same entity.
* Where is the data queried from?
** (should I mention this?)
* What defines good/bad reputation?
** (should I mention this?)
* Who provides the good/bad reputation?
** Impose/Emerge problem: reputation for an interaction can be calculated immediately or can be a function of time.
* Who do we trust for this information?
** Trusting the masses is generally a good way of ensuring trustworthiness. Imposed rules will not always fit every situation well - could potentially set bad reputation to a "good" entity.
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
** Do we maintain an ever-growing set of history items for interactions between entities? Do we look focus on the bad reputations?
* What entities are able to contribute to reputations?
* How do we access reputation about entities?
* Who is authorized to access particular reputations? How much to reveal? (Information flow)
* What assumptions will we make?
* Privacy issues? What will we reveal? Will centralized systems have a know-all mentality?
** Fine grained information will never be revealed (privacy concerns and user rights)
* Which history should I maintain? What to take as important, what to disregard?
* Immutable data structure. Who could add data? Who could remove data? Authority

===Reputation systems===
* record, aggregate, distribute information about an entity's behaviour in distributed applications

* reputation might be based on the entity's past ability to adhere to a license agreement (mutual contract between issuer and licensee)

===History-based access control systems===
* make decision based on an entity's past security-sensitive actions

===Examples of reputation systems (trust-informing technologies)===
* eBay - Feedback forum (positive, neutral, negative)

===Do reputation systems have some validity?===

Resnick et al. argue that reputation systems
foster an incentive for principals to well-behave because of “the expectation of
reciprocity or retaliation in future interactions

Abstractions are used to model the aggregated information of each entity. These abstractions may not encompass the full details of transactions and provide context to specific issues relating to feedback. In turn we can end up with ambiguous values.

So we need a system that provides sufficient information in order to verify the precise properties of a past behaviour.

* Krukow, K. A Logical Framework for Reputation Systems and History-based Access Control. School of Electronics and Computer Science University of Southampton, UK. (March 3, 2011) [http://www.brics.dk/~krukow/research/publications/online_papers/concrete-jcs.pdf]

====Abstract====
Reputation systems are meta systems that record, aggregate and distribute information about principals’ behaviour in distributed applications. Similarly, history-based access control systems make decisions based
on programs’ past security-sensitive actions. While the applications are
distinct, the two types of systems are fundamentally making decisions
based on information about the past behaviour of an entity.
A logical policy-centric framework for such behaviour-based decisionmaking is presented. In the framework, principals specify policies which
state precise requirements on the past behaviour of other principals that
must be fulﬁlled in order for interaction to take place. The framework consists of a formal model of behaviour, based on event structures; a declarative logical language for specifying properties of past behaviour; and
eﬃcient dynamic algorithms for checking whether a particular behaviour
satisﬁes a property from the language. It is shown how the framework can
be extended in several ways, most notably to encompass parameterized
events and quantiﬁcation over parameters. In an extended application, it
is illustrated how the framework can be applied for dynamic history-based
access control for safe execution of unknown and untrusted programs.

* Khosrow-Pour, M. Emerging trends and challenges in information technology management (March 7, 2011) [http://books.google.ca/books?id=ybzS-yylJfAC&lpg=PA822&ots=V7hn_RzqXA&dq=maintaining%20history%20in%20reputation%20systems&pg=PA822#v=onepage&q=maintaining%20history%20in%20reputation%20systems&f=false]

====Abstract====

* Bolton, G. et al. How Effective are Electronic Reputation Mechanisms? (March 10, 2011) [http://ccs.mit.edu/dell/reputation/BKOMSsub.pdf]

====Abstract====

Electronic reputation or “feedback” mechanisms aim to mitigate the moral hazard problems
associated with exchange among strangers by providing the type of information available in
more traditional close-knit groups, where members are frequently involved in one another’s
dealings. In this paper, we compare trading in a market with electronic feedback (as
implemented by many Internet markets) to a market without, as well as to a market in which the
same people interact with one another repeatedly (partners market). We find that, while the
feedback mechanism induces quite a substantial improvement in transaction efficiency, it also
exhibits a kind of public goods problem in that, unlike the partners market, the benefits of trust
and trustworthy behavior go to the whole community and are not completely internalized. We
discuss the implications of this perspective for improving these systems.

This portion of a reputation system answers the core question of how reputation is generated from the information exchanged between systems and how/where it is stored.

==Problem domain:==

This portion of a reputation system answers the core question of how reputation is generated from the information exchanged between systems and how/where it is stored.

Problem domain:

• Emerge vs. Impose reputation on the system?
• Probably both, how do we account for both systems?
• Do we maintain records based on a fixed set of imposed rules? Or do we build rules as the system emerges and reputations are formed?
• Where do you store the data?
• Distributed storage systems. Reputation in real-life is stored in interactions that an entity has with others. Reputation is not stored centrally. Reputation is most often a shared view of an entity by the masses, but sometimes an entities reputation can be disjoint among the masses: many different entities having differing views of reputation for the same entity.
• Who do we trust for this information?
• Trusting the masses is generally a good way of ensuring trustworthiness. Imposed rules will not always fit every situation well - could potentially set bad reputation to a "good" entity.
• Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
• Do we maintain an ever-growing set of history items for interactions between entities? Do we look focus on the bad reputations

Existing systems
• Peer-based systems (emerge)
• eBay - positive/negative rating system
• Youtube - like/dislike/spam comment system
• Policy-based systems (impose)
• Java - policy based security
• Android - policy based security
These two systems are on opposite ends of the Emerge-Impose spectrum.
EigenTrust system
The EigenTrust system utilizes a numerical scale for reputation storage.
Advantages:
• Numerical values are easy to compare.
• Little required storage space.
Disadvantages:
• Information is lost in the abstraction process.
o No concrete data
• Ambiguity
Storing concrete data
Hence, with the given information from the reputation system, we cannot generate an accurate profile of the entity.
We need a system that represents reputation in a concrete form
“If principal p gains access to resource r at time t, then the past behavior of p up until time t satisfies requirement ψr.”
Advantages
• A sufficient amount of information is available to come up with a profile of an entity
Disadvantages
• Storage space
Shmatikov and Talcott
Histories are sets of time-stamped events. Reputation is based on ability to adhere to licenses.
Licenses might permit certain actions OR require certain actions from being performed.
Advantages:
• Store data in concrete form
Disadvantages:
• No notion of sessions (logically connected set of events)
Representation of reputation
If we consider reputation information to encompass the events and actions of an entity, then we can model reputation as a set of events.
An interesting new problem is how to re-evaluate policies efficiently when new information becomes available.
This problem is known as “dynamic model-checking”.
Dynamic model-checking
We want a way of summarizing past reputations.
A solution here is to use: Havelund and Rosu, based on the technique of dynamic programming, used for runtime verification
• Given some information on an entity, how do we convert/abstract this to reputation?
• Is all the information necessary to maintain?
• Now that we have the reputation information, what can we do with it?
o What can we compare it with?
Implementation
Desired functionality of the system:
new()
- Append new reputation information
update()
- Update and summarize past behavior
- This is a reduce function
check()
- Analyze whether the given reputation satisfies the criteria of the policy

==Querying Reputation==

=== Problems ===

* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
***If you want to know someone's reputation, you either need to start asking around for it, imposing yourself. Or you need the data to be sent around, so you already have access to it; emergent.
* Where do you store the data?
***You need to know who has the data to ask them for it, or to go get it yourself.
* Where is the data queried from?
***First you need to know who's storing it. then you need to know if you're allowed to ask that node directly, do you ask a intermediary keeper of data. Will you even need to Query-- that is, do you already have all you need to know on hand? you need not get the latest updates on a node if every other node who's ever talked to it got DDOSed. (or do you?)
* What defines good/bad reputation?
***Should I make my own definition for bad reputation, and query if someone engaged in activities I consider bad, or should their be a global agreed upon reputation?
* Who provides the good/bad reputation?
***Who should I ask for information from?
* Who do we trust for this information?
***Whoever you trust, presumably their opinion on a given node is more important then a node you trust less.
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
***topically, would you bother asking for 10 year old reputation data on a node, if it's been a model citizen for the last 9?
* What entities are able to contribute to reputations?
***Should I ask everyone I trust for an opinion on a given node, or just certain keepers of trust data?
* How do we access reputation about entities?
***You query someone in the know who you trust and are allowed to query.
***you could say, ask everyone you know and trust, and ask them to ask people they know and trust, (and so on...if they're willing) until you find a node with the information you need.
***in a more centralized system you need to ask some kind of keeper of information for the information you want, and that keeper may or may not provide you with the reputation info you want.
* Who is authorized to access particular reputations? How much to reveal? (Information flow)
***The ability to control this would depend on how centralized a system you have. In a truly distributed system where every node has an opinion on any other node they've talked to you'll be able to find somebody who can tell you about the CIA node, but in a more centralized system the keepers of information might be less...willing to give Joe 6 cores information on who Iran is DDOSing.

===Maybe References===
http://www.kirkarts.com/wiki/images/1/13/Resnick_eBay.pdf - ''Trust Among Strangers in Internet Transactions:
Empirical Analysis of eBay’s Reputation System'' (maybe not too relevant)

http://portal.acm.org/citation.cfm?id=544741.544809 - ''An Evidential Model of Distributed Reputation Management''

http://portal.acm.org/citation.cfm?id=775152.775242&type=series%EF%BF%BD%C3%9C -- ''The EigenTrust Algorithm for Reputation Management in
P2P Networks''

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.4.2297&rep=rep1&type=pdf -- ''A Robust Reputation System for Mobile Ad-hoc
Networks''

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.125.8729&rep=rep1&type=pdf -- ''EigenRep: Reputation Management in P2P Networks''

http://www.chennaisunday.com/ieee%202010/Reputation%20Estimation%20and%20Query%20in%20Peer-to-Peer%20Networks.pdf -- ''Reputation Estimation and Query in Peer-to-Peer Networks''

Here is another paper that might be interesting for you. -- Lester
http://dcg.ethz.ch/publications/netecon06.pdf

==Possible implementations==
==Implementation Requirements==
==Conclusion==

==References==
* Joel Weise : "Public Key Infrastructure Overview " http://www.sun.com/blueprints/0801/publickey.pdf Accessed 2nd March 2011

* Security Glossary : http://www.cafesoft.com/support/security-glossary.html Accessed on 2nd March 2011

* Mattila, Anssi; and Mattila, Minna "What is the Effect of Product Attributes on Public-Key Infrastructure adoption? " http://internetjournals.net/journals/tir/2006/January/Paper%2003.pdf Accessed on 2nd March 2011

*Electronic Commerece Conference , PKI Sub-Group , Issue Paper : http://www.defense.gov/dodreform/ecwg/pki.pdf date accessed 5th March 2011

*SANS Institute InfoSec Reading Room, Common issues in PKI implementations - climbing the Slope of Enlightenment : http://www.sans.org/reading_room/whitepapers/authentication/common-issues-pki-implementations-climbing-slope-enlightenment_1198 date accessed 15th March 2011

DistOS-2011W Reputation

2011-04-08T23:15:29Z

Wahmed2: /* Problems */

==Members==
* Waheed Ahmed
* Trevor Gelowsky
** MSN: Gelowt@gmail.com
** E-Mail: tgelowsk@sce.carleton.ca
* Michael Du Plessis
* Nicolas Lessard (nick.lessard @t gmail.com / nlessard @t carleton.connect.ca)

==Our presentation==
Our current presentation can be viewed at the following link: https://docs.google.com/present/edit?id=0ASS7kj9hfc1aZGRiMjMzOHJfNGhnNzhuamRr&hl=en&authkey=CMHi3KAD

==Our Paper==

Our final paper can be found here: [[Distributed OS: Winter 2011 Reputation Systems Paper]]

==The problem==
* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
* Where do you store the data?
* Where is the data queried from?
* What defines good/bad reputation?
* Who provides the good/bad reputation?
* Who do we trust for this information?
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
* What entities are able to contribute to reputations?
* How do we access reputation about entities?
* Who is authorized to access particular reputations? How much to reveal? (Information flow)

==What technologies currently exist?==
* Digital signatures
** Certificates signed by trusted organizations

* Black hole- email, spam,
* Google - search reputation
* Credit bureaus
* Yellow pages
* Better business bureau
* CRC - criminal records

== What technologies don't currently exist?==

==Guaranteeing Authenticity/Public Key Infrastructure==

In our paper we must explain why PKI/Authentication fits into reputation. Why must it be handled by both Attribution and Reputation systems?

===Problem Domain===

This portion of a reputation system answers the core question of how reputation information being exchanged is guaranteed to be authentic.

*How do we ensure the information exchanged between peers authentic, and not tampered with?

*How to we attribute information exchanged?

*How do we do this in a highly decentralized, distributed system?

*How can we make sure the information is timely?

===Introduction===
In past few years, Internet has provided platform for a global market place and both business and private users realizes that the revolutionary communications opportunities provided by it will give way to large spectrum of business and private applications.Today online users face multitude of problems and issues like vulnerability to viruses , worms , exposure to sniffers, spoofing their private sessions not only this but also they are also subjected to invasion of privacy with multitude of spy ware available for monitoring how they behave. Today over the internet different kind of activities take place ranging from access to information to entertainment, financial services, product services and even socializing. The frequent usage of internet as an important business tool led to a major increase in deliberate abuse and criminal activities. All the organization operating electronically and trading expose their own information and IT systems to a wide range of security threats. The most common protocols like IP/TCP/UDP are the main targets of potential hackers. Its all because of IPs on which attacks are possible and don't have proper authentication mechanism for any incoming data over internet.

In order to build secure chain of trust Public-Key Infrastructure is used for internet based communication. It consists of various things like security policy , Certificate authority , registration authority , certificate distribution system PKI enabled applications.

===Uses and Need===
With development of modern e-commerce based businesses which has minimal customer face-to-face interactions is demanding more security and integrity. The online web based stores where huge amount of transactions take place needs to ensure customers that there information is confidential and processed through a secure channel. This is where implementation of PKI steps in to provide mechanisms to ensure trusted relationships are established and maintained. The specific security functions in which a PKI can provide foundation are confidentiality, integrity, non-repudiation,and authentication.

===Issues & Solutions===
I found out there are many different implementations of PKI , and they all focuses on their own issues and solutions. For example PKI used in DoD have following issues

*Lack of PKI-enabled eCommerce applications and lack of interoperability among PKI applications

*DoD is developing a single high assurance PKI

*Very High Cost Impact to the EC/EB community.

*The PKI community lacks metrics for mapping of trust models between the DoD :”high assurance” C2 and EC/EB domains

*Education of everyone (policy maker through user) to a common level of understanding is a huge challenge.

*While the purpose of using PKI in EC/EB is to provide additional trust to allow the Internet to serve as a vehicle for legally binding transactions , problems still exist with the methodologies associated with establishing a long-term burden of proof. Specifically, there are no widely adopted industry standards for maintenance of electronic signatures or for authenticated timestamps for record maintenance that have stood the test of time. These processes are untried and the case law has not yet been established to convince users that there are no issues with enforcement of these new processes. An additional barrier to EC/EB within this space is the current DoD Certificate policy in which DoD accepts

===Common Issues With PKI Implementation===

*Commercial Off-The-Shelf (COTS) versus Customised applications : The choice between COTS or customised products is usually one of cost versus usability. In case of usability the thing to be focused should be error messages. If PKI is built int o applications (transparent to users) than its fine if not than user will require to have some understanding of the use of keys, certificates, Certificate Revocation Lists (CRLs) and directories/certificate repositories so that they can make informed decisions.

*Token Logistics (smart card): The point where keys and certificates are linked to their owner is a very critical point in a PKI. If a fraudulent certificate is issued by a registration officer and the certificate holder uses the certificate to commit a crime or prank, trust in the whole PKI hierarchy may be lost. The physical security requirements are high, and the registration officer, whether a person or a smartcard bureau, must be subject to strict security polices and practices. As it was problem with DoD mentioned in section above.

*Network issues - Traffic : There is no doubt that the implementation of PKI will add to the network load, although just how much depends on the system architecture. Potential additional traffic that should be considered includes: Certificate issuance, Email usage, CRLs , and Directory Replication

*Network issues - Encryption : Many organisations implement anti-virus software and content inspection on servers at the perimeter of their networks. Some have security policies that rejects or quarantines encrypted traffic. To provide user-to-user confidentiality, messages will traverse networks with their payload hidden from inspection by virus and content checking.

*Email address in certificate :In order to use certificates for S/MIME signed/encrypted email, the users’ email address must be in the certificate. Most people change their email addresses more frequently than the certificate. Unless a solution is built which allows users to keep the same email address over a long period, certificates would have to be re-issued every time a user changes email address. S/MIME v.3 stipulates that the receiving application must check the From: or Sender: field in the mail header and compare it to an email address in the sender’s certificate. If the check does not match, the mail application should perform another explicit check to ensure that the person who signed the message is indeed the person who sent it. As usual, the ‘devil is in the detail’ when it comes to implementation.

*Certificate Validity Checking:CRLs have been the conventional method of providing certificate validity checking. CRLs do not scale very well as discussed earlier, but are usually kept for backward compatibility, archiving/historical verification and for use in off-line mode. The other issue with CRLs is that they are generally issued at certain intervals of 6, 12 or 24 hours, causing a time lag from the time a certificate is revoked until it appears on the published CRL. This may present a security risk, as a certificate may verify correctly after it has been reported as compromised and revoked; (however some would argue that the time from actual compromise until the discovery and reporting of it would in most cases be a more significant lag). The Online Certificate Status Protocol (OCSP) (RFC2560) allows a client to query an OCSP responder for the current status of a certificate. This saves searching through a large CRL and can save bandwidth if the CRL would normally be downloaded - although it may increase network traffic. Most OCSP responders are based on CRLs and thus do not solve the problem of time lag as outlined above.

*Availability and storage of reliable user information : For an identity certificate scheme, names in certificates need to be unique, meaningful - and correct. Few large user communities have all their member details in a central and accurate database or directory, and the exercise of consolidating, checking and updating all user data can turn into a massive and expensive exercise.

*Archiving/historic verification : Digital signatures need to be verifiable even after the keys used to sign have expired. Likewise, we need to be able to verify that the certificate was valid at the time the datawas signed. This means we would need to archive: the signed file,the public key certificate of the signer, the CRL that was valid at the time of signing, a reliable timestamp to prove the accuracy of the time of signing and, the hardware environment that can run the software that was used at the time

===What are the solution to these problems?===

*Identity Based Encryption:enables senders to encrypt messages for recipients w/o requiring that a recipients key first be established, certified, and published.

*Certificate-based encryption:it incorporates IBE methods, but uses double encryption so that its CA cant decrypt on behalf of the user.

*Certificateless Public Key Cryptography:it incorporates IBE methods, using partial private keys so PKG cant decrypt on behalf of the user.

*Distributed Computation:There exists methods that distribute cryptographic operations so that the cooperative contribution of a number of entities is required in order to perform an operation such as a signature or decryption. It helps in tighter protection at servers vs clients, but implies that the users mist fully trust servers to apply keys appropriately.

*Alternative Validation Strategies:Hash tree:it offers compact protected representations of the status of large number of certiticates.Highly valued if PKI is operated large scale more benifical than Certificate Revovation List. CRL reflect ststus information at fixed intervals.

==Dissemination==

===The Problem Domain===

===Random Ramblings on Reputation Management and Distribution===

Publish/Subscribe?

This system has unique distribution requirements as compared to most distributed systems in general. In this system, we cannot assume that there will be a universally agreed-upon definition of good, or bad. Similarly, the system must be self-policing. It would be up to each and every group of autonomous systems to decide which updates to accept and reject. Updates themselves also should not cause the network to DDoS itself. Lastly, it would be impossible for every system to know what the reputation for a given system is. Therefore the system must disseminate information in some way that is query-able and localizes reputation information where required.

To this end, we need a way of spreading information that while reliable, does not depend on one universally agreed-upon set of reputations.

For example, on an internet-scale operating system it would be entirely reasonable for one group of systems to not want to accept updates, or want to avoid communication with a given series of systems.

Any solution would assume that the problems of attribution are solved.

===Current Examples of Reputation Dissemination===

The first protocol that immediately comes to mind in this situation is a gossip-based protocol. These protocols are designed to operate in highly decentralized, large-scale systems.

Here's a nice overview:
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4537308 "Reputation management in distributed systems"

Examples are as follows:
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4228013 "Gossip-based Reputation Aggregation for Unstructured Peer-to-Peer Networks"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=5569965 "Improving Accuracy and Coverage in an Internet-Deployed Reputation Mechanism"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4459326 "GossipTrust for Fast Reputation Aggregation in Peer-to-Peer Networks"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4777496 "Adaptive trust management in P2P networks using gossip protocol"

Another possibility is using "Reputation chains"
* http://dx.doi.org.proxy.library.carleton.ca/10.1109/TKDE.2009.45 "P2P Reputation Management Using Distributed Identities and Decentralized Recommendation Chains"

==Maintaining History==

===Problem domain===

* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
*** Do we maintain records based on a fixed set of imposed rules? Or do we build rules as the system emerges and reputations are formed?
* Where do you store the data?
** Distributed storage systems. Reputation in real-life is stored in interactions that an entity has with others. Reputation is not stored centrally. Reputation is most often a shared view of an entity by the masses, but sometimes an entities reputation can be disjoint among the masses: many different entities having differing views of reputation for the same entity.
* Where is the data queried from?
** (should I mention this?)
* What defines good/bad reputation?
** (should I mention this?)
* Who provides the good/bad reputation?
** Impose/Emerge problem: reputation for an interaction can be calculated immediately or can be a function of time.
* Who do we trust for this information?
** Trusting the masses is generally a good way of ensuring trustworthiness. Imposed rules will not always fit every situation well - could potentially set bad reputation to a "good" entity.
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
** Do we maintain an ever-growing set of history items for interactions between entities? Do we look focus on the bad reputations?
* What entities are able to contribute to reputations?
* How do we access reputation about entities?
* Who is authorized to access particular reputations? How much to reveal? (Information flow)
* What assumptions will we make?
* Privacy issues? What will we reveal? Will centralized systems have a know-all mentality?
** Fine grained information will never be revealed (privacy concerns and user rights)
* Which history should I maintain? What to take as important, what to disregard?
* Immutable data structure. Who could add data? Who could remove data? Authority

===Reputation systems===
* record, aggregate, distribute information about an entity's behaviour in distributed applications

* reputation might be based on the entity's past ability to adhere to a license agreement (mutual contract between issuer and licensee)

===History-based access control systems===
* make decision based on an entity's past security-sensitive actions

===Examples of reputation systems (trust-informing technologies)===
* eBay - Feedback forum (positive, neutral, negative)

===Do reputation systems have some validity?===

Resnick et al. argue that reputation systems
foster an incentive for principals to well-behave because of “the expectation of
reciprocity or retaliation in future interactions

Abstractions are used to model the aggregated information of each entity. These abstractions may not encompass the full details of transactions and provide context to specific issues relating to feedback. In turn we can end up with ambiguous values.

So we need a system that provides sufficient information in order to verify the precise properties of a past behaviour.

* Krukow, K. A Logical Framework for Reputation Systems and History-based Access Control. School of Electronics and Computer Science University of Southampton, UK. (March 3, 2011) [http://www.brics.dk/~krukow/research/publications/online_papers/concrete-jcs.pdf]

====Abstract====
Reputation systems are meta systems that record, aggregate and distribute information about principals’ behaviour in distributed applications. Similarly, history-based access control systems make decisions based
on programs’ past security-sensitive actions. While the applications are
distinct, the two types of systems are fundamentally making decisions
based on information about the past behaviour of an entity.
A logical policy-centric framework for such behaviour-based decisionmaking is presented. In the framework, principals specify policies which
state precise requirements on the past behaviour of other principals that
must be fulﬁlled in order for interaction to take place. The framework consists of a formal model of behaviour, based on event structures; a declarative logical language for specifying properties of past behaviour; and
eﬃcient dynamic algorithms for checking whether a particular behaviour
satisﬁes a property from the language. It is shown how the framework can
be extended in several ways, most notably to encompass parameterized
events and quantiﬁcation over parameters. In an extended application, it
is illustrated how the framework can be applied for dynamic history-based
access control for safe execution of unknown and untrusted programs.

* Khosrow-Pour, M. Emerging trends and challenges in information technology management (March 7, 2011) [http://books.google.ca/books?id=ybzS-yylJfAC&lpg=PA822&ots=V7hn_RzqXA&dq=maintaining%20history%20in%20reputation%20systems&pg=PA822#v=onepage&q=maintaining%20history%20in%20reputation%20systems&f=false]

====Abstract====

* Bolton, G. et al. How Effective are Electronic Reputation Mechanisms? (March 10, 2011) [http://ccs.mit.edu/dell/reputation/BKOMSsub.pdf]

====Abstract====

Electronic reputation or “feedback” mechanisms aim to mitigate the moral hazard problems
associated with exchange among strangers by providing the type of information available in
more traditional close-knit groups, where members are frequently involved in one another’s
dealings. In this paper, we compare trading in a market with electronic feedback (as
implemented by many Internet markets) to a market without, as well as to a market in which the
same people interact with one another repeatedly (partners market). We find that, while the
feedback mechanism induces quite a substantial improvement in transaction efficiency, it also
exhibits a kind of public goods problem in that, unlike the partners market, the benefits of trust
and trustworthy behavior go to the whole community and are not completely internalized. We
discuss the implications of this perspective for improving these systems.

This portion of a reputation system answers the core question of how reputation is generated from the information exchanged between systems and how/where it is stored.

==Problem domain:==

This portion of a reputation system answers the core question of how reputation is generated from the information exchanged between systems and how/where it is stored.

Problem domain:

• Emerge vs. Impose reputation on the system?
• Probably both, how do we account for both systems?
• Do we maintain records based on a fixed set of imposed rules? Or do we build rules as the system emerges and reputations are formed?
• Where do you store the data?
• Distributed storage systems. Reputation in real-life is stored in interactions that an entity has with others. Reputation is not stored centrally. Reputation is most often a shared view of an entity by the masses, but sometimes an entities reputation can be disjoint among the masses: many different entities having differing views of reputation for the same entity.
• Who do we trust for this information?
• Trusting the masses is generally a good way of ensuring trustworthiness. Imposed rules will not always fit every situation well - could potentially set bad reputation to a "good" entity.
• Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
• Do we maintain an ever-growing set of history items for interactions between entities? Do we look focus on the bad reputations

Existing systems
• Peer-based systems (emerge)
• eBay - positive/negative rating system
• Youtube - like/dislike/spam comment system
• Policy-based systems (impose)
• Java - policy based security
• Android - policy based security
These two systems are on opposite ends of the Emerge-Impose spectrum.
EigenTrust system
The EigenTrust system utilizes a numerical scale for reputation storage.
Advantages:
• Numerical values are easy to compare.
• Little required storage space.
Disadvantages:
• Information is lost in the abstraction process.
o No concrete data
• Ambiguity
Storing concrete data
Hence, with the given information from the reputation system, we cannot generate an accurate profile of the entity.
We need a system that represents reputation in a concrete form
“If principal p gains access to resource r at time t, then the past behavior of p up until time t satisfies requirement ψr.”
Advantages
• A sufficient amount of information is available to come up with a profile of an entity
Disadvantages
• Storage space
Shmatikov and Talcott
Histories are sets of time-stamped events. Reputation is based on ability to adhere to licenses.
Licenses might permit certain actions OR require certain actions from being performed.
Advantages:
• Store data in concrete form
Disadvantages:
• No notion of sessions (logically connected set of events)
Representation of reputation
If we consider reputation information to encompass the events and actions of an entity, then we can model reputation as a set of events.
An interesting new problem is how to re-evaluate policies efficiently when new information becomes available.
This problem is known as “dynamic model-checking”.
Dynamic model-checking
We want a way of summarizing past reputations.
A solution here is to use: Havelund and Rosu, based on the technique of dynamic programming, used for runtime verification
• Given some information on an entity, how do we convert/abstract this to reputation?
• Is all the information necessary to maintain?
• Now that we have the reputation information, what can we do with it?
o What can we compare it with?
Implementation
Desired functionality of the system:
new()
- Append new reputation information
update()
- Update and summarize past behavior
- This is a reduce function
check()
- Analyze whether the given reputation satisfies the criteria of the policy

==Querying Reputation==

=== Problems ===

* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
***If you want to know someone's reputation, you either need to start asking around for it, imposing yourself. Or you need the data to be sent around, so you already have access to it; emergent.
* Where do you store the data?
***You need to know who has the data to ask them for it, or to go get it yourself.
* Where is the data queried from?
***First you need to know who's storing it. then you need to know if you're allowed to ask that node directly, do you ask a intermediary keeper of data. Will you even need to Query-- that is, do you already have all you need to know on hand? you need not get the latest updates on a node if every other node who's ever talked to it got DDOSed. (or do you?)
* What defines good/bad reputation?
***Should I make my own definition for bad reputation, and query if someone engaged in activities I consider bad, or should their be a global agreed upon reputation?
* Who provides the good/bad reputation?
***Who should I ask for information from?
* Who do we trust for this information?
***Whoever you trust, presumably their opinion on a given node is more important then a node you trust less.
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
***topically, would you bother asking for 10 year old reputation data on a node, if it's been a model citizen for the last 9?
* What entities are able to contribute to reputations?
***Should I ask everyone I trust for an opinion on a given node, or just certain keepers of trust data?
* How do we access reputation about entities?
***You query someone in the know who you trust and are allowed to query.
***you could say, ask everyone you know and trust, and ask them to ask people they know and trust, (and so on...if they're willing) until you find a node with the information you need.
***in a more centralized system you need to ask some kind of keeper of information for the information you want, and that keeper may or may not provide you with the reputation info you want.
* Who is authorized to access particular reputations? How much to reveal? (Information flow)
***The ability to control this would depend on how centralized a system you have. In a truly distributed system where every node has an opinion on any other node they've talked to you'll be able to find somebody who can tell you about the CIA node, but in a more centralized system the keepers of information might be less...willing to give Joe 6 cores information on who Iran is DDOSing.

===Maybe References===
http://www.kirkarts.com/wiki/images/1/13/Resnick_eBay.pdf - ''Trust Among Strangers in Internet Transactions:
Empirical Analysis of eBay’s Reputation System'' (maybe not too relevant)

http://portal.acm.org/citation.cfm?id=544741.544809 - ''An Evidential Model of Distributed Reputation Management''

http://portal.acm.org/citation.cfm?id=775152.775242&type=series%EF%BF%BD%C3%9C -- ''The EigenTrust Algorithm for Reputation Management in
P2P Networks''

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.4.2297&rep=rep1&type=pdf -- ''A Robust Reputation System for Mobile Ad-hoc
Networks''

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.125.8729&rep=rep1&type=pdf -- ''EigenRep: Reputation Management in P2P Networks''

http://www.chennaisunday.com/ieee%202010/Reputation%20Estimation%20and%20Query%20in%20Peer-to-Peer%20Networks.pdf -- ''Reputation Estimation and Query in Peer-to-Peer Networks''

Here is another paper that might be interesting for you. -- Lester
http://dcg.ethz.ch/publications/netecon06.pdf

==Possible implementations==
==Implementation Requirements==
==Conclusion==

==References==
* Joel Weise : "Public Key Infrastructure Overview " http://www.sun.com/blueprints/0801/publickey.pdf Accessed 2nd March 2011

* Security Glossary : http://www.cafesoft.com/support/security-glossary.html Accessed on 2nd March 2011

* Mattila, Anssi; and Mattila, Minna "What is the Effect of Product Attributes on Public-Key Infrastructure adoption? " http://internetjournals.net/journals/tir/2006/January/Paper%2003.pdf Accessed on 2nd March 2011

*Electronic Commerece Conference , PKI Sub-Group , Issue Paper : http://www.defense.gov/dodreform/ecwg/pki.pdf date accessed 5th March 2011

*SANS Institute InfoSec Reading Room, Common issues in PKI implementations - climbing the Slope of Enlightenment : http://www.sans.org/reading_room/whitepapers/authentication/common-issues-pki-implementations-climbing-slope-enlightenment_1198 date accessed 15th March 2011

DistOS-2011W Reputation

2011-04-08T23:13:18Z

Wahmed2: /* Introduction */

==Members==
* Waheed Ahmed
* Trevor Gelowsky
** MSN: Gelowt@gmail.com
** E-Mail: tgelowsk@sce.carleton.ca
* Michael Du Plessis
* Nicolas Lessard (nick.lessard @t gmail.com / nlessard @t carleton.connect.ca)

==Our presentation==
Our current presentation can be viewed at the following link: https://docs.google.com/present/edit?id=0ASS7kj9hfc1aZGRiMjMzOHJfNGhnNzhuamRr&hl=en&authkey=CMHi3KAD

==Our Paper==

Our final paper can be found here: [[Distributed OS: Winter 2011 Reputation Systems Paper]]

==The problem==
* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
* Where do you store the data?
* Where is the data queried from?
* What defines good/bad reputation?
* Who provides the good/bad reputation?
* Who do we trust for this information?
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
* What entities are able to contribute to reputations?
* How do we access reputation about entities?
* Who is authorized to access particular reputations? How much to reveal? (Information flow)

==What technologies currently exist?==
* Digital signatures
** Certificates signed by trusted organizations

* Black hole- email, spam,
* Google - search reputation
* Credit bureaus
* Yellow pages
* Better business bureau
* CRC - criminal records

== What technologies don't currently exist?==

==Guaranteeing Authenticity/Public Key Infrastructure==

In our paper we must explain why PKI/Authentication fits into reputation. Why must it be handled by both Attribution and Reputation systems?

===Problems===

===Introduction===
In past few years, Internet has provided platform for a global market place and both business and private users realizes that the revolutionary communications opportunities provided by it will give way to large spectrum of business and private applications.Today online users face multitude of problems and issues like vulnerability to viruses , worms , exposure to sniffers, spoofing their private sessions not only this but also they are also subjected to invasion of privacy with multitude of spy ware available for monitoring how they behave. Today over the internet different kind of activities take place ranging from access to information to entertainment, financial services, product services and even socializing. The frequent usage of internet as an important business tool led to a major increase in deliberate abuse and criminal activities. All the organization operating electronically and trading expose their own information and IT systems to a wide range of security threats. The most common protocols like IP/TCP/UDP are the main targets of potential hackers. Its all because of IPs on which attacks are possible and don't have proper authentication mechanism for any incoming data over internet.

In order to build secure chain of trust Public-Key Infrastructure is used for internet based communication. It consists of various things like security policy , Certificate authority , registration authority , certificate distribution system PKI enabled applications.

===Uses and Need===
With development of modern e-commerce based businesses which has minimal customer face-to-face interactions is demanding more security and integrity. The online web based stores where huge amount of transactions take place needs to ensure customers that there information is confidential and processed through a secure channel. This is where implementation of PKI steps in to provide mechanisms to ensure trusted relationships are established and maintained. The specific security functions in which a PKI can provide foundation are confidentiality, integrity, non-repudiation,and authentication.

===Issues & Solutions===
I found out there are many different implementations of PKI , and they all focuses on their own issues and solutions. For example PKI used in DoD have following issues

*Lack of PKI-enabled eCommerce applications and lack of interoperability among PKI applications

*DoD is developing a single high assurance PKI

*Very High Cost Impact to the EC/EB community.

*The PKI community lacks metrics for mapping of trust models between the DoD :”high assurance” C2 and EC/EB domains

*Education of everyone (policy maker through user) to a common level of understanding is a huge challenge.

*While the purpose of using PKI in EC/EB is to provide additional trust to allow the Internet to serve as a vehicle for legally binding transactions , problems still exist with the methodologies associated with establishing a long-term burden of proof. Specifically, there are no widely adopted industry standards for maintenance of electronic signatures or for authenticated timestamps for record maintenance that have stood the test of time. These processes are untried and the case law has not yet been established to convince users that there are no issues with enforcement of these new processes. An additional barrier to EC/EB within this space is the current DoD Certificate policy in which DoD accepts

===Common Issues With PKI Implementation===

*Commercial Off-The-Shelf (COTS) versus Customised applications : The choice between COTS or customised products is usually one of cost versus usability. In case of usability the thing to be focused should be error messages. If PKI is built int o applications (transparent to users) than its fine if not than user will require to have some understanding of the use of keys, certificates, Certificate Revocation Lists (CRLs) and directories/certificate repositories so that they can make informed decisions.

*Token Logistics (smart card): The point where keys and certificates are linked to their owner is a very critical point in a PKI. If a fraudulent certificate is issued by a registration officer and the certificate holder uses the certificate to commit a crime or prank, trust in the whole PKI hierarchy may be lost. The physical security requirements are high, and the registration officer, whether a person or a smartcard bureau, must be subject to strict security polices and practices. As it was problem with DoD mentioned in section above.

*Network issues - Traffic : There is no doubt that the implementation of PKI will add to the network load, although just how much depends on the system architecture. Potential additional traffic that should be considered includes: Certificate issuance, Email usage, CRLs , and Directory Replication

*Network issues - Encryption : Many organisations implement anti-virus software and content inspection on servers at the perimeter of their networks. Some have security policies that rejects or quarantines encrypted traffic. To provide user-to-user confidentiality, messages will traverse networks with their payload hidden from inspection by virus and content checking.

*Email address in certificate :In order to use certificates for S/MIME signed/encrypted email, the users’ email address must be in the certificate. Most people change their email addresses more frequently than the certificate. Unless a solution is built which allows users to keep the same email address over a long period, certificates would have to be re-issued every time a user changes email address. S/MIME v.3 stipulates that the receiving application must check the From: or Sender: field in the mail header and compare it to an email address in the sender’s certificate. If the check does not match, the mail application should perform another explicit check to ensure that the person who signed the message is indeed the person who sent it. As usual, the ‘devil is in the detail’ when it comes to implementation.

*Certificate Validity Checking:CRLs have been the conventional method of providing certificate validity checking. CRLs do not scale very well as discussed earlier, but are usually kept for backward compatibility, archiving/historical verification and for use in off-line mode. The other issue with CRLs is that they are generally issued at certain intervals of 6, 12 or 24 hours, causing a time lag from the time a certificate is revoked until it appears on the published CRL. This may present a security risk, as a certificate may verify correctly after it has been reported as compromised and revoked; (however some would argue that the time from actual compromise until the discovery and reporting of it would in most cases be a more significant lag). The Online Certificate Status Protocol (OCSP) (RFC2560) allows a client to query an OCSP responder for the current status of a certificate. This saves searching through a large CRL and can save bandwidth if the CRL would normally be downloaded - although it may increase network traffic. Most OCSP responders are based on CRLs and thus do not solve the problem of time lag as outlined above.

*Availability and storage of reliable user information : For an identity certificate scheme, names in certificates need to be unique, meaningful - and correct. Few large user communities have all their member details in a central and accurate database or directory, and the exercise of consolidating, checking and updating all user data can turn into a massive and expensive exercise.

*Archiving/historic verification : Digital signatures need to be verifiable even after the keys used to sign have expired. Likewise, we need to be able to verify that the certificate was valid at the time the datawas signed. This means we would need to archive: the signed file,the public key certificate of the signer, the CRL that was valid at the time of signing, a reliable timestamp to prove the accuracy of the time of signing and, the hardware environment that can run the software that was used at the time

===What are the solution to these problems?===

*Identity Based Encryption:enables senders to encrypt messages for recipients w/o requiring that a recipients key first be established, certified, and published.

*Certificate-based encryption:it incorporates IBE methods, but uses double encryption so that its CA cant decrypt on behalf of the user.

*Certificateless Public Key Cryptography:it incorporates IBE methods, using partial private keys so PKG cant decrypt on behalf of the user.

*Distributed Computation:There exists methods that distribute cryptographic operations so that the cooperative contribution of a number of entities is required in order to perform an operation such as a signature or decryption. It helps in tighter protection at servers vs clients, but implies that the users mist fully trust servers to apply keys appropriately.

*Alternative Validation Strategies:Hash tree:it offers compact protected representations of the status of large number of certiticates.Highly valued if PKI is operated large scale more benifical than Certificate Revovation List. CRL reflect ststus information at fixed intervals.

==Dissemination==

===The Problem Domain===

===Random Ramblings on Reputation Management and Distribution===

Publish/Subscribe?

This system has unique distribution requirements as compared to most distributed systems in general. In this system, we cannot assume that there will be a universally agreed-upon definition of good, or bad. Similarly, the system must be self-policing. It would be up to each and every group of autonomous systems to decide which updates to accept and reject. Updates themselves also should not cause the network to DDoS itself. Lastly, it would be impossible for every system to know what the reputation for a given system is. Therefore the system must disseminate information in some way that is query-able and localizes reputation information where required.

To this end, we need a way of spreading information that while reliable, does not depend on one universally agreed-upon set of reputations.

For example, on an internet-scale operating system it would be entirely reasonable for one group of systems to not want to accept updates, or want to avoid communication with a given series of systems.

Any solution would assume that the problems of attribution are solved.

===Current Examples of Reputation Dissemination===

The first protocol that immediately comes to mind in this situation is a gossip-based protocol. These protocols are designed to operate in highly decentralized, large-scale systems.

Here's a nice overview:
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4537308 "Reputation management in distributed systems"

Examples are as follows:
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4228013 "Gossip-based Reputation Aggregation for Unstructured Peer-to-Peer Networks"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=5569965 "Improving Accuracy and Coverage in an Internet-Deployed Reputation Mechanism"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4459326 "GossipTrust for Fast Reputation Aggregation in Peer-to-Peer Networks"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4777496 "Adaptive trust management in P2P networks using gossip protocol"

Another possibility is using "Reputation chains"
* http://dx.doi.org.proxy.library.carleton.ca/10.1109/TKDE.2009.45 "P2P Reputation Management Using Distributed Identities and Decentralized Recommendation Chains"

==Maintaining History==

===Problem domain===

* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
*** Do we maintain records based on a fixed set of imposed rules? Or do we build rules as the system emerges and reputations are formed?
* Where do you store the data?
** Distributed storage systems. Reputation in real-life is stored in interactions that an entity has with others. Reputation is not stored centrally. Reputation is most often a shared view of an entity by the masses, but sometimes an entities reputation can be disjoint among the masses: many different entities having differing views of reputation for the same entity.
* Where is the data queried from?
** (should I mention this?)
* What defines good/bad reputation?
** (should I mention this?)
* Who provides the good/bad reputation?
** Impose/Emerge problem: reputation for an interaction can be calculated immediately or can be a function of time.
* Who do we trust for this information?
** Trusting the masses is generally a good way of ensuring trustworthiness. Imposed rules will not always fit every situation well - could potentially set bad reputation to a "good" entity.
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
** Do we maintain an ever-growing set of history items for interactions between entities? Do we look focus on the bad reputations?
* What entities are able to contribute to reputations?
* How do we access reputation about entities?
* Who is authorized to access particular reputations? How much to reveal? (Information flow)
* What assumptions will we make?
* Privacy issues? What will we reveal? Will centralized systems have a know-all mentality?
** Fine grained information will never be revealed (privacy concerns and user rights)
* Which history should I maintain? What to take as important, what to disregard?
* Immutable data structure. Who could add data? Who could remove data? Authority

===Reputation systems===
* record, aggregate, distribute information about an entity's behaviour in distributed applications

* reputation might be based on the entity's past ability to adhere to a license agreement (mutual contract between issuer and licensee)

===History-based access control systems===
* make decision based on an entity's past security-sensitive actions

===Examples of reputation systems (trust-informing technologies)===
* eBay - Feedback forum (positive, neutral, negative)

===Do reputation systems have some validity?===

Resnick et al. argue that reputation systems
foster an incentive for principals to well-behave because of “the expectation of
reciprocity or retaliation in future interactions

Abstractions are used to model the aggregated information of each entity. These abstractions may not encompass the full details of transactions and provide context to specific issues relating to feedback. In turn we can end up with ambiguous values.

So we need a system that provides sufficient information in order to verify the precise properties of a past behaviour.

* Krukow, K. A Logical Framework for Reputation Systems and History-based Access Control. School of Electronics and Computer Science University of Southampton, UK. (March 3, 2011) [http://www.brics.dk/~krukow/research/publications/online_papers/concrete-jcs.pdf]

====Abstract====
Reputation systems are meta systems that record, aggregate and distribute information about principals’ behaviour in distributed applications. Similarly, history-based access control systems make decisions based
on programs’ past security-sensitive actions. While the applications are
distinct, the two types of systems are fundamentally making decisions
based on information about the past behaviour of an entity.
A logical policy-centric framework for such behaviour-based decisionmaking is presented. In the framework, principals specify policies which
state precise requirements on the past behaviour of other principals that
must be fulﬁlled in order for interaction to take place. The framework consists of a formal model of behaviour, based on event structures; a declarative logical language for specifying properties of past behaviour; and
eﬃcient dynamic algorithms for checking whether a particular behaviour
satisﬁes a property from the language. It is shown how the framework can
be extended in several ways, most notably to encompass parameterized
events and quantiﬁcation over parameters. In an extended application, it
is illustrated how the framework can be applied for dynamic history-based
access control for safe execution of unknown and untrusted programs.

* Khosrow-Pour, M. Emerging trends and challenges in information technology management (March 7, 2011) [http://books.google.ca/books?id=ybzS-yylJfAC&lpg=PA822&ots=V7hn_RzqXA&dq=maintaining%20history%20in%20reputation%20systems&pg=PA822#v=onepage&q=maintaining%20history%20in%20reputation%20systems&f=false]

====Abstract====

* Bolton, G. et al. How Effective are Electronic Reputation Mechanisms? (March 10, 2011) [http://ccs.mit.edu/dell/reputation/BKOMSsub.pdf]

====Abstract====

Electronic reputation or “feedback” mechanisms aim to mitigate the moral hazard problems
associated with exchange among strangers by providing the type of information available in
more traditional close-knit groups, where members are frequently involved in one another’s
dealings. In this paper, we compare trading in a market with electronic feedback (as
implemented by many Internet markets) to a market without, as well as to a market in which the
same people interact with one another repeatedly (partners market). We find that, while the
feedback mechanism induces quite a substantial improvement in transaction efficiency, it also
exhibits a kind of public goods problem in that, unlike the partners market, the benefits of trust
and trustworthy behavior go to the whole community and are not completely internalized. We
discuss the implications of this perspective for improving these systems.

This portion of a reputation system answers the core question of how reputation is generated from the information exchanged between systems and how/where it is stored.

==Problem domain:==

This portion of a reputation system answers the core question of how reputation is generated from the information exchanged between systems and how/where it is stored.

Problem domain:

• Emerge vs. Impose reputation on the system?
• Probably both, how do we account for both systems?
• Do we maintain records based on a fixed set of imposed rules? Or do we build rules as the system emerges and reputations are formed?
• Where do you store the data?
• Distributed storage systems. Reputation in real-life is stored in interactions that an entity has with others. Reputation is not stored centrally. Reputation is most often a shared view of an entity by the masses, but sometimes an entities reputation can be disjoint among the masses: many different entities having differing views of reputation for the same entity.
• Who do we trust for this information?
• Trusting the masses is generally a good way of ensuring trustworthiness. Imposed rules will not always fit every situation well - could potentially set bad reputation to a "good" entity.
• Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
• Do we maintain an ever-growing set of history items for interactions between entities? Do we look focus on the bad reputations

Existing systems
• Peer-based systems (emerge)
• eBay - positive/negative rating system
• Youtube - like/dislike/spam comment system
• Policy-based systems (impose)
• Java - policy based security
• Android - policy based security
These two systems are on opposite ends of the Emerge-Impose spectrum.
EigenTrust system
The EigenTrust system utilizes a numerical scale for reputation storage.
Advantages:
• Numerical values are easy to compare.
• Little required storage space.
Disadvantages:
• Information is lost in the abstraction process.
o No concrete data
• Ambiguity
Storing concrete data
Hence, with the given information from the reputation system, we cannot generate an accurate profile of the entity.
We need a system that represents reputation in a concrete form
“If principal p gains access to resource r at time t, then the past behavior of p up until time t satisfies requirement ψr.”
Advantages
• A sufficient amount of information is available to come up with a profile of an entity
Disadvantages
• Storage space
Shmatikov and Talcott
Histories are sets of time-stamped events. Reputation is based on ability to adhere to licenses.
Licenses might permit certain actions OR require certain actions from being performed.
Advantages:
• Store data in concrete form
Disadvantages:
• No notion of sessions (logically connected set of events)
Representation of reputation
If we consider reputation information to encompass the events and actions of an entity, then we can model reputation as a set of events.
An interesting new problem is how to re-evaluate policies efficiently when new information becomes available.
This problem is known as “dynamic model-checking”.
Dynamic model-checking
We want a way of summarizing past reputations.
A solution here is to use: Havelund and Rosu, based on the technique of dynamic programming, used for runtime verification
• Given some information on an entity, how do we convert/abstract this to reputation?
• Is all the information necessary to maintain?
• Now that we have the reputation information, what can we do with it?
o What can we compare it with?
Implementation
Desired functionality of the system:
new()
- Append new reputation information
update()
- Update and summarize past behavior
- This is a reduce function
check()
- Analyze whether the given reputation satisfies the criteria of the policy

==Querying Reputation==

=== Problems ===

* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
***If you want to know someone's reputation, you either need to start asking around for it, imposing yourself. Or you need the data to be sent around, so you already have access to it; emergent.
* Where do you store the data?
***You need to know who has the data to ask them for it, or to go get it yourself.
* Where is the data queried from?
***First you need to know who's storing it. then you need to know if you're allowed to ask that node directly, do you ask a intermediary keeper of data. Will you even need to Query-- that is, do you already have all you need to know on hand? you need not get the latest updates on a node if every other node who's ever talked to it got DDOSed. (or do you?)
* What defines good/bad reputation?
***Should I make my own definition for bad reputation, and query if someone engaged in activities I consider bad, or should their be a global agreed upon reputation?
* Who provides the good/bad reputation?
***Who should I ask for information from?
* Who do we trust for this information?
***Whoever you trust, presumably their opinion on a given node is more important then a node you trust less.
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
***topically, would you bother asking for 10 year old reputation data on a node, if it's been a model citizen for the last 9?
* What entities are able to contribute to reputations?
***Should I ask everyone I trust for an opinion on a given node, or just certain keepers of trust data?
* How do we access reputation about entities?
***You query someone in the know who you trust and are allowed to query.
***you could say, ask everyone you know and trust, and ask them to ask people they know and trust, (and so on...if they're willing) until you find a node with the information you need.
***in a more centralized system you need to ask some kind of keeper of information for the information you want, and that keeper may or may not provide you with the reputation info you want.
* Who is authorized to access particular reputations? How much to reveal? (Information flow)
***The ability to control this would depend on how centralized a system you have. In a truly distributed system where every node has an opinion on any other node they've talked to you'll be able to find somebody who can tell you about the CIA node, but in a more centralized system the keepers of information might be less...willing to give Joe 6 cores information on who Iran is DDOSing.

===Maybe References===
http://www.kirkarts.com/wiki/images/1/13/Resnick_eBay.pdf - ''Trust Among Strangers in Internet Transactions:
Empirical Analysis of eBay’s Reputation System'' (maybe not too relevant)

http://portal.acm.org/citation.cfm?id=544741.544809 - ''An Evidential Model of Distributed Reputation Management''

http://portal.acm.org/citation.cfm?id=775152.775242&type=series%EF%BF%BD%C3%9C -- ''The EigenTrust Algorithm for Reputation Management in
P2P Networks''

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.4.2297&rep=rep1&type=pdf -- ''A Robust Reputation System for Mobile Ad-hoc
Networks''

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.125.8729&rep=rep1&type=pdf -- ''EigenRep: Reputation Management in P2P Networks''

http://www.chennaisunday.com/ieee%202010/Reputation%20Estimation%20and%20Query%20in%20Peer-to-Peer%20Networks.pdf -- ''Reputation Estimation and Query in Peer-to-Peer Networks''

Here is another paper that might be interesting for you. -- Lester
http://dcg.ethz.ch/publications/netecon06.pdf

==Possible implementations==
==Implementation Requirements==
==Conclusion==

==References==
* Joel Weise : "Public Key Infrastructure Overview " http://www.sun.com/blueprints/0801/publickey.pdf Accessed 2nd March 2011

* Security Glossary : http://www.cafesoft.com/support/security-glossary.html Accessed on 2nd March 2011

* Mattila, Anssi; and Mattila, Minna "What is the Effect of Product Attributes on Public-Key Infrastructure adoption? " http://internetjournals.net/journals/tir/2006/January/Paper%2003.pdf Accessed on 2nd March 2011

*Electronic Commerece Conference , PKI Sub-Group , Issue Paper : http://www.defense.gov/dodreform/ecwg/pki.pdf date accessed 5th March 2011

*SANS Institute InfoSec Reading Room, Common issues in PKI implementations - climbing the Slope of Enlightenment : http://www.sans.org/reading_room/whitepapers/authentication/common-issues-pki-implementations-climbing-slope-enlightenment_1198 date accessed 15th March 2011

DistOS-2011W Reputation

2011-04-08T19:52:27Z

Wahmed2: /* Introduction */

==Members==
* Waheed Ahmed
* Trevor Gelowsky
** MSN: Gelowt@gmail.com
** E-Mail: tgelowsk@sce.carleton.ca
* Michael Du Plessis
* Nicolas Lessard (nick.lessard @t gmail.com / nlessard @t carleton.connect.ca)

==Our presentation==
Our current presentation can be viewed at the following link: https://docs.google.com/present/edit?id=0ASS7kj9hfc1aZGRiMjMzOHJfNGhnNzhuamRr&hl=en&authkey=CMHi3KAD

==Our Paper==

Our final paper can be found here: [[Distributed OS: Winter 2011 Reputation Systems Paper]]

==The problem==
* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
* Where do you store the data?
* Where is the data queried from?
* What defines good/bad reputation?
* Who provides the good/bad reputation?
* Who do we trust for this information?
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
* What entities are able to contribute to reputations?
* How do we access reputation about entities?
* Who is authorized to access particular reputations? How much to reveal? (Information flow)

==What technologies currently exist?==
* Digital signatures
** Certificates signed by trusted organizations

* Black hole- email, spam,
* Google - search reputation
* Credit bureaus
* Yellow pages
* Better business bureau
* CRC - criminal records

== What technologies don't currently exist?==

==Guaranteeing Authenticity/Public Key Infrastructure==

In our paper we must explain why PKI/Authentication fits into reputation. Why must it be handled by both Attribution and Reputation systems?

===Problems===

===Introduction===
In past few years, Internet has provided platform for a global market place and both business and private users realizes that the revolutionary communications opportunities provided by it will give way to large spectrum of business and private applications.Today online users face multitude of problems and issues like vulnerability to viruses , worms , exposure to sniffers, spoofing their private sessions not only this but also they are also subjected to invasion of privacy with multitude of spy ware available for monitoring how they behave. Today over the internet different kind of activities take place ranging from access to information to entertainment, financial services, product services and even socializing. The frequent usage of internet as an important business tool led to a major increase in deliberate abuse and criminal activities. All the organization operating electronically and trading expose their own information and IT systems to a wide range of security threats. The most common protocols like IP/TCP/UDP are the main targets of potential hackers. Its all because of IPs on which attacks are possible and don't have proper authentication mechanism for any incoming data over internet. In order to build secure chain of trust Public-Key Infrastructure is used for internet based communication. It consists of various things like security policy , Certificate authority , registration authority , certificate distribution system PKI enabled applications.

===Uses and Need===
With development of modern e-commerce based businesses which has minimal customer face-to-face interactions is demanding more security and integrity. The online web based stores where huge amount of transactions take place needs to ensure customers that there information is confidential and processed through a secure channel. This is where implementation of PKI steps in to provide mechanisms to ensure trusted relationships are established and maintained. The specific security functions in which a PKI can provide foundation are confidentiality, integrity, non-repudiation,and authentication.

===Issues & Solutions===
I found out there are many different implementations of PKI , and they all focuses on their own issues and solutions. For example PKI used in DoD have following issues

*Lack of PKI-enabled eCommerce applications and lack of interoperability among PKI applications

*DoD is developing a single high assurance PKI

*Very High Cost Impact to the EC/EB community.

*The PKI community lacks metrics for mapping of trust models between the DoD :”high assurance” C2 and EC/EB domains

*Education of everyone (policy maker through user) to a common level of understanding is a huge challenge.

*While the purpose of using PKI in EC/EB is to provide additional trust to allow the Internet to serve as a vehicle for legally binding transactions , problems still exist with the methodologies associated with establishing a long-term burden of proof. Specifically, there are no widely adopted industry standards for maintenance of electronic signatures or for authenticated timestamps for record maintenance that have stood the test of time. These processes are untried and the case law has not yet been established to convince users that there are no issues with enforcement of these new processes. An additional barrier to EC/EB within this space is the current DoD Certificate policy in which DoD accepts

===Common Issues With PKI Implementation===

*Commercial Off-The-Shelf (COTS) versus Customised applications : The choice between COTS or customised products is usually one of cost versus usability. In case of usability the thing to be focused should be error messages. If PKI is built int o applications (transparent to users) than its fine if not than user will require to have some understanding of the use of keys, certificates, Certificate Revocation Lists (CRLs) and directories/certificate repositories so that they can make informed decisions.

*Token Logistics (smart card): The point where keys and certificates are linked to their owner is a very critical point in a PKI. If a fraudulent certificate is issued by a registration officer and the certificate holder uses the certificate to commit a crime or prank, trust in the whole PKI hierarchy may be lost. The physical security requirements are high, and the registration officer, whether a person or a smartcard bureau, must be subject to strict security polices and practices. As it was problem with DoD mentioned in section above.

*Network issues - Traffic : There is no doubt that the implementation of PKI will add to the network load, although just how much depends on the system architecture. Potential additional traffic that should be considered includes: Certificate issuance, Email usage, CRLs , and Directory Replication

*Network issues - Encryption : Many organisations implement anti-virus software and content inspection on servers at the perimeter of their networks. Some have security policies that rejects or quarantines encrypted traffic. To provide user-to-user confidentiality, messages will traverse networks with their payload hidden from inspection by virus and content checking.

*Email address in certificate :In order to use certificates for S/MIME signed/encrypted email, the users’ email address must be in the certificate. Most people change their email addresses more frequently than the certificate. Unless a solution is built which allows users to keep the same email address over a long period, certificates would have to be re-issued every time a user changes email address. S/MIME v.3 stipulates that the receiving application must check the From: or Sender: field in the mail header and compare it to an email address in the sender’s certificate. If the check does not match, the mail application should perform another explicit check to ensure that the person who signed the message is indeed the person who sent it. As usual, the ‘devil is in the detail’ when it comes to implementation.

*Certificate Validity Checking:CRLs have been the conventional method of providing certificate validity checking. CRLs do not scale very well as discussed earlier, but are usually kept for backward compatibility, archiving/historical verification and for use in off-line mode. The other issue with CRLs is that they are generally issued at certain intervals of 6, 12 or 24 hours, causing a time lag from the time a certificate is revoked until it appears on the published CRL. This may present a security risk, as a certificate may verify correctly after it has been reported as compromised and revoked; (however some would argue that the time from actual compromise until the discovery and reporting of it would in most cases be a more significant lag). The Online Certificate Status Protocol (OCSP) (RFC2560) allows a client to query an OCSP responder for the current status of a certificate. This saves searching through a large CRL and can save bandwidth if the CRL would normally be downloaded - although it may increase network traffic. Most OCSP responders are based on CRLs and thus do not solve the problem of time lag as outlined above.

*Availability and storage of reliable user information : For an identity certificate scheme, names in certificates need to be unique, meaningful - and correct. Few large user communities have all their member details in a central and accurate database or directory, and the exercise of consolidating, checking and updating all user data can turn into a massive and expensive exercise.

*Archiving/historic verification : Digital signatures need to be verifiable even after the keys used to sign have expired. Likewise, we need to be able to verify that the certificate was valid at the time the datawas signed. This means we would need to archive: the signed file,the public key certificate of the signer, the CRL that was valid at the time of signing, a reliable timestamp to prove the accuracy of the time of signing and, the hardware environment that can run the software that was used at the time

===What are the solution to these problems?===

*Identity Based Encryption:enables senders to encrypt messages for recipients w/o requiring that a recipients key first be established, certified, and published.

*Certificate-based encryption:it incorporates IBE methods, but uses double encryption so that its CA cant decrypt on behalf of the user.

*Certificateless Public Key Cryptography:it incorporates IBE methods, using partial private keys so PKG cant decrypt on behalf of the user.

*Distributed Computation:There exists methods that distribute cryptographic operations so that the cooperative contribution of a number of entities is required in order to perform an operation such as a signature or decryption. It helps in tighter protection at servers vs clients, but implies that the users mist fully trust servers to apply keys appropriately.

*Alternative Validation Strategies:Hash tree:it offers compact protected representations of the status of large number of certiticates.Highly valued if PKI is operated large scale more benifical than Certificate Revovation List. CRL reflect ststus information at fixed intervals.

==Dissemination==

===The Problem Domain===

===Random Ramblings on Reputation Management and Distribution===

Publish/Subscribe?

This system has unique distribution requirements as compared to most distributed systems in general. In this system, we cannot assume that there will be a universally agreed-upon definition of good, or bad. Similarly, the system must be self-policing. It would be up to each and every group of autonomous systems to decide which updates to accept and reject. Updates themselves also should not cause the network to DDoS itself. Lastly, it would be impossible for every system to know what the reputation for a given system is. Therefore the system must disseminate information in some way that is query-able and localizes reputation information where required.

To this end, we need a way of spreading information that while reliable, does not depend on one universally agreed-upon set of reputations.

For example, on an internet-scale operating system it would be entirely reasonable for one group of systems to not want to accept updates, or want to avoid communication with a given series of systems.

Any solution would assume that the problems of attribution are solved.

===Current Examples of Reputation Dissemination===

The first protocol that immediately comes to mind in this situation is a gossip-based protocol. These protocols are designed to operate in highly decentralized, large-scale systems.

Here's a nice overview:
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4537308 "Reputation management in distributed systems"

Examples are as follows:
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4228013 "Gossip-based Reputation Aggregation for Unstructured Peer-to-Peer Networks"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=5569965 "Improving Accuracy and Coverage in an Internet-Deployed Reputation Mechanism"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4459326 "GossipTrust for Fast Reputation Aggregation in Peer-to-Peer Networks"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4777496 "Adaptive trust management in P2P networks using gossip protocol"

Another possibility is using "Reputation chains"
* http://dx.doi.org.proxy.library.carleton.ca/10.1109/TKDE.2009.45 "P2P Reputation Management Using Distributed Identities and Decentralized Recommendation Chains"

==Maintaining History==

===Problem domain===

* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
*** Do we maintain records based on a fixed set of imposed rules? Or do we build rules as the system emerges and reputations are formed?
* Where do you store the data?
** Distributed storage systems. Reputation in real-life is stored in interactions that an entity has with others. Reputation is not stored centrally. Reputation is most often a shared view of an entity by the masses, but sometimes an entities reputation can be disjoint among the masses: many different entities having differing views of reputation for the same entity.
* Where is the data queried from?
** (should I mention this?)
* What defines good/bad reputation?
** (should I mention this?)
* Who provides the good/bad reputation?
** Impose/Emerge problem: reputation for an interaction can be calculated immediately or can be a function of time.
* Who do we trust for this information?
** Trusting the masses is generally a good way of ensuring trustworthiness. Imposed rules will not always fit every situation well - could potentially set bad reputation to a "good" entity.
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
** Do we maintain an ever-growing set of history items for interactions between entities? Do we look focus on the bad reputations?
* What entities are able to contribute to reputations?
* How do we access reputation about entities?
* Who is authorized to access particular reputations? How much to reveal? (Information flow)
* What assumptions will we make?
* Privacy issues? What will we reveal? Will centralized systems have a know-all mentality?
** Fine grained information will never be revealed (privacy concerns and user rights)
* Which history should I maintain? What to take as important, what to disregard?
* Immutable data structure. Who could add data? Who could remove data? Authority

===Reputation systems===
* record, aggregate, distribute information about an entity's behaviour in distributed applications

* reputation might be based on the entity's past ability to adhere to a license agreement (mutual contract between issuer and licensee)

===History-based access control systems===
* make decision based on an entity's past security-sensitive actions

===Examples of reputation systems (trust-informing technologies)===
* eBay - Feedback forum (positive, neutral, negative)

===Do reputation systems have some validity?===

Resnick et al. argue that reputation systems
foster an incentive for principals to well-behave because of “the expectation of
reciprocity or retaliation in future interactions

Abstractions are used to model the aggregated information of each entity. These abstractions may not encompass the full details of transactions and provide context to specific issues relating to feedback. In turn we can end up with ambiguous values.

So we need a system that provides sufficient information in order to verify the precise properties of a past behaviour.

* Krukow, K. A Logical Framework for Reputation Systems and History-based Access Control. School of Electronics and Computer Science University of Southampton, UK. (March 3, 2011) [http://www.brics.dk/~krukow/research/publications/online_papers/concrete-jcs.pdf]

====Abstract====
Reputation systems are meta systems that record, aggregate and distribute information about principals’ behaviour in distributed applications. Similarly, history-based access control systems make decisions based
on programs’ past security-sensitive actions. While the applications are
distinct, the two types of systems are fundamentally making decisions
based on information about the past behaviour of an entity.
A logical policy-centric framework for such behaviour-based decisionmaking is presented. In the framework, principals specify policies which
state precise requirements on the past behaviour of other principals that
must be fulﬁlled in order for interaction to take place. The framework consists of a formal model of behaviour, based on event structures; a declarative logical language for specifying properties of past behaviour; and
eﬃcient dynamic algorithms for checking whether a particular behaviour
satisﬁes a property from the language. It is shown how the framework can
be extended in several ways, most notably to encompass parameterized
events and quantiﬁcation over parameters. In an extended application, it
is illustrated how the framework can be applied for dynamic history-based
access control for safe execution of unknown and untrusted programs.

* Khosrow-Pour, M. Emerging trends and challenges in information technology management (March 7, 2011) [http://books.google.ca/books?id=ybzS-yylJfAC&lpg=PA822&ots=V7hn_RzqXA&dq=maintaining%20history%20in%20reputation%20systems&pg=PA822#v=onepage&q=maintaining%20history%20in%20reputation%20systems&f=false]

====Abstract====

* Bolton, G. et al. How Effective are Electronic Reputation Mechanisms? (March 10, 2011) [http://ccs.mit.edu/dell/reputation/BKOMSsub.pdf]

====Abstract====

Electronic reputation or “feedback” mechanisms aim to mitigate the moral hazard problems
associated with exchange among strangers by providing the type of information available in
more traditional close-knit groups, where members are frequently involved in one another’s
dealings. In this paper, we compare trading in a market with electronic feedback (as
implemented by many Internet markets) to a market without, as well as to a market in which the
same people interact with one another repeatedly (partners market). We find that, while the
feedback mechanism induces quite a substantial improvement in transaction efficiency, it also
exhibits a kind of public goods problem in that, unlike the partners market, the benefits of trust
and trustworthy behavior go to the whole community and are not completely internalized. We
discuss the implications of this perspective for improving these systems.

This portion of a reputation system answers the core question of how reputation is generated from the information exchanged between systems and how/where it is stored.

==Problem domain:==

This portion of a reputation system answers the core question of how reputation is generated from the information exchanged between systems and how/where it is stored.

Problem domain:

• Emerge vs. Impose reputation on the system?
• Probably both, how do we account for both systems?
• Do we maintain records based on a fixed set of imposed rules? Or do we build rules as the system emerges and reputations are formed?
• Where do you store the data?
• Distributed storage systems. Reputation in real-life is stored in interactions that an entity has with others. Reputation is not stored centrally. Reputation is most often a shared view of an entity by the masses, but sometimes an entities reputation can be disjoint among the masses: many different entities having differing views of reputation for the same entity.
• Who do we trust for this information?
• Trusting the masses is generally a good way of ensuring trustworthiness. Imposed rules will not always fit every situation well - could potentially set bad reputation to a "good" entity.
• Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
• Do we maintain an ever-growing set of history items for interactions between entities? Do we look focus on the bad reputations

Existing systems
• Peer-based systems (emerge)
• eBay - positive/negative rating system
• Youtube - like/dislike/spam comment system
• Policy-based systems (impose)
• Java - policy based security
• Android - policy based security
These two systems are on opposite ends of the Emerge-Impose spectrum.
EigenTrust system
The EigenTrust system utilizes a numerical scale for reputation storage.
Advantages:
• Numerical values are easy to compare.
• Little required storage space.
Disadvantages:
• Information is lost in the abstraction process.
o No concrete data
• Ambiguity
Storing concrete data
Hence, with the given information from the reputation system, we cannot generate an accurate profile of the entity.
We need a system that represents reputation in a concrete form
“If principal p gains access to resource r at time t, then the past behavior of p up until time t satisfies requirement ψr.”
Advantages
• A sufficient amount of information is available to come up with a profile of an entity
Disadvantages
• Storage space
Shmatikov and Talcott
Histories are sets of time-stamped events. Reputation is based on ability to adhere to licenses.
Licenses might permit certain actions OR require certain actions from being performed.
Advantages:
• Store data in concrete form
Disadvantages:
• No notion of sessions (logically connected set of events)
Representation of reputation
If we consider reputation information to encompass the events and actions of an entity, then we can model reputation as a set of events.
An interesting new problem is how to re-evaluate policies efficiently when new information becomes available.
This problem is known as “dynamic model-checking”.
Dynamic model-checking
We want a way of summarizing past reputations.
A solution here is to use: Havelund and Rosu, based on the technique of dynamic programming, used for runtime verification
• Given some information on an entity, how do we convert/abstract this to reputation?
• Is all the information necessary to maintain?
• Now that we have the reputation information, what can we do with it?
o What can we compare it with?
Implementation
Desired functionality of the system:
new()
- Append new reputation information
update()
- Update and summarize past behavior
- This is a reduce function
check()
- Analyze whether the given reputation satisfies the criteria of the policy

==Querying Reputation==

=== Problems ===

* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
***If you want to know someone's reputation, you either need to start asking around for it, imposing yourself. Or you need the data to be sent around, so you already have access to it; emergent.
* Where do you store the data?
***You need to know who has the data to ask them for it, or to go get it yourself.
* Where is the data queried from?
***First you need to know who's storing it. then you need to know if you're allowed to ask that node directly, do you ask a intermediary keeper of data. Will you even need to Query-- that is, do you already have all you need to know on hand? you need not get the latest updates on a node if every other node who's ever talked to it got DDOSed. (or do you?)
* What defines good/bad reputation?
***Should I make my own definition for bad reputation, and query if someone engaged in activities I consider bad, or should their be a global agreed upon reputation?
* Who provides the good/bad reputation?
***Who should I ask for information from?
* Who do we trust for this information?
***Whoever you trust, presumably their opinion on a given node is more important then a node you trust less.
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
***topically, would you bother asking for 10 year old reputation data on a node, if it's been a model citizen for the last 9?
* What entities are able to contribute to reputations?
***Should I ask everyone I trust for an opinion on a given node, or just certain keepers of trust data?
* How do we access reputation about entities?
***You query someone in the know who you trust and are allowed to query.
***you could say, ask everyone you know and trust, and ask them to ask people they know and trust, (and so on...if they're willing) until you find a node with the information you need.
***in a more centralized system you need to ask some kind of keeper of information for the information you want, and that keeper may or may not provide you with the reputation info you want.
* Who is authorized to access particular reputations? How much to reveal? (Information flow)
***The ability to control this would depend on how centralized a system you have. In a truly distributed system where every node has an opinion on any other node they've talked to you'll be able to find somebody who can tell you about the CIA node, but in a more centralized system the keepers of information might be less...willing to give Joe 6 cores information on who Iran is DDOSing.

===Maybe References===
http://www.kirkarts.com/wiki/images/1/13/Resnick_eBay.pdf - ''Trust Among Strangers in Internet Transactions:
Empirical Analysis of eBay’s Reputation System'' (maybe not too relevant)

http://portal.acm.org/citation.cfm?id=544741.544809 - ''An Evidential Model of Distributed Reputation Management''

http://portal.acm.org/citation.cfm?id=775152.775242&type=series%EF%BF%BD%C3%9C -- ''The EigenTrust Algorithm for Reputation Management in
P2P Networks''

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.4.2297&rep=rep1&type=pdf -- ''A Robust Reputation System for Mobile Ad-hoc
Networks''

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.125.8729&rep=rep1&type=pdf -- ''EigenRep: Reputation Management in P2P Networks''

http://www.chennaisunday.com/ieee%202010/Reputation%20Estimation%20and%20Query%20in%20Peer-to-Peer%20Networks.pdf -- ''Reputation Estimation and Query in Peer-to-Peer Networks''

Here is another paper that might be interesting for you. -- Lester
http://dcg.ethz.ch/publications/netecon06.pdf

==Possible implementations==
==Implementation Requirements==
==Conclusion==

==References==
* Joel Weise : "Public Key Infrastructure Overview " http://www.sun.com/blueprints/0801/publickey.pdf Accessed 2nd March 2011

* Security Glossary : http://www.cafesoft.com/support/security-glossary.html Accessed on 2nd March 2011

* Mattila, Anssi; and Mattila, Minna "What is the Effect of Product Attributes on Public-Key Infrastructure adoption? " http://internetjournals.net/journals/tir/2006/January/Paper%2003.pdf Accessed on 2nd March 2011

*Electronic Commerece Conference , PKI Sub-Group , Issue Paper : http://www.defense.gov/dodreform/ecwg/pki.pdf date accessed 5th March 2011

*SANS Institute InfoSec Reading Room, Common issues in PKI implementations - climbing the Slope of Enlightenment : http://www.sans.org/reading_room/whitepapers/authentication/common-issues-pki-implementations-climbing-slope-enlightenment_1198 date accessed 15th March 2011

DistOS-2011W Reputation

2011-04-08T19:26:53Z

Wahmed2: /* Introduction */

==Members==
* Waheed Ahmed
* Trevor Gelowsky
** MSN: Gelowt@gmail.com
** E-Mail: tgelowsk@sce.carleton.ca
* Michael Du Plessis
* Nicolas Lessard (nick.lessard @t gmail.com / nlessard @t carleton.connect.ca)

==Our presentation==
Our current presentation can be viewed at the following link: https://docs.google.com/present/edit?id=0ASS7kj9hfc1aZGRiMjMzOHJfNGhnNzhuamRr&hl=en&authkey=CMHi3KAD

==Our Paper==

Our final paper can be found here: [[Distributed OS: Winter 2011 Reputation Systems Paper]]

==The problem==
* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
* Where do you store the data?
* Where is the data queried from?
* What defines good/bad reputation?
* Who provides the good/bad reputation?
* Who do we trust for this information?
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
* What entities are able to contribute to reputations?
* How do we access reputation about entities?
* Who is authorized to access particular reputations? How much to reveal? (Information flow)

==What technologies currently exist?==
* Digital signatures
** Certificates signed by trusted organizations

* Black hole- email, spam,
* Google - search reputation
* Credit bureaus
* Yellow pages
* Better business bureau
* CRC - criminal records

== What technologies don't currently exist?==

==Guaranteeing Authenticity/Public Key Infrastructure==

In our paper we must explain why PKI/Authentication fits into reputation. Why must it be handled by both Attribution and Reputation systems?

===Problems===

===Introduction===
In past few years, Internet has provided platform for a global market place and both business and private users realizes that the revolutionary communications opportunities provided by it will give way to large spectrum of business and private applications.Today online users face multitude of problems and issues like vulnerability to viruses , worms , exposure to sniffers, spoofing their private sessions not only this but also they are also subjected to invasion of privacy with multitude of spy ware available for monitoring how they behave. In order to build secure chain of trust Public-Key Infrastructure is used for internet based communication. It consists of various things like security policy , Certificate authority , registration authority , certificate distribution system PKI enabled applications.

===Uses and Need===
With development of modern e-commerce based businesses which has minimal customer face-to-face interactions is demanding more security and integrity. The online web based stores where huge amount of transactions take place needs to ensure customers that there information is confidential and processed through a secure channel. This is where implementation of PKI steps in to provide mechanisms to ensure trusted relationships are established and maintained. The specific security functions in which a PKI can provide foundation are confidentiality, integrity, non-repudiation,and authentication.

===Issues & Solutions===
I found out there are many different implementations of PKI , and they all focuses on their own issues and solutions. For example PKI used in DoD have following issues

*Lack of PKI-enabled eCommerce applications and lack of interoperability among PKI applications

*DoD is developing a single high assurance PKI

*Very High Cost Impact to the EC/EB community.

*The PKI community lacks metrics for mapping of trust models between the DoD :”high assurance” C2 and EC/EB domains

*Education of everyone (policy maker through user) to a common level of understanding is a huge challenge.

*While the purpose of using PKI in EC/EB is to provide additional trust to allow the Internet to serve as a vehicle for legally binding transactions , problems still exist with the methodologies associated with establishing a long-term burden of proof. Specifically, there are no widely adopted industry standards for maintenance of electronic signatures or for authenticated timestamps for record maintenance that have stood the test of time. These processes are untried and the case law has not yet been established to convince users that there are no issues with enforcement of these new processes. An additional barrier to EC/EB within this space is the current DoD Certificate policy in which DoD accepts

===Common Issues With PKI Implementation===

*Commercial Off-The-Shelf (COTS) versus Customised applications : The choice between COTS or customised products is usually one of cost versus usability. In case of usability the thing to be focused should be error messages. If PKI is built int o applications (transparent to users) than its fine if not than user will require to have some understanding of the use of keys, certificates, Certificate Revocation Lists (CRLs) and directories/certificate repositories so that they can make informed decisions.

*Token Logistics (smart card): The point where keys and certificates are linked to their owner is a very critical point in a PKI. If a fraudulent certificate is issued by a registration officer and the certificate holder uses the certificate to commit a crime or prank, trust in the whole PKI hierarchy may be lost. The physical security requirements are high, and the registration officer, whether a person or a smartcard bureau, must be subject to strict security polices and practices. As it was problem with DoD mentioned in section above.

*Network issues - Traffic : There is no doubt that the implementation of PKI will add to the network load, although just how much depends on the system architecture. Potential additional traffic that should be considered includes: Certificate issuance, Email usage, CRLs , and Directory Replication

*Network issues - Encryption : Many organisations implement anti-virus software and content inspection on servers at the perimeter of their networks. Some have security policies that rejects or quarantines encrypted traffic. To provide user-to-user confidentiality, messages will traverse networks with their payload hidden from inspection by virus and content checking.

*Email address in certificate :In order to use certificates for S/MIME signed/encrypted email, the users’ email address must be in the certificate. Most people change their email addresses more frequently than the certificate. Unless a solution is built which allows users to keep the same email address over a long period, certificates would have to be re-issued every time a user changes email address. S/MIME v.3 stipulates that the receiving application must check the From: or Sender: field in the mail header and compare it to an email address in the sender’s certificate. If the check does not match, the mail application should perform another explicit check to ensure that the person who signed the message is indeed the person who sent it. As usual, the ‘devil is in the detail’ when it comes to implementation.

*Certificate Validity Checking:CRLs have been the conventional method of providing certificate validity checking. CRLs do not scale very well as discussed earlier, but are usually kept for backward compatibility, archiving/historical verification and for use in off-line mode. The other issue with CRLs is that they are generally issued at certain intervals of 6, 12 or 24 hours, causing a time lag from the time a certificate is revoked until it appears on the published CRL. This may present a security risk, as a certificate may verify correctly after it has been reported as compromised and revoked; (however some would argue that the time from actual compromise until the discovery and reporting of it would in most cases be a more significant lag). The Online Certificate Status Protocol (OCSP) (RFC2560) allows a client to query an OCSP responder for the current status of a certificate. This saves searching through a large CRL and can save bandwidth if the CRL would normally be downloaded - although it may increase network traffic. Most OCSP responders are based on CRLs and thus do not solve the problem of time lag as outlined above.

*Availability and storage of reliable user information : For an identity certificate scheme, names in certificates need to be unique, meaningful - and correct. Few large user communities have all their member details in a central and accurate database or directory, and the exercise of consolidating, checking and updating all user data can turn into a massive and expensive exercise.

*Archiving/historic verification : Digital signatures need to be verifiable even after the keys used to sign have expired. Likewise, we need to be able to verify that the certificate was valid at the time the datawas signed. This means we would need to archive: the signed file,the public key certificate of the signer, the CRL that was valid at the time of signing, a reliable timestamp to prove the accuracy of the time of signing and, the hardware environment that can run the software that was used at the time

===What are the solution to these problems?===

*Identity Based Encryption:enables senders to encrypt messages for recipients w/o requiring that a recipients key first be established, certified, and published.

*Certificate-based encryption:it incorporates IBE methods, but uses double encryption so that its CA cant decrypt on behalf of the user.

*Certificateless Public Key Cryptography:it incorporates IBE methods, using partial private keys so PKG cant decrypt on behalf of the user.

*Distributed Computation:There exists methods that distribute cryptographic operations so that the cooperative contribution of a number of entities is required in order to perform an operation such as a signature or decryption. It helps in tighter protection at servers vs clients, but implies that the users mist fully trust servers to apply keys appropriately.

*Alternative Validation Strategies:Hash tree:it offers compact protected representations of the status of large number of certiticates.Highly valued if PKI is operated large scale more benifical than Certificate Revovation List. CRL reflect ststus information at fixed intervals.

==Dissemination==

===The Problem Domain===

===Random Ramblings on Reputation Management and Distribution===

Publish/Subscribe?

This system has unique distribution requirements as compared to most distributed systems in general. In this system, we cannot assume that there will be a universally agreed-upon definition of good, or bad. Similarly, the system must be self-policing. It would be up to each and every group of autonomous systems to decide which updates to accept and reject. Updates themselves also should not cause the network to DDoS itself. Lastly, it would be impossible for every system to know what the reputation for a given system is. Therefore the system must disseminate information in some way that is query-able and localizes reputation information where required.

To this end, we need a way of spreading information that while reliable, does not depend on one universally agreed-upon set of reputations.

For example, on an internet-scale operating system it would be entirely reasonable for one group of systems to not want to accept updates, or want to avoid communication with a given series of systems.

Any solution would assume that the problems of attribution are solved.

===Current Examples of Reputation Dissemination===

The first protocol that immediately comes to mind in this situation is a gossip-based protocol. These protocols are designed to operate in highly decentralized, large-scale systems.

Here's a nice overview:
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4537308 "Reputation management in distributed systems"

Examples are as follows:
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4228013 "Gossip-based Reputation Aggregation for Unstructured Peer-to-Peer Networks"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=5569965 "Improving Accuracy and Coverage in an Internet-Deployed Reputation Mechanism"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4459326 "GossipTrust for Fast Reputation Aggregation in Peer-to-Peer Networks"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4777496 "Adaptive trust management in P2P networks using gossip protocol"

Another possibility is using "Reputation chains"
* http://dx.doi.org.proxy.library.carleton.ca/10.1109/TKDE.2009.45 "P2P Reputation Management Using Distributed Identities and Decentralized Recommendation Chains"

==Maintaining History==

===Problem domain===

* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
*** Do we maintain records based on a fixed set of imposed rules? Or do we build rules as the system emerges and reputations are formed?
* Where do you store the data?
** Distributed storage systems. Reputation in real-life is stored in interactions that an entity has with others. Reputation is not stored centrally. Reputation is most often a shared view of an entity by the masses, but sometimes an entities reputation can be disjoint among the masses: many different entities having differing views of reputation for the same entity.
* Where is the data queried from?
** (should I mention this?)
* What defines good/bad reputation?
** (should I mention this?)
* Who provides the good/bad reputation?
** Impose/Emerge problem: reputation for an interaction can be calculated immediately or can be a function of time.
* Who do we trust for this information?
** Trusting the masses is generally a good way of ensuring trustworthiness. Imposed rules will not always fit every situation well - could potentially set bad reputation to a "good" entity.
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
** Do we maintain an ever-growing set of history items for interactions between entities? Do we look focus on the bad reputations?
* What entities are able to contribute to reputations?
* How do we access reputation about entities?
* Who is authorized to access particular reputations? How much to reveal? (Information flow)
* What assumptions will we make?
* Privacy issues? What will we reveal? Will centralized systems have a know-all mentality?
** Fine grained information will never be revealed (privacy concerns and user rights)
* Which history should I maintain? What to take as important, what to disregard?
* Immutable data structure. Who could add data? Who could remove data? Authority

===Reputation systems===
* record, aggregate, distribute information about an entity's behaviour in distributed applications

* reputation might be based on the entity's past ability to adhere to a license agreement (mutual contract between issuer and licensee)

===History-based access control systems===
* make decision based on an entity's past security-sensitive actions

===Examples of reputation systems (trust-informing technologies)===
* eBay - Feedback forum (positive, neutral, negative)

===Do reputation systems have some validity?===

Resnick et al. argue that reputation systems
foster an incentive for principals to well-behave because of “the expectation of
reciprocity or retaliation in future interactions

Abstractions are used to model the aggregated information of each entity. These abstractions may not encompass the full details of transactions and provide context to specific issues relating to feedback. In turn we can end up with ambiguous values.

So we need a system that provides sufficient information in order to verify the precise properties of a past behaviour.

* Krukow, K. A Logical Framework for Reputation Systems and History-based Access Control. School of Electronics and Computer Science University of Southampton, UK. (March 3, 2011) [http://www.brics.dk/~krukow/research/publications/online_papers/concrete-jcs.pdf]

====Abstract====
Reputation systems are meta systems that record, aggregate and distribute information about principals’ behaviour in distributed applications. Similarly, history-based access control systems make decisions based
on programs’ past security-sensitive actions. While the applications are
distinct, the two types of systems are fundamentally making decisions
based on information about the past behaviour of an entity.
A logical policy-centric framework for such behaviour-based decisionmaking is presented. In the framework, principals specify policies which
state precise requirements on the past behaviour of other principals that
must be fulﬁlled in order for interaction to take place. The framework consists of a formal model of behaviour, based on event structures; a declarative logical language for specifying properties of past behaviour; and
eﬃcient dynamic algorithms for checking whether a particular behaviour
satisﬁes a property from the language. It is shown how the framework can
be extended in several ways, most notably to encompass parameterized
events and quantiﬁcation over parameters. In an extended application, it
is illustrated how the framework can be applied for dynamic history-based
access control for safe execution of unknown and untrusted programs.

* Khosrow-Pour, M. Emerging trends and challenges in information technology management (March 7, 2011) [http://books.google.ca/books?id=ybzS-yylJfAC&lpg=PA822&ots=V7hn_RzqXA&dq=maintaining%20history%20in%20reputation%20systems&pg=PA822#v=onepage&q=maintaining%20history%20in%20reputation%20systems&f=false]

====Abstract====

* Bolton, G. et al. How Effective are Electronic Reputation Mechanisms? (March 10, 2011) [http://ccs.mit.edu/dell/reputation/BKOMSsub.pdf]

====Abstract====

Electronic reputation or “feedback” mechanisms aim to mitigate the moral hazard problems
associated with exchange among strangers by providing the type of information available in
more traditional close-knit groups, where members are frequently involved in one another’s
dealings. In this paper, we compare trading in a market with electronic feedback (as
implemented by many Internet markets) to a market without, as well as to a market in which the
same people interact with one another repeatedly (partners market). We find that, while the
feedback mechanism induces quite a substantial improvement in transaction efficiency, it also
exhibits a kind of public goods problem in that, unlike the partners market, the benefits of trust
and trustworthy behavior go to the whole community and are not completely internalized. We
discuss the implications of this perspective for improving these systems.

This portion of a reputation system answers the core question of how reputation is generated from the information exchanged between systems and how/where it is stored.

==Problem domain:==

This portion of a reputation system answers the core question of how reputation is generated from the information exchanged between systems and how/where it is stored.

Problem domain:

• Emerge vs. Impose reputation on the system?
• Probably both, how do we account for both systems?
• Do we maintain records based on a fixed set of imposed rules? Or do we build rules as the system emerges and reputations are formed?
• Where do you store the data?
• Distributed storage systems. Reputation in real-life is stored in interactions that an entity has with others. Reputation is not stored centrally. Reputation is most often a shared view of an entity by the masses, but sometimes an entities reputation can be disjoint among the masses: many different entities having differing views of reputation for the same entity.
• Who do we trust for this information?
• Trusting the masses is generally a good way of ensuring trustworthiness. Imposed rules will not always fit every situation well - could potentially set bad reputation to a "good" entity.
• Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
• Do we maintain an ever-growing set of history items for interactions between entities? Do we look focus on the bad reputations

Existing systems
• Peer-based systems (emerge)
• eBay - positive/negative rating system
• Youtube - like/dislike/spam comment system
• Policy-based systems (impose)
• Java - policy based security
• Android - policy based security
These two systems are on opposite ends of the Emerge-Impose spectrum.
EigenTrust system
The EigenTrust system utilizes a numerical scale for reputation storage.
Advantages:
• Numerical values are easy to compare.
• Little required storage space.
Disadvantages:
• Information is lost in the abstraction process.
o No concrete data
• Ambiguity
Storing concrete data
Hence, with the given information from the reputation system, we cannot generate an accurate profile of the entity.
We need a system that represents reputation in a concrete form
“If principal p gains access to resource r at time t, then the past behavior of p up until time t satisfies requirement ψr.”
Advantages
• A sufficient amount of information is available to come up with a profile of an entity
Disadvantages
• Storage space
Shmatikov and Talcott
Histories are sets of time-stamped events. Reputation is based on ability to adhere to licenses.
Licenses might permit certain actions OR require certain actions from being performed.
Advantages:
• Store data in concrete form
Disadvantages:
• No notion of sessions (logically connected set of events)
Representation of reputation
If we consider reputation information to encompass the events and actions of an entity, then we can model reputation as a set of events.
An interesting new problem is how to re-evaluate policies efficiently when new information becomes available.
This problem is known as “dynamic model-checking”.
Dynamic model-checking
We want a way of summarizing past reputations.
A solution here is to use: Havelund and Rosu, based on the technique of dynamic programming, used for runtime verification
• Given some information on an entity, how do we convert/abstract this to reputation?
• Is all the information necessary to maintain?
• Now that we have the reputation information, what can we do with it?
o What can we compare it with?
Implementation
Desired functionality of the system:
new()
- Append new reputation information
update()
- Update and summarize past behavior
- This is a reduce function
check()
- Analyze whether the given reputation satisfies the criteria of the policy

==Querying Reputation==

=== Problems ===

* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
***If you want to know someone's reputation, you either need to start asking around for it, imposing yourself. Or you need the data to be sent around, so you already have access to it; emergent.
* Where do you store the data?
***You need to know who has the data to ask them for it, or to go get it yourself.
* Where is the data queried from?
***First you need to know who's storing it. then you need to know if you're allowed to ask that node directly, do you ask a intermediary keeper of data. Will you even need to Query-- that is, do you already have all you need to know on hand? you need not get the latest updates on a node if every other node who's ever talked to it got DDOSed. (or do you?)
* What defines good/bad reputation?
***Should I make my own definition for bad reputation, and query if someone engaged in activities I consider bad, or should their be a global agreed upon reputation?
* Who provides the good/bad reputation?
***Who should I ask for information from?
* Who do we trust for this information?
***Whoever you trust, presumably their opinion on a given node is more important then a node you trust less.
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
***topically, would you bother asking for 10 year old reputation data on a node, if it's been a model citizen for the last 9?
* What entities are able to contribute to reputations?
***Should I ask everyone I trust for an opinion on a given node, or just certain keepers of trust data?
* How do we access reputation about entities?
***You query someone in the know who you trust and are allowed to query.
***you could say, ask everyone you know and trust, and ask them to ask people they know and trust, (and so on...if they're willing) until you find a node with the information you need.
***in a more centralized system you need to ask some kind of keeper of information for the information you want, and that keeper may or may not provide you with the reputation info you want.
* Who is authorized to access particular reputations? How much to reveal? (Information flow)
***The ability to control this would depend on how centralized a system you have. In a truly distributed system where every node has an opinion on any other node they've talked to you'll be able to find somebody who can tell you about the CIA node, but in a more centralized system the keepers of information might be less...willing to give Joe 6 cores information on who Iran is DDOSing.

===Maybe References===
http://www.kirkarts.com/wiki/images/1/13/Resnick_eBay.pdf - ''Trust Among Strangers in Internet Transactions:
Empirical Analysis of eBay’s Reputation System'' (maybe not too relevant)

http://portal.acm.org/citation.cfm?id=544741.544809 - ''An Evidential Model of Distributed Reputation Management''

http://portal.acm.org/citation.cfm?id=775152.775242&type=series%EF%BF%BD%C3%9C -- ''The EigenTrust Algorithm for Reputation Management in
P2P Networks''

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.4.2297&rep=rep1&type=pdf -- ''A Robust Reputation System for Mobile Ad-hoc
Networks''

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.125.8729&rep=rep1&type=pdf -- ''EigenRep: Reputation Management in P2P Networks''

http://www.chennaisunday.com/ieee%202010/Reputation%20Estimation%20and%20Query%20in%20Peer-to-Peer%20Networks.pdf -- ''Reputation Estimation and Query in Peer-to-Peer Networks''

Here is another paper that might be interesting for you. -- Lester
http://dcg.ethz.ch/publications/netecon06.pdf

==Possible implementations==
==Implementation Requirements==
==Conclusion==

==References==
* Joel Weise : "Public Key Infrastructure Overview " http://www.sun.com/blueprints/0801/publickey.pdf Accessed 2nd March 2011

* Security Glossary : http://www.cafesoft.com/support/security-glossary.html Accessed on 2nd March 2011

* Mattila, Anssi; and Mattila, Minna "What is the Effect of Product Attributes on Public-Key Infrastructure adoption? " http://internetjournals.net/journals/tir/2006/January/Paper%2003.pdf Accessed on 2nd March 2011

*Electronic Commerece Conference , PKI Sub-Group , Issue Paper : http://www.defense.gov/dodreform/ecwg/pki.pdf date accessed 5th March 2011

*SANS Institute InfoSec Reading Room, Common issues in PKI implementations - climbing the Slope of Enlightenment : http://www.sans.org/reading_room/whitepapers/authentication/common-issues-pki-implementations-climbing-slope-enlightenment_1198 date accessed 15th March 2011

DistOS-2011W Reputation

2011-03-31T16:36:08Z

Wahmed2: /* Guaranteeing Authenticity/Public Key Infrastructure */

==Members==
* Waheed Ahmed
* Trevor Gelowsky
** MSN: Gelowt@gmail.com
** E-Mail: tgelowsk@sce.carleton.ca
* Michael Du Plessis
* Nicolas Lessard (nick.lessard @t gmail.com / nlessard @t carleton.connect.ca)

==Our presentation==
Our current presentation can be viewed at the following link: https://docs.google.com/present/edit?id=0AbY-UrFwVEEVZHgyZmJoel8yMmhkZzI3aGM1&hl=en&authkey=CIDatKoH

==Our Paper==

Our final paper can be found here: [[Distributed OS: Winter 2011 Reputation Systems Paper]]

==The problem==
* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
* Where do you store the data?
* Where is the data queried from?
* What defines good/bad reputation?
* Who provides the good/bad reputation?
* Who do we trust for this information?
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
* What entities are able to contribute to reputations?
* How do we access reputation about entities?
* Who is authorized to access particular reputations? How much to reveal? (Information flow)

==What technologies currently exist?==
* Digital signatures
** Certificates signed by trusted organizations

* Black hole- email, spam,
* Google - search reputation
* Credit bureaus
* Yellow pages
* Better business bureau
* CRC - criminal records

== What technologies don't currently exist?==

==Guaranteeing Authenticity/Public Key Infrastructure==

In our paper we must explain why PKI/Authentication fits into reputation. Why must it be handled by both Attribution and Reputation systems?

===Problems===

===Introduction===
In order to build secure chain of trust Public-Key Infrastructure is used for internet based communication. It consists of various things like security policy , Certificate authority , registration authority , certificate distribution system PKI enabled applications.

===Uses and Need===
With development of modern e-commerce based businesses which has minimal customer face-to-face interactions is demanding more security and integrity. The online web based stores where huge amount of transactions take place needs to ensure customers that there information is confidential and processed through a secure channel. This is where implementation of PKI steps in to provide mechanisms to ensure trusted relationships are established and maintained. The specific security functions in which a PKI can provide foundation are confidentiality, integrity, non-repudiation,and authentication.

===Issues & Solutions===
I found out there are many different implementations of PKI , and they all focuses on their own issues and solutions. For example PKI used in DoD have following issues

*Lack of PKI-enabled eCommerce applications and lack of interoperability among PKI applications

*DoD is developing a single high assurance PKI

*Very High Cost Impact to the EC/EB community.

*The PKI community lacks metrics for mapping of trust models between the DoD :”high assurance” C2 and EC/EB domains

*Education of everyone (policy maker through user) to a common level of understanding is a huge challenge.

*While the purpose of using PKI in EC/EB is to provide additional trust to allow the Internet to serve as a vehicle for legally binding transactions , problems still exist with the methodologies associated with establishing a long-term burden of proof. Specifically, there are no widely adopted industry standards for maintenance of electronic signatures or for authenticated timestamps for record maintenance that have stood the test of time. These processes are untried and the case law has not yet been established to convince users that there are no issues with enforcement of these new processes. An additional barrier to EC/EB within this space is the current DoD Certificate policy in which DoD accepts

===Common Issues With PKI Implementation===

*Commercial Off-The-Shelf (COTS) versus Customised applications : The choice between COTS or customised products is usually one of cost versus usability. In case of usability the thing to be focused should be error messages. If PKI is built int o applications (transparent to users) than its fine if not than user will require to have some understanding of the use of keys, certificates, Certificate Revocation Lists (CRLs) and directories/certificate repositories so that they can make informed decisions.

*Token Logistics (smart card): The point where keys and certificates are linked to their owner is a very critical point in a PKI. If a fraudulent certificate is issued by a registration officer and the certificate holder uses the certificate to commit a crime or prank, trust in the whole PKI hierarchy may be lost. The physical security requirements are high, and the registration officer, whether a person or a smartcard bureau, must be subject to strict security polices and practices. As it was problem with DoD mentioned in section above.

*Network issues - Traffic : There is no doubt that the implementation of PKI will add to the network load, although just how much depends on the system architecture. Potential additional traffic that should be considered includes: Certificate issuance, Email usage, CRLs , and Directory Replication

*Network issues - Encryption : Many organisations implement anti-virus software and content inspection on servers at the perimeter of their networks. Some have security policies that rejects or quarantines encrypted traffic. To provide user-to-user confidentiality, messages will traverse networks with their payload hidden from inspection by virus and content checking.

*Email address in certificate :In order to use certificates for S/MIME signed/encrypted email, the users’ email address must be in the certificate. Most people change their email addresses more frequently than the certificate. Unless a solution is built which allows users to keep the same email address over a long period, certificates would have to be re-issued every time a user changes email address. S/MIME v.3 stipulates that the receiving application must check the From: or Sender: field in the mail header and compare it to an email address in the sender’s certificate. If the check does not match, the mail application should perform another explicit check to ensure that the person who signed the message is indeed the person who sent it. As usual, the ‘devil is in the detail’ when it comes to implementation.

*Certificate Validity Checking:CRLs have been the conventional method of providing certificate validity checking. CRLs do not scale very well as discussed earlier, but are usually kept for backward compatibility, archiving/historical verification and for use in off-line mode. The other issue with CRLs is that they are generally issued at certain intervals of 6, 12 or 24 hours, causing a time lag from the time a certificate is revoked until it appears on the published CRL. This may present a security risk, as a certificate may verify correctly after it has been reported as compromised and revoked; (however some would argue that the time from actual compromise until the discovery and reporting of it would in most cases be a more significant lag). The Online Certificate Status Protocol (OCSP) (RFC2560) allows a client to query an OCSP responder for the current status of a certificate. This saves searching through a large CRL and can save bandwidth if the CRL would normally be downloaded - although it may increase network traffic. Most OCSP responders are based on CRLs and thus do not solve the problem of time lag as outlined above.

*Availability and storage of reliable user information : For an identity certificate scheme, names in certificates need to be unique, meaningful - and correct. Few large user communities have all their member details in a central and accurate database or directory, and the exercise of consolidating, checking and updating all user data can turn into a massive and expensive exercise.

*Archiving/historic verification : Digital signatures need to be verifiable even after the keys used to sign have expired. Likewise, we need to be able to verify that the certificate was valid at the time the datawas signed. This means we would need to archive: the signed file,the public key certificate of the signer, the CRL that was valid at the time of signing, a reliable timestamp to prove the accuracy of the time of signing and, the hardware environment that can run the software that was used at the time

===What are the solution to these problems?===

*Identity Based Encryption:enables senders to encrypt messages for recipients w/o requiring that a recipients key first be established, certified, and published.

*Certificate-based encryption:it incorporates IBE methods, but uses double encryption so that its CA cant decrypt on behalf of the user.

*Certificateless Public Key Cryptography:it incorporates IBE methods, using partial private keys so PKG cant decrypt on behalf of the user.

*Distributed Computation:There exists methods that distribute cryptographic operations so that the cooperative contribution of a number of entities is required in order to perform an operation such as a signature or decryption. It helps in tighter protection at servers vs clients, but implies that the users mist fully trust servers to apply keys appropriately.

*Alternative Validation Strategies:Hash tree:it offers compact protected representations of the status of large number of certiticates.Highly valued if PKI is operated large scale more benifical than Certificate Revovation List. CRL reflect ststus information at fixed intervals.

==Dissemination==

===The Problem Domain===

===Random Ramblings on Reputation Management and Distribution===

Publish/Subscribe?

This system has unique distribution requirements as compared to most distributed systems in general. In this system, we cannot assume that there will be a universally agreed-upon definition of good, or bad. Similarly, the system must be self-policing. It would be up to each and every group of autonomous systems to decide which updates to accept and reject. Updates themselves also should not cause the network to DDoS itself. Lastly, it would be impossible for every system to know what the reputation for a given system is. Therefore the system must disseminate information in some way that is query-able and localizes reputation information where required.

To this end, we need a way of spreading information that while reliable, does not depend on one universally agreed-upon set of reputations.

For example, on an internet-scale operating system it would be entirely reasonable for one group of systems to not want to accept updates, or want to avoid communication with a given series of systems.

Any solution would assume that the problems of attribution are solved.

===Current Examples of Reputation Dissemination===

The first protocol that immediately comes to mind in this situation is a gossip-based protocol. These protocols are designed to operate in highly decentralized, large-scale systems.

Here's a nice overview:
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4537308 "Reputation management in distributed systems"

Examples are as follows:
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4228013 "Gossip-based Reputation Aggregation for Unstructured Peer-to-Peer Networks"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=5569965 "Improving Accuracy and Coverage in an Internet-Deployed Reputation Mechanism"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4459326 "GossipTrust for Fast Reputation Aggregation in Peer-to-Peer Networks"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4777496 "Adaptive trust management in P2P networks using gossip protocol"

Another possibility is using "Reputation chains"
* http://dx.doi.org.proxy.library.carleton.ca/10.1109/TKDE.2009.45 "P2P Reputation Management Using Distributed Identities and Decentralized Recommendation Chains"

==Maintaining History==

===Problem domain===

* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
*** Do we maintain records based on a fixed set of imposed rules? Or do we build rules as the system emerges and reputations are formed?
* Where do you store the data?
** Distributed storage systems. Reputation in real-life is stored in interactions that an entity has with others. Reputation is not stored centrally. Reputation is most often a shared view of an entity by the masses, but sometimes an entities reputation can be disjoint among the masses: many different entities having differing views of reputation for the same entity.
* Where is the data queried from?
** (should I mention this?)
* What defines good/bad reputation?
** (should I mention this?)
* Who provides the good/bad reputation?
** Impose/Emerge problem: reputation for an interaction can be calculated immediately or can be a function of time.
* Who do we trust for this information?
** Trusting the masses is generally a good way of ensuring trustworthiness. Imposed rules will not always fit every situation well - could potentially set bad reputation to a "good" entity.
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
** Do we maintain an ever-growing set of history items for interactions between entities? Do we look focus on the bad reputations?
* What entities are able to contribute to reputations?
* How do we access reputation about entities?
* Who is authorized to access particular reputations? How much to reveal? (Information flow)
* What assumptions will we make?
* Privacy issues? What will we reveal? Will centralized systems have a know-all mentality?
** Fine grained information will never be revealed (privacy concerns and user rights)
* Which history should I maintain? What to take as important, what to disregard?
* Immutable data structure. Who could add data? Who could remove data? Authority

===Reputation systems===
* record, aggregate, distribute information about an entity's behaviour in distributed applications

* reputation might be based on the entity's past ability to adhere to a license agreement (mutual contract between issuer and licensee)

===History-based access control systems===
* make decision based on an entity's past security-sensitive actions

===Examples of reputation systems (trust-informing technologies)===
* eBay - Feedback forum (positive, neutral, negative)

===Do reputation systems have some validity?===

Resnick et al. argue that reputation systems
foster an incentive for principals to well-behave because of “the expectation of
reciprocity or retaliation in future interactions

Abstractions are used to model the aggregated information of each entity. These abstractions may not encompass the full details of transactions and provide context to specific issues relating to feedback. In turn we can end up with ambiguous values.

So we need a system that provides sufficient information in order to verify the precise properties of a past behaviour.

* Krukow, K. A Logical Framework for Reputation Systems and History-based Access Control. School of Electronics and Computer Science University of Southampton, UK. (March 3, 2011) [http://www.brics.dk/~krukow/research/publications/online_papers/concrete-jcs.pdf]

====Abstract====
Reputation systems are meta systems that record, aggregate and distribute information about principals’ behaviour in distributed applications. Similarly, history-based access control systems make decisions based
on programs’ past security-sensitive actions. While the applications are
distinct, the two types of systems are fundamentally making decisions
based on information about the past behaviour of an entity.
A logical policy-centric framework for such behaviour-based decisionmaking is presented. In the framework, principals specify policies which
state precise requirements on the past behaviour of other principals that
must be fulﬁlled in order for interaction to take place. The framework consists of a formal model of behaviour, based on event structures; a declarative logical language for specifying properties of past behaviour; and
eﬃcient dynamic algorithms for checking whether a particular behaviour
satisﬁes a property from the language. It is shown how the framework can
be extended in several ways, most notably to encompass parameterized
events and quantiﬁcation over parameters. In an extended application, it
is illustrated how the framework can be applied for dynamic history-based
access control for safe execution of unknown and untrusted programs.

* Khosrow-Pour, M. Emerging trends and challenges in information technology management (March 7, 2011) [http://books.google.ca/books?id=ybzS-yylJfAC&lpg=PA822&ots=V7hn_RzqXA&dq=maintaining%20history%20in%20reputation%20systems&pg=PA822#v=onepage&q=maintaining%20history%20in%20reputation%20systems&f=false]

====Abstract====

* Bolton, G. et al. How Effective are Electronic Reputation Mechanisms? (March 10, 2011) [http://ccs.mit.edu/dell/reputation/BKOMSsub.pdf]

====Abstract====

Electronic reputation or “feedback” mechanisms aim to mitigate the moral hazard problems
associated with exchange among strangers by providing the type of information available in
more traditional close-knit groups, where members are frequently involved in one another’s
dealings. In this paper, we compare trading in a market with electronic feedback (as
implemented by many Internet markets) to a market without, as well as to a market in which the
same people interact with one another repeatedly (partners market). We find that, while the
feedback mechanism induces quite a substantial improvement in transaction efficiency, it also
exhibits a kind of public goods problem in that, unlike the partners market, the benefits of trust
and trustworthy behavior go to the whole community and are not completely internalized. We
discuss the implications of this perspective for improving these systems.

==Querying Reputation==

=== Problems ===

* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
***If you want to know someone's reputation, you either need to start asking around for it, imposing yourself. Or you need the data to be sent around, so you already have access to it; emergent.
* Where do you store the data?
***You need to know who has the data to ask them for it, or to go get it yourself.
* Where is the data queried from?
***First you need to know who's storing it. then you need to know if you're allowed to ask that node directly, do you ask a intermediary keeper of data. Will you even need to Query-- that is, do you already have all you need to know on hand? you need not get the latest updates on a node if every other node who's ever talked to it got DDOSed. (or do you?)
* What defines good/bad reputation?
***Should I make my own definition for bad reputation, and query if someone engaged in activities I consider bad, or should their be a global agreed upon reputation?
* Who provides the good/bad reputation?
***Who should I ask for information from?
* Who do we trust for this information?
***Whoever you trust, presumably their opinion on a given node is more important then a node you trust less.
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
***topically, would you bother asking for 10 year old reputation data on a node, if it's been a model citizen for the last 9?
* What entities are able to contribute to reputations?
***Should I ask everyone I trust for an opinion on a given node, or just certain keepers of trust data?
* How do we access reputation about entities?
***You query someone in the know who you trust and are allowed to query.
***you could say, ask everyone you know and trust, and ask them to ask people they know and trust, (and so on...if they're willing) until you find a node with the information you need.
***in a more centralized system you need to ask some kind of keeper of information for the information you want, and that keeper may or may not provide you with the reputation info you want.
* Who is authorized to access particular reputations? How much to reveal? (Information flow)
***The ability to control this would depend on how centralized a system you have. In a truly distributed system where every node has an opinion on any other node they've talked to you'll be able to find somebody who can tell you about the CIA node, but in a more centralized system the keepers of information might be less...willing to give Joe 6 cores information on who Iran is DDOSing.

===Maybe References===
http://www.kirkarts.com/wiki/images/1/13/Resnick_eBay.pdf - ''Trust Among Strangers in Internet Transactions:
Empirical Analysis of eBay’s Reputation System'' (maybe not too relevant)

http://portal.acm.org/citation.cfm?id=544741.544809 - ''An Evidential Model of Distributed Reputation Management''

http://portal.acm.org/citation.cfm?id=775152.775242&type=series%EF%BF%BD%C3%9C -- ''The EigenTrust Algorithm for Reputation Management in
P2P Networks''

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.4.2297&rep=rep1&type=pdf -- ''A Robust Reputation System for Mobile Ad-hoc
Networks''

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.125.8729&rep=rep1&type=pdf -- ''EigenRep: Reputation Management in P2P Networks''

http://www.chennaisunday.com/ieee%202010/Reputation%20Estimation%20and%20Query%20in%20Peer-to-Peer%20Networks.pdf -- ''Reputation Estimation and Query in Peer-to-Peer Networks''

Here is another paper that might be interesting for you. -- Lester
http://dcg.ethz.ch/publications/netecon06.pdf

==Possible implementations==
==Implementation Requirements==
==Conclusion==

==References==
* Joel Weise : "Public Key Infrastructure Overview " http://www.sun.com/blueprints/0801/publickey.pdf Accessed 2nd March 2011

* Security Glossary : http://www.cafesoft.com/support/security-glossary.html Accessed on 2nd March 2011

* Mattila, Anssi; and Mattila, Minna "What is the Effect of Product Attributes on Public-Key Infrastructure adoption? " http://internetjournals.net/journals/tir/2006/January/Paper%2003.pdf Accessed on 2nd March 2011

*Electronic Commerece Conference , PKI Sub-Group , Issue Paper : http://www.defense.gov/dodreform/ecwg/pki.pdf date accessed 5th March 2011

*SANS Institute InfoSec Reading Room, Common issues in PKI implementations - climbing the Slope of Enlightenment : http://www.sans.org/reading_room/whitepapers/authentication/common-issues-pki-implementations-climbing-slope-enlightenment_1198 date accessed 15th March 2011

DistOS-2011W Reputation

2011-03-27T19:59:12Z

Wahmed2: /* Common Issues With PKI Implementation */

DistOS-2011W Reputation

2011-03-17T15:37:54Z

Wahmed2: /* References */

==Members==
* Waheed Ahmed
* Trevor Gelowsky
** MSN: Gelowt@gmail.com
** E-Mail: tgelowsk@sce.carleton.ca
* Michael Du Plessis
* Nicolas Lessard (nick.lessard @t gmail.com / nlessard @t carleton.connect.ca)

==The problem==
* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
* Where do you store the data?
* Where is the data queried from?
* What defines good/bad reputation?
* Who provides the good/bad reputation?
* Who do we trust for this information?
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
* What entities are able to contribute to reputations?
* How do we access reputation about entities?
* Who is authorized to access particular reputations? How much to reveal? (Information flow)

==What technologies currently exist?==
* Digital signatures
** Certificates signed by trusted organizations

* Black hole- email, spam,
* Google - search reputation
* Credit bureaus
* Yellow pages
* Better business bureau
* CRC - criminal records

== What technologies don't currently exist?==

==Guaranteeing Authenticity/Public Key Infrastructure==

===Problems===

===Introduction===
In order to build secure chain of trust Public-Key Infrastructure is used for internet based communication. It consists of various things like security policy , Certificate authority , registration authority , certificate distribution system PKI enabled applications.

===Uses and Need===
With development of modern e-commerce based businesses which has minimal customer face-to-face interactions is demanding more security and integrity. The online web based stores where huge amount of transactions take place needs to ensure customers that there information is confidential and processed through a secure channel. This is where implementation of PKI steps in to provide mechanisms to ensure trusted relationships are established and maintained. The specific security functions in which a PKI can provide foundation are confidentiality, integrity, non-repudiation,and authentication.

===Issues & Solutions===
I found out there are many different implementations of PKI , and they all focuses on their own issues and solutions. For example PKI used in DoD have following issues

*Lack of PKI-enabled eCommerce applications and lack of interoperability among PKI applications

*DoD is developing a single high assurance PKI

*Very High Cost Impact to the EC/EB community.

*The PKI community lacks metrics for mapping of trust models between the DoD :”high assurance” C2 and EC/EB domains

*Education of everyone (policy maker through user) to a common level of understanding is a huge challenge.

*While the purpose of using PKI in EC/EB is to provide additional trust to allow the Internet to serve as a vehicle for legally binding transactions , problems still exist with the methodologies associated with establishing a long-term burden of proof. Specifically, there are no widely adopted industry standards for maintenance of electronic signatures or for authenticated timestamps for record maintenance that have stood the test of time. These processes are untried and the case law has not yet been established to convince users that there are no issues with enforcement of these new processes. An additional barrier to EC/EB within this space is the current DoD Certificate policy in which DoD accepts

===Common Issues With PKI Implementation===

*Commercial Off-The-Shelf (COTS) versus Customised applications : The choice between COTS or customised products is usually one of cost versus usability. In case of usability the thing to be focused should be error messages. If PKI is built int o applications (transparent to users) than its fine if not than user will require to have some understanding of the use of keys, certificates, Certificate Revocation Lists (CRLs)
and directories/certificate repositories so that they can make informed decisions.

*Token Logistics (smart card): The point where keys and certificates are linked to their owner is a very critical point in a PKI. If a fraudulent certificate is issued by a registration officer and the certificate holder uses the certificate to commit a crime or prank, trust in the whole PKI hierarchy may be lost. The physical security requirements are high, and the registration officer, whether a person or a smartcard bureau, must be subject to strict security polices and practices. As it was problem with DoD mentioned in section above.

*Network issues - Traffic : There is no doubt that the implementation of PKI will add to the network load, although just how much depends on the system architecture. Potential additional traffic that should be considered includes: Certificate issuance, Email usage, CRLs , and Directory Replication

*Network issues - Encryption : Many organisations implement anti-virus software and content inspection on servers at the perimeter of their networks. Some have security policies that rejects or quarantines encrypted traffic. To provide user-to-user confidentiality, messages will traverse networks with their payload hidden from inspection by virus and content checking.

*Email address in certificate :In order to use certificates for S/MIME signed/encrypted email, the users’ email address must be in the certificate. Most people change their email addresses more frequently than the certificate. Unless a solution is built which allows users to keep the same email address over a long period, certificates would have to be re-issued every time a user changes email address. S/MIME v.3 stipulates that the receiving application must check the From: or Sender: field in the mail header and compare it to an email address in the sender’s certificate. If the check does not match, the mail application should perform another explicit check to ensure that the person who signed the message is indeed the person who sent it. As usual, the ‘devil is in the detail’ when it comes to implementation.

*Certificate Validity Checking:CRLs have been the conventional method of providing certificate validity checking. CRLs do not scale very well as discussed earlier, but are usually kept for backward compatibility, archiving/historical verification and for use in off-line mode. The other issue with CRLs is that they are generally issued at certain intervals of 6, 12 or 24 hours, causing a time lag from the time a certificate is revoked until it appears on the published CRL. This may present a security risk, as a certificate may verify correctly
after it has been reported as compromised and revoked; (however some would argue that the time from actual compromise until the discovery and reporting of it would in
most cases be a more significant lag). The Online Certificate Status Protocol (OCSP) (RFC2560) allows a client to query an OCSP responder for the current status of a certificate. This saves searching through a large CRL and can save bandwidth if the CRL would normally be downloaded - although it may increase network traffic. Most OCSP responders are based on CRLs
and thus do not solve the problem of time lag as outlined above.

==Dissemination==

===The Problem Domain===

===Random Ramblings on Reputation Management and Distribution===

Publish/Subscribe?

This system has unique distribution requirements as compared to most distributed systems in general. In this system, we cannot assume that there will be a universally agreed-upon definition of good, or bad. Similarly, the system must be self-policing. It would be up to each and every group of autonomous systems to decide which updates to accept and reject. Updates themselves also should not cause the network to DDoS itself. Lastly, it would be impossible for every system to know what the reputation for a given system is. Therefore the system must disseminate information in some way that is query-able and localizes reputation information where required.

To this end, we need a way of spreading information that while reliable, does not depend on one universally agreed-upon set of reputations.

For example, on an internet-scale operating system it would be entirely reasonable for one group of systems to not want to accept updates, or want to avoid communication with a given series of systems.

Any solution would assume that the problems of attribution are solved.

===Current Examples of Reputation Dissemination===

The first protocol that immediately comes to mind in this situation is a gossip-based protocol. These protocols are designed to operate in highly decentralized, large-scale systems.

Here's a nice overview:
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4537308 "Reputation management in distributed systems"

Examples are as follows:
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4228013 "Gossip-based Reputation Aggregation for Unstructured Peer-to-Peer Networks"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=5569965 "Improving Accuracy and Coverage in an Internet-Deployed Reputation Mechanism"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4459326 "GossipTrust for Fast Reputation Aggregation in Peer-to-Peer Networks"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4777496 "Adaptive trust management in P2P networks using gossip protocol"

Another possibility is using "Reputation chains"
* http://dx.doi.org.proxy.library.carleton.ca/10.1109/TKDE.2009.45 "P2P Reputation Management Using Distributed Identities and Decentralized Recommendation Chains"

==Maintaining History==

Problem domain:
Which history should I maintain? What to take as important, what to disregard?

Immutable data structure
Who could add data?
Who could remove data?
Authority

===Reputation systems===
* record, aggregate, distribute information about an entity's behaviour in distributed applications

* reputation might be based on the entity's past ability to adhere to a license agreement (mutual contract between issuer and licensee)

===History-based access control systems===
* make decision based on an entity's past security-sensitive actions

===Examples of reputation systems (trust-informing technologies)===
* eBay - Feedback forum (positive, neutral, negative)

===Do reputation systems have some validity?===

Resnick et al. argue that reputation systems
foster an incentive for principals to well-behave because of “the expectation of
reciprocity or retaliation in future interactions

Abstractions are used to model the aggregated information of each entity. These abstractions may not encompass the full details of transactions and provide context to specific issues relating to feedback. In turn we can end up with ambiguous values.

So we need a system that provides sufficient information in order to verify the precise properties of a past behaviour.

* Krukow, K. A Logical Framework for Reputation Systems and History-based Access Control. School of Electronics and Computer Science University of Southampton, UK. (March 3, 2011) [http://www.brics.dk/~krukow/research/publications/online_papers/concrete-jcs.pdf]

====Abstract====
Reputation systems are meta systems that record, aggregate and distribute information about principals’ behaviour in distributed applications. Similarly, history-based access control systems make decisions based
on programs’ past security-sensitive actions. While the applications are
distinct, the two types of systems are fundamentally making decisions
based on information about the past behaviour of an entity.
A logical policy-centric framework for such behaviour-based decisionmaking is presented. In the framework, principals specify policies which
state precise requirements on the past behaviour of other principals that
must be fulﬁlled in order for interaction to take place. The framework consists of a formal model of behaviour, based on event structures; a declarative logical language for specifying properties of past behaviour; and
eﬃcient dynamic algorithms for checking whether a particular behaviour
satisﬁes a property from the language. It is shown how the framework can
be extended in several ways, most notably to encompass parameterized
events and quantiﬁcation over parameters. In an extended application, it
is illustrated how the framework can be applied for dynamic history-based
access control for safe execution of unknown and untrusted programs.

* Khosrow-Pour, M. Emerging trends and challenges in information technology management (March 7, 2011) [http://books.google.ca/books?id=ybzS-yylJfAC&lpg=PA822&ots=V7hn_RzqXA&dq=maintaining%20history%20in%20reputation%20systems&pg=PA822#v=onepage&q=maintaining%20history%20in%20reputation%20systems&f=false]

====Abstract====

* Bolton, G. et al. How Effective are Electronic Reputation Mechanisms? (March 10, 2011) [http://ccs.mit.edu/dell/reputation/BKOMSsub.pdf]

====Abstract====

Electronic reputation or “feedback” mechanisms aim to mitigate the moral hazard problems
associated with exchange among strangers by providing the type of information available in
more traditional close-knit groups, where members are frequently involved in one another’s
dealings. In this paper, we compare trading in a market with electronic feedback (as
implemented by many Internet markets) to a market without, as well as to a market in which the
same people interact with one another repeatedly (partners market). We find that, while the
feedback mechanism induces quite a substantial improvement in transaction efficiency, it also
exhibits a kind of public goods problem in that, unlike the partners market, the benefits of trust
and trustworthy behavior go to the whole community and are not completely internalized. We
discuss the implications of this perspective for improving these systems.

==Querying Reputation==

=== Problems ===

* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
***If you want to know someone's reputation, you either need to start asking around for it, imposing yourself. Or you need the data to be sent around, so you already have access to it; emergent.
* Where do you store the data?
***You need to know who has the data to ask them for it, or to go get it yourself.
* Where is the data queried from?
***First you need to know who's storing it. then you need to know if you're allowed to ask that node directly, do you ask a intermediary keeper of data. Will you even need to Query-- that is, do you already have all you need to know on hand? you need not get the latest updates on a node if every other node who's ever talked to it got DDOSed. (or do you?)
* What defines good/bad reputation?
***Should I make my own definition for bad reputation, and query if someone engaged in activities I consider bad, or should their be a global agreed upon reputation?
* Who provides the good/bad reputation?
***Who should I ask for information from?
* Who do we trust for this information?
***Whoever you trust, presumably their opinion on a given node is more important then a node you trust less.
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
***topically, would you bother asking for 10 year old reputation data on a node, if it's been a model citizen for the last 9?
* What entities are able to contribute to reputations?
***Should I ask everyone I trust for an opinion on a given node, or just certain keepers of trust data?
* How do we access reputation about entities?
***You query someone in the know who you trust and are allowed to query.
***you could say, ask everyone you know and trust, and ask them to ask people they know and trust, (and so on...if they're willing) until you find a node with the information you need.
***in a more centralized system you need to ask some kind of keeper of information for the information you want, and that keeper may or may not provide you with the reputation info you want.
* Who is authorized to access particular reputations? How much to reveal? (Information flow)
***The ability to control this would depend on how centralized a system you have. In a truly distributed system where every node has an opinion on any other node they've talked to you'll be able to find somebody who can tell you about the CIA node, but in a more centralized system the keepers of information might be less...willing to give Joe 6 cores information on who Iran is DDOSing.

===Maybe References===
http://www.kirkarts.com/wiki/images/1/13/Resnick_eBay.pdf - ''Trust Among Strangers in Internet Transactions:
Empirical Analysis of eBay’s Reputation System'' (maybe not too relevant)

http://portal.acm.org/citation.cfm?id=544741.544809 - ''An Evidential Model of Distributed Reputation Management''

http://portal.acm.org/citation.cfm?id=775152.775242&type=series%EF%BF%BD%C3%9C -- ''The EigenTrust Algorithm for Reputation Management in
P2P Networks''

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.4.2297&rep=rep1&type=pdf -- ''A Robust Reputation System for Mobile Ad-hoc
Networks''

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.125.8729&rep=rep1&type=pdf -- ''EigenRep: Reputation Management in P2P Networks''

http://www.chennaisunday.com/ieee%202010/Reputation%20Estimation%20and%20Query%20in%20Peer-to-Peer%20Networks.pdf -- ''Reputation Estimation and Query in Peer-to-Peer Networks''

Here is another paper that might be interesting for you. -- Lester
http://dcg.ethz.ch/publications/netecon06.pdf

==Possible implementations==
==Conclusion==

==References==
* Joel Weise : "Public Key Infrastructure Overview " http://www.sun.com/blueprints/0801/publickey.pdf Accessed 2nd March 2011

* Security Glossary : http://www.cafesoft.com/support/security-glossary.html Accessed on 2nd March 2011

* Mattila, Anssi; and Mattila, Minna "What is the Effect of Product Attributes on Public-Key Infrastructure adoption? " http://internetjournals.net/journals/tir/2006/January/Paper%2003.pdf Accessed on 2nd March 2011

*Electronic Commerece Conference , PKI Sub-Group , Issue Paper : http://www.defense.gov/dodreform/ecwg/pki.pdf date accessed 5th March 2011

*SANS Institute InfoSec Reading Room, Common issues in PKI implementations - climbing the Slope of Enlightenment : http://www.sans.org/reading_room/whitepapers/authentication/common-issues-pki-implementations-climbing-slope-enlightenment_1198 date accessed 15th March 2011

DistOS-2011W Reputation

2011-03-17T15:36:03Z

Wahmed2: /* Common Issues With PKI Implementation */

==Members==
* Waheed Ahmed
* Trevor Gelowsky
** MSN: Gelowt@gmail.com
** E-Mail: tgelowsk@sce.carleton.ca
* Michael Du Plessis
* Nicolas Lessard (nick.lessard @t gmail.com / nlessard @t carleton.connect.ca)

==The problem==
* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
* Where do you store the data?
* Where is the data queried from?
* What defines good/bad reputation?
* Who provides the good/bad reputation?
* Who do we trust for this information?
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
* What entities are able to contribute to reputations?
* How do we access reputation about entities?
* Who is authorized to access particular reputations? How much to reveal? (Information flow)

==What technologies currently exist?==
* Digital signatures
** Certificates signed by trusted organizations

* Black hole- email, spam,
* Google - search reputation
* Credit bureaus
* Yellow pages
* Better business bureau
* CRC - criminal records

== What technologies don't currently exist?==

==Guaranteeing Authenticity/Public Key Infrastructure==

===Problems===

===Introduction===
In order to build secure chain of trust Public-Key Infrastructure is used for internet based communication. It consists of various things like security policy , Certificate authority , registration authority , certificate distribution system PKI enabled applications.

===Uses and Need===
With development of modern e-commerce based businesses which has minimal customer face-to-face interactions is demanding more security and integrity. The online web based stores where huge amount of transactions take place needs to ensure customers that there information is confidential and processed through a secure channel. This is where implementation of PKI steps in to provide mechanisms to ensure trusted relationships are established and maintained. The specific security functions in which a PKI can provide foundation are confidentiality, integrity, non-repudiation,and authentication.

===Issues & Solutions===
I found out there are many different implementations of PKI , and they all focuses on their own issues and solutions. For example PKI used in DoD have following issues

*Lack of PKI-enabled eCommerce applications and lack of interoperability among PKI applications

*DoD is developing a single high assurance PKI

*Very High Cost Impact to the EC/EB community.

*The PKI community lacks metrics for mapping of trust models between the DoD :”high assurance” C2 and EC/EB domains

*Education of everyone (policy maker through user) to a common level of understanding is a huge challenge.

*While the purpose of using PKI in EC/EB is to provide additional trust to allow the Internet to serve as a vehicle for legally binding transactions , problems still exist with the methodologies associated with establishing a long-term burden of proof. Specifically, there are no widely adopted industry standards for maintenance of electronic signatures or for authenticated timestamps for record maintenance that have stood the test of time. These processes are untried and the case law has not yet been established to convince users that there are no issues with enforcement of these new processes. An additional barrier to EC/EB within this space is the current DoD Certificate policy in which DoD accepts

===Common Issues With PKI Implementation===

*Commercial Off-The-Shelf (COTS) versus Customised applications : The choice between COTS or customised products is usually one of cost versus usability. In case of usability the thing to be focused should be error messages. If PKI is built int o applications (transparent to users) than its fine if not than user will require to have some understanding of the use of keys, certificates, Certificate Revocation Lists (CRLs)
and directories/certificate repositories so that they can make informed decisions.

*Token Logistics (smart card): The point where keys and certificates are linked to their owner is a very critical point in a PKI. If a fraudulent certificate is issued by a registration officer and the certificate holder uses the certificate to commit a crime or prank, trust in the whole PKI hierarchy may be lost. The physical security requirements are high, and the registration officer, whether a person or a smartcard bureau, must be subject to strict security polices and practices. As it was problem with DoD mentioned in section above.

*Network issues - Traffic : There is no doubt that the implementation of PKI will add to the network load, although just how much depends on the system architecture. Potential additional traffic that should be considered includes: Certificate issuance, Email usage, CRLs , and Directory Replication

*Network issues - Encryption : Many organisations implement anti-virus software and content inspection on servers at the perimeter of their networks. Some have security policies that rejects or quarantines encrypted traffic. To provide user-to-user confidentiality, messages will traverse networks with their payload hidden from inspection by virus and content checking.

*Email address in certificate :In order to use certificates for S/MIME signed/encrypted email, the users’ email address must be in the certificate. Most people change their email addresses more frequently than the certificate. Unless a solution is built which allows users to keep the same email address over a long period, certificates would have to be re-issued every time a user changes email address. S/MIME v.3 stipulates that the receiving application must check the From: or Sender: field in the mail header and compare it to an email address in the sender’s certificate. If the check does not match, the mail application should perform another explicit check to ensure that the person who signed the message is indeed the person who sent it. As usual, the ‘devil is in the detail’ when it comes to implementation.

*Certificate Validity Checking:CRLs have been the conventional method of providing certificate validity checking. CRLs do not scale very well as discussed earlier, but are usually kept for backward compatibility, archiving/historical verification and for use in off-line mode. The other issue with CRLs is that they are generally issued at certain intervals of 6, 12 or 24 hours, causing a time lag from the time a certificate is revoked until it appears on the published CRL. This may present a security risk, as a certificate may verify correctly
after it has been reported as compromised and revoked; (however some would argue that the time from actual compromise until the discovery and reporting of it would in
most cases be a more significant lag). The Online Certificate Status Protocol (OCSP) (RFC2560) allows a client to query an OCSP responder for the current status of a certificate. This saves searching through a large CRL and can save bandwidth if the CRL would normally be downloaded - although it may increase network traffic. Most OCSP responders are based on CRLs
and thus do not solve the problem of time lag as outlined above.

==Dissemination==

===The Problem Domain===

===Random Ramblings on Reputation Management and Distribution===

Publish/Subscribe?

This system has unique distribution requirements as compared to most distributed systems in general. In this system, we cannot assume that there will be a universally agreed-upon definition of good, or bad. Similarly, the system must be self-policing. It would be up to each and every group of autonomous systems to decide which updates to accept and reject. Updates themselves also should not cause the network to DDoS itself. Lastly, it would be impossible for every system to know what the reputation for a given system is. Therefore the system must disseminate information in some way that is query-able and localizes reputation information where required.

To this end, we need a way of spreading information that while reliable, does not depend on one universally agreed-upon set of reputations.

For example, on an internet-scale operating system it would be entirely reasonable for one group of systems to not want to accept updates, or want to avoid communication with a given series of systems.

Any solution would assume that the problems of attribution are solved.

===Current Examples of Reputation Dissemination===

The first protocol that immediately comes to mind in this situation is a gossip-based protocol. These protocols are designed to operate in highly decentralized, large-scale systems.

Here's a nice overview:
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4537308 "Reputation management in distributed systems"

Examples are as follows:
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4228013 "Gossip-based Reputation Aggregation for Unstructured Peer-to-Peer Networks"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=5569965 "Improving Accuracy and Coverage in an Internet-Deployed Reputation Mechanism"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4459326 "GossipTrust for Fast Reputation Aggregation in Peer-to-Peer Networks"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4777496 "Adaptive trust management in P2P networks using gossip protocol"

Another possibility is using "Reputation chains"
* http://dx.doi.org.proxy.library.carleton.ca/10.1109/TKDE.2009.45 "P2P Reputation Management Using Distributed Identities and Decentralized Recommendation Chains"

==Maintaining History==

Problem domain:
Which history should I maintain? What to take as important, what to disregard?

Immutable data structure
Who could add data?
Who could remove data?
Authority

===Reputation systems===
* record, aggregate, distribute information about an entity's behaviour in distributed applications

* reputation might be based on the entity's past ability to adhere to a license agreement (mutual contract between issuer and licensee)

===History-based access control systems===
* make decision based on an entity's past security-sensitive actions

===Examples of reputation systems (trust-informing technologies)===
* eBay - Feedback forum (positive, neutral, negative)

===Do reputation systems have some validity?===

Resnick et al. argue that reputation systems
foster an incentive for principals to well-behave because of “the expectation of
reciprocity or retaliation in future interactions

Abstractions are used to model the aggregated information of each entity. These abstractions may not encompass the full details of transactions and provide context to specific issues relating to feedback. In turn we can end up with ambiguous values.

So we need a system that provides sufficient information in order to verify the precise properties of a past behaviour.

* Krukow, K. A Logical Framework for Reputation Systems and History-based Access Control. School of Electronics and Computer Science University of Southampton, UK. (March 3, 2011) [http://www.brics.dk/~krukow/research/publications/online_papers/concrete-jcs.pdf]

====Abstract====
Reputation systems are meta systems that record, aggregate and distribute information about principals’ behaviour in distributed applications. Similarly, history-based access control systems make decisions based
on programs’ past security-sensitive actions. While the applications are
distinct, the two types of systems are fundamentally making decisions
based on information about the past behaviour of an entity.
A logical policy-centric framework for such behaviour-based decisionmaking is presented. In the framework, principals specify policies which
state precise requirements on the past behaviour of other principals that
must be fulﬁlled in order for interaction to take place. The framework consists of a formal model of behaviour, based on event structures; a declarative logical language for specifying properties of past behaviour; and
eﬃcient dynamic algorithms for checking whether a particular behaviour
satisﬁes a property from the language. It is shown how the framework can
be extended in several ways, most notably to encompass parameterized
events and quantiﬁcation over parameters. In an extended application, it
is illustrated how the framework can be applied for dynamic history-based
access control for safe execution of unknown and untrusted programs.

* Khosrow-Pour, M. Emerging trends and challenges in information technology management (March 7, 2011) [http://books.google.ca/books?id=ybzS-yylJfAC&lpg=PA822&ots=V7hn_RzqXA&dq=maintaining%20history%20in%20reputation%20systems&pg=PA822#v=onepage&q=maintaining%20history%20in%20reputation%20systems&f=false]

====Abstract====

* Bolton, G. et al. How Effective are Electronic Reputation Mechanisms? (March 10, 2011) [http://ccs.mit.edu/dell/reputation/BKOMSsub.pdf]

====Abstract====

Electronic reputation or “feedback” mechanisms aim to mitigate the moral hazard problems
associated with exchange among strangers by providing the type of information available in
more traditional close-knit groups, where members are frequently involved in one another’s
dealings. In this paper, we compare trading in a market with electronic feedback (as
implemented by many Internet markets) to a market without, as well as to a market in which the
same people interact with one another repeatedly (partners market). We find that, while the
feedback mechanism induces quite a substantial improvement in transaction efficiency, it also
exhibits a kind of public goods problem in that, unlike the partners market, the benefits of trust
and trustworthy behavior go to the whole community and are not completely internalized. We
discuss the implications of this perspective for improving these systems.

==Querying Reputation==

=== Problems ===

* Emerge vs. Impose reputation on the system?
** Probably both, how do we account for both systems?
***If you want to know someone's reputation, you either need to start asking around for it, imposing yourself. Or you need the data to be sent around, so you already have access to it; emergent.
* Where do you store the data?
***You need to know who has the data to ask them for it, or to go get it yourself.
* Where is the data queried from?
***First you need to know who's storing it. then you need to know if you're allowed to ask that node directly, do you ask a intermediary keeper of data. Will you even need to Query-- that is, do you already have all you need to know on hand? you need not get the latest updates on a node if every other node who's ever talked to it got DDOSed. (or do you?)
* What defines good/bad reputation?
***Should I make my own definition for bad reputation, and query if someone engaged in activities I consider bad, or should their be a global agreed upon reputation?
* Who provides the good/bad reputation?
***Who should I ask for information from?
* Who do we trust for this information?
***Whoever you trust, presumably their opinion on a given node is more important then a node you trust less.
* Should reputation be mutable? Can we be pardoned, or can reputations be reversed?
***topically, would you bother asking for 10 year old reputation data on a node, if it's been a model citizen for the last 9?
* What entities are able to contribute to reputations?
***Should I ask everyone I trust for an opinion on a given node, or just certain keepers of trust data?
* How do we access reputation about entities?
***You query someone in the know who you trust and are allowed to query.
***you could say, ask everyone you know and trust, and ask them to ask people they know and trust, (and so on...if they're willing) until you find a node with the information you need.
***in a more centralized system you need to ask some kind of keeper of information for the information you want, and that keeper may or may not provide you with the reputation info you want.
* Who is authorized to access particular reputations? How much to reveal? (Information flow)
***The ability to control this would depend on how centralized a system you have. In a truly distributed system where every node has an opinion on any other node they've talked to you'll be able to find somebody who can tell you about the CIA node, but in a more centralized system the keepers of information might be less...willing to give Joe 6 cores information on who Iran is DDOSing.

===Maybe References===
http://www.kirkarts.com/wiki/images/1/13/Resnick_eBay.pdf - ''Trust Among Strangers in Internet Transactions:
Empirical Analysis of eBay’s Reputation System'' (maybe not too relevant)

http://portal.acm.org/citation.cfm?id=544741.544809 - ''An Evidential Model of Distributed Reputation Management''

http://portal.acm.org/citation.cfm?id=775152.775242&type=series%EF%BF%BD%C3%9C -- ''The EigenTrust Algorithm for Reputation Management in
P2P Networks''

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.4.2297&rep=rep1&type=pdf -- ''A Robust Reputation System for Mobile Ad-hoc
Networks''

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.125.8729&rep=rep1&type=pdf -- ''EigenRep: Reputation Management in P2P Networks''

http://www.chennaisunday.com/ieee%202010/Reputation%20Estimation%20and%20Query%20in%20Peer-to-Peer%20Networks.pdf -- ''Reputation Estimation and Query in Peer-to-Peer Networks''

Here is another paper that might be interesting for you. -- Lester
http://dcg.ethz.ch/publications/netecon06.pdf

==Possible implementations==
==Conclusion==

==References==
* Joel Weise : "Public Key Infrastructure Overview " http://www.sun.com/blueprints/0801/publickey.pdf Accessed 2nd March 2011

* Security Glossary : http://www.cafesoft.com/support/security-glossary.html Accessed on 2nd March 2011

* Mattila, Anssi; and Mattila, Minna "What is the Effect of Product Attributes on Public-Key Infrastructure adoption? " http://internetjournals.net/journals/tir/2006/January/Paper%2003.pdf Accessed on 2nd March 2011

*Electronic Commerece Conference , PKI Sub-Group , Issue Paper : http://www.defense.gov/dodreform/ecwg/pki.pdf date accessed 5th March 2011

DistOS-2011W Reputation

2011-03-15T14:36:50Z

Wahmed2: /* Public-key infrastructure */

==Members==
* Waheed Ahmed
* Trevor Gelowsky
** MSN: Gelowt@gmail.com
** E-Mail: tgelowsk@sce.carleton.ca
* Michael Du Plessis
* Nicolas Lessard

==The problem==
* Emerge vs. Impose reputation on the system
* Where do you store the data?
* Where is the data queried from?
* What defines good/bad reputation?

==What technologies currently exist?==
* Digital signatures
** Certificates signed by trusted organizations

* Black hole- email, spam,
* Google - search reputation
* Credit bureaus
* Yellow pages
* Better business bureau
* CRC - criminal records

== What technologies don't currently exist?==

==Public-key infrastructure==

===Introduction===
In order to build secure chain of trust Public-Key Infrastructure is used for internet based communication. It consists of various things like security policy , Certificate authority , registration authority , certificate distribution system PKI enabled applications.

===Uses and Need===
With development of modern e-commerce based businesses which has minimal customer face-to-face interactions is demanding more security and integrity. The online web based stores where huge amount of transactions take place needs to ensure customers that there information is confidential and processed through a secure channel. This is where implementation of PKI steps in to provide mechanisms to ensure trusted relationships are established and maintained. The specific security functions in which a PKI can provide foundation are confidentiality, integrity, non-repudiation,and authentication.

===Issues & Solutions===
I found out there are many different implementations of PKI , and they all focuses on their own issues and solutions. For example PKI used in DoD have following issues

*Lack of PKI-enabled eCommerce applications and lack of interoperability among PKI applications

*DoD is developing a single high assurance PKI

*Very High Cost Impact to the EC/EB community.

*The PKI community lacks metrics for mapping of trust models between the DoD :”high assurance” C2 and EC/EB domains

*Education of everyone (policy maker through user) to a common level of understanding is a huge challenge.

*While the purpose of using PKI in EC/EB is to provide additional trust to allow the Internet to serve as a vehicle for legally binding transactions , problems still exist with the methodologies associated with establishing a long-term burden of proof. Specifically, there are no widely adopted industry standards for maintenance of electronic signatures or for authenticated timestamps for record maintenance that have stood the test of time. These processes are untried and the case law has not yet been established to convince users that there are no issues with enforcement of these new processes. An additional barrier to EC/EB within this space is the current DoD Certificate policy in which DoD accepts

===Common Issues With PKI Implementation===

*Commercial Off-The-Shelf (COTS) versus Customised applications : The choice between COTS or customised products is usually one of cost versus usability. In case of usability the thing to be focused should be error messages. If PKI is built int o applications (transparent to users) than its fine if not than user will require to have some understanding of the use of keys, certificates, Certificate Revocation Lists (CRLs)
and directories/certificate repositories so that they can make informed decisions.

*Token Logistics (smart card): The point where keys and certificates are linked to their owner is a very critical point in a PKI. If a fraudulent certificate is issued by a registration officer and the certificate holder uses the certificate to commit a crime or prank, trust in the whole PKI hierarchy may be lost. The physical security requirements are high, and the registration officer, whether a person or a smartcard bureau, must be subject to strict security polices and practices. As it was problem with DoD mentioned in section above.

*Network issues - Traffic : There is no doubt that the implementation of PKI will add to the network load, although just how much depends on the system architecture. Potential additional traffic that should be considered includes: Certificate issuance, Email usage, CRLs , and Directory Replication

*Network issues - Encryption : Many organisations implement anti-virus software and content inspection on servers at the perimeter of their networks. Some have security policies that rejects or quarantines encrypted traffic. To provide user-to-user confidentiality, messages will traverse networks with their payload hidden from inspection by virus and content checking.

*Email address in certificate :

==Dissemination==

===Random Ramblings on Reputation Management and Distribution===

This system has unique distribution requirements as compared to most distributed systems in general. In this system, we cannot assume that there will be a universally agreed-upon definition of good, or bad. Similarly, the system must be self-policing. It would be up to each and every group of autonomous systems to decide which updates to accept and reject. Updates themselves also should not cause the network to DDoS itself. Lastly, it would be impossible for every system to know what the reputation for a given system is. Therefore the system must disseminate information in some way that is query-able and localizes reputation information where required.

To this end, we need a way of spreading information that while reliable, does not depend on one universally agreed-upon set of reputations.

For example, on an internet-scale operating system it would be entirely reasonable for one group of systems to not want to accept updates, or want to avoid communication with a given series of systems.

Any solution would assume that the problems of attribution are solved.

===Current Examples of Reputation Dissemination===

The first protocol that immediately comes to mind in this situation is a gossip-based protocol. These protocols are designed to operate in highly decentralized, large-scale systems.

Here's a nice overview:
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4537308 "Reputation management in distributed systems"

Examples are as follows:
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4228013 "Gossip-based Reputation Aggregation for Unstructured Peer-to-Peer Networks"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=5569965 "Improving Accuracy and Coverage in an Internet-Deployed Reputation Mechanism"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4459326 "GossipTrust for Fast Reputation Aggregation in Peer-to-Peer Networks"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4777496 "Adaptive trust management in P2P networks using gossip protocol"

Another possibility is using "Reputation chains"
* http://dx.doi.org.proxy.library.carleton.ca/10.1109/TKDE.2009.45 "P2P Reputation Management Using Distributed Identities and Decentralized Recommendation Chains"

==Maintaining History==

===Reputation systems===
* record, aggregate, distribute information about an entity's behaviour in distributed applications

* reputation might be based on the entity's past ability to adhere to a license agreement (mutual contract between issuer and licensee)

===History-based access control systems===
* make decision based on an entity's past security-sensitive actions

===Examples of reputation systems (trust-informing technologies)===
* eBay - Feedback forum (positive, neutral, negative)

===Do reputation systems have some validity?===

Resnick et al. argue that reputation systems
foster an incentive for principals to well-behave because of “the expectation of
reciprocity or retaliation in future interactions

Abstractions are used to model the aggregated information of each entity. These abstractions may not encompass the full details of transactions and provide context to specific issues relating to feedback. In turn we can end up with ambiguous values.

So we need a system that provides sufficient information in order to verify the precise properties of a past behaviour.

* Krukow, K. A Logical Framework for Reputation Systems and History-based Access Control. School of Electronics and Computer Science University of Southampton, UK. (March 3, 2011) [http://www.brics.dk/~krukow/research/publications/online_papers/concrete-jcs.pdf]

====Abstract====
Reputation systems are meta systems that record, aggregate and distribute information about principals’ behaviour in distributed applications. Similarly, history-based access control systems make decisions based
on programs’ past security-sensitive actions. While the applications are
distinct, the two types of systems are fundamentally making decisions
based on information about the past behaviour of an entity.
A logical policy-centric framework for such behaviour-based decisionmaking is presented. In the framework, principals specify policies which
state precise requirements on the past behaviour of other principals that
must be fulﬁlled in order for interaction to take place. The framework consists of a formal model of behaviour, based on event structures; a declarative logical language for specifying properties of past behaviour; and
eﬃcient dynamic algorithms for checking whether a particular behaviour
satisﬁes a property from the language. It is shown how the framework can
be extended in several ways, most notably to encompass parameterized
events and quantiﬁcation over parameters. In an extended application, it
is illustrated how the framework can be applied for dynamic history-based
access control for safe execution of unknown and untrusted programs.

* Khosrow-Pour, M. Emerging trends and challenges in information technology management (March 7, 2011) [http://books.google.ca/books?id=ybzS-yylJfAC&lpg=PA822&ots=V7hn_RzqXA&dq=maintaining%20history%20in%20reputation%20systems&pg=PA822#v=onepage&q=maintaining%20history%20in%20reputation%20systems&f=false]

====Abstract====

* Bolton, G. et al. How Effective are Electronic Reputation Mechanisms? (March 10, 2011) [http://ccs.mit.edu/dell/reputation/BKOMSsub.pdf]

====Abstract====

Electronic reputation or “feedback” mechanisms aim to mitigate the moral hazard problems
associated with exchange among strangers by providing the type of information available in
more traditional close-knit groups, where members are frequently involved in one another’s
dealings. In this paper, we compare trading in a market with electronic feedback (as
implemented by many Internet markets) to a market without, as well as to a market in which the
same people interact with one another repeatedly (partners market). We find that, while the
feedback mechanism induces quite a substantial improvement in transaction efficiency, it also
exhibits a kind of public goods problem in that, unlike the partners market, the benefits of trust
and trustworthy behavior go to the whole community and are not completely internalized. We
discuss the implications of this perspective for improving these systems.

==Querying Reputation==

Since this won't be the actual page the paper is written on, I'm going to dump possibly relevant links here. If they actually get used I'll make them into proper references.

http://www.kirkarts.com/wiki/images/1/13/Resnick_eBay.pdf - ''Trust Among Strangers in Internet Transactions:
Empirical Analysis of eBay’s Reputation System'' (maybe not too relevant)

http://portal.acm.org/citation.cfm?id=544741.544809 - ''An Evidential Model of Distributed Reputation Management''

http://portal.acm.org/citation.cfm?id=775152.775242&type=series%EF%BF%BD%C3%9C -- ''The EigenTrust Algorithm for Reputation Management in
P2P Networks''

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.4.2297&rep=rep1&type=pdf -- ''A Robust Reputation System for Mobile Ad-hoc
Networks''

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.125.8729&rep=rep1&type=pdf -- ''EigenRep: Reputation Management in P2P Networks''

http://www.chennaisunday.com/ieee%202010/Reputation%20Estimation%20and%20Query%20in%20Peer-to-Peer%20Networks.pdf -- ''Reputation Estimation and Query in Peer-to-Peer Networks''

Here is another paper that might be interesting for you. -- Lester
http://dcg.ethz.ch/publications/netecon06.pdf

==Possible implementations==
==Conclusion==

==References==
* Joel Weise : "Public Key Infrastructure Overview " http://www.sun.com/blueprints/0801/publickey.pdf Accessed 2nd March 2011

* Security Glossary : http://www.cafesoft.com/support/security-glossary.html Accessed on 2nd March 2011

* Mattila, Anssi; and Mattila, Minna "What is the Effect of Product Attributes on Public-Key Infrastructure adoption? " http://internetjournals.net/journals/tir/2006/January/Paper%2003.pdf Accessed on 2nd March 2011

*Electronic Commerece Conference , PKI Sub-Group , Issue Paper : http://www.defense.gov/dodreform/ecwg/pki.pdf date accessed 5th March 2011

DistOS-2011W Reputation

2011-03-10T17:17:17Z

Wahmed2: /* References */

==Members==
* Waheed Ahmed
* Trevor Gelowsky
** MSN: Gelowt@gmail.com
** E-Mail: tgelowsk@sce.carleton.ca
** Current Status: Snowed-in somewhere in the outskirts of Orleans...
* Michael Du Plessis
* Nicolas Lessard

==The problem==
Emerge vs. Impose reputation on the system
==What currently exists?==
* Digital signatures
** Certificates signed by trusted organizations
==Public-key infrastructure==

===Introduction===
In order to build secure chain of trust Public-Key Infrastructure is used for internet based communication. It consists of various things like security policy , Certificate authority , registration authority , certificate distribution system PKI enabled applications.

===Uses and Need===
With development of modern e-commerce based businesses which has minimal customer face-to-face interactions is demanding more security and integrity. The online web based stores where huge amount of transactions take place needs to ensure customers that there information is confidential and processed through a secure channel. This is where implementation of PKI steps in to provide mechanisms to ensure trusted relationships are established and maintained. The specific security functions in which a PKI can provide foundation are confidentiality, integrity, non-repudiation,and authentication.

===Issues & Solutions===
I found out there are many different implementations of PKI , and they all focuses on their own issues and solutions. For example PKI used in DoD have following issues

*Lack of PKI-enabled eCommerce applications and lack of interoperability among PKI applications

*DoD is developing a single high assurance PKI

*Very High Cost Impact to the EC/EB community.

*The PKI community lacks metrics for mapping of trust models between the DoD :”high assurance” C2 and EC/EB domains

*Education of everyone (policy maker through user) to a common level of understanding is a huge challenge.

*While the purpose of using PKI in EC/EB is to provide additional trust to allow the Internet to serve as a vehicle for legally binding transactions , problems still exist with the methodologies associated with establishing a long-term burden of proof. Specifically, there are no widely adopted industry standards for maintenance of electronic signatures or for authenticated timestamps for record maintenance that have stood the test of time. These processes are untried and the case law has not yet been established to convince users that there are no issues with enforcement of these new processes. An additional barrier to EC/EB within this space is the current DoD Certificate policy in which DoD accepts

==Dissemination==

===Random Ramblings on Reputation Management and Distribution===

This system has unique distribution requirements as compared to most distributed systems in general. In this system, we cannot assume that there will be a universally agreed-upon definition of good, or bad. Similarly, the system must be self-policing. It would be up to each and every group of autonomous systems to decide which updates to accept and reject. Updates themselves also should not cause the network to DDoS itself. Lastly, it would be impossible for every system to know what the reputation for a given system is. Therefore the system must disseminate information in some way that is query-able and localizes reputation information where required.

To this end, we need a way of spreading information that while reliable, does not depend on one universally agreed-upon set of reputations.

For example, on an internet-scale operating system it would be entirely reasonable for one group of systems to not want to accept updates, or want to avoid communication with a given series of systems.

Any solution would assume that the problems of attribution are solved.

===Current Examples of Reputation Dissemination===

The first protocol that immediately comes to mind in this situation is a gossip-based protocol. These protocols are designed to operate in highly decentralized, large-scale systems.

Here's a nice overview:
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4537308 "Reputation management in distributed systems"

Examples are as follows:
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4228013 "Gossip-based Reputation Aggregation for Unstructured Peer-to-Peer Networks"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=5569965 "Improving Accuracy and Coverage in an Internet-Deployed Reputation Mechanism"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4459326 "GossipTrust for Fast Reputation Aggregation in Peer-to-Peer Networks"
* http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4777496 "Adaptive trust management in P2P networks using gossip protocol"

Another possibility is using "Reputation chains"
* http://dx.doi.org.proxy.library.carleton.ca/10.1109/TKDE.2009.45 "P2P Reputation Management Using Distributed Identities and Decentralized Recommendation Chains"

==Maintaining History==

===Reputation systems===
* record, aggregate, distribute information about an entity's behaviour in distributed applications

* reputation might be based on the entity's past ability to adhere to a license agreement (mutual contract between issuer and licensee)

===History-based access control systems===
* make decision based on an entity's past security-sensitive actions

===Examples of reputation systems (trust-informing technologies)===
* eBay - Feedback forum (positive, neutral, negative)

===Do reputation systems have some validity?===

Resnick et al. argue that reputation systems
foster an incentive for principals to well-behave because of “the expectation of
reciprocity or retaliation in future interactions

Abstractions are used to model the aggregated information of each entity. These abstractions may not encompass the full details of transactions and provide context to specific issues relating to feedback. In turn we can end up with ambiguous values.

So we need a system that provides sufficient information in order to verify the precise properties of a past behaviour.

* Krukow, K. A Logical Framework for Reputation Systems and History-based Access Control. School of Electronics and Computer Science University of Southampton, UK. (March 3, 2011) [http://www.brics.dk/~krukow/research/publications/online_papers/concrete-jcs.pdf]

* Khosrow-Pour, M. Emerging trends and challenges in information technology management (March 7, 2011) [http://books.google.ca/books?id=ybzS-yylJfAC&lpg=PA822&ots=V7hn_RzqXA&dq=maintaining%20history%20in%20reputation%20systems&pg=PA822#v=onepage&q=maintaining%20history%20in%20reputation%20systems&f=false]

==Querying Reputation==

Since this won't be the actual page the paper is written on, I'm going to dump possibly relevant links here. If they actually get used I'll make them into proper references.

http://www.kirkarts.com/wiki/images/1/13/Resnick_eBay.pdf - ''Trust Among Strangers in Internet Transactions:
Empirical Analysis of eBay’s Reputation System'' (maybe not too relevant)

http://portal.acm.org/citation.cfm?id=544741.544809 - ''An Evidential Model of Distributed Reputation Management''

http://portal.acm.org/citation.cfm?id=775152.775242&type=series%EF%BF%BD%C3%9C -- ''The EigenTrust Algorithm for Reputation Management in
P2P Networks''

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.4.2297&rep=rep1&type=pdf -- ''A Robust Reputation System for Mobile Ad-hoc
Networks''

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.125.8729&rep=rep1&type=pdf -- ''EigenRep: Reputation Management in P2P Networks''

http://www.chennaisunday.com/ieee%202010/Reputation%20Estimation%20and%20Query%20in%20Peer-to-Peer%20Networks.pdf -- ''Reputation Estimation and Query in Peer-to-Peer Networks''

Here is another paper that might be interesting for you. -- Lester
http://dcg.ethz.ch/publications/netecon06.pdf

==Possible implementations==
==Conclusion==

==References==
* Joel Weise : "Public Key Infrastructure Overview " http://www.sun.com/blueprints/0801/publickey.pdf Accessed 2nd March 2011

* Security Glossary : http://www.cafesoft.com/support/security-glossary.html Accessed on 2nd March 2011

* Mattila, Anssi; and Mattila, Minna "What is the Effect of Product Attributes on Public-Key Infrastructure adoption? " http://internetjournals.net/journals/tir/2006/January/Paper%2003.pdf Accessed on 2nd March 2011

*Electronic Commerece Conference , PKI Sub-Group , Issue Paper : http://www.defense.gov/dodreform/ecwg/pki.pdf date accessed 5th March 2011

DistOS-2011W Reputation

2011-03-10T17:15:49Z

Wahmed2: /* Public-key infrastructure */

DistOS-2011W Reputation

2011-03-03T16:22:21Z

Wahmed2: /* Uses and Need */

==Members==
* Waheed Ahmed
* Trevor Gelowsky
* Michael Du Plessis
* Nicolas Lessard

==The problem==
Emerge vs. Impose reputation on the system
==What currently exists?==
* Digital signatures
** Certificates signed by trusted organizations
==Public-key infrastructure==

===Introduction===
In order to build secure chain of trust Public-Key Infrastructure is used for internet based communication. It consists of various things like security policy , Certificate authority , registration authority , certificate distribution system PKI enabled applications.

===Uses and Need===
With development of modern e-commerce based businesses which has minimal customer face-to-face interactions is demanding more security and integrity. The online web based stores where huge amount of transactions take place needs to ensure customers that there information is confidential and processed through a secure channel. This is where implementation of PKI steps in to provide mechanisms to ensure trusted relationships are established and maintained. The specific security functions in which a PKI can provide foundation are confidentiality, integrity, non-repudiation,and authentication.

==Dissemination==
==Maintaining History==
==Querying Reputation==
==Possible implementations==
==Conclusion==

==References==
* Joel Weise : "Public Key Infrastructure Overview " http://www.sun.com/blueprints/0801/publickey.pdf Accessed 2nd March 2011

* Security Glossary : http://www.cafesoft.com/support/security-glossary.html Accessed on 2nd March 2011

* Mattila, Anssi; and Mattila, Minna "What is the Effect of Product Attributes on Public-Key Infrastructure adoption? " http://internetjournals.net/journals/tir/2006/January/Paper%2003.pdf Accessed on 2nd March 2011

DistOS-2011W Reputation

2011-03-03T16:21:50Z

Wahmed2: /* References */

==Members==
* Waheed Ahmed
* Trevor Gelowsky
* Michael Du Plessis
* Nicolas Lessard

==The problem==
Emerge vs. Impose reputation on the system
==What currently exists?==
* Digital signatures
** Certificates signed by trusted organizations
==Public-key infrastructure==

===Introduction===
In order to build secure chain of trust Public-Key Infrastructure is used for internet based communication. It consists of various things like security policy , Certificate authority , registration authority , certificate distribution system PKI enabled applications.

====Uses and Need====
With development of modern e-commerce based businesses which has minimal customer face-to-face interactions is demanding more security and integrity. The online web based stores where huge amount of transactions take place needs to ensure customers that there information is confidential and processed through a secure channel. This is where implementation of PKI steps in to provide mechanisms to ensure trusted relationships are established and maintained. The specific security functions in which a PKI can provide foundation are confidentiality, integrity, non-repudiation,and authentication.

==Dissemination==
==Maintaining History==
==Querying Reputation==
==Possible implementations==
==Conclusion==

==References==
* Joel Weise : "Public Key Infrastructure Overview " http://www.sun.com/blueprints/0801/publickey.pdf Accessed 2nd March 2011

* Security Glossary : http://www.cafesoft.com/support/security-glossary.html Accessed on 2nd March 2011

* Mattila, Anssi; and Mattila, Minna "What is the Effect of Product Attributes on Public-Key Infrastructure adoption? " http://internetjournals.net/journals/tir/2006/January/Paper%2003.pdf Accessed on 2nd March 2011

DistOS-2011W Reputation

2011-03-03T16:13:55Z

Wahmed2: /* References */

==Members==
* Waheed Ahmed
* Trevor Gelowsky
* Michael Du Plessis
* Nicolas Lessard

==The problem==
Emerge vs. Impose reputation on the system
==What currently exists?==
* Digital signatures
** Certificates signed by trusted organizations
==Public-key infrastructure==

===Introduction===
In order to build secure chain of trust Public-Key Infrastructure is used for internet based communication. It consists of various things like security policy , Certificate authority , registration authority , certificate distribution system PKI enabled applications.

====Uses and Need====
With development of modern e-commerce based businesses which has minimal customer face-to-face interactions is demanding more security and integrity. The online web based stores where huge amount of transactions take place needs to ensure customers that there information is confidential and processed through a secure channel. This is where implementation of PKI steps in to provide mechanisms to ensure trusted relationships are established and maintained. The specific security functions in which a PKI can provide foundation are confidentiality, integrity, non-repudiation,and authentication.

==Dissemination==
==Maintaining History==
==Querying Reputation==
==Possible implementations==
==Conclusion==

==References==
* Joel Weise : "Public Key Infrastructure Overview " http://www.sun.com/blueprints/0801/publickey.pdf Accessed 2nd March 2011

* Security Glossary : http://www.cafesoft.com/support/security-glossary.html Accessed on 2nd March 2011

DistOS-2011W Reputation

2011-03-03T16:09:53Z

Wahmed2: /* Public-key infrastructure */

==Members==
* Waheed Ahmed
* Trevor Gelowsky
* Michael Du Plessis
* Nicolas Lessard

==The problem==
Emerge vs. Impose reputation on the system
==What currently exists?==
* Digital signatures
** Certificates signed by trusted organizations
==Public-key infrastructure==

===Introduction===
In order to build secure chain of trust Public-Key Infrastructure is used for internet based communication. It consists of various things like security policy , Certificate authority , registration authority , certificate distribution system PKI enabled applications.

====Uses and Need====
With development of modern e-commerce based businesses which has minimal customer face-to-face interactions is demanding more security and integrity. The online web based stores where huge amount of transactions take place needs to ensure customers that there information is confidential and processed through a secure channel. This is where implementation of PKI steps in to provide mechanisms to ensure trusted relationships are established and maintained. The specific security functions in which a PKI can provide foundation are confidentiality, integrity, non-repudiation,and authentication.

==Dissemination==
==Maintaining History==
==Querying Reputation==
==Possible implementations==
==Conclusion==

==References==

DistOS-2011W NTP

2011-03-03T15:44:48Z

Wahmed2: /* References */

=Introduction:Context/Background=

Network Time Protocol (NTP) synchronizes clocks of hosts and routers. According to NIST estimates there are 10-20 million NTP servers and clients deployed in the internet and its tributaries all over the world. NTP software has been ported to almost every workstation and server platform available today – from PCs to Cray’s – UNIX, Windows, VMS and embedded systems, even home routers and battery backup systems.

NTP provides nominal accuracies of low tens of milliseconds on WANs, sub milliseconds on LANs and sub microseconds using a precision time source such as cesium oscillator or GPS receiver. Our time protocol was designed over the LAN and our goal was to achieve sub milliseconds of accuracy.

== Importance==

Precision of time is very important in the computer world. Following are some example why it is so important to maintain a precise and accurate time.

Network monitoring, measurement and control, Distributed database transaction journaling and logging, Secure document timestamps with cryptographic certification, Radio and TV programming lunch and monitoring, Stock market buy and sell orders, Distributed network gaming and training, etc.

== Definition of the problem ==

My goal was to acquire time accuracy in the Linux lab assuming that all the computers do not have internet connection. To time synchronize all the computers I had to build our own hierarchy of NTP servers and clients.

== Summary of the result ==

An efficient hierarchy of NTP server and client has been implemented. Precision of
Sub milliseconds were achieved.

==Outline of the report==
Detailed background information and the structure of our time protocol are provided in Section
2. The development setup and code explanation is reviewed in Section 3. Details of which server/client contains which files are given in section 4. My contributions in this implementation are described in Section 5.

= Detailed context/background information=

In this section the background information of Network Time protocol and the hierarchical model and the comparison with the original NTP designed by Dave Mills is reviewed. For unambiguity I will call the protocol as Linux lab time protocol (LLTP) and the original Network time protocol as NTP.

== NTP timestamps==

The 64-bit timestamps used by NTP consist of a 32-bit second part and a 32-bit fractional second part. I followed the same time structure for our LLTP which gives us a time scale of 232 seconds (136 years) and a theoretical resolution of 2-32 seconds (233 picoseconds).

== Structure==

NTP uses a hierarchical, semi-layered system of levels of clock sources; each level of this hierarchy is termed a stratum and assigned a layer number starting with 0 at the top. The stratum level defines its distance from the reference clock and exists to prevent cyclical dependencies in hierarchy.

My target is to develop a time synchronizing software for a small office or lab with a large number of computer where a very few computer has internet connectivity but all the computers are connected together in a local network. As a result not all the computer is able to connect to internet for time accuracy. LLTP uses similar hierarchical, semi-layered sources as NTP. Each layer of this hierarchy is termed as level and assigned a layer number starting with 1.

== Level 1 server==

Computers with internet connectivity are the level 1 server for our LLTP

Level 1 server

synchronizes its system clock with the NTP servers on the internet. In this hierarchy, only a fixed (5) number of level 1 servers are used. Unlike NTP we don’t have the privilege to use a 10 of thousands of level 1 and level 2 servers, and unlike NTP our LLTP has only 3 level structures.

Picture below shows the connectivity and the hierarchy of LLTP

[[File:LTTPHireachy.jpg]]

== Level 2 server and client==

Level 2 servers run two different programs. A client program to synchronize its system clock with the level1 servers and a server program that waits for any level 3 client for time request. Upon the request it sends its synchronized system time to the client. Level 2 servers also synchronize its system clock with the other level2 servers.

Each level2 server is connected to 4 level1 servers. It polls time from each level1 server and chooses the best time and synchronizes its system clock. How the best time is achieved is described in the section 3 below with code explanation. There are maximum 5 level2 servers with our LLTP model. Each level2 server is connected to 4 different level1 servers. None of the level2 server has the same level1 server set.

== Level 3 client==

Bottom of the LLTP hierarchy is level3 client. It only runs a client program that polls time from the level2 servers. It can only connect/request time from one of our level2 servers at a time. It also requires the knowledge of the IP address of the level2 server. For the load management purpose level 3 clients are not able to connect to a Level1 servers.

= Development Details=

== Server==

As it is mentioned in section-2, there are two servers in our LLTP model. One is level 1 and the other is level 2. Both servers wait for a client with time requests and upon receiving one send the time string. Code structure for both the servers is almost similar with minor technical differences which are explained later in this section. Both servers use a infinite for loop for waiting for the
client. They use a fixed but different port number for the client connection.

[[File:clientntp.jpg]]

Time requests sent to Level 1 server have to contain a message no more than 1 character and it has to be an integer which is sent by the level2 client to distinguish the level1 servers from one another. Level1 server will take that message received from the client and add that at the end of the time string sent to the client. If the level1 server receives a time request message with greater size of 1than it will not return any time value. For the level2 server the maximum message size is
128 characters and it do not have to be an integer. Level2 server discards the message sent from the level3 client. A unique feature of level2 server is that a level3 client time request with “exit” message can shut down the level2 server. This feature is not implemented in level1 servers.

[[File:level1servercode.jpg]]

[[File:level2servercode.jpg]]

Both servers send time as a string which handles the problem with big and little Endean.

== Client==

There are also two clients for our LLTP, level2 client and level3 client. Level3 client is very straight forward where the level3 is a bit more complicated.

Level3 client sends a time request to the level2 server. It cannot send time request to a level1 server as the port number for both server are different and the port numbers are hard coded for both the client (leve3 client/level2 server uses port 123 where level2 client/level1 server uses port 4003). Level2 server also has a restriction over the message sent by the client which is explained above in the server section. Upon receiving the time string back from the server which contains the seconds and milliseconds part together as a string separated by a comma (‘,’), it separates this two parts and checks for the latency and calls function named set_time(long[] ) which sets the system clock time with the new time.

[[File:level3clientcode.jpg]]

Level2 client dose almost the same things but in a broader and sophisticated way. It polls time from four different levle1 time server at the same time. When it sends the time request, it sends a unique message as the server ID to identify the time received from different level1 server which helps to fix the latency later.

[[File:level2clientcode.jpg]]

Level2 client also uses a time out of 1 second. If it dose not receive the time string back from the server then it will ignore the time from that server and choose the best time from the ones it received from the other servers. Received time strings from the different servers also have the server id added at the end of the time string. Using that information level2 client saves the received time in order and fixes the latency accordingly. It keeps all the latency fixed time in an array with the invalid time being -1. Then it calls a different function called marzullo(long []), which takes the array of latency fixed times and returns the best time depending on the Marzullo’s algorithm. Function marzullo() is explained in the section 3.4. Then it uses the set_time(long []) method to set the system clock time.

[[File:level2clientcode2.jpg]]

== Daemon client and server==

The level2 client and server runs as a daemon process to give them the privilege to run continuously in the background. As both the client and server in level2 does not need any user interference and need to run all the time to update the system clock of the running computer from the level1 server.

As it is mentioned above the client process for level2 run continuously and it updates its system clock every 1 minute (polls time from the level1 server every 60 seconds). Server daemon process is built in the same manner but unlike the client process it does not have a waiting time.

[[File:codefordaemon.jpg]]

== Marzullo’s algorithm and function marzullo(long []) ==

Marzullo’s algorithm, invented by Keith Marzullo, is an agreement algorithm used to select sources for estimating accurate time from a number of noisy time sources. A modified version of his algorithm, intersection algorithm, is use in NTP. We used the simplified version of this algorithm to choose the best possible time from our time sources (level1 servers).

Our function long* marzullo (long []) takes an array of four times from four different level1 servers and return a pointer to an array with the best seconds and milliseconds. First it takes the given array of times and makes two entries for each time as ±2 milliseconds. For each entry it also adds a type of -1 for -2millisecons and +1 for +2 milliseconds. As a result for 4 time entry it produces an array of 8 time entries with type as ±1. For our LLTP we took ±2milliseconds as our error range. After adding the type the entry will look like <entry – 2, -1> for the beginning of the range and <entry+2, +1> for the end of the range [entry±2].

[[File:IntersectionMarzullasalg.jpg]]

The description of the algorithm uses the following variables: best (largest number of overlapping intervals found), cnt (current number of overlapping intervals), best_start and

best_end (the beginning and end of best interval found so far). We used a two dimensional array to keep the second, millisecond and type of the range. The structure of the array is as follows:

time_entry [i][0] = second part of the time
entry time_entry [i][1] = millisecond part of the time
entry time_entry [i][2] = type of the range (i.e. ±1)

Here ‘Time_entry is the name of the array and ‘i’ is used as an index to traverse the array. After creating the array it follows the following step to determine the accurate time. Then it sends back the best chosen time to the level2 client program.

[[File:section3last.jpg]]

= List of files=

Level1 server:

DieWithError.c
get-time.c
L1Server.c

Level2 server:

daemon_server.c
DieWithError.c
get-time.c
L2Server.c

Level2 client:

daemon_client.c
DieWithError.c
get-time.c
L2Client.c
marzullo.c
print-time.c
selection_sort.c
set-time.c

Level3 client:

DieWithError.c
get-time.c
print-time.c
set-time.c
UDPTimeClient.c

=Discussion/Output=

The most intresting part of testing this was to see how distrubted architect of NTP client-server works and how could it be improved. I learnt alot by setting up and running them and seeing the results. It was fun improving little bit on exsisting implementation of NTP.

Output form different levels of clients and servers are given in this section. Output from level1 server:

[[File:output1.jpg]]

[[File:output2.jpg]]

[[File:output3.jpg]]

[[File:output4.jpg]]

=Conclusion=

Overall, there has been lots of work going on NTP , due to high demand of distributed clinet-server architect based application. This was carried on a very small linux based lab , in future I guess there will be need to test on large industry based scale , where many systems can be connected and than test to see how the delay from clients to server causes disturbance in algorithm. There will be issues for sure since as the information travels it takes time and that delayed time will need to be adjusted more and will be focus of future work. The most important feature of NTP relevant to this paper is its scalability: its phase lock loop clock discipline is designed for small scale networks so that NTP can maintain low offsets among clocks, even if propagation delays increase significantly.

They main problem of NTP is under heavy load, constantly changing offsets may cause frequent clock re-adjustments by NTP. Each time it re-adjust, packets transmitted at later time may have smaller time stamps than packets transmitted earlier resulting in interfere with path monitoring. Future plan is to Global Positioning System receivers for clock synchronization among edge routers.

Whether its traffic in air or ground or buying selling things carefully coordinated , reliable and accurate time is vital. In some cases ill-gotten time might cause DNS caches to to expire and entire internet to implode on the root servers , which was one of the threat on the eve of millennium in 1999. Critical data files might expire before they are created, and an electronic message might arrive before it was sent. Reliable and accurate computer time is necessary for any real-time distributed computer application, which is what much of our public infrastructure has become.

=References=

*Ntp home page: http://ntp.org Acessed on 13th Feb 2011

*Wikipedia for network time protocol: http://en.wikipedia.org/wiki/Network_Time_Protocol Accessed on 13th Feb 2011

*Wikipedia for Marzullo’s algorithm: http://en.wikipedia.org/wiki/Marzullo's_algorithm Accessed on 13th Feb 2011

*Linux Home Networking : "How To : NTP Server" : http://www.linuxhomenetworking.com/wiki/index.php/Quick_HOWTO_:_Ch24_:_The_NTP_Server Accessed on 15 Feb 2011

DistOS-2011W NTP

2011-03-01T05:19:07Z

Wahmed2: /* Conclusion */

DistOS-2011W NTP

2011-02-28T19:45:25Z

Wahmed2: /* Conclusion */

=Introduction:Context/Background=

Network Time Protocol (NTP) synchronizes clocks of hosts and routers. According to NIST estimates there are 10-20 million NTP servers and clients deployed in the internet and its tributaries all over the world. NTP software has been ported to almost every workstation and server platform available today – from PCs to Cray’s – UNIX, Windows, VMS and embedded systems, even home routers and battery backup systems.

NTP provides nominal accuracies of low tens of milliseconds on WANs, sub milliseconds on LANs and sub microseconds using a precision time source such as cesium oscillator or GPS receiver. Our time protocol was designed over the LAN and our goal was to achieve sub milliseconds of accuracy.

== Importance==

Precision of time is very important in the computer world. Following are some example why it is so important to maintain a precise and accurate time.

Network monitoring, measurement and control, Distributed database transaction journaling and logging, Secure document timestamps with cryptographic certification, Radio and TV programming lunch and monitoring, Stock market buy and sell orders, Distributed network gaming and training, etc.

== Definition of the problem ==

My goal was to acquire time accuracy in the Linux lab assuming that all the computers do not have internet connection. To time synchronize all the computers I had to build our own hierarchy of NTP servers and clients.

== Summary of the result ==

An efficient hierarchy of NTP server and client has been implemented. Precision of
Sub milliseconds were achieved.

==Outline of the report==
Detailed background information and the structure of our time protocol are provided in Section
2. The development setup and code explanation is reviewed in Section 3. Details of which server/client contains which files are given in section 4. My contributions in this implementation are described in Section 5.

= Detailed context/background information=

In this section the background information of Network Time protocol and the hierarchical model and the comparison with the original NTP designed by Dave Mills is reviewed. For unambiguity I will call the protocol as Linux lab time protocol (LLTP) and the original Network time protocol as NTP.

== NTP timestamps==

The 64-bit timestamps used by NTP consist of a 32-bit second part and a 32-bit fractional second part. I followed the same time structure for our LLTP which gives us a time scale of 232 seconds (136 years) and a theoretical resolution of 2-32 seconds (233 picoseconds).

== Structure==

NTP uses a hierarchical, semi-layered system of levels of clock sources; each level of this hierarchy is termed a stratum and assigned a layer number starting with 0 at the top. The stratum level defines its distance from the reference clock and exists to prevent cyclical dependencies in hierarchy.

My target is to develop a time synchronizing software for a small office or lab with a large number of computer where a very few computer has internet connectivity but all the computers are connected together in a local network. As a result not all the computer is able to connect to internet for time accuracy. LLTP uses similar hierarchical, semi-layered sources as NTP. Each layer of this hierarchy is termed as level and assigned a layer number starting with 1.

== Level 1 server==

Computers with internet connectivity are the level 1 server for our LLTP

Level 1 server

synchronizes its system clock with the NTP servers on the internet. In this hierarchy, only a fixed (5) number of level 1 servers are used. Unlike NTP we don’t have the privilege to use a 10 of thousands of level 1 and level 2 servers, and unlike NTP our LLTP has only 3 level structures.

Picture below shows the connectivity and the hierarchy of LLTP

[[File:LTTPHireachy.jpg]]

== Level 2 server and client==

Level 2 servers run two different programs. A client program to synchronize its system clock with the level1 servers and a server program that waits for any level 3 client for time request. Upon the request it sends its synchronized system time to the client. Level 2 servers also synchronize its system clock with the other level2 servers.

Each level2 server is connected to 4 level1 servers. It polls time from each level1 server and chooses the best time and synchronizes its system clock. How the best time is achieved is described in the section 3 below with code explanation. There are maximum 5 level2 servers with our LLTP model. Each level2 server is connected to 4 different level1 servers. None of the level2 server has the same level1 server set.

== Level 3 client==

Bottom of the LLTP hierarchy is level3 client. It only runs a client program that polls time from the level2 servers. It can only connect/request time from one of our level2 servers at a time. It also requires the knowledge of the IP address of the level2 server. For the load management purpose level 3 clients are not able to connect to a Level1 servers.

= Development Details=

== Server==

As it is mentioned in section-2, there are two servers in our LLTP model. One is level 1 and the other is level 2. Both servers wait for a client with time requests and upon receiving one send the time string. Code structure for both the servers is almost similar with minor technical differences which are explained later in this section. Both servers use a infinite for loop for waiting for the
client. They use a fixed but different port number for the client connection.

[[File:clientntp.jpg]]

Time requests sent to Level 1 server have to contain a message no more than 1 character and it has to be an integer which is sent by the level2 client to distinguish the level1 servers from one another. Level1 server will take that message received from the client and add that at the end of the time string sent to the client. If the level1 server receives a time request message with greater size of 1than it will not return any time value. For the level2 server the maximum message size is
128 characters and it do not have to be an integer. Level2 server discards the message sent from the level3 client. A unique feature of level2 server is that a level3 client time request with “exit” message can shut down the level2 server. This feature is not implemented in level1 servers.

[[File:level1servercode.jpg]]

[[File:level2servercode.jpg]]

Both servers send time as a string which handles the problem with big and little Endean.

== Client==

There are also two clients for our LLTP, level2 client and level3 client. Level3 client is very straight forward where the level3 is a bit more complicated.

Level3 client sends a time request to the level2 server. It cannot send time request to a level1 server as the port number for both server are different and the port numbers are hard coded for both the client (leve3 client/level2 server uses port 123 where level2 client/level1 server uses port 4003). Level2 server also has a restriction over the message sent by the client which is explained above in the server section. Upon receiving the time string back from the server which contains the seconds and milliseconds part together as a string separated by a comma (‘,’), it separates this two parts and checks for the latency and calls function named set_time(long[] ) which sets the system clock time with the new time.

[[File:level3clientcode.jpg]]

Level2 client dose almost the same things but in a broader and sophisticated way. It polls time from four different levle1 time server at the same time. When it sends the time request, it sends a unique message as the server ID to identify the time received from different level1 server which helps to fix the latency later.

[[File:level2clientcode.jpg]]

Level2 client also uses a time out of 1 second. If it dose not receive the time string back from the server then it will ignore the time from that server and choose the best time from the ones it received from the other servers. Received time strings from the different servers also have the server id added at the end of the time string. Using that information level2 client saves the received time in order and fixes the latency accordingly. It keeps all the latency fixed time in an array with the invalid time being -1. Then it calls a different function called marzullo(long []), which takes the array of latency fixed times and returns the best time depending on the Marzullo’s algorithm. Function marzullo() is explained in the section 3.4. Then it uses the set_time(long []) method to set the system clock time.

[[File:level2clientcode2.jpg]]

== Daemon client and server==

The level2 client and server runs as a daemon process to give them the privilege to run continuously in the background. As both the client and server in level2 does not need any user interference and need to run all the time to update the system clock of the running computer from the level1 server.

As it is mentioned above the client process for level2 run continuously and it updates its system clock every 1 minute (polls time from the level1 server every 60 seconds). Server daemon process is built in the same manner but unlike the client process it does not have a waiting time.

[[File:codefordaemon.jpg]]

== Marzullo’s algorithm and function marzullo(long []) ==

Marzullo’s algorithm, invented by Keith Marzullo, is an agreement algorithm used to select sources for estimating accurate time from a number of noisy time sources. A modified version of his algorithm, intersection algorithm, is use in NTP. We used the simplified version of this algorithm to choose the best possible time from our time sources (level1 servers).

Our function long* marzullo (long []) takes an array of four times from four different level1 servers and return a pointer to an array with the best seconds and milliseconds. First it takes the given array of times and makes two entries for each time as ±2 milliseconds. For each entry it also adds a type of -1 for -2millisecons and +1 for +2 milliseconds. As a result for 4 time entry it produces an array of 8 time entries with type as ±1. For our LLTP we took ±2milliseconds as our error range. After adding the type the entry will look like <entry – 2, -1> for the beginning of the range and <entry+2, +1> for the end of the range [entry±2].

[[File:IntersectionMarzullasalg.jpg]]

The description of the algorithm uses the following variables: best (largest number of overlapping intervals found), cnt (current number of overlapping intervals), best_start and

best_end (the beginning and end of best interval found so far). We used a two dimensional array to keep the second, millisecond and type of the range. The structure of the array is as follows:

time_entry [i][0] = second part of the time
entry time_entry [i][1] = millisecond part of the time
entry time_entry [i][2] = type of the range (i.e. ±1)

Here ‘Time_entry is the name of the array and ‘i’ is used as an index to traverse the array. After creating the array it follows the following step to determine the accurate time. Then it sends back the best chosen time to the level2 client program.

[[File:section3last.jpg]]

= List of files=

Level1 server:

DieWithError.c
get-time.c
L1Server.c

Level2 server:

daemon_server.c
DieWithError.c
get-time.c
L2Server.c

Level2 client:

daemon_client.c
DieWithError.c
get-time.c
L2Client.c
marzullo.c
print-time.c
selection_sort.c
set-time.c

Level3 client:

DieWithError.c
get-time.c
print-time.c
set-time.c
UDPTimeClient.c

=Discussion/Output=

The most intresting part of testing this was to see how distrubted architect of NTP client-server works and how could it be improved. I learnt alot by setting up and running them and seeing the results. It was fun improving little bit on exsisting implementation of NTP.

Output form different levels of clients and servers are given in this section. Output from level1 server:

[[File:output1.jpg]]

[[File:output2.jpg]]

[[File:output3.jpg]]

[[File:output4.jpg]]

=Conclusion=

Overall, there has been lots of work going on NTP , due to high demand of distributed clinet-server architect based application. This was carried on a very small linux based lab , in future I guess there will be need to test on large industry based scale , where many systems can be connected and than test to see how the delay from clients to server causes disturbance in algorithm. There will be issues for sure since as the information travels it takes time and that delayed time will need to be adjusted more and will be focus of future work. The most important feature of NTP relevant to this paper is its scalability: its phase lock loop clock discipline is designed for small scale networks so that NTP can maintain low offsets among clocks, even if propagation delays increase significantly.

They main problem of NTP is under heavy load, constantly changing offsets may cause frequent clock re-adjustments by NTP. Each time it re-adjust, packets transmitted at later time may have smaller time stamps than packets transmitted earlier resulting in interfere with path monitoring. Future plan is to Global Positioning System receivers for clock synchronization among edge routers.

=References=

*Ntp home page: http://ntp.org Acessed on 13th Feb 2011

*Wikipedia for network time protocol: http://en.wikipedia.org/wiki/Network_Time_Protocol Accessed on 13th Feb 2011

*Wikipedia for Marzullo’s algorithm: http://en.wikipedia.org/wiki/Marzullo's_algorithm Accessed on 13th Feb 2011

DistOS-2011W NTP

2011-02-22T18:38:30Z

Wahmed2: /* 3.2 Client */

=Introduction:Context/Background=

Network Time Protocol (NTP) synchronizes clocks of hosts and routers. According to NIST estimates there are 10-20 million NTP servers and clients deployed in the internet and its tributaries all over the world. NTP software has been ported to almost every workstation and server platform available today – from PCs to Cray’s – UNIX, Windows, VMS and embedded systems, even home routers and battery backup systems.

NTP provides nominal accuracies of low tens of milliseconds on WANs, sub milliseconds on LANs and sub microseconds using a precision time source such as cesium oscillator or GPS receiver. Our time protocol was designed over the LAN and our goal was to achieve sub milliseconds of accuracy.

== Importance==

Precision of time is very important in the computer world. Following are some example why it is so important to maintain a precise and accurate time.

Network monitoring, measurement and control, Distributed database transaction journaling and logging, Secure document timestamps with cryptographic certification, Radio and TV programming lunch and monitoring, Stock market buy and sell orders, Distributed network gaming and training, etc.

== Definition of the problem ==

My goal was to acquire time accuracy in the Linux lab assuming that all the computers do not have internet connection. To time synchronize all the computers I had to build our own hierarchy of NTP servers and clients.

== Summary of the result ==

An efficient hierarchy of NTP server and client has been implemented. Precision of
Sub milliseconds were achieved.

==Outline of the report==
Detailed background information and the structure of our time protocol are provided in Section
2. The development setup and code explanation is reviewed in Section 3. Details of which server/client contains which files are given in section 4. My contributions in this implementation are described in Section 5.

= Detailed context/background information=

In this section the background information of Network Time protocol and the hierarchical model and the comparison with the original NTP designed by Dave Mills is reviewed. For unambiguity I will call the protocol as Linux lab time protocol (LLTP) and the original Network time protocol as NTP.

== NTP timestamps==

The 64-bit timestamps used by NTP consist of a 32-bit second part and a 32-bit fractional second part. I followed the same time structure for our LLTP which gives us a time scale of 232 seconds (136 years) and a theoretical resolution of 2-32 seconds (233 picoseconds).

== Structure==

NTP uses a hierarchical, semi-layered system of levels of clock sources; each level of this hierarchy is termed a stratum and assigned a layer number starting with 0 at the top. The stratum level defines its distance from the reference clock and exists to prevent cyclical dependencies in hierarchy.

My target is to develop a time synchronizing software for a small office or lab with a large number of computer where a very few computer has internet connectivity but all the computers are connected together in a local network. As a result not all the computer is able to connect to internet for time accuracy. LLTP uses similar hierarchical, semi-layered sources as NTP. Each layer of this hierarchy is termed as level and assigned a layer number starting with 1.

== Level 1 server==

Computers with internet connectivity are the level 1 server for our LLTP

Level 1 server

synchronizes its system clock with the NTP servers on the internet. In this hierarchy, only a fixed (5) number of level 1 servers are used. Unlike NTP we don’t have the privilege to use a 10 of thousands of level 1 and level 2 servers, and unlike NTP our LLTP has only 3 level structures.

Picture below shows the connectivity and the hierarchy of LLTP

[[File:LTTPHireachy.jpg]]

== Level 2 server and client==

Level 2 servers run two different programs. A client program to synchronize its system clock with the level1 servers and a server program that waits for any level 3 client for time request. Upon the request it sends its synchronized system time to the client. Level 2 servers also synchronize its system clock with the other level2 servers.

Each level2 server is connected to 4 level1 servers. It polls time from each level1 server and chooses the best time and synchronizes its system clock. How the best time is achieved is described in the section 3 below with code explanation. There are maximum 5 level2 servers with our LLTP model. Each level2 server is connected to 4 different level1 servers. None of the level2 server has the same level1 server set.

== Level 3 client==

Bottom of the LLTP hierarchy is level3 client. It only runs a client program that polls time from the level2 servers. It can only connect/request time from one of our level2 servers at a time. It also requires the knowledge of the IP address of the level2 server. For the load management purpose level 3 clients are not able to connect to a Level1 servers.

= Development Details=

== Server==

As it is mentioned in section-2, there are two servers in our LLTP model. One is level 1 and the other is level 2. Both servers wait for a client with time requests and upon receiving one send the time string. Code structure for both the servers is almost similar with minor technical differences which are explained later in this section. Both servers use a infinite for loop for waiting for the
client. They use a fixed but different port number for the client connection.

[[File:clientntp.jpg]]

Time requests sent to Level 1 server have to contain a message no more than 1 character and it has to be an integer which is sent by the level2 client to distinguish the level1 servers from one another. Level1 server will take that message received from the client and add that at the end of the time string sent to the client. If the level1 server receives a time request message with greater size of 1than it will not return any time value. For the level2 server the maximum message size is
128 characters and it do not have to be an integer. Level2 server discards the message sent from the level3 client. A unique feature of level2 server is that a level3 client time request with “exit” message can shut down the level2 server. This feature is not implemented in level1 servers.

[[File:level1servercode.jpg]]

[[File:level2servercode.jpg]]

Both servers send time as a string which handles the problem with big and little Endean.

== Client==

There are also two clients for our LLTP, level2 client and level3 client. Level3 client is very straight forward where the level3 is a bit more complicated.

Level3 client sends a time request to the level2 server. It cannot send time request to a level1 server as the port number for both server are different and the port numbers are hard coded for both the client (leve3 client/level2 server uses port 123 where level2 client/level1 server uses port 4003). Level2 server also has a restriction over the message sent by the client which is explained above in the server section. Upon receiving the time string back from the server which contains the seconds and milliseconds part together as a string separated by a comma (‘,’), it separates this two parts and checks for the latency and calls function named set_time(long[] ) which sets the system clock time with the new time.

[[File:level3clientcode.jpg]]

Level2 client dose almost the same things but in a broader and sophisticated way. It polls time from four different levle1 time server at the same time. When it sends the time request, it sends a unique message as the server ID to identify the time received from different level1 server which helps to fix the latency later.

[[File:level2clientcode.jpg]]

Level2 client also uses a time out of 1 second. If it dose not receive the time string back from the server then it will ignore the time from that server and choose the best time from the ones it received from the other servers. Received time strings from the different servers also have the server id added at the end of the time string. Using that information level2 client saves the received time in order and fixes the latency accordingly. It keeps all the latency fixed time in an array with the invalid time being -1. Then it calls a different function called marzullo(long []), which takes the array of latency fixed times and returns the best time depending on the Marzullo’s algorithm. Function marzullo() is explained in the section 3.4. Then it uses the set_time(long []) method to set the system clock time.

[[File:level2clientcode2.jpg]]

== Daemon client and server==

The level2 client and server runs as a daemon process to give them the privilege to run continuously in the background. As both the client and server in level2 does not need any user interference and need to run all the time to update the system clock of the running computer from the level1 server.

As it is mentioned above the client process for level2 run continuously and it updates its system clock every 1 minute (polls time from the level1 server every 60 seconds). Server daemon process is built in the same manner but unlike the client process it does not have a waiting time.

[[File:codefordaemon.jpg]]

== Marzullo’s algorithm and function marzullo(long []) ==

Marzullo’s algorithm, invented by Keith Marzullo, is an agreement algorithm used to select sources for estimating accurate time from a number of noisy time sources. A modified version of his algorithm, intersection algorithm, is use in NTP. We used the simplified version of this algorithm to choose the best possible time from our time sources (level1 servers).

Our function long* marzullo (long []) takes an array of four times from four different level1 servers and return a pointer to an array with the best seconds and milliseconds. First it takes the given array of times and makes two entries for each time as ±2 milliseconds. For each entry it also adds a type of -1 for -2millisecons and +1 for +2 milliseconds. As a result for 4 time entry it produces an array of 8 time entries with type as ±1. For our LLTP we took ±2milliseconds as our error range. After adding the type the entry will look like <entry – 2, -1> for the beginning of the range and <entry+2, +1> for the end of the range [entry±2].

[[File:IntersectionMarzullasalg.jpg]]

The description of the algorithm uses the following variables: best (largest number of overlapping intervals found), cnt (current number of overlapping intervals), best_start and

best_end (the beginning and end of best interval found so far). We used a two dimensional array to keep the second, millisecond and type of the range. The structure of the array is as follows:

time_entry [i][0] = second part of the time
entry time_entry [i][1] = millisecond part of the time
entry time_entry [i][2] = type of the range (i.e. ±1)

Here ‘Time_entry is the name of the array and ‘i’ is used as an index to traverse the array. After creating the array it follows the following step to determine the accurate time. Then it sends back the best chosen time to the level2 client program.

[[File:section3last.jpg]]

= List of files=

Level1 server:

DieWithError.c
get-time.c
L1Server.c

Level2 server:

daemon_server.c
DieWithError.c
get-time.c
L2Server.c

Level2 client:

daemon_client.c
DieWithError.c
get-time.c
L2Client.c
marzullo.c
print-time.c
selection_sort.c
set-time.c

Level3 client:

DieWithError.c
get-time.c
print-time.c
set-time.c
UDPTimeClient.c

=Discussion/Output=

The most intresting part of testing this was to see how distrubted architect of NTP client-server works and how could it be improved. I learnt alot by setting up and running them and seeing the results. It was fun improving little bit on exsisting implementation of NTP.

Output form different levels of clients and servers are given in this section. Output from level1 server:

[[File:output1.jpg]]

[[File:output2.jpg]]

[[File:output3.jpg]]

[[File:output4.jpg]]

=Conclusion=

Overall, there has been lots of work going on NTP , due to high demand of distributed clinet-server architect based application. This was carried on a very small linux based lab , in future i guess there will be need to test on large industry based scale , where many systems can be connected and than test to see how the delay from clients to server causes disturbance in algorithm. There will be issues for sure since as the information travels it takes time and that delayed time will need to be adjusted more and will be focus of future work.

=References=

*Ntp home page: http://ntp.org Acessed on 13th Feb 2011

*Wikipedia for network time protocol: http://en.wikipedia.org/wiki/Network_Time_Protocol Accessed on 13th Feb 2011

*Wikipedia for Marzullo’s algorithm: http://en.wikipedia.org/wiki/Marzullo's_algorithm Accessed on 13th Feb 2011

DistOS-2011W NTP

2011-02-22T18:38:05Z

Wahmed2: /* Development Details */

=Introduction:Context/Background=

Network Time Protocol (NTP) synchronizes clocks of hosts and routers. According to NIST estimates there are 10-20 million NTP servers and clients deployed in the internet and its tributaries all over the world. NTP software has been ported to almost every workstation and server platform available today – from PCs to Cray’s – UNIX, Windows, VMS and embedded systems, even home routers and battery backup systems.

NTP provides nominal accuracies of low tens of milliseconds on WANs, sub milliseconds on LANs and sub microseconds using a precision time source such as cesium oscillator or GPS receiver. Our time protocol was designed over the LAN and our goal was to achieve sub milliseconds of accuracy.

== Importance==

Precision of time is very important in the computer world. Following are some example why it is so important to maintain a precise and accurate time.

Network monitoring, measurement and control, Distributed database transaction journaling and logging, Secure document timestamps with cryptographic certification, Radio and TV programming lunch and monitoring, Stock market buy and sell orders, Distributed network gaming and training, etc.

== Definition of the problem ==

My goal was to acquire time accuracy in the Linux lab assuming that all the computers do not have internet connection. To time synchronize all the computers I had to build our own hierarchy of NTP servers and clients.

== Summary of the result ==

An efficient hierarchy of NTP server and client has been implemented. Precision of
Sub milliseconds were achieved.

==Outline of the report==
Detailed background information and the structure of our time protocol are provided in Section
2. The development setup and code explanation is reviewed in Section 3. Details of which server/client contains which files are given in section 4. My contributions in this implementation are described in Section 5.

= Detailed context/background information=

In this section the background information of Network Time protocol and the hierarchical model and the comparison with the original NTP designed by Dave Mills is reviewed. For unambiguity I will call the protocol as Linux lab time protocol (LLTP) and the original Network time protocol as NTP.

== NTP timestamps==

The 64-bit timestamps used by NTP consist of a 32-bit second part and a 32-bit fractional second part. I followed the same time structure for our LLTP which gives us a time scale of 232 seconds (136 years) and a theoretical resolution of 2-32 seconds (233 picoseconds).

== Structure==

NTP uses a hierarchical, semi-layered system of levels of clock sources; each level of this hierarchy is termed a stratum and assigned a layer number starting with 0 at the top. The stratum level defines its distance from the reference clock and exists to prevent cyclical dependencies in hierarchy.

My target is to develop a time synchronizing software for a small office or lab with a large number of computer where a very few computer has internet connectivity but all the computers are connected together in a local network. As a result not all the computer is able to connect to internet for time accuracy. LLTP uses similar hierarchical, semi-layered sources as NTP. Each layer of this hierarchy is termed as level and assigned a layer number starting with 1.

== Level 1 server==

Computers with internet connectivity are the level 1 server for our LLTP

Level 1 server

synchronizes its system clock with the NTP servers on the internet. In this hierarchy, only a fixed (5) number of level 1 servers are used. Unlike NTP we don’t have the privilege to use a 10 of thousands of level 1 and level 2 servers, and unlike NTP our LLTP has only 3 level structures.

Picture below shows the connectivity and the hierarchy of LLTP

[[File:LTTPHireachy.jpg]]

== Level 2 server and client==

Level 2 servers run two different programs. A client program to synchronize its system clock with the level1 servers and a server program that waits for any level 3 client for time request. Upon the request it sends its synchronized system time to the client. Level 2 servers also synchronize its system clock with the other level2 servers.

Each level2 server is connected to 4 level1 servers. It polls time from each level1 server and chooses the best time and synchronizes its system clock. How the best time is achieved is described in the section 3 below with code explanation. There are maximum 5 level2 servers with our LLTP model. Each level2 server is connected to 4 different level1 servers. None of the level2 server has the same level1 server set.

== Level 3 client==

Bottom of the LLTP hierarchy is level3 client. It only runs a client program that polls time from the level2 servers. It can only connect/request time from one of our level2 servers at a time. It also requires the knowledge of the IP address of the level2 server. For the load management purpose level 3 clients are not able to connect to a Level1 servers.

= Development Details=

== Server==

As it is mentioned in section-2, there are two servers in our LLTP model. One is level 1 and the other is level 2. Both servers wait for a client with time requests and upon receiving one send the time string. Code structure for both the servers is almost similar with minor technical differences which are explained later in this section. Both servers use a infinite for loop for waiting for the
client. They use a fixed but different port number for the client connection.

[[File:clientntp.jpg]]

Time requests sent to Level 1 server have to contain a message no more than 1 character and it has to be an integer which is sent by the level2 client to distinguish the level1 servers from one another. Level1 server will take that message received from the client and add that at the end of the time string sent to the client. If the level1 server receives a time request message with greater size of 1than it will not return any time value. For the level2 server the maximum message size is
128 characters and it do not have to be an integer. Level2 server discards the message sent from the level3 client. A unique feature of level2 server is that a level3 client time request with “exit” message can shut down the level2 server. This feature is not implemented in level1 servers.

[[File:level1servercode.jpg]]

[[File:level2servercode.jpg]]

Both servers send time as a string which handles the problem with big and little Endean.

==3.2 Client==

There are also two clients for our LLTP, level2 client and level3 client. Level3 client is very straight forward where the level3 is a bit more complicated.

Level3 client sends a time request to the level2 server. It cannot send time request to a level1 server as the port number for both server are different and the port numbers are hard coded for both the client (leve3 client/level2 server uses port 123 where level2 client/level1 server uses port 4003). Level2 server also has a restriction over the message sent by the client which is explained above in the server section. Upon receiving the time string back from the server which contains the seconds and milliseconds part together as a string separated by a comma (‘,’), it separates this two parts and checks for the latency and calls function named set_time(long[] ) which sets the system clock time with the new time.

[[File:level3clientcode.jpg]]

Level2 client dose almost the same things but in a broader and sophisticated way. It polls time from four different levle1 time server at the same time. When it sends the time request, it sends a unique message as the server ID to identify the time received from different level1 server which helps to fix the latency later.

[[File:level2clientcode.jpg]]

Level2 client also uses a time out of 1 second. If it dose not receive the time string back from the server then it will ignore the time from that server and choose the best time from the ones it received from the other servers. Received time strings from the different servers also have the server id added at the end of the time string. Using that information level2 client saves the received time in order and fixes the latency accordingly. It keeps all the latency fixed time in an array with the invalid time being -1. Then it calls a different function called marzullo(long []), which takes the array of latency fixed times and returns the best time depending on the Marzullo’s algorithm. Function marzullo() is explained in the section 3.4. Then it uses the set_time(long []) method to set the system clock time.

[[File:level2clientcode2.jpg]]

== Daemon client and server==

The level2 client and server runs as a daemon process to give them the privilege to run continuously in the background. As both the client and server in level2 does not need any user interference and need to run all the time to update the system clock of the running computer from the level1 server.

As it is mentioned above the client process for level2 run continuously and it updates its system clock every 1 minute (polls time from the level1 server every 60 seconds). Server daemon process is built in the same manner but unlike the client process it does not have a waiting time.

[[File:codefordaemon.jpg]]

== Marzullo’s algorithm and function marzullo(long []) ==

Marzullo’s algorithm, invented by Keith Marzullo, is an agreement algorithm used to select sources for estimating accurate time from a number of noisy time sources. A modified version of his algorithm, intersection algorithm, is use in NTP. We used the simplified version of this algorithm to choose the best possible time from our time sources (level1 servers).

Our function long* marzullo (long []) takes an array of four times from four different level1 servers and return a pointer to an array with the best seconds and milliseconds. First it takes the given array of times and makes two entries for each time as ±2 milliseconds. For each entry it also adds a type of -1 for -2millisecons and +1 for +2 milliseconds. As a result for 4 time entry it produces an array of 8 time entries with type as ±1. For our LLTP we took ±2milliseconds as our error range. After adding the type the entry will look like <entry – 2, -1> for the beginning of the range and <entry+2, +1> for the end of the range [entry±2].

[[File:IntersectionMarzullasalg.jpg]]

The description of the algorithm uses the following variables: best (largest number of overlapping intervals found), cnt (current number of overlapping intervals), best_start and

best_end (the beginning and end of best interval found so far). We used a two dimensional array to keep the second, millisecond and type of the range. The structure of the array is as follows:

time_entry [i][0] = second part of the time
entry time_entry [i][1] = millisecond part of the time
entry time_entry [i][2] = type of the range (i.e. ±1)

Here ‘Time_entry is the name of the array and ‘i’ is used as an index to traverse the array. After creating the array it follows the following step to determine the accurate time. Then it sends back the best chosen time to the level2 client program.

[[File:section3last.jpg]]

= List of files=

Level1 server:

DieWithError.c
get-time.c
L1Server.c

Level2 server:

daemon_server.c
DieWithError.c
get-time.c
L2Server.c

Level2 client:

daemon_client.c
DieWithError.c
get-time.c
L2Client.c
marzullo.c
print-time.c
selection_sort.c
set-time.c

Level3 client:

DieWithError.c
get-time.c
print-time.c
set-time.c
UDPTimeClient.c

=Discussion/Output=

The most intresting part of testing this was to see how distrubted architect of NTP client-server works and how could it be improved. I learnt alot by setting up and running them and seeing the results. It was fun improving little bit on exsisting implementation of NTP.

Output form different levels of clients and servers are given in this section. Output from level1 server:

[[File:output1.jpg]]

[[File:output2.jpg]]

[[File:output3.jpg]]

[[File:output4.jpg]]

=Conclusion=

Overall, there has been lots of work going on NTP , due to high demand of distributed clinet-server architect based application. This was carried on a very small linux based lab , in future i guess there will be need to test on large industry based scale , where many systems can be connected and than test to see how the delay from clients to server causes disturbance in algorithm. There will be issues for sure since as the information travels it takes time and that delayed time will need to be adjusted more and will be focus of future work.

=References=

*Ntp home page: http://ntp.org Acessed on 13th Feb 2011

*Wikipedia for network time protocol: http://en.wikipedia.org/wiki/Network_Time_Protocol Accessed on 13th Feb 2011

*Wikipedia for Marzullo’s algorithm: http://en.wikipedia.org/wiki/Marzullo's_algorithm Accessed on 13th Feb 2011

DistOS-2011W NTP

2011-02-22T18:35:25Z

Wahmed2: /* 2.5 Level 3 client */

=Introduction:Context/Background=

Network Time Protocol (NTP) synchronizes clocks of hosts and routers. According to NIST estimates there are 10-20 million NTP servers and clients deployed in the internet and its tributaries all over the world. NTP software has been ported to almost every workstation and server platform available today – from PCs to Cray’s – UNIX, Windows, VMS and embedded systems, even home routers and battery backup systems.

NTP provides nominal accuracies of low tens of milliseconds on WANs, sub milliseconds on LANs and sub microseconds using a precision time source such as cesium oscillator or GPS receiver. Our time protocol was designed over the LAN and our goal was to achieve sub milliseconds of accuracy.

== Importance==

Precision of time is very important in the computer world. Following are some example why it is so important to maintain a precise and accurate time.

Network monitoring, measurement and control, Distributed database transaction journaling and logging, Secure document timestamps with cryptographic certification, Radio and TV programming lunch and monitoring, Stock market buy and sell orders, Distributed network gaming and training, etc.

== Definition of the problem ==

My goal was to acquire time accuracy in the Linux lab assuming that all the computers do not have internet connection. To time synchronize all the computers I had to build our own hierarchy of NTP servers and clients.

== Summary of the result ==

An efficient hierarchy of NTP server and client has been implemented. Precision of
Sub milliseconds were achieved.

==Outline of the report==
Detailed background information and the structure of our time protocol are provided in Section
2. The development setup and code explanation is reviewed in Section 3. Details of which server/client contains which files are given in section 4. My contributions in this implementation are described in Section 5.

= Detailed context/background information=

In this section the background information of Network Time protocol and the hierarchical model and the comparison with the original NTP designed by Dave Mills is reviewed. For unambiguity I will call the protocol as Linux lab time protocol (LLTP) and the original Network time protocol as NTP.

== NTP timestamps==

The 64-bit timestamps used by NTP consist of a 32-bit second part and a 32-bit fractional second part. I followed the same time structure for our LLTP which gives us a time scale of 232 seconds (136 years) and a theoretical resolution of 2-32 seconds (233 picoseconds).

== Structure==

NTP uses a hierarchical, semi-layered system of levels of clock sources; each level of this hierarchy is termed a stratum and assigned a layer number starting with 0 at the top. The stratum level defines its distance from the reference clock and exists to prevent cyclical dependencies in hierarchy.

My target is to develop a time synchronizing software for a small office or lab with a large number of computer where a very few computer has internet connectivity but all the computers are connected together in a local network. As a result not all the computer is able to connect to internet for time accuracy. LLTP uses similar hierarchical, semi-layered sources as NTP. Each layer of this hierarchy is termed as level and assigned a layer number starting with 1.

== Level 1 server==

Computers with internet connectivity are the level 1 server for our LLTP

Level 1 server

synchronizes its system clock with the NTP servers on the internet. In this hierarchy, only a fixed (5) number of level 1 servers are used. Unlike NTP we don’t have the privilege to use a 10 of thousands of level 1 and level 2 servers, and unlike NTP our LLTP has only 3 level structures.

Picture below shows the connectivity and the hierarchy of LLTP

[[File:LTTPHireachy.jpg]]

== Level 2 server and client==

Level 2 servers run two different programs. A client program to synchronize its system clock with the level1 servers and a server program that waits for any level 3 client for time request. Upon the request it sends its synchronized system time to the client. Level 2 servers also synchronize its system clock with the other level2 servers.

Each level2 server is connected to 4 level1 servers. It polls time from each level1 server and chooses the best time and synchronizes its system clock. How the best time is achieved is described in the section 3 below with code explanation. There are maximum 5 level2 servers with our LLTP model. Each level2 server is connected to 4 different level1 servers. None of the level2 server has the same level1 server set.

== Level 3 client==

Bottom of the LLTP hierarchy is level3 client. It only runs a client program that polls time from the level2 servers. It can only connect/request time from one of our level2 servers at a time. It also requires the knowledge of the IP address of the level2 server. For the load management purpose level 3 clients are not able to connect to a Level1 servers.

= Development Details=

*3.1 Server

As it is mentioned in section-2, there are two servers in our LLTP model. One is level 1 and the other is level 2. Both servers wait for a client with time requests and upon receiving one send the time string. Code structure for both the servers is almost similar with minor technical differences which are explained later in this section. Both servers use a infinite for loop for waiting for the
client. They use a fixed but different port number for the client connection.

[[File:clientntp.jpg]]

Time requests sent to Level 1 server have to contain a message no more than 1 character and it has to be an integer which is sent by the level2 client to distinguish the level1 servers from one another. Level1 server will take that message received from the client and add that at the end of the time string sent to the client. If the level1 server receives a time request message with greater size of 1than it will not return any time value. For the level2 server the maximum message size is
128 characters and it do not have to be an integer. Level2 server discards the message sent from the level3 client. A unique feature of level2 server is that a level3 client time request with “exit” message can shut down the level2 server. This feature is not implemented in level1 servers.

[[File:level1servercode.jpg]]

[[File:level2servercode.jpg]]

Both servers send time as a string which handles the problem with big and little Endean.

*3.2 Client

There are also two clients for our LLTP, level2 client and level3 client. Level3 client is very straight forward where the level3 is a bit more complicated.

Level3 client sends a time request to the level2 server. It cannot send time request to a level1 server as the port number for both server are different and the port numbers are hard coded for both the client (leve3 client/level2 server uses port 123 where level2 client/level1 server uses port 4003). Level2 server also has a restriction over the message sent by the client which is explained above in the server section. Upon receiving the time string back from the server which contains the seconds and milliseconds part together as a string separated by a comma (‘,’), it separates this two parts and checks for the latency and calls function named set_time(long[] ) which sets the system clock time with the new time.

[[File:level3clientcode.jpg]]

Level2 client dose almost the same things but in a broader and sophisticated way. It polls time from four different levle1 time server at the same time. When it sends the time request, it sends a unique message as the server ID to identify the time received from different level1 server which helps to fix the latency later.

[[File:level2clientcode.jpg]]

Level2 client also uses a time out of 1 second. If it dose not receive the time string back from the server then it will ignore the time from that server and choose the best time from the ones it received from the other servers. Received time strings from the different servers also have the server id added at the end of the time string. Using that information level2 client saves the received time in order and fixes the latency accordingly. It keeps all the latency fixed time in an array with the invalid time being -1. Then it calls a different function called marzullo(long []), which takes the array of latency fixed times and returns the best time depending on the Marzullo’s algorithm. Function marzullo() is explained in the section 3.4. Then it uses the set_time(long []) method to set the system clock time.

[[File:level2clientcode2.jpg]]

*3.3 Daemon client and server

The level2 client and server runs as a daemon process to give them the privilege to run continuously in the background. As both the client and server in level2 does not need any user interference and need to run all the time to update the system clock of the running computer from the level1 server.

As it is mentioned above the client process for level2 run continuously and it updates its system clock every 1 minute (polls time from the level1 server every 60 seconds). Server daemon process is built in the same manner but unlike the client process it does not have a waiting time.

[[File:codefordaemon.jpg]]

*3.4 Marzullo’s algorithm and function marzullo(long [])

Marzullo’s algorithm, invented by Keith Marzullo, is an agreement algorithm used to select sources for estimating accurate time from a number of noisy time sources. A modified version of his algorithm, intersection algorithm, is use in NTP. We used the simplified version of this algorithm to choose the best possible time from our time sources (level1 servers).

Our function long* marzullo (long []) takes an array of four times from four different level1 servers and return a pointer to an array with the best seconds and milliseconds. First it takes the given array of times and makes two entries for each time as ±2 milliseconds. For each entry it also adds a type of -1 for -2millisecons and +1 for +2 milliseconds. As a result for 4 time entry it produces an array of 8 time entries with type as ±1. For our LLTP we took ±2milliseconds as our error range. After adding the type the entry will look like <entry – 2, -1> for the beginning of the range and <entry+2, +1> for the end of the range [entry±2].

[[File:IntersectionMarzullasalg.jpg]]

The description of the algorithm uses the following variables: best (largest number of overlapping intervals found), cnt (current number of overlapping intervals), best_start and

best_end (the beginning and end of best interval found so far). We used a two dimensional array to keep the second, millisecond and type of the range. The structure of the array is as follows:

time_entry [i][0] = second part of the time
entry time_entry [i][1] = millisecond part of the time
entry time_entry [i][2] = type of the range (i.e. ±1)

Here ‘Time_entry is the name of the array and ‘i’ is used as an index to traverse the array. After creating the array it follows the following step to determine the accurate time. Then it sends back the best chosen time to the level2 client program.

[[File:section3last.jpg]]

= List of files=

Level1 server:

DieWithError.c
get-time.c
L1Server.c

Level2 server:

daemon_server.c
DieWithError.c
get-time.c
L2Server.c

Level2 client:

daemon_client.c
DieWithError.c
get-time.c
L2Client.c
marzullo.c
print-time.c
selection_sort.c
set-time.c

Level3 client:

DieWithError.c
get-time.c
print-time.c
set-time.c
UDPTimeClient.c

=Discussion/Output=

The most intresting part of testing this was to see how distrubted architect of NTP client-server works and how could it be improved. I learnt alot by setting up and running them and seeing the results. It was fun improving little bit on exsisting implementation of NTP.

Output form different levels of clients and servers are given in this section. Output from level1 server:

[[File:output1.jpg]]

[[File:output2.jpg]]

[[File:output3.jpg]]

[[File:output4.jpg]]

=Conclusion=

Overall, there has been lots of work going on NTP , due to high demand of distributed clinet-server architect based application. This was carried on a very small linux based lab , in future i guess there will be need to test on large industry based scale , where many systems can be connected and than test to see how the delay from clients to server causes disturbance in algorithm. There will be issues for sure since as the information travels it takes time and that delayed time will need to be adjusted more and will be focus of future work.

=References=

*Ntp home page: http://ntp.org Acessed on 13th Feb 2011

*Wikipedia for network time protocol: http://en.wikipedia.org/wiki/Network_Time_Protocol Accessed on 13th Feb 2011

*Wikipedia for Marzullo’s algorithm: http://en.wikipedia.org/wiki/Marzullo's_algorithm Accessed on 13th Feb 2011

DistOS-2011W NTP

2011-02-22T18:35:17Z

Wahmed2: /* 2.4 Level 2 server and client */

=Introduction:Context/Background=

Network Time Protocol (NTP) synchronizes clocks of hosts and routers. According to NIST estimates there are 10-20 million NTP servers and clients deployed in the internet and its tributaries all over the world. NTP software has been ported to almost every workstation and server platform available today – from PCs to Cray’s – UNIX, Windows, VMS and embedded systems, even home routers and battery backup systems.

NTP provides nominal accuracies of low tens of milliseconds on WANs, sub milliseconds on LANs and sub microseconds using a precision time source such as cesium oscillator or GPS receiver. Our time protocol was designed over the LAN and our goal was to achieve sub milliseconds of accuracy.

== Importance==

Precision of time is very important in the computer world. Following are some example why it is so important to maintain a precise and accurate time.

Network monitoring, measurement and control, Distributed database transaction journaling and logging, Secure document timestamps with cryptographic certification, Radio and TV programming lunch and monitoring, Stock market buy and sell orders, Distributed network gaming and training, etc.

== Definition of the problem ==

My goal was to acquire time accuracy in the Linux lab assuming that all the computers do not have internet connection. To time synchronize all the computers I had to build our own hierarchy of NTP servers and clients.

== Summary of the result ==

An efficient hierarchy of NTP server and client has been implemented. Precision of
Sub milliseconds were achieved.

==Outline of the report==
Detailed background information and the structure of our time protocol are provided in Section
2. The development setup and code explanation is reviewed in Section 3. Details of which server/client contains which files are given in section 4. My contributions in this implementation are described in Section 5.

= Detailed context/background information=

In this section the background information of Network Time protocol and the hierarchical model and the comparison with the original NTP designed by Dave Mills is reviewed. For unambiguity I will call the protocol as Linux lab time protocol (LLTP) and the original Network time protocol as NTP.

== NTP timestamps==

The 64-bit timestamps used by NTP consist of a 32-bit second part and a 32-bit fractional second part. I followed the same time structure for our LLTP which gives us a time scale of 232 seconds (136 years) and a theoretical resolution of 2-32 seconds (233 picoseconds).

== Structure==

NTP uses a hierarchical, semi-layered system of levels of clock sources; each level of this hierarchy is termed a stratum and assigned a layer number starting with 0 at the top. The stratum level defines its distance from the reference clock and exists to prevent cyclical dependencies in hierarchy.

My target is to develop a time synchronizing software for a small office or lab with a large number of computer where a very few computer has internet connectivity but all the computers are connected together in a local network. As a result not all the computer is able to connect to internet for time accuracy. LLTP uses similar hierarchical, semi-layered sources as NTP. Each layer of this hierarchy is termed as level and assigned a layer number starting with 1.

== Level 1 server==

Computers with internet connectivity are the level 1 server for our LLTP

Level 1 server

synchronizes its system clock with the NTP servers on the internet. In this hierarchy, only a fixed (5) number of level 1 servers are used. Unlike NTP we don’t have the privilege to use a 10 of thousands of level 1 and level 2 servers, and unlike NTP our LLTP has only 3 level structures.

Picture below shows the connectivity and the hierarchy of LLTP

[[File:LTTPHireachy.jpg]]

== Level 2 server and client==

Level 2 servers run two different programs. A client program to synchronize its system clock with the level1 servers and a server program that waits for any level 3 client for time request. Upon the request it sends its synchronized system time to the client. Level 2 servers also synchronize its system clock with the other level2 servers.

Each level2 server is connected to 4 level1 servers. It polls time from each level1 server and chooses the best time and synchronizes its system clock. How the best time is achieved is described in the section 3 below with code explanation. There are maximum 5 level2 servers with our LLTP model. Each level2 server is connected to 4 different level1 servers. None of the level2 server has the same level1 server set.

==2.5 Level 3 client==

Bottom of the LLTP hierarchy is level3 client. It only runs a client program that polls time from the level2 servers. It can only connect/request time from one of our level2 servers at a time. It also requires the knowledge of the IP address of the level2 server. For the load management purpose level 3 clients are not able to connect to a Level1 servers.

= Development Details=

*3.1 Server

As it is mentioned in section-2, there are two servers in our LLTP model. One is level 1 and the other is level 2. Both servers wait for a client with time requests and upon receiving one send the time string. Code structure for both the servers is almost similar with minor technical differences which are explained later in this section. Both servers use a infinite for loop for waiting for the
client. They use a fixed but different port number for the client connection.

[[File:clientntp.jpg]]

Time requests sent to Level 1 server have to contain a message no more than 1 character and it has to be an integer which is sent by the level2 client to distinguish the level1 servers from one another. Level1 server will take that message received from the client and add that at the end of the time string sent to the client. If the level1 server receives a time request message with greater size of 1than it will not return any time value. For the level2 server the maximum message size is
128 characters and it do not have to be an integer. Level2 server discards the message sent from the level3 client. A unique feature of level2 server is that a level3 client time request with “exit” message can shut down the level2 server. This feature is not implemented in level1 servers.

[[File:level1servercode.jpg]]

[[File:level2servercode.jpg]]

Both servers send time as a string which handles the problem with big and little Endean.

*3.2 Client

There are also two clients for our LLTP, level2 client and level3 client. Level3 client is very straight forward where the level3 is a bit more complicated.

Level3 client sends a time request to the level2 server. It cannot send time request to a level1 server as the port number for both server are different and the port numbers are hard coded for both the client (leve3 client/level2 server uses port 123 where level2 client/level1 server uses port 4003). Level2 server also has a restriction over the message sent by the client which is explained above in the server section. Upon receiving the time string back from the server which contains the seconds and milliseconds part together as a string separated by a comma (‘,’), it separates this two parts and checks for the latency and calls function named set_time(long[] ) which sets the system clock time with the new time.

[[File:level3clientcode.jpg]]

Level2 client dose almost the same things but in a broader and sophisticated way. It polls time from four different levle1 time server at the same time. When it sends the time request, it sends a unique message as the server ID to identify the time received from different level1 server which helps to fix the latency later.

[[File:level2clientcode.jpg]]

Level2 client also uses a time out of 1 second. If it dose not receive the time string back from the server then it will ignore the time from that server and choose the best time from the ones it received from the other servers. Received time strings from the different servers also have the server id added at the end of the time string. Using that information level2 client saves the received time in order and fixes the latency accordingly. It keeps all the latency fixed time in an array with the invalid time being -1. Then it calls a different function called marzullo(long []), which takes the array of latency fixed times and returns the best time depending on the Marzullo’s algorithm. Function marzullo() is explained in the section 3.4. Then it uses the set_time(long []) method to set the system clock time.

[[File:level2clientcode2.jpg]]

*3.3 Daemon client and server

The level2 client and server runs as a daemon process to give them the privilege to run continuously in the background. As both the client and server in level2 does not need any user interference and need to run all the time to update the system clock of the running computer from the level1 server.

As it is mentioned above the client process for level2 run continuously and it updates its system clock every 1 minute (polls time from the level1 server every 60 seconds). Server daemon process is built in the same manner but unlike the client process it does not have a waiting time.

[[File:codefordaemon.jpg]]

*3.4 Marzullo’s algorithm and function marzullo(long [])

Marzullo’s algorithm, invented by Keith Marzullo, is an agreement algorithm used to select sources for estimating accurate time from a number of noisy time sources. A modified version of his algorithm, intersection algorithm, is use in NTP. We used the simplified version of this algorithm to choose the best possible time from our time sources (level1 servers).

Our function long* marzullo (long []) takes an array of four times from four different level1 servers and return a pointer to an array with the best seconds and milliseconds. First it takes the given array of times and makes two entries for each time as ±2 milliseconds. For each entry it also adds a type of -1 for -2millisecons and +1 for +2 milliseconds. As a result for 4 time entry it produces an array of 8 time entries with type as ±1. For our LLTP we took ±2milliseconds as our error range. After adding the type the entry will look like <entry – 2, -1> for the beginning of the range and <entry+2, +1> for the end of the range [entry±2].

[[File:IntersectionMarzullasalg.jpg]]

The description of the algorithm uses the following variables: best (largest number of overlapping intervals found), cnt (current number of overlapping intervals), best_start and

best_end (the beginning and end of best interval found so far). We used a two dimensional array to keep the second, millisecond and type of the range. The structure of the array is as follows:

time_entry [i][0] = second part of the time
entry time_entry [i][1] = millisecond part of the time
entry time_entry [i][2] = type of the range (i.e. ±1)

Here ‘Time_entry is the name of the array and ‘i’ is used as an index to traverse the array. After creating the array it follows the following step to determine the accurate time. Then it sends back the best chosen time to the level2 client program.

[[File:section3last.jpg]]

= List of files=

Level1 server:

DieWithError.c
get-time.c
L1Server.c

Level2 server:

daemon_server.c
DieWithError.c
get-time.c
L2Server.c

Level2 client:

daemon_client.c
DieWithError.c
get-time.c
L2Client.c
marzullo.c
print-time.c
selection_sort.c
set-time.c

Level3 client:

DieWithError.c
get-time.c
print-time.c
set-time.c
UDPTimeClient.c

=Discussion/Output=

The most intresting part of testing this was to see how distrubted architect of NTP client-server works and how could it be improved. I learnt alot by setting up and running them and seeing the results. It was fun improving little bit on exsisting implementation of NTP.

Output form different levels of clients and servers are given in this section. Output from level1 server:

[[File:output1.jpg]]

[[File:output2.jpg]]

[[File:output3.jpg]]

[[File:output4.jpg]]

=Conclusion=

Overall, there has been lots of work going on NTP , due to high demand of distributed clinet-server architect based application. This was carried on a very small linux based lab , in future i guess there will be need to test on large industry based scale , where many systems can be connected and than test to see how the delay from clients to server causes disturbance in algorithm. There will be issues for sure since as the information travels it takes time and that delayed time will need to be adjusted more and will be focus of future work.

=References=

*Ntp home page: http://ntp.org Acessed on 13th Feb 2011

*Wikipedia for network time protocol: http://en.wikipedia.org/wiki/Network_Time_Protocol Accessed on 13th Feb 2011

*Wikipedia for Marzullo’s algorithm: http://en.wikipedia.org/wiki/Marzullo's_algorithm Accessed on 13th Feb 2011

DistOS-2011W NTP

2011-02-22T18:35:04Z

Wahmed2: /* 2.3 Level 1 server */

=Introduction:Context/Background=

Network Time Protocol (NTP) synchronizes clocks of hosts and routers. According to NIST estimates there are 10-20 million NTP servers and clients deployed in the internet and its tributaries all over the world. NTP software has been ported to almost every workstation and server platform available today – from PCs to Cray’s – UNIX, Windows, VMS and embedded systems, even home routers and battery backup systems.

NTP provides nominal accuracies of low tens of milliseconds on WANs, sub milliseconds on LANs and sub microseconds using a precision time source such as cesium oscillator or GPS receiver. Our time protocol was designed over the LAN and our goal was to achieve sub milliseconds of accuracy.

== Importance==

Precision of time is very important in the computer world. Following are some example why it is so important to maintain a precise and accurate time.

Network monitoring, measurement and control, Distributed database transaction journaling and logging, Secure document timestamps with cryptographic certification, Radio and TV programming lunch and monitoring, Stock market buy and sell orders, Distributed network gaming and training, etc.

== Definition of the problem ==

My goal was to acquire time accuracy in the Linux lab assuming that all the computers do not have internet connection. To time synchronize all the computers I had to build our own hierarchy of NTP servers and clients.

== Summary of the result ==

An efficient hierarchy of NTP server and client has been implemented. Precision of
Sub milliseconds were achieved.

==Outline of the report==
Detailed background information and the structure of our time protocol are provided in Section
2. The development setup and code explanation is reviewed in Section 3. Details of which server/client contains which files are given in section 4. My contributions in this implementation are described in Section 5.

= Detailed context/background information=

In this section the background information of Network Time protocol and the hierarchical model and the comparison with the original NTP designed by Dave Mills is reviewed. For unambiguity I will call the protocol as Linux lab time protocol (LLTP) and the original Network time protocol as NTP.

== NTP timestamps==

The 64-bit timestamps used by NTP consist of a 32-bit second part and a 32-bit fractional second part. I followed the same time structure for our LLTP which gives us a time scale of 232 seconds (136 years) and a theoretical resolution of 2-32 seconds (233 picoseconds).

== Structure==

NTP uses a hierarchical, semi-layered system of levels of clock sources; each level of this hierarchy is termed a stratum and assigned a layer number starting with 0 at the top. The stratum level defines its distance from the reference clock and exists to prevent cyclical dependencies in hierarchy.

My target is to develop a time synchronizing software for a small office or lab with a large number of computer where a very few computer has internet connectivity but all the computers are connected together in a local network. As a result not all the computer is able to connect to internet for time accuracy. LLTP uses similar hierarchical, semi-layered sources as NTP. Each layer of this hierarchy is termed as level and assigned a layer number starting with 1.

== Level 1 server==

Computers with internet connectivity are the level 1 server for our LLTP

Level 1 server

synchronizes its system clock with the NTP servers on the internet. In this hierarchy, only a fixed (5) number of level 1 servers are used. Unlike NTP we don’t have the privilege to use a 10 of thousands of level 1 and level 2 servers, and unlike NTP our LLTP has only 3 level structures.

Picture below shows the connectivity and the hierarchy of LLTP

[[File:LTTPHireachy.jpg]]

==2.4 Level 2 server and client==

Level 2 servers run two different programs. A client program to synchronize its system clock with the level1 servers and a server program that waits for any level 3 client for time request. Upon the request it sends its synchronized system time to the client. Level 2 servers also synchronize its system clock with the other level2 servers.

Each level2 server is connected to 4 level1 servers. It polls time from each level1 server and chooses the best time and synchronizes its system clock. How the best time is achieved is described in the section 3 below with code explanation. There are maximum 5 level2 servers with our LLTP model. Each level2 server is connected to 4 different level1 servers. None of the level2 server has the same level1 server set.

==2.5 Level 3 client==

Bottom of the LLTP hierarchy is level3 client. It only runs a client program that polls time from the level2 servers. It can only connect/request time from one of our level2 servers at a time. It also requires the knowledge of the IP address of the level2 server. For the load management purpose level 3 clients are not able to connect to a Level1 servers.

= Development Details=

*3.1 Server

As it is mentioned in section-2, there are two servers in our LLTP model. One is level 1 and the other is level 2. Both servers wait for a client with time requests and upon receiving one send the time string. Code structure for both the servers is almost similar with minor technical differences which are explained later in this section. Both servers use a infinite for loop for waiting for the
client. They use a fixed but different port number for the client connection.

[[File:clientntp.jpg]]

Time requests sent to Level 1 server have to contain a message no more than 1 character and it has to be an integer which is sent by the level2 client to distinguish the level1 servers from one another. Level1 server will take that message received from the client and add that at the end of the time string sent to the client. If the level1 server receives a time request message with greater size of 1than it will not return any time value. For the level2 server the maximum message size is
128 characters and it do not have to be an integer. Level2 server discards the message sent from the level3 client. A unique feature of level2 server is that a level3 client time request with “exit” message can shut down the level2 server. This feature is not implemented in level1 servers.

[[File:level1servercode.jpg]]

[[File:level2servercode.jpg]]

Both servers send time as a string which handles the problem with big and little Endean.

*3.2 Client

There are also two clients for our LLTP, level2 client and level3 client. Level3 client is very straight forward where the level3 is a bit more complicated.

Level3 client sends a time request to the level2 server. It cannot send time request to a level1 server as the port number for both server are different and the port numbers are hard coded for both the client (leve3 client/level2 server uses port 123 where level2 client/level1 server uses port 4003). Level2 server also has a restriction over the message sent by the client which is explained above in the server section. Upon receiving the time string back from the server which contains the seconds and milliseconds part together as a string separated by a comma (‘,’), it separates this two parts and checks for the latency and calls function named set_time(long[] ) which sets the system clock time with the new time.

[[File:level3clientcode.jpg]]

Level2 client dose almost the same things but in a broader and sophisticated way. It polls time from four different levle1 time server at the same time. When it sends the time request, it sends a unique message as the server ID to identify the time received from different level1 server which helps to fix the latency later.

[[File:level2clientcode.jpg]]

Level2 client also uses a time out of 1 second. If it dose not receive the time string back from the server then it will ignore the time from that server and choose the best time from the ones it received from the other servers. Received time strings from the different servers also have the server id added at the end of the time string. Using that information level2 client saves the received time in order and fixes the latency accordingly. It keeps all the latency fixed time in an array with the invalid time being -1. Then it calls a different function called marzullo(long []), which takes the array of latency fixed times and returns the best time depending on the Marzullo’s algorithm. Function marzullo() is explained in the section 3.4. Then it uses the set_time(long []) method to set the system clock time.

[[File:level2clientcode2.jpg]]

*3.3 Daemon client and server

The level2 client and server runs as a daemon process to give them the privilege to run continuously in the background. As both the client and server in level2 does not need any user interference and need to run all the time to update the system clock of the running computer from the level1 server.

As it is mentioned above the client process for level2 run continuously and it updates its system clock every 1 minute (polls time from the level1 server every 60 seconds). Server daemon process is built in the same manner but unlike the client process it does not have a waiting time.

[[File:codefordaemon.jpg]]

*3.4 Marzullo’s algorithm and function marzullo(long [])

Marzullo’s algorithm, invented by Keith Marzullo, is an agreement algorithm used to select sources for estimating accurate time from a number of noisy time sources. A modified version of his algorithm, intersection algorithm, is use in NTP. We used the simplified version of this algorithm to choose the best possible time from our time sources (level1 servers).

Our function long* marzullo (long []) takes an array of four times from four different level1 servers and return a pointer to an array with the best seconds and milliseconds. First it takes the given array of times and makes two entries for each time as ±2 milliseconds. For each entry it also adds a type of -1 for -2millisecons and +1 for +2 milliseconds. As a result for 4 time entry it produces an array of 8 time entries with type as ±1. For our LLTP we took ±2milliseconds as our error range. After adding the type the entry will look like <entry – 2, -1> for the beginning of the range and <entry+2, +1> for the end of the range [entry±2].

[[File:IntersectionMarzullasalg.jpg]]

The description of the algorithm uses the following variables: best (largest number of overlapping intervals found), cnt (current number of overlapping intervals), best_start and

best_end (the beginning and end of best interval found so far). We used a two dimensional array to keep the second, millisecond and type of the range. The structure of the array is as follows:

time_entry [i][0] = second part of the time
entry time_entry [i][1] = millisecond part of the time
entry time_entry [i][2] = type of the range (i.e. ±1)

Here ‘Time_entry is the name of the array and ‘i’ is used as an index to traverse the array. After creating the array it follows the following step to determine the accurate time. Then it sends back the best chosen time to the level2 client program.

[[File:section3last.jpg]]

= List of files=

Level1 server:

DieWithError.c
get-time.c
L1Server.c

Level2 server:

daemon_server.c
DieWithError.c
get-time.c
L2Server.c

Level2 client:

daemon_client.c
DieWithError.c
get-time.c
L2Client.c
marzullo.c
print-time.c
selection_sort.c
set-time.c

Level3 client:

DieWithError.c
get-time.c
print-time.c
set-time.c
UDPTimeClient.c

=Discussion/Output=

The most intresting part of testing this was to see how distrubted architect of NTP client-server works and how could it be improved. I learnt alot by setting up and running them and seeing the results. It was fun improving little bit on exsisting implementation of NTP.

Output form different levels of clients and servers are given in this section. Output from level1 server:

[[File:output1.jpg]]

[[File:output2.jpg]]

[[File:output3.jpg]]

[[File:output4.jpg]]

=Conclusion=

Overall, there has been lots of work going on NTP , due to high demand of distributed clinet-server architect based application. This was carried on a very small linux based lab , in future i guess there will be need to test on large industry based scale , where many systems can be connected and than test to see how the delay from clients to server causes disturbance in algorithm. There will be issues for sure since as the information travels it takes time and that delayed time will need to be adjusted more and will be focus of future work.

=References=

*Ntp home page: http://ntp.org Acessed on 13th Feb 2011

*Wikipedia for network time protocol: http://en.wikipedia.org/wiki/Network_Time_Protocol Accessed on 13th Feb 2011

*Wikipedia for Marzullo’s algorithm: http://en.wikipedia.org/wiki/Marzullo's_algorithm Accessed on 13th Feb 2011

DistOS-2011W NTP

2011-02-22T18:34:55Z

Wahmed2: /* 2.2 Structure */

=Introduction:Context/Background=

Network Time Protocol (NTP) synchronizes clocks of hosts and routers. According to NIST estimates there are 10-20 million NTP servers and clients deployed in the internet and its tributaries all over the world. NTP software has been ported to almost every workstation and server platform available today – from PCs to Cray’s – UNIX, Windows, VMS and embedded systems, even home routers and battery backup systems.

NTP provides nominal accuracies of low tens of milliseconds on WANs, sub milliseconds on LANs and sub microseconds using a precision time source such as cesium oscillator or GPS receiver. Our time protocol was designed over the LAN and our goal was to achieve sub milliseconds of accuracy.

== Importance==

Precision of time is very important in the computer world. Following are some example why it is so important to maintain a precise and accurate time.

Network monitoring, measurement and control, Distributed database transaction journaling and logging, Secure document timestamps with cryptographic certification, Radio and TV programming lunch and monitoring, Stock market buy and sell orders, Distributed network gaming and training, etc.

== Definition of the problem ==

My goal was to acquire time accuracy in the Linux lab assuming that all the computers do not have internet connection. To time synchronize all the computers I had to build our own hierarchy of NTP servers and clients.

== Summary of the result ==

An efficient hierarchy of NTP server and client has been implemented. Precision of
Sub milliseconds were achieved.

==Outline of the report==
Detailed background information and the structure of our time protocol are provided in Section
2. The development setup and code explanation is reviewed in Section 3. Details of which server/client contains which files are given in section 4. My contributions in this implementation are described in Section 5.

= Detailed context/background information=

In this section the background information of Network Time protocol and the hierarchical model and the comparison with the original NTP designed by Dave Mills is reviewed. For unambiguity I will call the protocol as Linux lab time protocol (LLTP) and the original Network time protocol as NTP.

== NTP timestamps==

The 64-bit timestamps used by NTP consist of a 32-bit second part and a 32-bit fractional second part. I followed the same time structure for our LLTP which gives us a time scale of 232 seconds (136 years) and a theoretical resolution of 2-32 seconds (233 picoseconds).

== Structure==

NTP uses a hierarchical, semi-layered system of levels of clock sources; each level of this hierarchy is termed a stratum and assigned a layer number starting with 0 at the top. The stratum level defines its distance from the reference clock and exists to prevent cyclical dependencies in hierarchy.

My target is to develop a time synchronizing software for a small office or lab with a large number of computer where a very few computer has internet connectivity but all the computers are connected together in a local network. As a result not all the computer is able to connect to internet for time accuracy. LLTP uses similar hierarchical, semi-layered sources as NTP. Each layer of this hierarchy is termed as level and assigned a layer number starting with 1.

==2.3 Level 1 server==

Computers with internet connectivity are the level 1 server for our LLTP

Level 1 server

synchronizes its system clock with the NTP servers on the internet. In this hierarchy, only a fixed (5) number of level 1 servers are used. Unlike NTP we don’t have the privilege to use a 10 of thousands of level 1 and level 2 servers, and unlike NTP our LLTP has only 3 level structures.

Picture below shows the connectivity and the hierarchy of LLTP

[[File:LTTPHireachy.jpg]]

==2.4 Level 2 server and client==

Level 2 servers run two different programs. A client program to synchronize its system clock with the level1 servers and a server program that waits for any level 3 client for time request. Upon the request it sends its synchronized system time to the client. Level 2 servers also synchronize its system clock with the other level2 servers.

Each level2 server is connected to 4 level1 servers. It polls time from each level1 server and chooses the best time and synchronizes its system clock. How the best time is achieved is described in the section 3 below with code explanation. There are maximum 5 level2 servers with our LLTP model. Each level2 server is connected to 4 different level1 servers. None of the level2 server has the same level1 server set.

==2.5 Level 3 client==

Bottom of the LLTP hierarchy is level3 client. It only runs a client program that polls time from the level2 servers. It can only connect/request time from one of our level2 servers at a time. It also requires the knowledge of the IP address of the level2 server. For the load management purpose level 3 clients are not able to connect to a Level1 servers.

= Development Details=

*3.1 Server

As it is mentioned in section-2, there are two servers in our LLTP model. One is level 1 and the other is level 2. Both servers wait for a client with time requests and upon receiving one send the time string. Code structure for both the servers is almost similar with minor technical differences which are explained later in this section. Both servers use a infinite for loop for waiting for the
client. They use a fixed but different port number for the client connection.

[[File:clientntp.jpg]]

Time requests sent to Level 1 server have to contain a message no more than 1 character and it has to be an integer which is sent by the level2 client to distinguish the level1 servers from one another. Level1 server will take that message received from the client and add that at the end of the time string sent to the client. If the level1 server receives a time request message with greater size of 1than it will not return any time value. For the level2 server the maximum message size is
128 characters and it do not have to be an integer. Level2 server discards the message sent from the level3 client. A unique feature of level2 server is that a level3 client time request with “exit” message can shut down the level2 server. This feature is not implemented in level1 servers.

[[File:level1servercode.jpg]]

[[File:level2servercode.jpg]]

Both servers send time as a string which handles the problem with big and little Endean.

*3.2 Client

There are also two clients for our LLTP, level2 client and level3 client. Level3 client is very straight forward where the level3 is a bit more complicated.

Level3 client sends a time request to the level2 server. It cannot send time request to a level1 server as the port number for both server are different and the port numbers are hard coded for both the client (leve3 client/level2 server uses port 123 where level2 client/level1 server uses port 4003). Level2 server also has a restriction over the message sent by the client which is explained above in the server section. Upon receiving the time string back from the server which contains the seconds and milliseconds part together as a string separated by a comma (‘,’), it separates this two parts and checks for the latency and calls function named set_time(long[] ) which sets the system clock time with the new time.

[[File:level3clientcode.jpg]]

Level2 client dose almost the same things but in a broader and sophisticated way. It polls time from four different levle1 time server at the same time. When it sends the time request, it sends a unique message as the server ID to identify the time received from different level1 server which helps to fix the latency later.

[[File:level2clientcode.jpg]]

Level2 client also uses a time out of 1 second. If it dose not receive the time string back from the server then it will ignore the time from that server and choose the best time from the ones it received from the other servers. Received time strings from the different servers also have the server id added at the end of the time string. Using that information level2 client saves the received time in order and fixes the latency accordingly. It keeps all the latency fixed time in an array with the invalid time being -1. Then it calls a different function called marzullo(long []), which takes the array of latency fixed times and returns the best time depending on the Marzullo’s algorithm. Function marzullo() is explained in the section 3.4. Then it uses the set_time(long []) method to set the system clock time.

[[File:level2clientcode2.jpg]]

*3.3 Daemon client and server

The level2 client and server runs as a daemon process to give them the privilege to run continuously in the background. As both the client and server in level2 does not need any user interference and need to run all the time to update the system clock of the running computer from the level1 server.

As it is mentioned above the client process for level2 run continuously and it updates its system clock every 1 minute (polls time from the level1 server every 60 seconds). Server daemon process is built in the same manner but unlike the client process it does not have a waiting time.

[[File:codefordaemon.jpg]]

*3.4 Marzullo’s algorithm and function marzullo(long [])

Marzullo’s algorithm, invented by Keith Marzullo, is an agreement algorithm used to select sources for estimating accurate time from a number of noisy time sources. A modified version of his algorithm, intersection algorithm, is use in NTP. We used the simplified version of this algorithm to choose the best possible time from our time sources (level1 servers).

Our function long* marzullo (long []) takes an array of four times from four different level1 servers and return a pointer to an array with the best seconds and milliseconds. First it takes the given array of times and makes two entries for each time as ±2 milliseconds. For each entry it also adds a type of -1 for -2millisecons and +1 for +2 milliseconds. As a result for 4 time entry it produces an array of 8 time entries with type as ±1. For our LLTP we took ±2milliseconds as our error range. After adding the type the entry will look like <entry – 2, -1> for the beginning of the range and <entry+2, +1> for the end of the range [entry±2].

[[File:IntersectionMarzullasalg.jpg]]

The description of the algorithm uses the following variables: best (largest number of overlapping intervals found), cnt (current number of overlapping intervals), best_start and

best_end (the beginning and end of best interval found so far). We used a two dimensional array to keep the second, millisecond and type of the range. The structure of the array is as follows:

time_entry [i][0] = second part of the time
entry time_entry [i][1] = millisecond part of the time
entry time_entry [i][2] = type of the range (i.e. ±1)

Here ‘Time_entry is the name of the array and ‘i’ is used as an index to traverse the array. After creating the array it follows the following step to determine the accurate time. Then it sends back the best chosen time to the level2 client program.

[[File:section3last.jpg]]

= List of files=

Level1 server:

DieWithError.c
get-time.c
L1Server.c

Level2 server:

daemon_server.c
DieWithError.c
get-time.c
L2Server.c

Level2 client:

daemon_client.c
DieWithError.c
get-time.c
L2Client.c
marzullo.c
print-time.c
selection_sort.c
set-time.c

Level3 client:

DieWithError.c
get-time.c
print-time.c
set-time.c
UDPTimeClient.c

=Discussion/Output=

The most intresting part of testing this was to see how distrubted architect of NTP client-server works and how could it be improved. I learnt alot by setting up and running them and seeing the results. It was fun improving little bit on exsisting implementation of NTP.

Output form different levels of clients and servers are given in this section. Output from level1 server:

[[File:output1.jpg]]

[[File:output2.jpg]]

[[File:output3.jpg]]

[[File:output4.jpg]]

=Conclusion=

Overall, there has been lots of work going on NTP , due to high demand of distributed clinet-server architect based application. This was carried on a very small linux based lab , in future i guess there will be need to test on large industry based scale , where many systems can be connected and than test to see how the delay from clients to server causes disturbance in algorithm. There will be issues for sure since as the information travels it takes time and that delayed time will need to be adjusted more and will be focus of future work.

=References=

*Ntp home page: http://ntp.org Acessed on 13th Feb 2011

*Wikipedia for network time protocol: http://en.wikipedia.org/wiki/Network_Time_Protocol Accessed on 13th Feb 2011

*Wikipedia for Marzullo’s algorithm: http://en.wikipedia.org/wiki/Marzullo's_algorithm Accessed on 13th Feb 2011

DistOS-2011W NTP

2011-02-22T18:34:45Z

Wahmed2: /* 2.1 NTP timestamps */

=Introduction:Context/Background=

Network Time Protocol (NTP) synchronizes clocks of hosts and routers. According to NIST estimates there are 10-20 million NTP servers and clients deployed in the internet and its tributaries all over the world. NTP software has been ported to almost every workstation and server platform available today – from PCs to Cray’s – UNIX, Windows, VMS and embedded systems, even home routers and battery backup systems.

NTP provides nominal accuracies of low tens of milliseconds on WANs, sub milliseconds on LANs and sub microseconds using a precision time source such as cesium oscillator or GPS receiver. Our time protocol was designed over the LAN and our goal was to achieve sub milliseconds of accuracy.

== Importance==

Precision of time is very important in the computer world. Following are some example why it is so important to maintain a precise and accurate time.

Network monitoring, measurement and control, Distributed database transaction journaling and logging, Secure document timestamps with cryptographic certification, Radio and TV programming lunch and monitoring, Stock market buy and sell orders, Distributed network gaming and training, etc.

== Definition of the problem ==

My goal was to acquire time accuracy in the Linux lab assuming that all the computers do not have internet connection. To time synchronize all the computers I had to build our own hierarchy of NTP servers and clients.

== Summary of the result ==

An efficient hierarchy of NTP server and client has been implemented. Precision of
Sub milliseconds were achieved.

==Outline of the report==
Detailed background information and the structure of our time protocol are provided in Section
2. The development setup and code explanation is reviewed in Section 3. Details of which server/client contains which files are given in section 4. My contributions in this implementation are described in Section 5.

= Detailed context/background information=

In this section the background information of Network Time protocol and the hierarchical model and the comparison with the original NTP designed by Dave Mills is reviewed. For unambiguity I will call the protocol as Linux lab time protocol (LLTP) and the original Network time protocol as NTP.

== NTP timestamps==

The 64-bit timestamps used by NTP consist of a 32-bit second part and a 32-bit fractional second part. I followed the same time structure for our LLTP which gives us a time scale of 232 seconds (136 years) and a theoretical resolution of 2-32 seconds (233 picoseconds).

==2.2 Structure==

NTP uses a hierarchical, semi-layered system of levels of clock sources; each level of this hierarchy is termed a stratum and assigned a layer number starting with 0 at the top. The stratum level defines its distance from the reference clock and exists to prevent cyclical dependencies in hierarchy.

My target is to develop a time synchronizing software for a small office or lab with a large number of computer where a very few computer has internet connectivity but all the computers are connected together in a local network. As a result not all the computer is able to connect to internet for time accuracy. LLTP uses similar hierarchical, semi-layered sources as NTP. Each layer of this hierarchy is termed as level and assigned a layer number starting with 1.

==2.3 Level 1 server==

Computers with internet connectivity are the level 1 server for our LLTP

Level 1 server

synchronizes its system clock with the NTP servers on the internet. In this hierarchy, only a fixed (5) number of level 1 servers are used. Unlike NTP we don’t have the privilege to use a 10 of thousands of level 1 and level 2 servers, and unlike NTP our LLTP has only 3 level structures.

Picture below shows the connectivity and the hierarchy of LLTP

[[File:LTTPHireachy.jpg]]

==2.4 Level 2 server and client==

Level 2 servers run two different programs. A client program to synchronize its system clock with the level1 servers and a server program that waits for any level 3 client for time request. Upon the request it sends its synchronized system time to the client. Level 2 servers also synchronize its system clock with the other level2 servers.

Each level2 server is connected to 4 level1 servers. It polls time from each level1 server and chooses the best time and synchronizes its system clock. How the best time is achieved is described in the section 3 below with code explanation. There are maximum 5 level2 servers with our LLTP model. Each level2 server is connected to 4 different level1 servers. None of the level2 server has the same level1 server set.

==2.5 Level 3 client==

Bottom of the LLTP hierarchy is level3 client. It only runs a client program that polls time from the level2 servers. It can only connect/request time from one of our level2 servers at a time. It also requires the knowledge of the IP address of the level2 server. For the load management purpose level 3 clients are not able to connect to a Level1 servers.

= Development Details=

*3.1 Server

As it is mentioned in section-2, there are two servers in our LLTP model. One is level 1 and the other is level 2. Both servers wait for a client with time requests and upon receiving one send the time string. Code structure for both the servers is almost similar with minor technical differences which are explained later in this section. Both servers use a infinite for loop for waiting for the
client. They use a fixed but different port number for the client connection.

[[File:clientntp.jpg]]

Time requests sent to Level 1 server have to contain a message no more than 1 character and it has to be an integer which is sent by the level2 client to distinguish the level1 servers from one another. Level1 server will take that message received from the client and add that at the end of the time string sent to the client. If the level1 server receives a time request message with greater size of 1than it will not return any time value. For the level2 server the maximum message size is
128 characters and it do not have to be an integer. Level2 server discards the message sent from the level3 client. A unique feature of level2 server is that a level3 client time request with “exit” message can shut down the level2 server. This feature is not implemented in level1 servers.

[[File:level1servercode.jpg]]

[[File:level2servercode.jpg]]

Both servers send time as a string which handles the problem with big and little Endean.

*3.2 Client

There are also two clients for our LLTP, level2 client and level3 client. Level3 client is very straight forward where the level3 is a bit more complicated.

Level3 client sends a time request to the level2 server. It cannot send time request to a level1 server as the port number for both server are different and the port numbers are hard coded for both the client (leve3 client/level2 server uses port 123 where level2 client/level1 server uses port 4003). Level2 server also has a restriction over the message sent by the client which is explained above in the server section. Upon receiving the time string back from the server which contains the seconds and milliseconds part together as a string separated by a comma (‘,’), it separates this two parts and checks for the latency and calls function named set_time(long[] ) which sets the system clock time with the new time.

[[File:level3clientcode.jpg]]

Level2 client dose almost the same things but in a broader and sophisticated way. It polls time from four different levle1 time server at the same time. When it sends the time request, it sends a unique message as the server ID to identify the time received from different level1 server which helps to fix the latency later.

[[File:level2clientcode.jpg]]

Level2 client also uses a time out of 1 second. If it dose not receive the time string back from the server then it will ignore the time from that server and choose the best time from the ones it received from the other servers. Received time strings from the different servers also have the server id added at the end of the time string. Using that information level2 client saves the received time in order and fixes the latency accordingly. It keeps all the latency fixed time in an array with the invalid time being -1. Then it calls a different function called marzullo(long []), which takes the array of latency fixed times and returns the best time depending on the Marzullo’s algorithm. Function marzullo() is explained in the section 3.4. Then it uses the set_time(long []) method to set the system clock time.

[[File:level2clientcode2.jpg]]

*3.3 Daemon client and server

The level2 client and server runs as a daemon process to give them the privilege to run continuously in the background. As both the client and server in level2 does not need any user interference and need to run all the time to update the system clock of the running computer from the level1 server.

As it is mentioned above the client process for level2 run continuously and it updates its system clock every 1 minute (polls time from the level1 server every 60 seconds). Server daemon process is built in the same manner but unlike the client process it does not have a waiting time.

[[File:codefordaemon.jpg]]

*3.4 Marzullo’s algorithm and function marzullo(long [])

Marzullo’s algorithm, invented by Keith Marzullo, is an agreement algorithm used to select sources for estimating accurate time from a number of noisy time sources. A modified version of his algorithm, intersection algorithm, is use in NTP. We used the simplified version of this algorithm to choose the best possible time from our time sources (level1 servers).

Our function long* marzullo (long []) takes an array of four times from four different level1 servers and return a pointer to an array with the best seconds and milliseconds. First it takes the given array of times and makes two entries for each time as ±2 milliseconds. For each entry it also adds a type of -1 for -2millisecons and +1 for +2 milliseconds. As a result for 4 time entry it produces an array of 8 time entries with type as ±1. For our LLTP we took ±2milliseconds as our error range. After adding the type the entry will look like <entry – 2, -1> for the beginning of the range and <entry+2, +1> for the end of the range [entry±2].

[[File:IntersectionMarzullasalg.jpg]]

The description of the algorithm uses the following variables: best (largest number of overlapping intervals found), cnt (current number of overlapping intervals), best_start and

best_end (the beginning and end of best interval found so far). We used a two dimensional array to keep the second, millisecond and type of the range. The structure of the array is as follows:

time_entry [i][0] = second part of the time
entry time_entry [i][1] = millisecond part of the time
entry time_entry [i][2] = type of the range (i.e. ±1)

Here ‘Time_entry is the name of the array and ‘i’ is used as an index to traverse the array. After creating the array it follows the following step to determine the accurate time. Then it sends back the best chosen time to the level2 client program.

[[File:section3last.jpg]]

= List of files=

Level1 server:

DieWithError.c
get-time.c
L1Server.c

Level2 server:

daemon_server.c
DieWithError.c
get-time.c
L2Server.c

Level2 client:

daemon_client.c
DieWithError.c
get-time.c
L2Client.c
marzullo.c
print-time.c
selection_sort.c
set-time.c

Level3 client:

DieWithError.c
get-time.c
print-time.c
set-time.c
UDPTimeClient.c

=Discussion/Output=

The most intresting part of testing this was to see how distrubted architect of NTP client-server works and how could it be improved. I learnt alot by setting up and running them and seeing the results. It was fun improving little bit on exsisting implementation of NTP.

Output form different levels of clients and servers are given in this section. Output from level1 server:

[[File:output1.jpg]]

[[File:output2.jpg]]

[[File:output3.jpg]]

[[File:output4.jpg]]

=Conclusion=

Overall, there has been lots of work going on NTP , due to high demand of distributed clinet-server architect based application. This was carried on a very small linux based lab , in future i guess there will be need to test on large industry based scale , where many systems can be connected and than test to see how the delay from clients to server causes disturbance in algorithm. There will be issues for sure since as the information travels it takes time and that delayed time will need to be adjusted more and will be focus of future work.

=References=

*Ntp home page: http://ntp.org Acessed on 13th Feb 2011

*Wikipedia for network time protocol: http://en.wikipedia.org/wiki/Network_Time_Protocol Accessed on 13th Feb 2011

*Wikipedia for Marzullo’s algorithm: http://en.wikipedia.org/wiki/Marzullo's_algorithm Accessed on 13th Feb 2011

DistOS-2011W NTP

2011-02-22T18:34:30Z

Wahmed2: /* Detailed context/background information */

=Introduction:Context/Background=

Network Time Protocol (NTP) synchronizes clocks of hosts and routers. According to NIST estimates there are 10-20 million NTP servers and clients deployed in the internet and its tributaries all over the world. NTP software has been ported to almost every workstation and server platform available today – from PCs to Cray’s – UNIX, Windows, VMS and embedded systems, even home routers and battery backup systems.

NTP provides nominal accuracies of low tens of milliseconds on WANs, sub milliseconds on LANs and sub microseconds using a precision time source such as cesium oscillator or GPS receiver. Our time protocol was designed over the LAN and our goal was to achieve sub milliseconds of accuracy.

== Importance==

Precision of time is very important in the computer world. Following are some example why it is so important to maintain a precise and accurate time.

Network monitoring, measurement and control, Distributed database transaction journaling and logging, Secure document timestamps with cryptographic certification, Radio and TV programming lunch and monitoring, Stock market buy and sell orders, Distributed network gaming and training, etc.

== Definition of the problem ==

My goal was to acquire time accuracy in the Linux lab assuming that all the computers do not have internet connection. To time synchronize all the computers I had to build our own hierarchy of NTP servers and clients.

== Summary of the result ==

An efficient hierarchy of NTP server and client has been implemented. Precision of
Sub milliseconds were achieved.

==Outline of the report==
Detailed background information and the structure of our time protocol are provided in Section
2. The development setup and code explanation is reviewed in Section 3. Details of which server/client contains which files are given in section 4. My contributions in this implementation are described in Section 5.

= Detailed context/background information=

In this section the background information of Network Time protocol and the hierarchical model and the comparison with the original NTP designed by Dave Mills is reviewed. For unambiguity I will call the protocol as Linux lab time protocol (LLTP) and the original Network time protocol as NTP.

==2.1 NTP timestamps==

The 64-bit timestamps used by NTP consist of a 32-bit second part and a 32-bit fractional second part. I followed the same time structure for our LLTP which gives us a time scale of 232 seconds (136 years) and a theoretical resolution of 2-32 seconds (233 picoseconds).

==2.2 Structure==

NTP uses a hierarchical, semi-layered system of levels of clock sources; each level of this hierarchy is termed a stratum and assigned a layer number starting with 0 at the top. The stratum level defines its distance from the reference clock and exists to prevent cyclical dependencies in hierarchy.

My target is to develop a time synchronizing software for a small office or lab with a large number of computer where a very few computer has internet connectivity but all the computers are connected together in a local network. As a result not all the computer is able to connect to internet for time accuracy. LLTP uses similar hierarchical, semi-layered sources as NTP. Each layer of this hierarchy is termed as level and assigned a layer number starting with 1.

==2.3 Level 1 server==

Computers with internet connectivity are the level 1 server for our LLTP

Level 1 server

synchronizes its system clock with the NTP servers on the internet. In this hierarchy, only a fixed (5) number of level 1 servers are used. Unlike NTP we don’t have the privilege to use a 10 of thousands of level 1 and level 2 servers, and unlike NTP our LLTP has only 3 level structures.

Picture below shows the connectivity and the hierarchy of LLTP

[[File:LTTPHireachy.jpg]]

==2.4 Level 2 server and client==

Level 2 servers run two different programs. A client program to synchronize its system clock with the level1 servers and a server program that waits for any level 3 client for time request. Upon the request it sends its synchronized system time to the client. Level 2 servers also synchronize its system clock with the other level2 servers.

Each level2 server is connected to 4 level1 servers. It polls time from each level1 server and chooses the best time and synchronizes its system clock. How the best time is achieved is described in the section 3 below with code explanation. There are maximum 5 level2 servers with our LLTP model. Each level2 server is connected to 4 different level1 servers. None of the level2 server has the same level1 server set.

==2.5 Level 3 client==

Bottom of the LLTP hierarchy is level3 client. It only runs a client program that polls time from the level2 servers. It can only connect/request time from one of our level2 servers at a time. It also requires the knowledge of the IP address of the level2 server. For the load management purpose level 3 clients are not able to connect to a Level1 servers.

= Development Details=

*3.1 Server

As it is mentioned in section-2, there are two servers in our LLTP model. One is level 1 and the other is level 2. Both servers wait for a client with time requests and upon receiving one send the time string. Code structure for both the servers is almost similar with minor technical differences which are explained later in this section. Both servers use a infinite for loop for waiting for the
client. They use a fixed but different port number for the client connection.

[[File:clientntp.jpg]]

Time requests sent to Level 1 server have to contain a message no more than 1 character and it has to be an integer which is sent by the level2 client to distinguish the level1 servers from one another. Level1 server will take that message received from the client and add that at the end of the time string sent to the client. If the level1 server receives a time request message with greater size of 1than it will not return any time value. For the level2 server the maximum message size is
128 characters and it do not have to be an integer. Level2 server discards the message sent from the level3 client. A unique feature of level2 server is that a level3 client time request with “exit” message can shut down the level2 server. This feature is not implemented in level1 servers.

[[File:level1servercode.jpg]]

[[File:level2servercode.jpg]]

Both servers send time as a string which handles the problem with big and little Endean.

*3.2 Client

There are also two clients for our LLTP, level2 client and level3 client. Level3 client is very straight forward where the level3 is a bit more complicated.

Level3 client sends a time request to the level2 server. It cannot send time request to a level1 server as the port number for both server are different and the port numbers are hard coded for both the client (leve3 client/level2 server uses port 123 where level2 client/level1 server uses port 4003). Level2 server also has a restriction over the message sent by the client which is explained above in the server section. Upon receiving the time string back from the server which contains the seconds and milliseconds part together as a string separated by a comma (‘,’), it separates this two parts and checks for the latency and calls function named set_time(long[] ) which sets the system clock time with the new time.

[[File:level3clientcode.jpg]]

Level2 client dose almost the same things but in a broader and sophisticated way. It polls time from four different levle1 time server at the same time. When it sends the time request, it sends a unique message as the server ID to identify the time received from different level1 server which helps to fix the latency later.

[[File:level2clientcode.jpg]]

Level2 client also uses a time out of 1 second. If it dose not receive the time string back from the server then it will ignore the time from that server and choose the best time from the ones it received from the other servers. Received time strings from the different servers also have the server id added at the end of the time string. Using that information level2 client saves the received time in order and fixes the latency accordingly. It keeps all the latency fixed time in an array with the invalid time being -1. Then it calls a different function called marzullo(long []), which takes the array of latency fixed times and returns the best time depending on the Marzullo’s algorithm. Function marzullo() is explained in the section 3.4. Then it uses the set_time(long []) method to set the system clock time.

[[File:level2clientcode2.jpg]]

*3.3 Daemon client and server

The level2 client and server runs as a daemon process to give them the privilege to run continuously in the background. As both the client and server in level2 does not need any user interference and need to run all the time to update the system clock of the running computer from the level1 server.

As it is mentioned above the client process for level2 run continuously and it updates its system clock every 1 minute (polls time from the level1 server every 60 seconds). Server daemon process is built in the same manner but unlike the client process it does not have a waiting time.

[[File:codefordaemon.jpg]]

*3.4 Marzullo’s algorithm and function marzullo(long [])

Marzullo’s algorithm, invented by Keith Marzullo, is an agreement algorithm used to select sources for estimating accurate time from a number of noisy time sources. A modified version of his algorithm, intersection algorithm, is use in NTP. We used the simplified version of this algorithm to choose the best possible time from our time sources (level1 servers).

Our function long* marzullo (long []) takes an array of four times from four different level1 servers and return a pointer to an array with the best seconds and milliseconds. First it takes the given array of times and makes two entries for each time as ±2 milliseconds. For each entry it also adds a type of -1 for -2millisecons and +1 for +2 milliseconds. As a result for 4 time entry it produces an array of 8 time entries with type as ±1. For our LLTP we took ±2milliseconds as our error range. After adding the type the entry will look like <entry – 2, -1> for the beginning of the range and <entry+2, +1> for the end of the range [entry±2].

[[File:IntersectionMarzullasalg.jpg]]

The description of the algorithm uses the following variables: best (largest number of overlapping intervals found), cnt (current number of overlapping intervals), best_start and

best_end (the beginning and end of best interval found so far). We used a two dimensional array to keep the second, millisecond and type of the range. The structure of the array is as follows:

time_entry [i][0] = second part of the time
entry time_entry [i][1] = millisecond part of the time
entry time_entry [i][2] = type of the range (i.e. ±1)

Here ‘Time_entry is the name of the array and ‘i’ is used as an index to traverse the array. After creating the array it follows the following step to determine the accurate time. Then it sends back the best chosen time to the level2 client program.

[[File:section3last.jpg]]

= List of files=

Level1 server:

DieWithError.c
get-time.c
L1Server.c

Level2 server:

daemon_server.c
DieWithError.c
get-time.c
L2Server.c

Level2 client:

daemon_client.c
DieWithError.c
get-time.c
L2Client.c
marzullo.c
print-time.c
selection_sort.c
set-time.c

Level3 client:

DieWithError.c
get-time.c
print-time.c
set-time.c
UDPTimeClient.c

=Discussion/Output=

The most intresting part of testing this was to see how distrubted architect of NTP client-server works and how could it be improved. I learnt alot by setting up and running them and seeing the results. It was fun improving little bit on exsisting implementation of NTP.

Output form different levels of clients and servers are given in this section. Output from level1 server:

[[File:output1.jpg]]

[[File:output2.jpg]]

[[File:output3.jpg]]

[[File:output4.jpg]]

=Conclusion=

Overall, there has been lots of work going on NTP , due to high demand of distributed clinet-server architect based application. This was carried on a very small linux based lab , in future i guess there will be need to test on large industry based scale , where many systems can be connected and than test to see how the delay from clients to server causes disturbance in algorithm. There will be issues for sure since as the information travels it takes time and that delayed time will need to be adjusted more and will be focus of future work.

=References=

*Ntp home page: http://ntp.org Acessed on 13th Feb 2011

*Wikipedia for network time protocol: http://en.wikipedia.org/wiki/Network_Time_Protocol Accessed on 13th Feb 2011

*Wikipedia for Marzullo’s algorithm: http://en.wikipedia.org/wiki/Marzullo's_algorithm Accessed on 13th Feb 2011

DistOS-2011W NTP

2011-02-22T18:33:38Z

Wahmed2: /* 1.5 Outline of the report */

=Introduction:Context/Background=

Network Time Protocol (NTP) synchronizes clocks of hosts and routers. According to NIST estimates there are 10-20 million NTP servers and clients deployed in the internet and its tributaries all over the world. NTP software has been ported to almost every workstation and server platform available today – from PCs to Cray’s – UNIX, Windows, VMS and embedded systems, even home routers and battery backup systems.

NTP provides nominal accuracies of low tens of milliseconds on WANs, sub milliseconds on LANs and sub microseconds using a precision time source such as cesium oscillator or GPS receiver. Our time protocol was designed over the LAN and our goal was to achieve sub milliseconds of accuracy.

== Importance==

Precision of time is very important in the computer world. Following are some example why it is so important to maintain a precise and accurate time.

Network monitoring, measurement and control, Distributed database transaction journaling and logging, Secure document timestamps with cryptographic certification, Radio and TV programming lunch and monitoring, Stock market buy and sell orders, Distributed network gaming and training, etc.

== Definition of the problem ==

My goal was to acquire time accuracy in the Linux lab assuming that all the computers do not have internet connection. To time synchronize all the computers I had to build our own hierarchy of NTP servers and clients.

== Summary of the result ==

An efficient hierarchy of NTP server and client has been implemented. Precision of
Sub milliseconds were achieved.

==Outline of the report==
Detailed background information and the structure of our time protocol are provided in Section
2. The development setup and code explanation is reviewed in Section 3. Details of which server/client contains which files are given in section 4. My contributions in this implementation are described in Section 5.

= Detailed context/background information=

In this section the background information of Network Time protocol and the hierarchical model and the comparison with the original NTP designed by Dave Mills is reviewed. For unambiguity I will call the protocol as Linux lab time protocol (LLTP) and the original Network time protocol as NTP.

*2.1 NTP timestamps

The 64-bit timestamps used by NTP consist of a 32-bit second part and a 32-bit fractional second part. I followed the same time structure for our LLTP which gives us a time scale of 232 seconds (136 years) and a theoretical resolution of 2-32 seconds (233 picoseconds).

*2.2 Structure

NTP uses a hierarchical, semi-layered system of levels of clock sources; each level of this hierarchy is termed a stratum and assigned a layer number starting with 0 at the top. The stratum level defines its distance from the reference clock and exists to prevent cyclical dependencies in hierarchy.

My target is to develop a time synchronizing software for a small office or lab with a large number of computer where a very few computer has internet connectivity but all the computers are connected together in a local network. As a result not all the computer is able to connect to internet for time accuracy. LLTP uses similar hierarchical, semi-layered sources as NTP. Each layer of this hierarchy is termed as level and assigned a layer number starting with 1.

*2.3 Level 1 server

Computers with internet connectivity are the level 1 server for our LLTP

Level 1 server

synchronizes its system clock with the NTP servers on the internet. In this hierarchy, only a fixed (5) number of level 1 servers are used. Unlike NTP we don’t have the privilege to use a 10 of thousands of level 1 and level 2 servers, and unlike NTP our LLTP has only 3 level structures.

Picture below shows the connectivity and the hierarchy of LLTP

[[File:LTTPHireachy.jpg]]

*2.4 Level 2 server and client

Level 2 servers run two different programs. A client program to synchronize its system clock with the level1 servers and a server program that waits for any level 3 client for time request. Upon the request it sends its synchronized system time to the client. Level 2 servers also synchronize its system clock with the other level2 servers.

Each level2 server is connected to 4 level1 servers. It polls time from each level1 server and chooses the best time and synchronizes its system clock. How the best time is achieved is described in the section 3 below with code explanation. There are maximum 5 level2 servers with our LLTP model. Each level2 server is connected to 4 different level1 servers. None of the level2 server has the same level1 server set.

*2.5 Level 3 client

Bottom of the LLTP hierarchy is level3 client. It only runs a client program that polls time from the level2 servers. It can only connect/request time from one of our level2 servers at a time. It also requires the knowledge of the IP address of the level2 server. For the load management purpose level 3 clients are not able to connect to a Level1 servers.

= Development Details=

*3.1 Server

As it is mentioned in section-2, there are two servers in our LLTP model. One is level 1 and the other is level 2. Both servers wait for a client with time requests and upon receiving one send the time string. Code structure for both the servers is almost similar with minor technical differences which are explained later in this section. Both servers use a infinite for loop for waiting for the
client. They use a fixed but different port number for the client connection.

[[File:clientntp.jpg]]

Time requests sent to Level 1 server have to contain a message no more than 1 character and it has to be an integer which is sent by the level2 client to distinguish the level1 servers from one another. Level1 server will take that message received from the client and add that at the end of the time string sent to the client. If the level1 server receives a time request message with greater size of 1than it will not return any time value. For the level2 server the maximum message size is
128 characters and it do not have to be an integer. Level2 server discards the message sent from the level3 client. A unique feature of level2 server is that a level3 client time request with “exit” message can shut down the level2 server. This feature is not implemented in level1 servers.

[[File:level1servercode.jpg]]

[[File:level2servercode.jpg]]

Both servers send time as a string which handles the problem with big and little Endean.

*3.2 Client

There are also two clients for our LLTP, level2 client and level3 client. Level3 client is very straight forward where the level3 is a bit more complicated.

Level3 client sends a time request to the level2 server. It cannot send time request to a level1 server as the port number for both server are different and the port numbers are hard coded for both the client (leve3 client/level2 server uses port 123 where level2 client/level1 server uses port 4003). Level2 server also has a restriction over the message sent by the client which is explained above in the server section. Upon receiving the time string back from the server which contains the seconds and milliseconds part together as a string separated by a comma (‘,’), it separates this two parts and checks for the latency and calls function named set_time(long[] ) which sets the system clock time with the new time.

[[File:level3clientcode.jpg]]

Level2 client dose almost the same things but in a broader and sophisticated way. It polls time from four different levle1 time server at the same time. When it sends the time request, it sends a unique message as the server ID to identify the time received from different level1 server which helps to fix the latency later.

[[File:level2clientcode.jpg]]

Level2 client also uses a time out of 1 second. If it dose not receive the time string back from the server then it will ignore the time from that server and choose the best time from the ones it received from the other servers. Received time strings from the different servers also have the server id added at the end of the time string. Using that information level2 client saves the received time in order and fixes the latency accordingly. It keeps all the latency fixed time in an array with the invalid time being -1. Then it calls a different function called marzullo(long []), which takes the array of latency fixed times and returns the best time depending on the Marzullo’s algorithm. Function marzullo() is explained in the section 3.4. Then it uses the set_time(long []) method to set the system clock time.

[[File:level2clientcode2.jpg]]

*3.3 Daemon client and server

The level2 client and server runs as a daemon process to give them the privilege to run continuously in the background. As both the client and server in level2 does not need any user interference and need to run all the time to update the system clock of the running computer from the level1 server.

As it is mentioned above the client process for level2 run continuously and it updates its system clock every 1 minute (polls time from the level1 server every 60 seconds). Server daemon process is built in the same manner but unlike the client process it does not have a waiting time.

[[File:codefordaemon.jpg]]

*3.4 Marzullo’s algorithm and function marzullo(long [])

Marzullo’s algorithm, invented by Keith Marzullo, is an agreement algorithm used to select sources for estimating accurate time from a number of noisy time sources. A modified version of his algorithm, intersection algorithm, is use in NTP. We used the simplified version of this algorithm to choose the best possible time from our time sources (level1 servers).

Our function long* marzullo (long []) takes an array of four times from four different level1 servers and return a pointer to an array with the best seconds and milliseconds. First it takes the given array of times and makes two entries for each time as ±2 milliseconds. For each entry it also adds a type of -1 for -2millisecons and +1 for +2 milliseconds. As a result for 4 time entry it produces an array of 8 time entries with type as ±1. For our LLTP we took ±2milliseconds as our error range. After adding the type the entry will look like <entry – 2, -1> for the beginning of the range and <entry+2, +1> for the end of the range [entry±2].

[[File:IntersectionMarzullasalg.jpg]]

The description of the algorithm uses the following variables: best (largest number of overlapping intervals found), cnt (current number of overlapping intervals), best_start and

best_end (the beginning and end of best interval found so far). We used a two dimensional array to keep the second, millisecond and type of the range. The structure of the array is as follows:

time_entry [i][0] = second part of the time
entry time_entry [i][1] = millisecond part of the time
entry time_entry [i][2] = type of the range (i.e. ±1)

Here ‘Time_entry is the name of the array and ‘i’ is used as an index to traverse the array. After creating the array it follows the following step to determine the accurate time. Then it sends back the best chosen time to the level2 client program.

[[File:section3last.jpg]]

= List of files=

Level1 server:

DieWithError.c
get-time.c
L1Server.c

Level2 server:

daemon_server.c
DieWithError.c
get-time.c
L2Server.c

Level2 client:

daemon_client.c
DieWithError.c
get-time.c
L2Client.c
marzullo.c
print-time.c
selection_sort.c
set-time.c

Level3 client:

DieWithError.c
get-time.c
print-time.c
set-time.c
UDPTimeClient.c

=Discussion/Output=

The most intresting part of testing this was to see how distrubted architect of NTP client-server works and how could it be improved. I learnt alot by setting up and running them and seeing the results. It was fun improving little bit on exsisting implementation of NTP.

Output form different levels of clients and servers are given in this section. Output from level1 server:

[[File:output1.jpg]]

[[File:output2.jpg]]

[[File:output3.jpg]]

[[File:output4.jpg]]

=Conclusion=

Overall, there has been lots of work going on NTP , due to high demand of distributed clinet-server architect based application. This was carried on a very small linux based lab , in future i guess there will be need to test on large industry based scale , where many systems can be connected and than test to see how the delay from clients to server causes disturbance in algorithm. There will be issues for sure since as the information travels it takes time and that delayed time will need to be adjusted more and will be focus of future work.

=References=

*Ntp home page: http://ntp.org Acessed on 13th Feb 2011

*Wikipedia for network time protocol: http://en.wikipedia.org/wiki/Network_Time_Protocol Accessed on 13th Feb 2011

*Wikipedia for Marzullo’s algorithm: http://en.wikipedia.org/wiki/Marzullo's_algorithm Accessed on 13th Feb 2011

DistOS-2011W NTP

2011-02-22T18:33:20Z

Wahmed2: /* Introduction */

=Introduction:Context/Background=

Network Time Protocol (NTP) synchronizes clocks of hosts and routers. According to NIST estimates there are 10-20 million NTP servers and clients deployed in the internet and its tributaries all over the world. NTP software has been ported to almost every workstation and server platform available today – from PCs to Cray’s – UNIX, Windows, VMS and embedded systems, even home routers and battery backup systems.

NTP provides nominal accuracies of low tens of milliseconds on WANs, sub milliseconds on LANs and sub microseconds using a precision time source such as cesium oscillator or GPS receiver. Our time protocol was designed over the LAN and our goal was to achieve sub milliseconds of accuracy.

== Importance==

Precision of time is very important in the computer world. Following are some example why it is so important to maintain a precise and accurate time.

Network monitoring, measurement and control, Distributed database transaction journaling and logging, Secure document timestamps with cryptographic certification, Radio and TV programming lunch and monitoring, Stock market buy and sell orders, Distributed network gaming and training, etc.

== Definition of the problem ==

My goal was to acquire time accuracy in the Linux lab assuming that all the computers do not have internet connection. To time synchronize all the computers I had to build our own hierarchy of NTP servers and clients.

== Summary of the result ==

An efficient hierarchy of NTP server and client has been implemented. Precision of
Sub milliseconds were achieved.

==1.5 Outline of the report==
Detailed background information and the structure of our time protocol are provided in Section
2. The development setup and code explanation is reviewed in Section 3. Details of which server/client contains which files are given in section 4. My contributions in this implementation are described in Section 5.

= Detailed context/background information=

In this section the background information of Network Time protocol and the hierarchical model and the comparison with the original NTP designed by Dave Mills is reviewed. For unambiguity I will call the protocol as Linux lab time protocol (LLTP) and the original Network time protocol as NTP.

*2.1 NTP timestamps

The 64-bit timestamps used by NTP consist of a 32-bit second part and a 32-bit fractional second part. I followed the same time structure for our LLTP which gives us a time scale of 232 seconds (136 years) and a theoretical resolution of 2-32 seconds (233 picoseconds).

*2.2 Structure

NTP uses a hierarchical, semi-layered system of levels of clock sources; each level of this hierarchy is termed a stratum and assigned a layer number starting with 0 at the top. The stratum level defines its distance from the reference clock and exists to prevent cyclical dependencies in hierarchy.

My target is to develop a time synchronizing software for a small office or lab with a large number of computer where a very few computer has internet connectivity but all the computers are connected together in a local network. As a result not all the computer is able to connect to internet for time accuracy. LLTP uses similar hierarchical, semi-layered sources as NTP. Each layer of this hierarchy is termed as level and assigned a layer number starting with 1.

*2.3 Level 1 server

Computers with internet connectivity are the level 1 server for our LLTP

Level 1 server

synchronizes its system clock with the NTP servers on the internet. In this hierarchy, only a fixed (5) number of level 1 servers are used. Unlike NTP we don’t have the privilege to use a 10 of thousands of level 1 and level 2 servers, and unlike NTP our LLTP has only 3 level structures.

Picture below shows the connectivity and the hierarchy of LLTP

[[File:LTTPHireachy.jpg]]

*2.4 Level 2 server and client

Level 2 servers run two different programs. A client program to synchronize its system clock with the level1 servers and a server program that waits for any level 3 client for time request. Upon the request it sends its synchronized system time to the client. Level 2 servers also synchronize its system clock with the other level2 servers.

Each level2 server is connected to 4 level1 servers. It polls time from each level1 server and chooses the best time and synchronizes its system clock. How the best time is achieved is described in the section 3 below with code explanation. There are maximum 5 level2 servers with our LLTP model. Each level2 server is connected to 4 different level1 servers. None of the level2 server has the same level1 server set.

*2.5 Level 3 client

Bottom of the LLTP hierarchy is level3 client. It only runs a client program that polls time from the level2 servers. It can only connect/request time from one of our level2 servers at a time. It also requires the knowledge of the IP address of the level2 server. For the load management purpose level 3 clients are not able to connect to a Level1 servers.

= Development Details=

*3.1 Server

As it is mentioned in section-2, there are two servers in our LLTP model. One is level 1 and the other is level 2. Both servers wait for a client with time requests and upon receiving one send the time string. Code structure for both the servers is almost similar with minor technical differences which are explained later in this section. Both servers use a infinite for loop for waiting for the
client. They use a fixed but different port number for the client connection.

[[File:clientntp.jpg]]

Time requests sent to Level 1 server have to contain a message no more than 1 character and it has to be an integer which is sent by the level2 client to distinguish the level1 servers from one another. Level1 server will take that message received from the client and add that at the end of the time string sent to the client. If the level1 server receives a time request message with greater size of 1than it will not return any time value. For the level2 server the maximum message size is
128 characters and it do not have to be an integer. Level2 server discards the message sent from the level3 client. A unique feature of level2 server is that a level3 client time request with “exit” message can shut down the level2 server. This feature is not implemented in level1 servers.

[[File:level1servercode.jpg]]

[[File:level2servercode.jpg]]

Both servers send time as a string which handles the problem with big and little Endean.

*3.2 Client

There are also two clients for our LLTP, level2 client and level3 client. Level3 client is very straight forward where the level3 is a bit more complicated.

Level3 client sends a time request to the level2 server. It cannot send time request to a level1 server as the port number for both server are different and the port numbers are hard coded for both the client (leve3 client/level2 server uses port 123 where level2 client/level1 server uses port 4003). Level2 server also has a restriction over the message sent by the client which is explained above in the server section. Upon receiving the time string back from the server which contains the seconds and milliseconds part together as a string separated by a comma (‘,’), it separates this two parts and checks for the latency and calls function named set_time(long[] ) which sets the system clock time with the new time.

[[File:level3clientcode.jpg]]

Level2 client dose almost the same things but in a broader and sophisticated way. It polls time from four different levle1 time server at the same time. When it sends the time request, it sends a unique message as the server ID to identify the time received from different level1 server which helps to fix the latency later.

[[File:level2clientcode.jpg]]

Level2 client also uses a time out of 1 second. If it dose not receive the time string back from the server then it will ignore the time from that server and choose the best time from the ones it received from the other servers. Received time strings from the different servers also have the server id added at the end of the time string. Using that information level2 client saves the received time in order and fixes the latency accordingly. It keeps all the latency fixed time in an array with the invalid time being -1. Then it calls a different function called marzullo(long []), which takes the array of latency fixed times and returns the best time depending on the Marzullo’s algorithm. Function marzullo() is explained in the section 3.4. Then it uses the set_time(long []) method to set the system clock time.

[[File:level2clientcode2.jpg]]

*3.3 Daemon client and server

The level2 client and server runs as a daemon process to give them the privilege to run continuously in the background. As both the client and server in level2 does not need any user interference and need to run all the time to update the system clock of the running computer from the level1 server.

As it is mentioned above the client process for level2 run continuously and it updates its system clock every 1 minute (polls time from the level1 server every 60 seconds). Server daemon process is built in the same manner but unlike the client process it does not have a waiting time.

[[File:codefordaemon.jpg]]

*3.4 Marzullo’s algorithm and function marzullo(long [])

Marzullo’s algorithm, invented by Keith Marzullo, is an agreement algorithm used to select sources for estimating accurate time from a number of noisy time sources. A modified version of his algorithm, intersection algorithm, is use in NTP. We used the simplified version of this algorithm to choose the best possible time from our time sources (level1 servers).

Our function long* marzullo (long []) takes an array of four times from four different level1 servers and return a pointer to an array with the best seconds and milliseconds. First it takes the given array of times and makes two entries for each time as ±2 milliseconds. For each entry it also adds a type of -1 for -2millisecons and +1 for +2 milliseconds. As a result for 4 time entry it produces an array of 8 time entries with type as ±1. For our LLTP we took ±2milliseconds as our error range. After adding the type the entry will look like <entry – 2, -1> for the beginning of the range and <entry+2, +1> for the end of the range [entry±2].

[[File:IntersectionMarzullasalg.jpg]]

The description of the algorithm uses the following variables: best (largest number of overlapping intervals found), cnt (current number of overlapping intervals), best_start and

best_end (the beginning and end of best interval found so far). We used a two dimensional array to keep the second, millisecond and type of the range. The structure of the array is as follows:

time_entry [i][0] = second part of the time
entry time_entry [i][1] = millisecond part of the time
entry time_entry [i][2] = type of the range (i.e. ±1)

Here ‘Time_entry is the name of the array and ‘i’ is used as an index to traverse the array. After creating the array it follows the following step to determine the accurate time. Then it sends back the best chosen time to the level2 client program.

[[File:section3last.jpg]]

= List of files=

Level1 server:

DieWithError.c
get-time.c
L1Server.c

Level2 server:

daemon_server.c
DieWithError.c
get-time.c
L2Server.c

Level2 client:

daemon_client.c
DieWithError.c
get-time.c
L2Client.c
marzullo.c
print-time.c
selection_sort.c
set-time.c

Level3 client:

DieWithError.c
get-time.c
print-time.c
set-time.c
UDPTimeClient.c

=Discussion/Output=

The most intresting part of testing this was to see how distrubted architect of NTP client-server works and how could it be improved. I learnt alot by setting up and running them and seeing the results. It was fun improving little bit on exsisting implementation of NTP.

Output form different levels of clients and servers are given in this section. Output from level1 server:

[[File:output1.jpg]]

[[File:output2.jpg]]

[[File:output3.jpg]]

[[File:output4.jpg]]

=Conclusion=

Overall, there has been lots of work going on NTP , due to high demand of distributed clinet-server architect based application. This was carried on a very small linux based lab , in future i guess there will be need to test on large industry based scale , where many systems can be connected and than test to see how the delay from clients to server causes disturbance in algorithm. There will be issues for sure since as the information travels it takes time and that delayed time will need to be adjusted more and will be focus of future work.

=References=

*Ntp home page: http://ntp.org Acessed on 13th Feb 2011

*Wikipedia for network time protocol: http://en.wikipedia.org/wiki/Network_Time_Protocol Accessed on 13th Feb 2011

*Wikipedia for Marzullo’s algorithm: http://en.wikipedia.org/wiki/Marzullo's_algorithm Accessed on 13th Feb 2011

DistOS-2011W NTP

2011-02-22T18:32:15Z

Wahmed2: /* 1.2 Importance */

=Introduction=

== Context/Background==

Network Time Protocol (NTP) synchronizes clocks of hosts and routers. According to NIST estimates there are 10-20 million NTP servers and clients deployed in the internet and its tributaries all over the world. NTP software has been ported to almost every workstation and server platform available today – from PCs to Cray’s – UNIX, Windows, VMS and embedded systems, even home routers and battery backup systems.

NTP provides nominal accuracies of low tens of milliseconds on WANs, sub milliseconds on LANs and sub microseconds using a precision time source such as cesium oscillator or GPS receiver. Our time protocol was designed over the LAN and our goal was to achieve sub milliseconds of accuracy.

== Importance==

Precision of time is very important in the computer world. Following are some example why it is so important to maintain a precise and accurate time.

Network monitoring, measurement and control, Distributed database transaction journaling and logging, Secure document timestamps with cryptographic certification, Radio and TV programming lunch and monitoring, Stock market buy and sell orders, Distributed network gaming and training, etc.

== Definition of the problem ==

My goal was to acquire time accuracy in the Linux lab assuming that all the computers do not have internet connection. To time synchronize all the computers I had to build our own hierarchy of NTP servers and clients.

== Summary of the result ==

An efficient hierarchy of NTP server and client has been implemented. Precision of
Sub milliseconds were achieved.

==1.5 Outline of the report==
Detailed background information and the structure of our time protocol are provided in Section
2. The development setup and code explanation is reviewed in Section 3. Details of which server/client contains which files are given in section 4. My contributions in this implementation are described in Section 5.

= Detailed context/background information=

In this section the background information of Network Time protocol and the hierarchical model and the comparison with the original NTP designed by Dave Mills is reviewed. For unambiguity I will call the protocol as Linux lab time protocol (LLTP) and the original Network time protocol as NTP.

*2.1 NTP timestamps

The 64-bit timestamps used by NTP consist of a 32-bit second part and a 32-bit fractional second part. I followed the same time structure for our LLTP which gives us a time scale of 232 seconds (136 years) and a theoretical resolution of 2-32 seconds (233 picoseconds).

*2.2 Structure

NTP uses a hierarchical, semi-layered system of levels of clock sources; each level of this hierarchy is termed a stratum and assigned a layer number starting with 0 at the top. The stratum level defines its distance from the reference clock and exists to prevent cyclical dependencies in hierarchy.

My target is to develop a time synchronizing software for a small office or lab with a large number of computer where a very few computer has internet connectivity but all the computers are connected together in a local network. As a result not all the computer is able to connect to internet for time accuracy. LLTP uses similar hierarchical, semi-layered sources as NTP. Each layer of this hierarchy is termed as level and assigned a layer number starting with 1.

*2.3 Level 1 server

Computers with internet connectivity are the level 1 server for our LLTP

Level 1 server

synchronizes its system clock with the NTP servers on the internet. In this hierarchy, only a fixed (5) number of level 1 servers are used. Unlike NTP we don’t have the privilege to use a 10 of thousands of level 1 and level 2 servers, and unlike NTP our LLTP has only 3 level structures.

Picture below shows the connectivity and the hierarchy of LLTP

[[File:LTTPHireachy.jpg]]

*2.4 Level 2 server and client

Level 2 servers run two different programs. A client program to synchronize its system clock with the level1 servers and a server program that waits for any level 3 client for time request. Upon the request it sends its synchronized system time to the client. Level 2 servers also synchronize its system clock with the other level2 servers.

Each level2 server is connected to 4 level1 servers. It polls time from each level1 server and chooses the best time and synchronizes its system clock. How the best time is achieved is described in the section 3 below with code explanation. There are maximum 5 level2 servers with our LLTP model. Each level2 server is connected to 4 different level1 servers. None of the level2 server has the same level1 server set.

*2.5 Level 3 client

Bottom of the LLTP hierarchy is level3 client. It only runs a client program that polls time from the level2 servers. It can only connect/request time from one of our level2 servers at a time. It also requires the knowledge of the IP address of the level2 server. For the load management purpose level 3 clients are not able to connect to a Level1 servers.

= Development Details=

*3.1 Server

As it is mentioned in section-2, there are two servers in our LLTP model. One is level 1 and the other is level 2. Both servers wait for a client with time requests and upon receiving one send the time string. Code structure for both the servers is almost similar with minor technical differences which are explained later in this section. Both servers use a infinite for loop for waiting for the
client. They use a fixed but different port number for the client connection.

[[File:clientntp.jpg]]

Time requests sent to Level 1 server have to contain a message no more than 1 character and it has to be an integer which is sent by the level2 client to distinguish the level1 servers from one another. Level1 server will take that message received from the client and add that at the end of the time string sent to the client. If the level1 server receives a time request message with greater size of 1than it will not return any time value. For the level2 server the maximum message size is
128 characters and it do not have to be an integer. Level2 server discards the message sent from the level3 client. A unique feature of level2 server is that a level3 client time request with “exit” message can shut down the level2 server. This feature is not implemented in level1 servers.

[[File:level1servercode.jpg]]

[[File:level2servercode.jpg]]

Both servers send time as a string which handles the problem with big and little Endean.

*3.2 Client

There are also two clients for our LLTP, level2 client and level3 client. Level3 client is very straight forward where the level3 is a bit more complicated.

Level3 client sends a time request to the level2 server. It cannot send time request to a level1 server as the port number for both server are different and the port numbers are hard coded for both the client (leve3 client/level2 server uses port 123 where level2 client/level1 server uses port 4003). Level2 server also has a restriction over the message sent by the client which is explained above in the server section. Upon receiving the time string back from the server which contains the seconds and milliseconds part together as a string separated by a comma (‘,’), it separates this two parts and checks for the latency and calls function named set_time(long[] ) which sets the system clock time with the new time.

[[File:level3clientcode.jpg]]

Level2 client dose almost the same things but in a broader and sophisticated way. It polls time from four different levle1 time server at the same time. When it sends the time request, it sends a unique message as the server ID to identify the time received from different level1 server which helps to fix the latency later.

[[File:level2clientcode.jpg]]

Level2 client also uses a time out of 1 second. If it dose not receive the time string back from the server then it will ignore the time from that server and choose the best time from the ones it received from the other servers. Received time strings from the different servers also have the server id added at the end of the time string. Using that information level2 client saves the received time in order and fixes the latency accordingly. It keeps all the latency fixed time in an array with the invalid time being -1. Then it calls a different function called marzullo(long []), which takes the array of latency fixed times and returns the best time depending on the Marzullo’s algorithm. Function marzullo() is explained in the section 3.4. Then it uses the set_time(long []) method to set the system clock time.

[[File:level2clientcode2.jpg]]

*3.3 Daemon client and server

The level2 client and server runs as a daemon process to give them the privilege to run continuously in the background. As both the client and server in level2 does not need any user interference and need to run all the time to update the system clock of the running computer from the level1 server.

As it is mentioned above the client process for level2 run continuously and it updates its system clock every 1 minute (polls time from the level1 server every 60 seconds). Server daemon process is built in the same manner but unlike the client process it does not have a waiting time.

[[File:codefordaemon.jpg]]

*3.4 Marzullo’s algorithm and function marzullo(long [])

Marzullo’s algorithm, invented by Keith Marzullo, is an agreement algorithm used to select sources for estimating accurate time from a number of noisy time sources. A modified version of his algorithm, intersection algorithm, is use in NTP. We used the simplified version of this algorithm to choose the best possible time from our time sources (level1 servers).

Our function long* marzullo (long []) takes an array of four times from four different level1 servers and return a pointer to an array with the best seconds and milliseconds. First it takes the given array of times and makes two entries for each time as ±2 milliseconds. For each entry it also adds a type of -1 for -2millisecons and +1 for +2 milliseconds. As a result for 4 time entry it produces an array of 8 time entries with type as ±1. For our LLTP we took ±2milliseconds as our error range. After adding the type the entry will look like <entry – 2, -1> for the beginning of the range and <entry+2, +1> for the end of the range [entry±2].

[[File:IntersectionMarzullasalg.jpg]]

The description of the algorithm uses the following variables: best (largest number of overlapping intervals found), cnt (current number of overlapping intervals), best_start and

best_end (the beginning and end of best interval found so far). We used a two dimensional array to keep the second, millisecond and type of the range. The structure of the array is as follows:

time_entry [i][0] = second part of the time
entry time_entry [i][1] = millisecond part of the time
entry time_entry [i][2] = type of the range (i.e. ±1)

Here ‘Time_entry is the name of the array and ‘i’ is used as an index to traverse the array. After creating the array it follows the following step to determine the accurate time. Then it sends back the best chosen time to the level2 client program.

[[File:section3last.jpg]]

= List of files=

Level1 server:

DieWithError.c
get-time.c
L1Server.c

Level2 server:

daemon_server.c
DieWithError.c
get-time.c
L2Server.c

Level2 client:

daemon_client.c
DieWithError.c
get-time.c
L2Client.c
marzullo.c
print-time.c
selection_sort.c
set-time.c

Level3 client:

DieWithError.c
get-time.c
print-time.c
set-time.c
UDPTimeClient.c

=Discussion/Output=

The most intresting part of testing this was to see how distrubted architect of NTP client-server works and how could it be improved. I learnt alot by setting up and running them and seeing the results. It was fun improving little bit on exsisting implementation of NTP.

Output form different levels of clients and servers are given in this section. Output from level1 server:

[[File:output1.jpg]]

[[File:output2.jpg]]

[[File:output3.jpg]]

[[File:output4.jpg]]

=Conclusion=

Overall, there has been lots of work going on NTP , due to high demand of distributed clinet-server architect based application. This was carried on a very small linux based lab , in future i guess there will be need to test on large industry based scale , where many systems can be connected and than test to see how the delay from clients to server causes disturbance in algorithm. There will be issues for sure since as the information travels it takes time and that delayed time will need to be adjusted more and will be focus of future work.

=References=

*Ntp home page: http://ntp.org Acessed on 13th Feb 2011

*Wikipedia for network time protocol: http://en.wikipedia.org/wiki/Network_Time_Protocol Accessed on 13th Feb 2011

*Wikipedia for Marzullo’s algorithm: http://en.wikipedia.org/wiki/Marzullo's_algorithm Accessed on 13th Feb 2011

DistOS-2011W NTP

2011-02-22T18:31:28Z

Wahmed2: /* 1.1 Context/Background */

=Introduction=

== Context/Background==

Network Time Protocol (NTP) synchronizes clocks of hosts and routers. According to NIST estimates there are 10-20 million NTP servers and clients deployed in the internet and its tributaries all over the world. NTP software has been ported to almost every workstation and server platform available today – from PCs to Cray’s – UNIX, Windows, VMS and embedded systems, even home routers and battery backup systems.

NTP provides nominal accuracies of low tens of milliseconds on WANs, sub milliseconds on LANs and sub microseconds using a precision time source such as cesium oscillator or GPS receiver. Our time protocol was designed over the LAN and our goal was to achieve sub milliseconds of accuracy.

==1.2 Importance==

Precision of time is very important in the computer world. Following are some example why it is so important to maintain a precise and accurate time.

Network monitoring, measurement and control, Distributed database transaction journaling and logging, Secure document timestamps with cryptographic certification, Radio and TV programming lunch and monitoring, Stock market buy and sell orders, Distributed network gaming and training, etc.

*1.3 Definition of the problem

My goal was to acquire time accuracy in the Linux lab assuming that all the computers do not have internet connection. To time synchronize all the computers I had to build our own hierarchy of NTP servers and clients.

*1.4 Summary of the result

An efficient hierarchy of NTP server and client has been implemented. Precision of
Sub milliseconds were achieved.

*1.5 Outline of the report
Detailed background information and the structure of our time protocol are provided in Section
2. The development setup and code explanation is reviewed in Section 3. Details of which server/client contains which files are given in section 4. My contributions in this implementation are described in Section 5.

= Detailed context/background information=

In this section the background information of Network Time protocol and the hierarchical model and the comparison with the original NTP designed by Dave Mills is reviewed. For unambiguity I will call the protocol as Linux lab time protocol (LLTP) and the original Network time protocol as NTP.

*2.1 NTP timestamps

The 64-bit timestamps used by NTP consist of a 32-bit second part and a 32-bit fractional second part. I followed the same time structure for our LLTP which gives us a time scale of 232 seconds (136 years) and a theoretical resolution of 2-32 seconds (233 picoseconds).

*2.2 Structure

NTP uses a hierarchical, semi-layered system of levels of clock sources; each level of this hierarchy is termed a stratum and assigned a layer number starting with 0 at the top. The stratum level defines its distance from the reference clock and exists to prevent cyclical dependencies in hierarchy.

My target is to develop a time synchronizing software for a small office or lab with a large number of computer where a very few computer has internet connectivity but all the computers are connected together in a local network. As a result not all the computer is able to connect to internet for time accuracy. LLTP uses similar hierarchical, semi-layered sources as NTP. Each layer of this hierarchy is termed as level and assigned a layer number starting with 1.

*2.3 Level 1 server

Computers with internet connectivity are the level 1 server for our LLTP

Level 1 server

synchronizes its system clock with the NTP servers on the internet. In this hierarchy, only a fixed (5) number of level 1 servers are used. Unlike NTP we don’t have the privilege to use a 10 of thousands of level 1 and level 2 servers, and unlike NTP our LLTP has only 3 level structures.

Picture below shows the connectivity and the hierarchy of LLTP

[[File:LTTPHireachy.jpg]]

*2.4 Level 2 server and client

Level 2 servers run two different programs. A client program to synchronize its system clock with the level1 servers and a server program that waits for any level 3 client for time request. Upon the request it sends its synchronized system time to the client. Level 2 servers also synchronize its system clock with the other level2 servers.

Each level2 server is connected to 4 level1 servers. It polls time from each level1 server and chooses the best time and synchronizes its system clock. How the best time is achieved is described in the section 3 below with code explanation. There are maximum 5 level2 servers with our LLTP model. Each level2 server is connected to 4 different level1 servers. None of the level2 server has the same level1 server set.

*2.5 Level 3 client

Bottom of the LLTP hierarchy is level3 client. It only runs a client program that polls time from the level2 servers. It can only connect/request time from one of our level2 servers at a time. It also requires the knowledge of the IP address of the level2 server. For the load management purpose level 3 clients are not able to connect to a Level1 servers.

= Development Details=

*3.1 Server

As it is mentioned in section-2, there are two servers in our LLTP model. One is level 1 and the other is level 2. Both servers wait for a client with time requests and upon receiving one send the time string. Code structure for both the servers is almost similar with minor technical differences which are explained later in this section. Both servers use a infinite for loop for waiting for the
client. They use a fixed but different port number for the client connection.

[[File:clientntp.jpg]]

Time requests sent to Level 1 server have to contain a message no more than 1 character and it has to be an integer which is sent by the level2 client to distinguish the level1 servers from one another. Level1 server will take that message received from the client and add that at the end of the time string sent to the client. If the level1 server receives a time request message with greater size of 1than it will not return any time value. For the level2 server the maximum message size is
128 characters and it do not have to be an integer. Level2 server discards the message sent from the level3 client. A unique feature of level2 server is that a level3 client time request with “exit” message can shut down the level2 server. This feature is not implemented in level1 servers.

[[File:level1servercode.jpg]]

[[File:level2servercode.jpg]]

Both servers send time as a string which handles the problem with big and little Endean.

*3.2 Client

There are also two clients for our LLTP, level2 client and level3 client. Level3 client is very straight forward where the level3 is a bit more complicated.

Level3 client sends a time request to the level2 server. It cannot send time request to a level1 server as the port number for both server are different and the port numbers are hard coded for both the client (leve3 client/level2 server uses port 123 where level2 client/level1 server uses port 4003). Level2 server also has a restriction over the message sent by the client which is explained above in the server section. Upon receiving the time string back from the server which contains the seconds and milliseconds part together as a string separated by a comma (‘,’), it separates this two parts and checks for the latency and calls function named set_time(long[] ) which sets the system clock time with the new time.

[[File:level3clientcode.jpg]]

Level2 client dose almost the same things but in a broader and sophisticated way. It polls time from four different levle1 time server at the same time. When it sends the time request, it sends a unique message as the server ID to identify the time received from different level1 server which helps to fix the latency later.

[[File:level2clientcode.jpg]]

Level2 client also uses a time out of 1 second. If it dose not receive the time string back from the server then it will ignore the time from that server and choose the best time from the ones it received from the other servers. Received time strings from the different servers also have the server id added at the end of the time string. Using that information level2 client saves the received time in order and fixes the latency accordingly. It keeps all the latency fixed time in an array with the invalid time being -1. Then it calls a different function called marzullo(long []), which takes the array of latency fixed times and returns the best time depending on the Marzullo’s algorithm. Function marzullo() is explained in the section 3.4. Then it uses the set_time(long []) method to set the system clock time.

[[File:level2clientcode2.jpg]]

*3.3 Daemon client and server

The level2 client and server runs as a daemon process to give them the privilege to run continuously in the background. As both the client and server in level2 does not need any user interference and need to run all the time to update the system clock of the running computer from the level1 server.

As it is mentioned above the client process for level2 run continuously and it updates its system clock every 1 minute (polls time from the level1 server every 60 seconds). Server daemon process is built in the same manner but unlike the client process it does not have a waiting time.

[[File:codefordaemon.jpg]]

*3.4 Marzullo’s algorithm and function marzullo(long [])

Marzullo’s algorithm, invented by Keith Marzullo, is an agreement algorithm used to select sources for estimating accurate time from a number of noisy time sources. A modified version of his algorithm, intersection algorithm, is use in NTP. We used the simplified version of this algorithm to choose the best possible time from our time sources (level1 servers).

Our function long* marzullo (long []) takes an array of four times from four different level1 servers and return a pointer to an array with the best seconds and milliseconds. First it takes the given array of times and makes two entries for each time as ±2 milliseconds. For each entry it also adds a type of -1 for -2millisecons and +1 for +2 milliseconds. As a result for 4 time entry it produces an array of 8 time entries with type as ±1. For our LLTP we took ±2milliseconds as our error range. After adding the type the entry will look like <entry – 2, -1> for the beginning of the range and <entry+2, +1> for the end of the range [entry±2].

[[File:IntersectionMarzullasalg.jpg]]

The description of the algorithm uses the following variables: best (largest number of overlapping intervals found), cnt (current number of overlapping intervals), best_start and

best_end (the beginning and end of best interval found so far). We used a two dimensional array to keep the second, millisecond and type of the range. The structure of the array is as follows:

time_entry [i][0] = second part of the time
entry time_entry [i][1] = millisecond part of the time
entry time_entry [i][2] = type of the range (i.e. ±1)

Here ‘Time_entry is the name of the array and ‘i’ is used as an index to traverse the array. After creating the array it follows the following step to determine the accurate time. Then it sends back the best chosen time to the level2 client program.

[[File:section3last.jpg]]

= List of files=

Level1 server:

DieWithError.c
get-time.c
L1Server.c

Level2 server:

daemon_server.c
DieWithError.c
get-time.c
L2Server.c

Level2 client:

daemon_client.c
DieWithError.c
get-time.c
L2Client.c
marzullo.c
print-time.c
selection_sort.c
set-time.c

Level3 client:

DieWithError.c
get-time.c
print-time.c
set-time.c
UDPTimeClient.c

=Discussion/Output=

The most intresting part of testing this was to see how distrubted architect of NTP client-server works and how could it be improved. I learnt alot by setting up and running them and seeing the results. It was fun improving little bit on exsisting implementation of NTP.

Output form different levels of clients and servers are given in this section. Output from level1 server:

[[File:output1.jpg]]

[[File:output2.jpg]]

[[File:output3.jpg]]

[[File:output4.jpg]]

=Conclusion=

Overall, there has been lots of work going on NTP , due to high demand of distributed clinet-server architect based application. This was carried on a very small linux based lab , in future i guess there will be need to test on large industry based scale , where many systems can be connected and than test to see how the delay from clients to server causes disturbance in algorithm. There will be issues for sure since as the information travels it takes time and that delayed time will need to be adjusted more and will be focus of future work.

=References=

*Ntp home page: http://ntp.org Acessed on 13th Feb 2011

*Wikipedia for network time protocol: http://en.wikipedia.org/wiki/Network_Time_Protocol Accessed on 13th Feb 2011

*Wikipedia for Marzullo’s algorithm: http://en.wikipedia.org/wiki/Marzullo's_algorithm Accessed on 13th Feb 2011

DistOS-2011W NTP

2011-02-22T18:31:08Z

Wahmed2: /* Introduction */

=Introduction=

==1.1 Context/Background==

Network Time Protocol (NTP) synchronizes clocks of hosts and routers. According to NIST estimates there are 10-20 million NTP servers and clients deployed in the internet and its tributaries all over the world. NTP software has been ported to almost every workstation and server platform available today – from PCs to Cray’s – UNIX, Windows, VMS and embedded systems, even home routers and battery backup systems.

NTP provides nominal accuracies of low tens of milliseconds on WANs, sub milliseconds on LANs and sub microseconds using a precision time source such as cesium oscillator or GPS receiver. Our time protocol was designed over the LAN and our goal was to achieve sub milliseconds of accuracy.

==1.2 Importance==

Precision of time is very important in the computer world. Following are some example why it is so important to maintain a precise and accurate time.

Network monitoring, measurement and control, Distributed database transaction journaling and logging, Secure document timestamps with cryptographic certification, Radio and TV programming lunch and monitoring, Stock market buy and sell orders, Distributed network gaming and training, etc.

*1.3 Definition of the problem

My goal was to acquire time accuracy in the Linux lab assuming that all the computers do not have internet connection. To time synchronize all the computers I had to build our own hierarchy of NTP servers and clients.

*1.4 Summary of the result

An efficient hierarchy of NTP server and client has been implemented. Precision of
Sub milliseconds were achieved.

*1.5 Outline of the report
Detailed background information and the structure of our time protocol are provided in Section
2. The development setup and code explanation is reviewed in Section 3. Details of which server/client contains which files are given in section 4. My contributions in this implementation are described in Section 5.

= Detailed context/background information=

In this section the background information of Network Time protocol and the hierarchical model and the comparison with the original NTP designed by Dave Mills is reviewed. For unambiguity I will call the protocol as Linux lab time protocol (LLTP) and the original Network time protocol as NTP.

*2.1 NTP timestamps

The 64-bit timestamps used by NTP consist of a 32-bit second part and a 32-bit fractional second part. I followed the same time structure for our LLTP which gives us a time scale of 232 seconds (136 years) and a theoretical resolution of 2-32 seconds (233 picoseconds).

*2.2 Structure

NTP uses a hierarchical, semi-layered system of levels of clock sources; each level of this hierarchy is termed a stratum and assigned a layer number starting with 0 at the top. The stratum level defines its distance from the reference clock and exists to prevent cyclical dependencies in hierarchy.

My target is to develop a time synchronizing software for a small office or lab with a large number of computer where a very few computer has internet connectivity but all the computers are connected together in a local network. As a result not all the computer is able to connect to internet for time accuracy. LLTP uses similar hierarchical, semi-layered sources as NTP. Each layer of this hierarchy is termed as level and assigned a layer number starting with 1.

*2.3 Level 1 server

Computers with internet connectivity are the level 1 server for our LLTP

Level 1 server

synchronizes its system clock with the NTP servers on the internet. In this hierarchy, only a fixed (5) number of level 1 servers are used. Unlike NTP we don’t have the privilege to use a 10 of thousands of level 1 and level 2 servers, and unlike NTP our LLTP has only 3 level structures.

Picture below shows the connectivity and the hierarchy of LLTP

[[File:LTTPHireachy.jpg]]

*2.4 Level 2 server and client

Level 2 servers run two different programs. A client program to synchronize its system clock with the level1 servers and a server program that waits for any level 3 client for time request. Upon the request it sends its synchronized system time to the client. Level 2 servers also synchronize its system clock with the other level2 servers.

Each level2 server is connected to 4 level1 servers. It polls time from each level1 server and chooses the best time and synchronizes its system clock. How the best time is achieved is described in the section 3 below with code explanation. There are maximum 5 level2 servers with our LLTP model. Each level2 server is connected to 4 different level1 servers. None of the level2 server has the same level1 server set.

*2.5 Level 3 client

Bottom of the LLTP hierarchy is level3 client. It only runs a client program that polls time from the level2 servers. It can only connect/request time from one of our level2 servers at a time. It also requires the knowledge of the IP address of the level2 server. For the load management purpose level 3 clients are not able to connect to a Level1 servers.

= Development Details=

*3.1 Server

As it is mentioned in section-2, there are two servers in our LLTP model. One is level 1 and the other is level 2. Both servers wait for a client with time requests and upon receiving one send the time string. Code structure for both the servers is almost similar with minor technical differences which are explained later in this section. Both servers use a infinite for loop for waiting for the
client. They use a fixed but different port number for the client connection.

[[File:clientntp.jpg]]

Time requests sent to Level 1 server have to contain a message no more than 1 character and it has to be an integer which is sent by the level2 client to distinguish the level1 servers from one another. Level1 server will take that message received from the client and add that at the end of the time string sent to the client. If the level1 server receives a time request message with greater size of 1than it will not return any time value. For the level2 server the maximum message size is
128 characters and it do not have to be an integer. Level2 server discards the message sent from the level3 client. A unique feature of level2 server is that a level3 client time request with “exit” message can shut down the level2 server. This feature is not implemented in level1 servers.

[[File:level1servercode.jpg]]

[[File:level2servercode.jpg]]

Both servers send time as a string which handles the problem with big and little Endean.

*3.2 Client

There are also two clients for our LLTP, level2 client and level3 client. Level3 client is very straight forward where the level3 is a bit more complicated.

Level3 client sends a time request to the level2 server. It cannot send time request to a level1 server as the port number for both server are different and the port numbers are hard coded for both the client (leve3 client/level2 server uses port 123 where level2 client/level1 server uses port 4003). Level2 server also has a restriction over the message sent by the client which is explained above in the server section. Upon receiving the time string back from the server which contains the seconds and milliseconds part together as a string separated by a comma (‘,’), it separates this two parts and checks for the latency and calls function named set_time(long[] ) which sets the system clock time with the new time.

[[File:level3clientcode.jpg]]

Level2 client dose almost the same things but in a broader and sophisticated way. It polls time from four different levle1 time server at the same time. When it sends the time request, it sends a unique message as the server ID to identify the time received from different level1 server which helps to fix the latency later.

[[File:level2clientcode.jpg]]

Level2 client also uses a time out of 1 second. If it dose not receive the time string back from the server then it will ignore the time from that server and choose the best time from the ones it received from the other servers. Received time strings from the different servers also have the server id added at the end of the time string. Using that information level2 client saves the received time in order and fixes the latency accordingly. It keeps all the latency fixed time in an array with the invalid time being -1. Then it calls a different function called marzullo(long []), which takes the array of latency fixed times and returns the best time depending on the Marzullo’s algorithm. Function marzullo() is explained in the section 3.4. Then it uses the set_time(long []) method to set the system clock time.

[[File:level2clientcode2.jpg]]

*3.3 Daemon client and server

The level2 client and server runs as a daemon process to give them the privilege to run continuously in the background. As both the client and server in level2 does not need any user interference and need to run all the time to update the system clock of the running computer from the level1 server.

As it is mentioned above the client process for level2 run continuously and it updates its system clock every 1 minute (polls time from the level1 server every 60 seconds). Server daemon process is built in the same manner but unlike the client process it does not have a waiting time.

[[File:codefordaemon.jpg]]

*3.4 Marzullo’s algorithm and function marzullo(long [])

Marzullo’s algorithm, invented by Keith Marzullo, is an agreement algorithm used to select sources for estimating accurate time from a number of noisy time sources. A modified version of his algorithm, intersection algorithm, is use in NTP. We used the simplified version of this algorithm to choose the best possible time from our time sources (level1 servers).

Our function long* marzullo (long []) takes an array of four times from four different level1 servers and return a pointer to an array with the best seconds and milliseconds. First it takes the given array of times and makes two entries for each time as ±2 milliseconds. For each entry it also adds a type of -1 for -2millisecons and +1 for +2 milliseconds. As a result for 4 time entry it produces an array of 8 time entries with type as ±1. For our LLTP we took ±2milliseconds as our error range. After adding the type the entry will look like <entry – 2, -1> for the beginning of the range and <entry+2, +1> for the end of the range [entry±2].

[[File:IntersectionMarzullasalg.jpg]]

The description of the algorithm uses the following variables: best (largest number of overlapping intervals found), cnt (current number of overlapping intervals), best_start and

best_end (the beginning and end of best interval found so far). We used a two dimensional array to keep the second, millisecond and type of the range. The structure of the array is as follows:

time_entry [i][0] = second part of the time
entry time_entry [i][1] = millisecond part of the time
entry time_entry [i][2] = type of the range (i.e. ±1)

Here ‘Time_entry is the name of the array and ‘i’ is used as an index to traverse the array. After creating the array it follows the following step to determine the accurate time. Then it sends back the best chosen time to the level2 client program.

[[File:section3last.jpg]]

= List of files=

Level1 server:

DieWithError.c
get-time.c
L1Server.c

Level2 server:

daemon_server.c
DieWithError.c
get-time.c
L2Server.c

Level2 client:

daemon_client.c
DieWithError.c
get-time.c
L2Client.c
marzullo.c
print-time.c
selection_sort.c
set-time.c

Level3 client:

DieWithError.c
get-time.c
print-time.c
set-time.c
UDPTimeClient.c

=Discussion/Output=

The most intresting part of testing this was to see how distrubted architect of NTP client-server works and how could it be improved. I learnt alot by setting up and running them and seeing the results. It was fun improving little bit on exsisting implementation of NTP.

Output form different levels of clients and servers are given in this section. Output from level1 server:

[[File:output1.jpg]]

[[File:output2.jpg]]

[[File:output3.jpg]]

[[File:output4.jpg]]

=Conclusion=

Overall, there has been lots of work going on NTP , due to high demand of distributed clinet-server architect based application. This was carried on a very small linux based lab , in future i guess there will be need to test on large industry based scale , where many systems can be connected and than test to see how the delay from clients to server causes disturbance in algorithm. There will be issues for sure since as the information travels it takes time and that delayed time will need to be adjusted more and will be focus of future work.

=References=

*Ntp home page: http://ntp.org Acessed on 13th Feb 2011

*Wikipedia for network time protocol: http://en.wikipedia.org/wiki/Network_Time_Protocol Accessed on 13th Feb 2011

*Wikipedia for Marzullo’s algorithm: http://en.wikipedia.org/wiki/Marzullo's_algorithm Accessed on 13th Feb 2011

DistOS-2011W NTP

2011-02-22T18:30:35Z

Wahmed2: /* Introduction */

=Introduction=

1. Introduction:

==1.1 Context/Background==

Network Time Protocol (NTP) synchronizes clocks of hosts and routers. According to NIST estimates there are 10-20 million NTP servers and clients deployed in the internet and its tributaries all over the world. NTP software has been ported to almost every workstation and server platform available today – from PCs to Cray’s – UNIX, Windows, VMS and embedded systems, even home routers and battery backup systems.

NTP provides nominal accuracies of low tens of milliseconds on WANs, sub milliseconds on LANs and sub microseconds using a precision time source such as cesium oscillator or GPS receiver. Our time protocol was designed over the LAN and our goal was to achieve sub milliseconds of accuracy.

*1.2 Importance

Precision of time is very important in the computer world. Following are some example why it is so important to maintain a precise and accurate time.

Network monitoring, measurement and control, Distributed database transaction journaling and logging, Secure document timestamps with cryptographic certification, Radio and TV programming lunch and monitoring, Stock market buy and sell orders, Distributed network gaming and training, etc.

*1.3 Definition of the problem

My goal was to acquire time accuracy in the Linux lab assuming that all the computers do not have internet connection. To time synchronize all the computers I had to build our own hierarchy of NTP servers and clients.

*1.4 Summary of the result

An efficient hierarchy of NTP server and client has been implemented. Precision of
Sub milliseconds were achieved.

*1.5 Outline of the report
Detailed background information and the structure of our time protocol are provided in Section
2. The development setup and code explanation is reviewed in Section 3. Details of which server/client contains which files are given in section 4. My contributions in this implementation are described in Section 5.

= Detailed context/background information=

In this section the background information of Network Time protocol and the hierarchical model and the comparison with the original NTP designed by Dave Mills is reviewed. For unambiguity I will call the protocol as Linux lab time protocol (LLTP) and the original Network time protocol as NTP.

*2.1 NTP timestamps

The 64-bit timestamps used by NTP consist of a 32-bit second part and a 32-bit fractional second part. I followed the same time structure for our LLTP which gives us a time scale of 232 seconds (136 years) and a theoretical resolution of 2-32 seconds (233 picoseconds).

*2.2 Structure

NTP uses a hierarchical, semi-layered system of levels of clock sources; each level of this hierarchy is termed a stratum and assigned a layer number starting with 0 at the top. The stratum level defines its distance from the reference clock and exists to prevent cyclical dependencies in hierarchy.

My target is to develop a time synchronizing software for a small office or lab with a large number of computer where a very few computer has internet connectivity but all the computers are connected together in a local network. As a result not all the computer is able to connect to internet for time accuracy. LLTP uses similar hierarchical, semi-layered sources as NTP. Each layer of this hierarchy is termed as level and assigned a layer number starting with 1.

*2.3 Level 1 server

Computers with internet connectivity are the level 1 server for our LLTP

Level 1 server

synchronizes its system clock with the NTP servers on the internet. In this hierarchy, only a fixed (5) number of level 1 servers are used. Unlike NTP we don’t have the privilege to use a 10 of thousands of level 1 and level 2 servers, and unlike NTP our LLTP has only 3 level structures.

Picture below shows the connectivity and the hierarchy of LLTP

[[File:LTTPHireachy.jpg]]

*2.4 Level 2 server and client

Level 2 servers run two different programs. A client program to synchronize its system clock with the level1 servers and a server program that waits for any level 3 client for time request. Upon the request it sends its synchronized system time to the client. Level 2 servers also synchronize its system clock with the other level2 servers.

Each level2 server is connected to 4 level1 servers. It polls time from each level1 server and chooses the best time and synchronizes its system clock. How the best time is achieved is described in the section 3 below with code explanation. There are maximum 5 level2 servers with our LLTP model. Each level2 server is connected to 4 different level1 servers. None of the level2 server has the same level1 server set.

*2.5 Level 3 client

Bottom of the LLTP hierarchy is level3 client. It only runs a client program that polls time from the level2 servers. It can only connect/request time from one of our level2 servers at a time. It also requires the knowledge of the IP address of the level2 server. For the load management purpose level 3 clients are not able to connect to a Level1 servers.

= Development Details=

*3.1 Server

As it is mentioned in section-2, there are two servers in our LLTP model. One is level 1 and the other is level 2. Both servers wait for a client with time requests and upon receiving one send the time string. Code structure for both the servers is almost similar with minor technical differences which are explained later in this section. Both servers use a infinite for loop for waiting for the
client. They use a fixed but different port number for the client connection.

[[File:clientntp.jpg]]

Time requests sent to Level 1 server have to contain a message no more than 1 character and it has to be an integer which is sent by the level2 client to distinguish the level1 servers from one another. Level1 server will take that message received from the client and add that at the end of the time string sent to the client. If the level1 server receives a time request message with greater size of 1than it will not return any time value. For the level2 server the maximum message size is
128 characters and it do not have to be an integer. Level2 server discards the message sent from the level3 client. A unique feature of level2 server is that a level3 client time request with “exit” message can shut down the level2 server. This feature is not implemented in level1 servers.

[[File:level1servercode.jpg]]

[[File:level2servercode.jpg]]

Both servers send time as a string which handles the problem with big and little Endean.

*3.2 Client

There are also two clients for our LLTP, level2 client and level3 client. Level3 client is very straight forward where the level3 is a bit more complicated.

Level3 client sends a time request to the level2 server. It cannot send time request to a level1 server as the port number for both server are different and the port numbers are hard coded for both the client (leve3 client/level2 server uses port 123 where level2 client/level1 server uses port 4003). Level2 server also has a restriction over the message sent by the client which is explained above in the server section. Upon receiving the time string back from the server which contains the seconds and milliseconds part together as a string separated by a comma (‘,’), it separates this two parts and checks for the latency and calls function named set_time(long[] ) which sets the system clock time with the new time.

[[File:level3clientcode.jpg]]

Level2 client dose almost the same things but in a broader and sophisticated way. It polls time from four different levle1 time server at the same time. When it sends the time request, it sends a unique message as the server ID to identify the time received from different level1 server which helps to fix the latency later.

[[File:level2clientcode.jpg]]

Level2 client also uses a time out of 1 second. If it dose not receive the time string back from the server then it will ignore the time from that server and choose the best time from the ones it received from the other servers. Received time strings from the different servers also have the server id added at the end of the time string. Using that information level2 client saves the received time in order and fixes the latency accordingly. It keeps all the latency fixed time in an array with the invalid time being -1. Then it calls a different function called marzullo(long []), which takes the array of latency fixed times and returns the best time depending on the Marzullo’s algorithm. Function marzullo() is explained in the section 3.4. Then it uses the set_time(long []) method to set the system clock time.

[[File:level2clientcode2.jpg]]

*3.3 Daemon client and server

The level2 client and server runs as a daemon process to give them the privilege to run continuously in the background. As both the client and server in level2 does not need any user interference and need to run all the time to update the system clock of the running computer from the level1 server.

As it is mentioned above the client process for level2 run continuously and it updates its system clock every 1 minute (polls time from the level1 server every 60 seconds). Server daemon process is built in the same manner but unlike the client process it does not have a waiting time.

[[File:codefordaemon.jpg]]

*3.4 Marzullo’s algorithm and function marzullo(long [])

Marzullo’s algorithm, invented by Keith Marzullo, is an agreement algorithm used to select sources for estimating accurate time from a number of noisy time sources. A modified version of his algorithm, intersection algorithm, is use in NTP. We used the simplified version of this algorithm to choose the best possible time from our time sources (level1 servers).

Our function long* marzullo (long []) takes an array of four times from four different level1 servers and return a pointer to an array with the best seconds and milliseconds. First it takes the given array of times and makes two entries for each time as ±2 milliseconds. For each entry it also adds a type of -1 for -2millisecons and +1 for +2 milliseconds. As a result for 4 time entry it produces an array of 8 time entries with type as ±1. For our LLTP we took ±2milliseconds as our error range. After adding the type the entry will look like <entry – 2, -1> for the beginning of the range and <entry+2, +1> for the end of the range [entry±2].

[[File:IntersectionMarzullasalg.jpg]]

The description of the algorithm uses the following variables: best (largest number of overlapping intervals found), cnt (current number of overlapping intervals), best_start and

best_end (the beginning and end of best interval found so far). We used a two dimensional array to keep the second, millisecond and type of the range. The structure of the array is as follows:

time_entry [i][0] = second part of the time
entry time_entry [i][1] = millisecond part of the time
entry time_entry [i][2] = type of the range (i.e. ±1)

Here ‘Time_entry is the name of the array and ‘i’ is used as an index to traverse the array. After creating the array it follows the following step to determine the accurate time. Then it sends back the best chosen time to the level2 client program.

[[File:section3last.jpg]]

= List of files=

Level1 server:

DieWithError.c
get-time.c
L1Server.c

Level2 server:

daemon_server.c
DieWithError.c
get-time.c
L2Server.c

Level2 client:

daemon_client.c
DieWithError.c
get-time.c
L2Client.c
marzullo.c
print-time.c
selection_sort.c
set-time.c

Level3 client:

DieWithError.c
get-time.c
print-time.c
set-time.c
UDPTimeClient.c

=Discussion/Output=

The most intresting part of testing this was to see how distrubted architect of NTP client-server works and how could it be improved. I learnt alot by setting up and running them and seeing the results. It was fun improving little bit on exsisting implementation of NTP.

Output form different levels of clients and servers are given in this section. Output from level1 server:

[[File:output1.jpg]]

[[File:output2.jpg]]

[[File:output3.jpg]]

[[File:output4.jpg]]

=Conclusion=

Overall, there has been lots of work going on NTP , due to high demand of distributed clinet-server architect based application. This was carried on a very small linux based lab , in future i guess there will be need to test on large industry based scale , where many systems can be connected and than test to see how the delay from clients to server causes disturbance in algorithm. There will be issues for sure since as the information travels it takes time and that delayed time will need to be adjusted more and will be focus of future work.

=References=

*Ntp home page: http://ntp.org Acessed on 13th Feb 2011

*Wikipedia for network time protocol: http://en.wikipedia.org/wiki/Network_Time_Protocol Accessed on 13th Feb 2011

*Wikipedia for Marzullo’s algorithm: http://en.wikipedia.org/wiki/Marzullo's_algorithm Accessed on 13th Feb 2011

DistOS-2011W NTP

2011-02-22T18:29:13Z

Wahmed2: /* 4. List of files */

=Introduction=

1. Introduction:

*1.1 Context/Background

Network Time Protocol (NTP) synchronizes clocks of hosts and routers. According to NIST estimates there are 10-20 million NTP servers and clients deployed in the internet and its tributaries all over the world. NTP software has been ported to almost every workstation and server platform available today – from PCs to Cray’s – UNIX, Windows, VMS and embedded systems, even home routers and battery backup systems.

NTP provides nominal accuracies of low tens of milliseconds on WANs, sub milliseconds on LANs and sub microseconds using a precision time source such as cesium oscillator or GPS receiver. Our time protocol was designed over the LAN and our goal was to achieve sub milliseconds of accuracy.

*1.2 Importance

Precision of time is very important in the computer world. Following are some example why it is so important to maintain a precise and accurate time.

Network monitoring, measurement and control, Distributed database transaction journaling and logging, Secure document timestamps with cryptographic certification, Radio and TV programming lunch and monitoring, Stock market buy and sell orders, Distributed network gaming and training, etc.

*1.3 Definition of the problem

My goal was to acquire time accuracy in the Linux lab assuming that all the computers do not have internet connection. To time synchronize all the computers I had to build our own hierarchy of NTP servers and clients.

*1.4 Summary of the result

An efficient hierarchy of NTP server and client has been implemented. Precision of
Sub milliseconds were achieved.

*1.5 Outline of the report
Detailed background information and the structure of our time protocol are provided in Section
2. The development setup and code explanation is reviewed in Section 3. Details of which server/client contains which files are given in section 4. My contributions in this implementation are described in Section 5.

= Detailed context/background information=

In this section the background information of Network Time protocol and the hierarchical model and the comparison with the original NTP designed by Dave Mills is reviewed. For unambiguity I will call the protocol as Linux lab time protocol (LLTP) and the original Network time protocol as NTP.

*2.1 NTP timestamps

The 64-bit timestamps used by NTP consist of a 32-bit second part and a 32-bit fractional second part. I followed the same time structure for our LLTP which gives us a time scale of 232 seconds (136 years) and a theoretical resolution of 2-32 seconds (233 picoseconds).

*2.2 Structure

NTP uses a hierarchical, semi-layered system of levels of clock sources; each level of this hierarchy is termed a stratum and assigned a layer number starting with 0 at the top. The stratum level defines its distance from the reference clock and exists to prevent cyclical dependencies in hierarchy.

My target is to develop a time synchronizing software for a small office or lab with a large number of computer where a very few computer has internet connectivity but all the computers are connected together in a local network. As a result not all the computer is able to connect to internet for time accuracy. LLTP uses similar hierarchical, semi-layered sources as NTP. Each layer of this hierarchy is termed as level and assigned a layer number starting with 1.

*2.3 Level 1 server

Computers with internet connectivity are the level 1 server for our LLTP

Level 1 server

synchronizes its system clock with the NTP servers on the internet. In this hierarchy, only a fixed (5) number of level 1 servers are used. Unlike NTP we don’t have the privilege to use a 10 of thousands of level 1 and level 2 servers, and unlike NTP our LLTP has only 3 level structures.

Picture below shows the connectivity and the hierarchy of LLTP

[[File:LTTPHireachy.jpg]]

*2.4 Level 2 server and client

Level 2 servers run two different programs. A client program to synchronize its system clock with the level1 servers and a server program that waits for any level 3 client for time request. Upon the request it sends its synchronized system time to the client. Level 2 servers also synchronize its system clock with the other level2 servers.

Each level2 server is connected to 4 level1 servers. It polls time from each level1 server and chooses the best time and synchronizes its system clock. How the best time is achieved is described in the section 3 below with code explanation. There are maximum 5 level2 servers with our LLTP model. Each level2 server is connected to 4 different level1 servers. None of the level2 server has the same level1 server set.

*2.5 Level 3 client

Bottom of the LLTP hierarchy is level3 client. It only runs a client program that polls time from the level2 servers. It can only connect/request time from one of our level2 servers at a time. It also requires the knowledge of the IP address of the level2 server. For the load management purpose level 3 clients are not able to connect to a Level1 servers.

= Development Details=

*3.1 Server

As it is mentioned in section-2, there are two servers in our LLTP model. One is level 1 and the other is level 2. Both servers wait for a client with time requests and upon receiving one send the time string. Code structure for both the servers is almost similar with minor technical differences which are explained later in this section. Both servers use a infinite for loop for waiting for the
client. They use a fixed but different port number for the client connection.

[[File:clientntp.jpg]]

Time requests sent to Level 1 server have to contain a message no more than 1 character and it has to be an integer which is sent by the level2 client to distinguish the level1 servers from one another. Level1 server will take that message received from the client and add that at the end of the time string sent to the client. If the level1 server receives a time request message with greater size of 1than it will not return any time value. For the level2 server the maximum message size is
128 characters and it do not have to be an integer. Level2 server discards the message sent from the level3 client. A unique feature of level2 server is that a level3 client time request with “exit” message can shut down the level2 server. This feature is not implemented in level1 servers.

[[File:level1servercode.jpg]]

[[File:level2servercode.jpg]]

Both servers send time as a string which handles the problem with big and little Endean.

*3.2 Client

There are also two clients for our LLTP, level2 client and level3 client. Level3 client is very straight forward where the level3 is a bit more complicated.

Level3 client sends a time request to the level2 server. It cannot send time request to a level1 server as the port number for both server are different and the port numbers are hard coded for both the client (leve3 client/level2 server uses port 123 where level2 client/level1 server uses port 4003). Level2 server also has a restriction over the message sent by the client which is explained above in the server section. Upon receiving the time string back from the server which contains the seconds and milliseconds part together as a string separated by a comma (‘,’), it separates this two parts and checks for the latency and calls function named set_time(long[] ) which sets the system clock time with the new time.

[[File:level3clientcode.jpg]]

Level2 client dose almost the same things but in a broader and sophisticated way. It polls time from four different levle1 time server at the same time. When it sends the time request, it sends a unique message as the server ID to identify the time received from different level1 server which helps to fix the latency later.

[[File:level2clientcode.jpg]]

Level2 client also uses a time out of 1 second. If it dose not receive the time string back from the server then it will ignore the time from that server and choose the best time from the ones it received from the other servers. Received time strings from the different servers also have the server id added at the end of the time string. Using that information level2 client saves the received time in order and fixes the latency accordingly. It keeps all the latency fixed time in an array with the invalid time being -1. Then it calls a different function called marzullo(long []), which takes the array of latency fixed times and returns the best time depending on the Marzullo’s algorithm. Function marzullo() is explained in the section 3.4. Then it uses the set_time(long []) method to set the system clock time.

[[File:level2clientcode2.jpg]]

*3.3 Daemon client and server

The level2 client and server runs as a daemon process to give them the privilege to run continuously in the background. As both the client and server in level2 does not need any user interference and need to run all the time to update the system clock of the running computer from the level1 server.

As it is mentioned above the client process for level2 run continuously and it updates its system clock every 1 minute (polls time from the level1 server every 60 seconds). Server daemon process is built in the same manner but unlike the client process it does not have a waiting time.

[[File:codefordaemon.jpg]]

*3.4 Marzullo’s algorithm and function marzullo(long [])

Marzullo’s algorithm, invented by Keith Marzullo, is an agreement algorithm used to select sources for estimating accurate time from a number of noisy time sources. A modified version of his algorithm, intersection algorithm, is use in NTP. We used the simplified version of this algorithm to choose the best possible time from our time sources (level1 servers).

Our function long* marzullo (long []) takes an array of four times from four different level1 servers and return a pointer to an array with the best seconds and milliseconds. First it takes the given array of times and makes two entries for each time as ±2 milliseconds. For each entry it also adds a type of -1 for -2millisecons and +1 for +2 milliseconds. As a result for 4 time entry it produces an array of 8 time entries with type as ±1. For our LLTP we took ±2milliseconds as our error range. After adding the type the entry will look like <entry – 2, -1> for the beginning of the range and <entry+2, +1> for the end of the range [entry±2].

[[File:IntersectionMarzullasalg.jpg]]

The description of the algorithm uses the following variables: best (largest number of overlapping intervals found), cnt (current number of overlapping intervals), best_start and

best_end (the beginning and end of best interval found so far). We used a two dimensional array to keep the second, millisecond and type of the range. The structure of the array is as follows:

time_entry [i][0] = second part of the time
entry time_entry [i][1] = millisecond part of the time
entry time_entry [i][2] = type of the range (i.e. ±1)

Here ‘Time_entry is the name of the array and ‘i’ is used as an index to traverse the array. After creating the array it follows the following step to determine the accurate time. Then it sends back the best chosen time to the level2 client program.

[[File:section3last.jpg]]

= List of files=

Level1 server:

DieWithError.c
get-time.c
L1Server.c

Level2 server:

daemon_server.c
DieWithError.c
get-time.c
L2Server.c

Level2 client:

daemon_client.c
DieWithError.c
get-time.c
L2Client.c
marzullo.c
print-time.c
selection_sort.c
set-time.c

Level3 client:

DieWithError.c
get-time.c
print-time.c
set-time.c
UDPTimeClient.c

=Discussion/Output=

The most intresting part of testing this was to see how distrubted architect of NTP client-server works and how could it be improved. I learnt alot by setting up and running them and seeing the results. It was fun improving little bit on exsisting implementation of NTP.

Output form different levels of clients and servers are given in this section. Output from level1 server:

[[File:output1.jpg]]

[[File:output2.jpg]]

[[File:output3.jpg]]

[[File:output4.jpg]]

=Conclusion=

Overall, there has been lots of work going on NTP , due to high demand of distributed clinet-server architect based application. This was carried on a very small linux based lab , in future i guess there will be need to test on large industry based scale , where many systems can be connected and than test to see how the delay from clients to server causes disturbance in algorithm. There will be issues for sure since as the information travels it takes time and that delayed time will need to be adjusted more and will be focus of future work.

=References=

*Ntp home page: http://ntp.org Acessed on 13th Feb 2011

*Wikipedia for network time protocol: http://en.wikipedia.org/wiki/Network_Time_Protocol Accessed on 13th Feb 2011

*Wikipedia for Marzullo’s algorithm: http://en.wikipedia.org/wiki/Marzullo's_algorithm Accessed on 13th Feb 2011

DistOS-2011W NTP

2011-02-22T18:28:28Z

Wahmed2: /* 3.1 Server */

=Introduction=

1. Introduction:

*1.1 Context/Background

Network Time Protocol (NTP) synchronizes clocks of hosts and routers. According to NIST estimates there are 10-20 million NTP servers and clients deployed in the internet and its tributaries all over the world. NTP software has been ported to almost every workstation and server platform available today – from PCs to Cray’s – UNIX, Windows, VMS and embedded systems, even home routers and battery backup systems.

NTP provides nominal accuracies of low tens of milliseconds on WANs, sub milliseconds on LANs and sub microseconds using a precision time source such as cesium oscillator or GPS receiver. Our time protocol was designed over the LAN and our goal was to achieve sub milliseconds of accuracy.

*1.2 Importance

Precision of time is very important in the computer world. Following are some example why it is so important to maintain a precise and accurate time.

Network monitoring, measurement and control, Distributed database transaction journaling and logging, Secure document timestamps with cryptographic certification, Radio and TV programming lunch and monitoring, Stock market buy and sell orders, Distributed network gaming and training, etc.

*1.3 Definition of the problem

My goal was to acquire time accuracy in the Linux lab assuming that all the computers do not have internet connection. To time synchronize all the computers I had to build our own hierarchy of NTP servers and clients.

*1.4 Summary of the result

An efficient hierarchy of NTP server and client has been implemented. Precision of
Sub milliseconds were achieved.

*1.5 Outline of the report
Detailed background information and the structure of our time protocol are provided in Section
2. The development setup and code explanation is reviewed in Section 3. Details of which server/client contains which files are given in section 4. My contributions in this implementation are described in Section 5.

= Detailed context/background information=

In this section the background information of Network Time protocol and the hierarchical model and the comparison with the original NTP designed by Dave Mills is reviewed. For unambiguity I will call the protocol as Linux lab time protocol (LLTP) and the original Network time protocol as NTP.

*2.1 NTP timestamps

The 64-bit timestamps used by NTP consist of a 32-bit second part and a 32-bit fractional second part. I followed the same time structure for our LLTP which gives us a time scale of 232 seconds (136 years) and a theoretical resolution of 2-32 seconds (233 picoseconds).

*2.2 Structure

NTP uses a hierarchical, semi-layered system of levels of clock sources; each level of this hierarchy is termed a stratum and assigned a layer number starting with 0 at the top. The stratum level defines its distance from the reference clock and exists to prevent cyclical dependencies in hierarchy.

My target is to develop a time synchronizing software for a small office or lab with a large number of computer where a very few computer has internet connectivity but all the computers are connected together in a local network. As a result not all the computer is able to connect to internet for time accuracy. LLTP uses similar hierarchical, semi-layered sources as NTP. Each layer of this hierarchy is termed as level and assigned a layer number starting with 1.

*2.3 Level 1 server

Computers with internet connectivity are the level 1 server for our LLTP

Level 1 server

synchronizes its system clock with the NTP servers on the internet. In this hierarchy, only a fixed (5) number of level 1 servers are used. Unlike NTP we don’t have the privilege to use a 10 of thousands of level 1 and level 2 servers, and unlike NTP our LLTP has only 3 level structures.

Picture below shows the connectivity and the hierarchy of LLTP

[[File:LTTPHireachy.jpg]]

*2.4 Level 2 server and client

Level 2 servers run two different programs. A client program to synchronize its system clock with the level1 servers and a server program that waits for any level 3 client for time request. Upon the request it sends its synchronized system time to the client. Level 2 servers also synchronize its system clock with the other level2 servers.

Each level2 server is connected to 4 level1 servers. It polls time from each level1 server and chooses the best time and synchronizes its system clock. How the best time is achieved is described in the section 3 below with code explanation. There are maximum 5 level2 servers with our LLTP model. Each level2 server is connected to 4 different level1 servers. None of the level2 server has the same level1 server set.

*2.5 Level 3 client

Bottom of the LLTP hierarchy is level3 client. It only runs a client program that polls time from the level2 servers. It can only connect/request time from one of our level2 servers at a time. It also requires the knowledge of the IP address of the level2 server. For the load management purpose level 3 clients are not able to connect to a Level1 servers.

= Development Details=

*3.1 Server

As it is mentioned in section-2, there are two servers in our LLTP model. One is level 1 and the other is level 2. Both servers wait for a client with time requests and upon receiving one send the time string. Code structure for both the servers is almost similar with minor technical differences which are explained later in this section. Both servers use a infinite for loop for waiting for the
client. They use a fixed but different port number for the client connection.

[[File:clientntp.jpg]]

Time requests sent to Level 1 server have to contain a message no more than 1 character and it has to be an integer which is sent by the level2 client to distinguish the level1 servers from one another. Level1 server will take that message received from the client and add that at the end of the time string sent to the client. If the level1 server receives a time request message with greater size of 1than it will not return any time value. For the level2 server the maximum message size is
128 characters and it do not have to be an integer. Level2 server discards the message sent from the level3 client. A unique feature of level2 server is that a level3 client time request with “exit” message can shut down the level2 server. This feature is not implemented in level1 servers.

[[File:level1servercode.jpg]]

[[File:level2servercode.jpg]]

Both servers send time as a string which handles the problem with big and little Endean.

*3.2 Client

There are also two clients for our LLTP, level2 client and level3 client. Level3 client is very straight forward where the level3 is a bit more complicated.

Level3 client sends a time request to the level2 server. It cannot send time request to a level1 server as the port number for both server are different and the port numbers are hard coded for both the client (leve3 client/level2 server uses port 123 where level2 client/level1 server uses port 4003). Level2 server also has a restriction over the message sent by the client which is explained above in the server section. Upon receiving the time string back from the server which contains the seconds and milliseconds part together as a string separated by a comma (‘,’), it separates this two parts and checks for the latency and calls function named set_time(long[] ) which sets the system clock time with the new time.

[[File:level3clientcode.jpg]]

Level2 client dose almost the same things but in a broader and sophisticated way. It polls time from four different levle1 time server at the same time. When it sends the time request, it sends a unique message as the server ID to identify the time received from different level1 server which helps to fix the latency later.

[[File:level2clientcode.jpg]]

Level2 client also uses a time out of 1 second. If it dose not receive the time string back from the server then it will ignore the time from that server and choose the best time from the ones it received from the other servers. Received time strings from the different servers also have the server id added at the end of the time string. Using that information level2 client saves the received time in order and fixes the latency accordingly. It keeps all the latency fixed time in an array with the invalid time being -1. Then it calls a different function called marzullo(long []), which takes the array of latency fixed times and returns the best time depending on the Marzullo’s algorithm. Function marzullo() is explained in the section 3.4. Then it uses the set_time(long []) method to set the system clock time.

[[File:level2clientcode2.jpg]]

*3.3 Daemon client and server

The level2 client and server runs as a daemon process to give them the privilege to run continuously in the background. As both the client and server in level2 does not need any user interference and need to run all the time to update the system clock of the running computer from the level1 server.

As it is mentioned above the client process for level2 run continuously and it updates its system clock every 1 minute (polls time from the level1 server every 60 seconds). Server daemon process is built in the same manner but unlike the client process it does not have a waiting time.

[[File:codefordaemon.jpg]]

*3.4 Marzullo’s algorithm and function marzullo(long [])

Marzullo’s algorithm, invented by Keith Marzullo, is an agreement algorithm used to select sources for estimating accurate time from a number of noisy time sources. A modified version of his algorithm, intersection algorithm, is use in NTP. We used the simplified version of this algorithm to choose the best possible time from our time sources (level1 servers).

Our function long* marzullo (long []) takes an array of four times from four different level1 servers and return a pointer to an array with the best seconds and milliseconds. First it takes the given array of times and makes two entries for each time as ±2 milliseconds. For each entry it also adds a type of -1 for -2millisecons and +1 for +2 milliseconds. As a result for 4 time entry it produces an array of 8 time entries with type as ±1. For our LLTP we took ±2milliseconds as our error range. After adding the type the entry will look like <entry – 2, -1> for the beginning of the range and <entry+2, +1> for the end of the range [entry±2].

[[File:IntersectionMarzullasalg.jpg]]

The description of the algorithm uses the following variables: best (largest number of overlapping intervals found), cnt (current number of overlapping intervals), best_start and

best_end (the beginning and end of best interval found so far). We used a two dimensional array to keep the second, millisecond and type of the range. The structure of the array is as follows:

time_entry [i][0] = second part of the time
entry time_entry [i][1] = millisecond part of the time
entry time_entry [i][2] = type of the range (i.e. ±1)

Here ‘Time_entry is the name of the array and ‘i’ is used as an index to traverse the array. After creating the array it follows the following step to determine the accurate time. Then it sends back the best chosen time to the level2 client program.

[[File:section3last.jpg]]

=4. List of files=

Level1 server:

DieWithError.c
get-time.c
L1Server.c

Level2 server:

daemon_server.c
DieWithError.c
get-time.c
L2Server.c

Level2 client:

daemon_client.c
DieWithError.c
get-time.c
L2Client.c
marzullo.c
print-time.c
selection_sort.c
set-time.c

Level3 client:

DieWithError.c
get-time.c
print-time.c
set-time.c
UDPTimeClient.c

=Discussion/Output=

The most intresting part of testing this was to see how distrubted architect of NTP client-server works and how could it be improved. I learnt alot by setting up and running them and seeing the results. It was fun improving little bit on exsisting implementation of NTP.

Output form different levels of clients and servers are given in this section. Output from level1 server:

[[File:output1.jpg]]

[[File:output2.jpg]]

[[File:output3.jpg]]

[[File:output4.jpg]]

=Conclusion=

Overall, there has been lots of work going on NTP , due to high demand of distributed clinet-server architect based application. This was carried on a very small linux based lab , in future i guess there will be need to test on large industry based scale , where many systems can be connected and than test to see how the delay from clients to server causes disturbance in algorithm. There will be issues for sure since as the information travels it takes time and that delayed time will need to be adjusted more and will be focus of future work.

=References=

*Ntp home page: http://ntp.org Acessed on 13th Feb 2011

*Wikipedia for network time protocol: http://en.wikipedia.org/wiki/Network_Time_Protocol Accessed on 13th Feb 2011

*Wikipedia for Marzullo’s algorithm: http://en.wikipedia.org/wiki/Marzullo's_algorithm Accessed on 13th Feb 2011

DistOS-2011W NTP

2011-02-22T18:27:14Z

Wahmed2: /* 3. Development Details */

=Introduction=

1. Introduction:

*1.1 Context/Background

Network Time Protocol (NTP) synchronizes clocks of hosts and routers. According to NIST estimates there are 10-20 million NTP servers and clients deployed in the internet and its tributaries all over the world. NTP software has been ported to almost every workstation and server platform available today – from PCs to Cray’s – UNIX, Windows, VMS and embedded systems, even home routers and battery backup systems.

NTP provides nominal accuracies of low tens of milliseconds on WANs, sub milliseconds on LANs and sub microseconds using a precision time source such as cesium oscillator or GPS receiver. Our time protocol was designed over the LAN and our goal was to achieve sub milliseconds of accuracy.

*1.2 Importance

Precision of time is very important in the computer world. Following are some example why it is so important to maintain a precise and accurate time.

Network monitoring, measurement and control, Distributed database transaction journaling and logging, Secure document timestamps with cryptographic certification, Radio and TV programming lunch and monitoring, Stock market buy and sell orders, Distributed network gaming and training, etc.

*1.3 Definition of the problem

My goal was to acquire time accuracy in the Linux lab assuming that all the computers do not have internet connection. To time synchronize all the computers I had to build our own hierarchy of NTP servers and clients.

*1.4 Summary of the result

An efficient hierarchy of NTP server and client has been implemented. Precision of
Sub milliseconds were achieved.

*1.5 Outline of the report
Detailed background information and the structure of our time protocol are provided in Section
2. The development setup and code explanation is reviewed in Section 3. Details of which server/client contains which files are given in section 4. My contributions in this implementation are described in Section 5.

= Detailed context/background information=

In this section the background information of Network Time protocol and the hierarchical model and the comparison with the original NTP designed by Dave Mills is reviewed. For unambiguity I will call the protocol as Linux lab time protocol (LLTP) and the original Network time protocol as NTP.

*2.1 NTP timestamps

The 64-bit timestamps used by NTP consist of a 32-bit second part and a 32-bit fractional second part. I followed the same time structure for our LLTP which gives us a time scale of 232 seconds (136 years) and a theoretical resolution of 2-32 seconds (233 picoseconds).

*2.2 Structure

NTP uses a hierarchical, semi-layered system of levels of clock sources; each level of this hierarchy is termed a stratum and assigned a layer number starting with 0 at the top. The stratum level defines its distance from the reference clock and exists to prevent cyclical dependencies in hierarchy.

My target is to develop a time synchronizing software for a small office or lab with a large number of computer where a very few computer has internet connectivity but all the computers are connected together in a local network. As a result not all the computer is able to connect to internet for time accuracy. LLTP uses similar hierarchical, semi-layered sources as NTP. Each layer of this hierarchy is termed as level and assigned a layer number starting with 1.

*2.3 Level 1 server

Computers with internet connectivity are the level 1 server for our LLTP

Level 1 server

synchronizes its system clock with the NTP servers on the internet. In this hierarchy, only a fixed (5) number of level 1 servers are used. Unlike NTP we don’t have the privilege to use a 10 of thousands of level 1 and level 2 servers, and unlike NTP our LLTP has only 3 level structures.

Picture below shows the connectivity and the hierarchy of LLTP

[[File:LTTPHireachy.jpg]]

*2.4 Level 2 server and client

Level 2 servers run two different programs. A client program to synchronize its system clock with the level1 servers and a server program that waits for any level 3 client for time request. Upon the request it sends its synchronized system time to the client. Level 2 servers also synchronize its system clock with the other level2 servers.

Each level2 server is connected to 4 level1 servers. It polls time from each level1 server and chooses the best time and synchronizes its system clock. How the best time is achieved is described in the section 3 below with code explanation. There are maximum 5 level2 servers with our LLTP model. Each level2 server is connected to 4 different level1 servers. None of the level2 server has the same level1 server set.

*2.5 Level 3 client

Bottom of the LLTP hierarchy is level3 client. It only runs a client program that polls time from the level2 servers. It can only connect/request time from one of our level2 servers at a time. It also requires the knowledge of the IP address of the level2 server. For the load management purpose level 3 clients are not able to connect to a Level1 servers.

= Development Details=

=3.1 Server=

As it is mentioned in section-2, there are two servers in our LLTP model. One is level 1 and the other is level 2. Both servers wait for a client with time requests and upon receiving one send the time string. Code structure for both the servers is almost similar with minor technical differences which are explained later in this section. Both servers use a infinite for loop for waiting for the
client. They use a fixed but different port number for the client connection.

[[File:clientntp.jpg]]

Time requests sent to Level 1 server have to contain a message no more than 1 character and it has to be an integer which is sent by the level2 client to distinguish the level1 servers from one another. Level1 server will take that message received from the client and add that at the end of the time string sent to the client. If the level1 server receives a time request message with greater size of 1than it will not return any time value. For the level2 server the maximum message size is
128 characters and it do not have to be an integer. Level2 server discards the message sent from the level3 client. A unique feature of level2 server is that a level3 client time request with “exit” message can shut down the level2 server. This feature is not implemented in level1 servers.

[[File:level1servercode.jpg]]

[[File:level2servercode.jpg]]

Both servers send time as a string which handles the problem with big and little Endean.

3.2 Client

There are also two clients for our LLTP, level2 client and level3 client. Level3 client is very straight forward where the level3 is a bit more complicated.

Level3 client sends a time request to the level2 server. It cannot send time request to a level1 server as the port number for both server are different and the port numbers are hard coded for both the client (leve3 client/level2 server uses port 123 where level2 client/level1 server uses port 4003). Level2 server also has a restriction over the message sent by the client which is explained above in the server section. Upon receiving the time string back from the server which contains the seconds and milliseconds part together as a string separated by a comma (‘,’), it separates this two parts and checks for the latency and calls function named set_time(long[] ) which sets the system clock time with the new time.

[[File:level3clientcode.jpg]]

Level2 client dose almost the same things but in a broader and sophisticated way. It polls time from four different levle1 time server at the same time. When it sends the time request, it sends a unique message as the server ID to identify the time received from different level1 server which helps to fix the latency later.

[[File:level2clientcode.jpg]]

Level2 client also uses a time out of 1 second. If it dose not receive the time string back from the server then it will ignore the time from that server and choose the best time from the ones it received from the other servers. Received time strings from the different servers also have the server id added at the end of the time string. Using that information level2 client saves the received time in order and fixes the latency accordingly. It keeps all the latency fixed time in an array with the invalid time being -1. Then it calls a different function called marzullo(long []), which takes the array of latency fixed times and returns the best time depending on the Marzullo’s algorithm. Function marzullo() is explained in the section 3.4. Then it uses the set_time(long []) method to set the system clock time.

[[File:level2clientcode2.jpg]]

3.3 Daemon client and server

The level2 client and server runs as a daemon process to give them the privilege to run continuously in the background. As both the client and server in level2 does not need any user interference and need to run all the time to update the system clock of the running computer from the level1 server.

As it is mentioned above the client process for level2 run continuously and it updates its system clock every 1 minute (polls time from the level1 server every 60 seconds). Server daemon process is built in the same manner but unlike the client process it does not have a waiting time.

[[File:codefordaemon.jpg]]

3.4 Marzullo’s algorithm and function marzullo(long [])

Marzullo’s algorithm, invented by Keith Marzullo, is an agreement algorithm used to select sources for estimating accurate time from a number of noisy time sources. A modified version of his algorithm, intersection algorithm, is use in NTP. We used the simplified version of this algorithm to choose the best possible time from our time sources (level1 servers).

Our function long* marzullo (long []) takes an array of four times from four different level1 servers and return a pointer to an array with the best seconds and milliseconds. First it takes the given array of times and makes two entries for each time as ±2 milliseconds. For each entry it also adds a type of -1 for -2millisecons and +1 for +2 milliseconds. As a result for 4 time entry it produces an array of 8 time entries with type as ±1. For our LLTP we took ±2milliseconds as our error range. After adding the type the entry will look like <entry – 2, -1> for the beginning of the range and <entry+2, +1> for the end of the range [entry±2].

[[File:IntersectionMarzullasalg.jpg]]

The description of the algorithm uses the following variables: best (largest number of overlapping intervals found), cnt (current number of overlapping intervals), best_start and

best_end (the beginning and end of best interval found so far). We used a two dimensional array to keep the second, millisecond and type of the range. The structure of the array is as follows:

time_entry [i][0] = second part of the time
entry time_entry [i][1] = millisecond part of the time
entry time_entry [i][2] = type of the range (i.e. ±1)

Here ‘Time_entry is the name of the array and ‘i’ is used as an index to traverse the array. After creating the array it follows the following step to determine the accurate time. Then it sends back the best chosen time to the level2 client program.

[[File:section3last.jpg]]

=4. List of files=

Level1 server:

DieWithError.c
get-time.c
L1Server.c

Level2 server:

daemon_server.c
DieWithError.c
get-time.c
L2Server.c

Level2 client:

daemon_client.c
DieWithError.c
get-time.c
L2Client.c
marzullo.c
print-time.c
selection_sort.c
set-time.c

Level3 client:

DieWithError.c
get-time.c
print-time.c
set-time.c
UDPTimeClient.c

=Discussion/Output=

The most intresting part of testing this was to see how distrubted architect of NTP client-server works and how could it be improved. I learnt alot by setting up and running them and seeing the results. It was fun improving little bit on exsisting implementation of NTP.

Output form different levels of clients and servers are given in this section. Output from level1 server:

[[File:output1.jpg]]

[[File:output2.jpg]]

[[File:output3.jpg]]

[[File:output4.jpg]]

=Conclusion=

Overall, there has been lots of work going on NTP , due to high demand of distributed clinet-server architect based application. This was carried on a very small linux based lab , in future i guess there will be need to test on large industry based scale , where many systems can be connected and than test to see how the delay from clients to server causes disturbance in algorithm. There will be issues for sure since as the information travels it takes time and that delayed time will need to be adjusted more and will be focus of future work.

=References=

*Ntp home page: http://ntp.org Acessed on 13th Feb 2011

*Wikipedia for network time protocol: http://en.wikipedia.org/wiki/Network_Time_Protocol Accessed on 13th Feb 2011

*Wikipedia for Marzullo’s algorithm: http://en.wikipedia.org/wiki/Marzullo's_algorithm Accessed on 13th Feb 2011

DistOS-2011W NTP

2011-02-22T18:25:10Z

Wahmed2: /* 2. Detailed context/background information */

=Introduction=

1. Introduction:

*1.1 Context/Background

Network Time Protocol (NTP) synchronizes clocks of hosts and routers. According to NIST estimates there are 10-20 million NTP servers and clients deployed in the internet and its tributaries all over the world. NTP software has been ported to almost every workstation and server platform available today – from PCs to Cray’s – UNIX, Windows, VMS and embedded systems, even home routers and battery backup systems.

NTP provides nominal accuracies of low tens of milliseconds on WANs, sub milliseconds on LANs and sub microseconds using a precision time source such as cesium oscillator or GPS receiver. Our time protocol was designed over the LAN and our goal was to achieve sub milliseconds of accuracy.

*1.2 Importance

Precision of time is very important in the computer world. Following are some example why it is so important to maintain a precise and accurate time.

Network monitoring, measurement and control, Distributed database transaction journaling and logging, Secure document timestamps with cryptographic certification, Radio and TV programming lunch and monitoring, Stock market buy and sell orders, Distributed network gaming and training, etc.

*1.3 Definition of the problem

My goal was to acquire time accuracy in the Linux lab assuming that all the computers do not have internet connection. To time synchronize all the computers I had to build our own hierarchy of NTP servers and clients.

*1.4 Summary of the result

An efficient hierarchy of NTP server and client has been implemented. Precision of
Sub milliseconds were achieved.

*1.5 Outline of the report
Detailed background information and the structure of our time protocol are provided in Section
2. The development setup and code explanation is reviewed in Section 3. Details of which server/client contains which files are given in section 4. My contributions in this implementation are described in Section 5.

= Detailed context/background information=

In this section the background information of Network Time protocol and the hierarchical model and the comparison with the original NTP designed by Dave Mills is reviewed. For unambiguity I will call the protocol as Linux lab time protocol (LLTP) and the original Network time protocol as NTP.

*2.1 NTP timestamps

The 64-bit timestamps used by NTP consist of a 32-bit second part and a 32-bit fractional second part. I followed the same time structure for our LLTP which gives us a time scale of 232 seconds (136 years) and a theoretical resolution of 2-32 seconds (233 picoseconds).

*2.2 Structure

NTP uses a hierarchical, semi-layered system of levels of clock sources; each level of this hierarchy is termed a stratum and assigned a layer number starting with 0 at the top. The stratum level defines its distance from the reference clock and exists to prevent cyclical dependencies in hierarchy.

My target is to develop a time synchronizing software for a small office or lab with a large number of computer where a very few computer has internet connectivity but all the computers are connected together in a local network. As a result not all the computer is able to connect to internet for time accuracy. LLTP uses similar hierarchical, semi-layered sources as NTP. Each layer of this hierarchy is termed as level and assigned a layer number starting with 1.

*2.3 Level 1 server

Computers with internet connectivity are the level 1 server for our LLTP

Level 1 server

synchronizes its system clock with the NTP servers on the internet. In this hierarchy, only a fixed (5) number of level 1 servers are used. Unlike NTP we don’t have the privilege to use a 10 of thousands of level 1 and level 2 servers, and unlike NTP our LLTP has only 3 level structures.

Picture below shows the connectivity and the hierarchy of LLTP

[[File:LTTPHireachy.jpg]]

*2.4 Level 2 server and client

Level 2 servers run two different programs. A client program to synchronize its system clock with the level1 servers and a server program that waits for any level 3 client for time request. Upon the request it sends its synchronized system time to the client. Level 2 servers also synchronize its system clock with the other level2 servers.

Each level2 server is connected to 4 level1 servers. It polls time from each level1 server and chooses the best time and synchronizes its system clock. How the best time is achieved is described in the section 3 below with code explanation. There are maximum 5 level2 servers with our LLTP model. Each level2 server is connected to 4 different level1 servers. None of the level2 server has the same level1 server set.

*2.5 Level 3 client

Bottom of the LLTP hierarchy is level3 client. It only runs a client program that polls time from the level2 servers. It can only connect/request time from one of our level2 servers at a time. It also requires the knowledge of the IP address of the level2 server. For the load management purpose level 3 clients are not able to connect to a Level1 servers.

=3. Development Details=

3.1 Server

As it is mentioned in section-2, there are two servers in our LLTP model. One is level 1 and the other is level 2. Both servers wait for a client with time requests and upon receiving one send the time string. Code structure for both the servers is almost similar with minor technical differences which are explained later in this section. Both servers use a infinite for loop for waiting for the
client. They use a fixed but different port number for the client connection.

[[File:clientntp.jpg]]

Time requests sent to Level 1 server have to contain a message no more than 1 character and it has to be an integer which is sent by the level2 client to distinguish the level1 servers from one another. Level1 server will take that message received from the client and add that at the end of the time string sent to the client. If the level1 server receives a time request message with greater size of 1than it will not return any time value. For the level2 server the maximum message size is
128 characters and it do not have to be an integer. Level2 server discards the message sent from the level3 client. A unique feature of level2 server is that a level3 client time request with “exit” message can shut down the level2 server. This feature is not implemented in level1 servers.

[[File:level1servercode.jpg]]

[[File:level2servercode.jpg]]

Both servers send time as a string which handles the problem with big and little Endean.

3.2 Client

There are also two clients for our LLTP, level2 client and level3 client. Level3 client is very straight forward where the level3 is a bit more complicated.

Level3 client sends a time request to the level2 server. It cannot send time request to a level1 server as the port number for both server are different and the port numbers are hard coded for both the client (leve3 client/level2 server uses port 123 where level2 client/level1 server uses port 4003). Level2 server also has a restriction over the message sent by the client which is explained above in the server section. Upon receiving the time string back from the server which contains the seconds and milliseconds part together as a string separated by a comma (‘,’), it separates this two parts and checks for the latency and calls function named set_time(long[] ) which sets the system clock time with the new time.

[[File:level3clientcode.jpg]]

Level2 client dose almost the same things but in a broader and sophisticated way. It polls time from four different levle1 time server at the same time. When it sends the time request, it sends a unique message as the server ID to identify the time received from different level1 server which helps to fix the latency later.

[[File:level2clientcode.jpg]]

Level2 client also uses a time out of 1 second. If it dose not receive the time string back from the server then it will ignore the time from that server and choose the best time from the ones it received from the other servers. Received time strings from the different servers also have the server id added at the end of the time string. Using that information level2 client saves the received time in order and fixes the latency accordingly. It keeps all the latency fixed time in an array with the invalid time being -1. Then it calls a different function called marzullo(long []), which takes the array of latency fixed times and returns the best time depending on the Marzullo’s algorithm. Function marzullo() is explained in the section 3.4. Then it uses the set_time(long []) method to set the system clock time.

[[File:level2clientcode2.jpg]]

3.3 Daemon client and server

The level2 client and server runs as a daemon process to give them the privilege to run continuously in the background. As both the client and server in level2 does not need any user interference and need to run all the time to update the system clock of the running computer from the level1 server.

As it is mentioned above the client process for level2 run continuously and it updates its system clock every 1 minute (polls time from the level1 server every 60 seconds). Server daemon process is built in the same manner but unlike the client process it does not have a waiting time.

[[File:codefordaemon.jpg]]

3.4 Marzullo’s algorithm and function marzullo(long [])

Marzullo’s algorithm, invented by Keith Marzullo, is an agreement algorithm used to select sources for estimating accurate time from a number of noisy time sources. A modified version of his algorithm, intersection algorithm, is use in NTP. We used the simplified version of this algorithm to choose the best possible time from our time sources (level1 servers).

Our function long* marzullo (long []) takes an array of four times from four different level1 servers and return a pointer to an array with the best seconds and milliseconds. First it takes the given array of times and makes two entries for each time as ±2 milliseconds. For each entry it also adds a type of -1 for -2millisecons and +1 for +2 milliseconds. As a result for 4 time entry it produces an array of 8 time entries with type as ±1. For our LLTP we took ±2milliseconds as our error range. After adding the type the entry will look like <entry – 2, -1> for the beginning of the range and <entry+2, +1> for the end of the range [entry±2].

[[File:IntersectionMarzullasalg.jpg]]

The description of the algorithm uses the following variables: best (largest number of overlapping intervals found), cnt (current number of overlapping intervals), best_start and

best_end (the beginning and end of best interval found so far). We used a two dimensional array to keep the second, millisecond and type of the range. The structure of the array is as follows:

time_entry [i][0] = second part of the time
entry time_entry [i][1] = millisecond part of the time
entry time_entry [i][2] = type of the range (i.e. ±1)

Here ‘Time_entry is the name of the array and ‘i’ is used as an index to traverse the array. After creating the array it follows the following step to determine the accurate time. Then it sends back the best chosen time to the level2 client program.

[[File:section3last.jpg]]

=4. List of files=

Level1 server:

DieWithError.c
get-time.c
L1Server.c

Level2 server:

daemon_server.c
DieWithError.c
get-time.c
L2Server.c

Level2 client:

daemon_client.c
DieWithError.c
get-time.c
L2Client.c
marzullo.c
print-time.c
selection_sort.c
set-time.c

Level3 client:

DieWithError.c
get-time.c
print-time.c
set-time.c
UDPTimeClient.c

=Discussion/Output=

The most intresting part of testing this was to see how distrubted architect of NTP client-server works and how could it be improved. I learnt alot by setting up and running them and seeing the results. It was fun improving little bit on exsisting implementation of NTP.

Output form different levels of clients and servers are given in this section. Output from level1 server:

[[File:output1.jpg]]

[[File:output2.jpg]]

[[File:output3.jpg]]

[[File:output4.jpg]]

=Conclusion=

Overall, there has been lots of work going on NTP , due to high demand of distributed clinet-server architect based application. This was carried on a very small linux based lab , in future i guess there will be need to test on large industry based scale , where many systems can be connected and than test to see how the delay from clients to server causes disturbance in algorithm. There will be issues for sure since as the information travels it takes time and that delayed time will need to be adjusted more and will be focus of future work.

=References=

*Ntp home page: http://ntp.org Acessed on 13th Feb 2011

*Wikipedia for network time protocol: http://en.wikipedia.org/wiki/Network_Time_Protocol Accessed on 13th Feb 2011

*Wikipedia for Marzullo’s algorithm: http://en.wikipedia.org/wiki/Marzullo's_algorithm Accessed on 13th Feb 2011

DistOS-2011W NTP

2011-02-22T18:24:03Z

Wahmed2: /* Introduction */

=Introduction=

1. Introduction:

*1.1 Context/Background

Network Time Protocol (NTP) synchronizes clocks of hosts and routers. According to NIST estimates there are 10-20 million NTP servers and clients deployed in the internet and its tributaries all over the world. NTP software has been ported to almost every workstation and server platform available today – from PCs to Cray’s – UNIX, Windows, VMS and embedded systems, even home routers and battery backup systems.

NTP provides nominal accuracies of low tens of milliseconds on WANs, sub milliseconds on LANs and sub microseconds using a precision time source such as cesium oscillator or GPS receiver. Our time protocol was designed over the LAN and our goal was to achieve sub milliseconds of accuracy.

*1.2 Importance

Precision of time is very important in the computer world. Following are some example why it is so important to maintain a precise and accurate time.

Network monitoring, measurement and control, Distributed database transaction journaling and logging, Secure document timestamps with cryptographic certification, Radio and TV programming lunch and monitoring, Stock market buy and sell orders, Distributed network gaming and training, etc.

*1.3 Definition of the problem

My goal was to acquire time accuracy in the Linux lab assuming that all the computers do not have internet connection. To time synchronize all the computers I had to build our own hierarchy of NTP servers and clients.

*1.4 Summary of the result

An efficient hierarchy of NTP server and client has been implemented. Precision of
Sub milliseconds were achieved.

*1.5 Outline of the report
Detailed background information and the structure of our time protocol are provided in Section
2. The development setup and code explanation is reviewed in Section 3. Details of which server/client contains which files are given in section 4. My contributions in this implementation are described in Section 5.

=2. Detailed context/background information=

In this section the background information of Network Time protocol and the hierarchical model and the comparison with the original NTP designed by Dave Mills is reviewed. For unambiguity I will call the protocol as Linux lab time protocol (LLTP) and the original Network time protocol as NTP.

2.1 NTP timestamps

The 64-bit timestamps used by NTP consist of a 32-bit second part and a 32-bit fractional second part. I followed the same time structure for our LLTP which gives us a time scale of 232 seconds (136 years) and a theoretical resolution of 2-32 seconds (233 picoseconds).

2.2 Structure

NTP uses a hierarchical, semi-layered system of levels of clock sources; each level of this hierarchy is termed a stratum and assigned a layer number starting with 0 at the top. The stratum level defines its distance from the reference clock and exists to prevent cyclical dependencies in hierarchy.

My target is to develop a time synchronizing software for a small office or lab with a large number of computer where a very few computer has internet connectivity but all the computers are connected together in a local network. As a result not all the computer is able to connect to internet for time accuracy. LLTP uses similar hierarchical, semi-layered sources as NTP. Each layer of this hierarchy is termed as level and assigned a layer number starting with 1.

2.3 Level 1 server

Computers with internet connectivity are the level 1 server for our LLTP

Level 1 server

synchronizes its system clock with the NTP servers on the internet. In this hierarchy, only a fixed (5) number of level 1 servers are used. Unlike NTP we don’t have the privilege to use a 10 of thousands of level 1 and level 2 servers, and unlike NTP our LLTP has only 3 level structures.

Picture below shows the connectivity and the hierarchy of LLTP

[[File:LTTPHireachy.jpg]]

2.4 Level 2 server and client

Level 2 servers run two different programs. A client program to synchronize its system clock with the level1 servers and a server program that waits for any level 3 client for time request. Upon the request it sends its synchronized system time to the client. Level 2 servers also synchronize its system clock with the other level2 servers.

Each level2 server is connected to 4 level1 servers. It polls time from each level1 server and chooses the best time and synchronizes its system clock. How the best time is achieved is described in the section 3 below with code explanation. There are maximum 5 level2 servers with our LLTP model. Each level2 server is connected to 4 different level1 servers. None of the level2 server has the same level1 server set.

2.5 Level 3 client

Bottom of the LLTP hierarchy is level3 client. It only runs a client program that polls time from the level2 servers. It can only connect/request time from one of our level2 servers at a time. It also requires the knowledge of the IP address of the level2 server. For the load management purpose level 3 clients are not able to connect to a Level1 servers.

=3. Development Details=

3.1 Server

As it is mentioned in section-2, there are two servers in our LLTP model. One is level 1 and the other is level 2. Both servers wait for a client with time requests and upon receiving one send the time string. Code structure for both the servers is almost similar with minor technical differences which are explained later in this section. Both servers use a infinite for loop for waiting for the
client. They use a fixed but different port number for the client connection.

[[File:clientntp.jpg]]

Time requests sent to Level 1 server have to contain a message no more than 1 character and it has to be an integer which is sent by the level2 client to distinguish the level1 servers from one another. Level1 server will take that message received from the client and add that at the end of the time string sent to the client. If the level1 server receives a time request message with greater size of 1than it will not return any time value. For the level2 server the maximum message size is
128 characters and it do not have to be an integer. Level2 server discards the message sent from the level3 client. A unique feature of level2 server is that a level3 client time request with “exit” message can shut down the level2 server. This feature is not implemented in level1 servers.

[[File:level1servercode.jpg]]

[[File:level2servercode.jpg]]

Both servers send time as a string which handles the problem with big and little Endean.

3.2 Client

There are also two clients for our LLTP, level2 client and level3 client. Level3 client is very straight forward where the level3 is a bit more complicated.

Level3 client sends a time request to the level2 server. It cannot send time request to a level1 server as the port number for both server are different and the port numbers are hard coded for both the client (leve3 client/level2 server uses port 123 where level2 client/level1 server uses port 4003). Level2 server also has a restriction over the message sent by the client which is explained above in the server section. Upon receiving the time string back from the server which contains the seconds and milliseconds part together as a string separated by a comma (‘,’), it separates this two parts and checks for the latency and calls function named set_time(long[] ) which sets the system clock time with the new time.

[[File:level3clientcode.jpg]]

Level2 client dose almost the same things but in a broader and sophisticated way. It polls time from four different levle1 time server at the same time. When it sends the time request, it sends a unique message as the server ID to identify the time received from different level1 server which helps to fix the latency later.

[[File:level2clientcode.jpg]]

Level2 client also uses a time out of 1 second. If it dose not receive the time string back from the server then it will ignore the time from that server and choose the best time from the ones it received from the other servers. Received time strings from the different servers also have the server id added at the end of the time string. Using that information level2 client saves the received time in order and fixes the latency accordingly. It keeps all the latency fixed time in an array with the invalid time being -1. Then it calls a different function called marzullo(long []), which takes the array of latency fixed times and returns the best time depending on the Marzullo’s algorithm. Function marzullo() is explained in the section 3.4. Then it uses the set_time(long []) method to set the system clock time.

[[File:level2clientcode2.jpg]]

3.3 Daemon client and server

The level2 client and server runs as a daemon process to give them the privilege to run continuously in the background. As both the client and server in level2 does not need any user interference and need to run all the time to update the system clock of the running computer from the level1 server.

As it is mentioned above the client process for level2 run continuously and it updates its system clock every 1 minute (polls time from the level1 server every 60 seconds). Server daemon process is built in the same manner but unlike the client process it does not have a waiting time.

[[File:codefordaemon.jpg]]

3.4 Marzullo’s algorithm and function marzullo(long [])

Marzullo’s algorithm, invented by Keith Marzullo, is an agreement algorithm used to select sources for estimating accurate time from a number of noisy time sources. A modified version of his algorithm, intersection algorithm, is use in NTP. We used the simplified version of this algorithm to choose the best possible time from our time sources (level1 servers).

Our function long* marzullo (long []) takes an array of four times from four different level1 servers and return a pointer to an array with the best seconds and milliseconds. First it takes the given array of times and makes two entries for each time as ±2 milliseconds. For each entry it also adds a type of -1 for -2millisecons and +1 for +2 milliseconds. As a result for 4 time entry it produces an array of 8 time entries with type as ±1. For our LLTP we took ±2milliseconds as our error range. After adding the type the entry will look like <entry – 2, -1> for the beginning of the range and <entry+2, +1> for the end of the range [entry±2].

[[File:IntersectionMarzullasalg.jpg]]

The description of the algorithm uses the following variables: best (largest number of overlapping intervals found), cnt (current number of overlapping intervals), best_start and

best_end (the beginning and end of best interval found so far). We used a two dimensional array to keep the second, millisecond and type of the range. The structure of the array is as follows:

time_entry [i][0] = second part of the time
entry time_entry [i][1] = millisecond part of the time
entry time_entry [i][2] = type of the range (i.e. ±1)

Here ‘Time_entry is the name of the array and ‘i’ is used as an index to traverse the array. After creating the array it follows the following step to determine the accurate time. Then it sends back the best chosen time to the level2 client program.

[[File:section3last.jpg]]

=4. List of files=

Level1 server:

DieWithError.c
get-time.c
L1Server.c

Level2 server:

daemon_server.c
DieWithError.c
get-time.c
L2Server.c

Level2 client:

daemon_client.c
DieWithError.c
get-time.c
L2Client.c
marzullo.c
print-time.c
selection_sort.c
set-time.c

Level3 client:

DieWithError.c
get-time.c
print-time.c
set-time.c
UDPTimeClient.c

=Discussion/Output=

The most intresting part of testing this was to see how distrubted architect of NTP client-server works and how could it be improved. I learnt alot by setting up and running them and seeing the results. It was fun improving little bit on exsisting implementation of NTP.

Output form different levels of clients and servers are given in this section. Output from level1 server:

[[File:output1.jpg]]

[[File:output2.jpg]]

[[File:output3.jpg]]

[[File:output4.jpg]]

=Conclusion=

Overall, there has been lots of work going on NTP , due to high demand of distributed clinet-server architect based application. This was carried on a very small linux based lab , in future i guess there will be need to test on large industry based scale , where many systems can be connected and than test to see how the delay from clients to server causes disturbance in algorithm. There will be issues for sure since as the information travels it takes time and that delayed time will need to be adjusted more and will be focus of future work.

=References=

*Ntp home page: http://ntp.org Acessed on 13th Feb 2011

*Wikipedia for network time protocol: http://en.wikipedia.org/wiki/Network_Time_Protocol Accessed on 13th Feb 2011

*Wikipedia for Marzullo’s algorithm: http://en.wikipedia.org/wiki/Marzullo's_algorithm Accessed on 13th Feb 2011

DistOS-2011W NTP

2011-02-21T21:05:37Z

Wahmed2: /* 3. Development Details */

=Introduction=

1. Introduction:

1.1 Context/Background

Network Time Protocol (NTP) synchronizes clocks of hosts and routers. According to NIST estimates there are 10-20 million NTP servers and clients deployed in the internet and its tributaries all over the world. NTP software has been ported to almost every workstation and server platform available today – from PCs to Cray’s – UNIX, Windows, VMS and embedded systems, even home routers and battery backup systems.

NTP provides nominal accuracies of low tens of milliseconds on WANs, sub milliseconds on LANs and sub microseconds using a precision time source such as cesium oscillator or GPS receiver. Our time protocol was designed over the LAN and our goal was to achieve sub milliseconds of accuracy.

1.2 Importance

Precision of time is very important in the computer world. Following are some example why it is so important to maintain a precise and accurate time.

Network monitoring, measurement and control, Distributed database transaction journaling and logging, Secure document timestamps with cryptographic certification, Radio and TV programming lunch and monitoring, Stock market buy and sell orders, Distributed network gaming and training, etc.

1.3 Definition of the problem

My goal was to acquire time accuracy in the Linux lab assuming that all the computers do not have internet connection. To time synchronize all the computers I had to build our own hierarchy of NTP servers and clients.

1.4 Summary of the result

An efficient hierarchy of NTP server and client has been implemented. Precision of
Sub milliseconds were achieved.

1.5 Outline of the report
Detailed background information and the structure of our time protocol are provided in Section
2. The development setup and code explanation is reviewed in Section 3. Details of which server/client contains which files are given in section 4. My contributions in this implementation are described in Section 5.

=2. Detailed context/background information=

In this section the background information of Network Time protocol and the hierarchical model and the comparison with the original NTP designed by Dave Mills is reviewed. For unambiguity I will call the protocol as Linux lab time protocol (LLTP) and the original Network time protocol as NTP.

2.1 NTP timestamps

The 64-bit timestamps used by NTP consist of a 32-bit second part and a 32-bit fractional second part. I followed the same time structure for our LLTP which gives us a time scale of 232 seconds (136 years) and a theoretical resolution of 2-32 seconds (233 picoseconds).

2.2 Structure

NTP uses a hierarchical, semi-layered system of levels of clock sources; each level of this hierarchy is termed a stratum and assigned a layer number starting with 0 at the top. The stratum level defines its distance from the reference clock and exists to prevent cyclical dependencies in hierarchy.

My target is to develop a time synchronizing software for a small office or lab with a large number of computer where a very few computer has internet connectivity but all the computers are connected together in a local network. As a result not all the computer is able to connect to internet for time accuracy. LLTP uses similar hierarchical, semi-layered sources as NTP. Each layer of this hierarchy is termed as level and assigned a layer number starting with 1.

2.3 Level 1 server

Computers with internet connectivity are the level 1 server for our LLTP

Level 1 server

synchronizes its system clock with the NTP servers on the internet. In this hierarchy, only a fixed (5) number of level 1 servers are used. Unlike NTP we don’t have the privilege to use a 10 of thousands of level 1 and level 2 servers, and unlike NTP our LLTP has only 3 level structures.

Picture below shows the connectivity and the hierarchy of LLTP

[[File:LTTPHireachy.jpg]]

2.4 Level 2 server and client

Level 2 servers run two different programs. A client program to synchronize its system clock with the level1 servers and a server program that waits for any level 3 client for time request. Upon the request it sends its synchronized system time to the client. Level 2 servers also synchronize its system clock with the other level2 servers.

Each level2 server is connected to 4 level1 servers. It polls time from each level1 server and chooses the best time and synchronizes its system clock. How the best time is achieved is described in the section 3 below with code explanation. There are maximum 5 level2 servers with our LLTP model. Each level2 server is connected to 4 different level1 servers. None of the level2 server has the same level1 server set.

2.5 Level 3 client

Bottom of the LLTP hierarchy is level3 client. It only runs a client program that polls time from the level2 servers. It can only connect/request time from one of our level2 servers at a time. It also requires the knowledge of the IP address of the level2 server. For the load management purpose level 3 clients are not able to connect to a Level1 servers.

=3. Development Details=

3.1 Server

As it is mentioned in section-2, there are two servers in our LLTP model. One is level 1 and the other is level 2. Both servers wait for a client with time requests and upon receiving one send the time string. Code structure for both the servers is almost similar with minor technical differences which are explained later in this section. Both servers use a infinite for loop for waiting for the
client. They use a fixed but different port number for the client connection.

[[File:clientntp.jpg]]

Time requests sent to Level 1 server have to contain a message no more than 1 character and it has to be an integer which is sent by the level2 client to distinguish the level1 servers from one another. Level1 server will take that message received from the client and add that at the end of the time string sent to the client. If the level1 server receives a time request message with greater size of 1than it will not return any time value. For the level2 server the maximum message size is
128 characters and it do not have to be an integer. Level2 server discards the message sent from the level3 client. A unique feature of level2 server is that a level3 client time request with “exit” message can shut down the level2 server. This feature is not implemented in level1 servers.

[[File:level1servercode.jpg]]

[[File:level2servercode.jpg]]

Both servers send time as a string which handles the problem with big and little Endean.

3.2 Client

There are also two clients for our LLTP, level2 client and level3 client. Level3 client is very straight forward where the level3 is a bit more complicated.

Level3 client sends a time request to the level2 server. It cannot send time request to a level1 server as the port number for both server are different and the port numbers are hard coded for both the client (leve3 client/level2 server uses port 123 where level2 client/level1 server uses port 4003). Level2 server also has a restriction over the message sent by the client which is explained above in the server section. Upon receiving the time string back from the server which contains the seconds and milliseconds part together as a string separated by a comma (‘,’), it separates this two parts and checks for the latency and calls function named set_time(long[] ) which sets the system clock time with the new time.

[[File:level3clientcode.jpg]]

Level2 client dose almost the same things but in a broader and sophisticated way. It polls time from four different levle1 time server at the same time. When it sends the time request, it sends a unique message as the server ID to identify the time received from different level1 server which helps to fix the latency later.

[[File:level2clientcode.jpg]]

Level2 client also uses a time out of 1 second. If it dose not receive the time string back from the server then it will ignore the time from that server and choose the best time from the ones it received from the other servers. Received time strings from the different servers also have the server id added at the end of the time string. Using that information level2 client saves the received time in order and fixes the latency accordingly. It keeps all the latency fixed time in an array with the invalid time being -1. Then it calls a different function called marzullo(long []), which takes the array of latency fixed times and returns the best time depending on the Marzullo’s algorithm. Function marzullo() is explained in the section 3.4. Then it uses the set_time(long []) method to set the system clock time.

[[File:level2clientcode2.jpg]]

3.3 Daemon client and server

The level2 client and server runs as a daemon process to give them the privilege to run continuously in the background. As both the client and server in level2 does not need any user interference and need to run all the time to update the system clock of the running computer from the level1 server.

As it is mentioned above the client process for level2 run continuously and it updates its system clock every 1 minute (polls time from the level1 server every 60 seconds). Server daemon process is built in the same manner but unlike the client process it does not have a waiting time.

[[File:codefordaemon.jpg]]

3.4 Marzullo’s algorithm and function marzullo(long [])

Marzullo’s algorithm, invented by Keith Marzullo, is an agreement algorithm used to select sources for estimating accurate time from a number of noisy time sources. A modified version of his algorithm, intersection algorithm, is use in NTP. We used the simplified version of this algorithm to choose the best possible time from our time sources (level1 servers).

Our function long* marzullo (long []) takes an array of four times from four different level1 servers and return a pointer to an array with the best seconds and milliseconds. First it takes the given array of times and makes two entries for each time as ±2 milliseconds. For each entry it also adds a type of -1 for -2millisecons and +1 for +2 milliseconds. As a result for 4 time entry it produces an array of 8 time entries with type as ±1. For our LLTP we took ±2milliseconds as our error range. After adding the type the entry will look like <entry – 2, -1> for the beginning of the range and <entry+2, +1> for the end of the range [entry±2].

[[File:IntersectionMarzullasalg.jpg]]

The description of the algorithm uses the following variables: best (largest number of overlapping intervals found), cnt (current number of overlapping intervals), best_start and

best_end (the beginning and end of best interval found so far). We used a two dimensional array to keep the second, millisecond and type of the range. The structure of the array is as follows:

time_entry [i][0] = second part of the time
entry time_entry [i][1] = millisecond part of the time
entry time_entry [i][2] = type of the range (i.e. ±1)

Here ‘Time_entry is the name of the array and ‘i’ is used as an index to traverse the array. After creating the array it follows the following step to determine the accurate time. Then it sends back the best chosen time to the level2 client program.

[[File:section3last.jpg]]

=4. List of files=

Level1 server:

DieWithError.c
get-time.c
L1Server.c

Level2 server:

daemon_server.c
DieWithError.c
get-time.c
L2Server.c

Level2 client:

daemon_client.c
DieWithError.c
get-time.c
L2Client.c
marzullo.c
print-time.c
selection_sort.c
set-time.c

Level3 client:

DieWithError.c
get-time.c
print-time.c
set-time.c
UDPTimeClient.c

=Discussion/Output=

The most intresting part of testing this was to see how distrubted architect of NTP client-server works and how could it be improved. I learnt alot by setting up and running them and seeing the results. It was fun improving little bit on exsisting implementation of NTP.

Output form different levels of clients and servers are given in this section. Output from level1 server:

[[File:output1.jpg]]

[[File:output2.jpg]]

[[File:output3.jpg]]

[[File:output4.jpg]]

=Conclusion=

Overall, there has been lots of work going on NTP , due to high demand of distributed clinet-server architect based application. This was carried on a very small linux based lab , in future i guess there will be need to test on large industry based scale , where many systems can be connected and than test to see how the delay from clients to server causes disturbance in algorithm. There will be issues for sure since as the information travels it takes time and that delayed time will need to be adjusted more and will be focus of future work.

=References=

*Ntp home page: http://ntp.org Acessed on 13th Feb 2011

*Wikipedia for network time protocol: http://en.wikipedia.org/wiki/Network_Time_Protocol Accessed on 13th Feb 2011

*Wikipedia for Marzullo’s algorithm: http://en.wikipedia.org/wiki/Marzullo's_algorithm Accessed on 13th Feb 2011

DistOS-2011W NTP

2011-02-21T21:03:05Z

Wahmed2: /* Conclusion */

=Introduction=

1. Introduction:

1.1 Context/Background

Network Time Protocol (NTP) synchronizes clocks of hosts and routers. According to NIST estimates there are 10-20 million NTP servers and clients deployed in the internet and its tributaries all over the world. NTP software has been ported to almost every workstation and server platform available today – from PCs to Cray’s – UNIX, Windows, VMS and embedded systems, even home routers and battery backup systems.

NTP provides nominal accuracies of low tens of milliseconds on WANs, sub milliseconds on LANs and sub microseconds using a precision time source such as cesium oscillator or GPS receiver. Our time protocol was designed over the LAN and our goal was to achieve sub milliseconds of accuracy.

1.2 Importance

Precision of time is very important in the computer world. Following are some example why it is so important to maintain a precise and accurate time.

Network monitoring, measurement and control, Distributed database transaction journaling and logging, Secure document timestamps with cryptographic certification, Radio and TV programming lunch and monitoring, Stock market buy and sell orders, Distributed network gaming and training, etc.

1.3 Definition of the problem

My goal was to acquire time accuracy in the Linux lab assuming that all the computers do not have internet connection. To time synchronize all the computers I had to build our own hierarchy of NTP servers and clients.

1.4 Summary of the result

An efficient hierarchy of NTP server and client has been implemented. Precision of
Sub milliseconds were achieved.

1.5 Outline of the report
Detailed background information and the structure of our time protocol are provided in Section
2. The development setup and code explanation is reviewed in Section 3. Details of which server/client contains which files are given in section 4. My contributions in this implementation are described in Section 5.

=2. Detailed context/background information=

In this section the background information of Network Time protocol and the hierarchical model and the comparison with the original NTP designed by Dave Mills is reviewed. For unambiguity I will call the protocol as Linux lab time protocol (LLTP) and the original Network time protocol as NTP.

2.1 NTP timestamps

The 64-bit timestamps used by NTP consist of a 32-bit second part and a 32-bit fractional second part. I followed the same time structure for our LLTP which gives us a time scale of 232 seconds (136 years) and a theoretical resolution of 2-32 seconds (233 picoseconds).

2.2 Structure

NTP uses a hierarchical, semi-layered system of levels of clock sources; each level of this hierarchy is termed a stratum and assigned a layer number starting with 0 at the top. The stratum level defines its distance from the reference clock and exists to prevent cyclical dependencies in hierarchy.

My target is to develop a time synchronizing software for a small office or lab with a large number of computer where a very few computer has internet connectivity but all the computers are connected together in a local network. As a result not all the computer is able to connect to internet for time accuracy. LLTP uses similar hierarchical, semi-layered sources as NTP. Each layer of this hierarchy is termed as level and assigned a layer number starting with 1.

2.3 Level 1 server

Computers with internet connectivity are the level 1 server for our LLTP

Level 1 server

synchronizes its system clock with the NTP servers on the internet. In this hierarchy, only a fixed (5) number of level 1 servers are used. Unlike NTP we don’t have the privilege to use a 10 of thousands of level 1 and level 2 servers, and unlike NTP our LLTP has only 3 level structures.

Picture below shows the connectivity and the hierarchy of LLTP

[[File:LTTPHireachy.jpg]]

2.4 Level 2 server and client

Level 2 servers run two different programs. A client program to synchronize its system clock with the level1 servers and a server program that waits for any level 3 client for time request. Upon the request it sends its synchronized system time to the client. Level 2 servers also synchronize its system clock with the other level2 servers.

Each level2 server is connected to 4 level1 servers. It polls time from each level1 server and chooses the best time and synchronizes its system clock. How the best time is achieved is described in the section 3 below with code explanation. There are maximum 5 level2 servers with our LLTP model. Each level2 server is connected to 4 different level1 servers. None of the level2 server has the same level1 server set.

2.5 Level 3 client

Bottom of the LLTP hierarchy is level3 client. It only runs a client program that polls time from the level2 servers. It can only connect/request time from one of our level2 servers at a time. It also requires the knowledge of the IP address of the level2 server. For the load management purpose level 3 clients are not able to connect to a Level1 servers.

=3. Development Details=

3.1 Server

As it is mentioned in section-2, there are two servers in our LLTP model. One is level 1 and the other is level 2. Both servers wait for a client with time requests and upon receiving one send the time string. Code structure for both the servers is almost similar with minor technical differences which are explained later in this section. Both servers use a infinite for loop for waiting for the
client. They use a fixed but different port number for the client connection.

[[File:Z:\clientntp.jpg]]

Time requests sent to Level 1 server have to contain a message no more than 1 character and it has to be an integer which is sent by the level2 client to distinguish the level1 servers from one another. Level1 server will take that message received from the client and add that at the end of the time string sent to the client. If the level1 server receives a time request message with greater size of 1than it will not return any time value. For the level2 server the maximum message size is
128 characters and it do not have to be an integer. Level2 server discards the message sent from the level3 client. A unique feature of level2 server is that a level3 client time request with “exit” message can shut down the level2 server. This feature is not implemented in level1 servers.

[[File:level1servercode.jpg]]

[[File:level2servercode.jpg]]

Both servers send time as a string which handles the problem with big and little Endean.

3.2 Client

There are also two clients for our LLTP, level2 client and level3 client. Level3 client is very straight forward where the level3 is a bit more complicated.

Level3 client sends a time request to the level2 server. It cannot send time request to a level1 server as the port number for both server are different and the port numbers are hard coded for both the client (leve3 client/level2 server uses port 123 where level2 client/level1 server uses port 4003). Level2 server also has a restriction over the message sent by the client which is explained above in the server section. Upon receiving the time string back from the server which contains the seconds and milliseconds part together as a string separated by a comma (‘,’), it separates this two parts and checks for the latency and calls function named set_time(long[] ) which sets the system clock time with the new time.

[[File:level3clientcode.jpg]]

Level2 client dose almost the same things but in a broader and sophisticated way. It polls time from four different levle1 time server at the same time. When it sends the time request, it sends a unique message as the server ID to identify the time received from different level1 server which helps to fix the latency later.

[[File:level2clientcode.jpg]]

Level2 client also uses a time out of 1 second. If it dose not receive the time string back from the server then it will ignore the time from that server and choose the best time from the ones it received from the other servers. Received time strings from the different servers also have the server id added at the end of the time string. Using that information level2 client saves the received time in order and fixes the latency accordingly. It keeps all the latency fixed time in an array with the invalid time being -1. Then it calls a different function called marzullo(long []), which takes the array of latency fixed times and returns the best time depending on the Marzullo’s algorithm. Function marzullo() is explained in the section 3.4. Then it uses the set_time(long []) method to set the system clock time.

[[File:level2clientcode2.jpg]]

3.3 Daemon client and server

The level2 client and server runs as a daemon process to give them the privilege to run continuously in the background. As both the client and server in level2 does not need any user interference and need to run all the time to update the system clock of the running computer from the level1 server.

As it is mentioned above the client process for level2 run continuously and it updates its system clock every 1 minute (polls time from the level1 server every 60 seconds). Server daemon process is built in the same manner but unlike the client process it does not have a waiting time.

[[File:codefordaemon.jpg]]

3.4 Marzullo’s algorithm and function marzullo(long [])

Marzullo’s algorithm, invented by Keith Marzullo, is an agreement algorithm used to select sources for estimating accurate time from a number of noisy time sources. A modified version of his algorithm, intersection algorithm, is use in NTP. We used the simplified version of this algorithm to choose the best possible time from our time sources (level1 servers).

Our function long* marzullo (long []) takes an array of four times from four different level1 servers and return a pointer to an array with the best seconds and milliseconds. First it takes the given array of times and makes two entries for each time as ±2 milliseconds. For each entry it also adds a type of -1 for -2millisecons and +1 for +2 milliseconds. As a result for 4 time entry it produces an array of 8 time entries with type as ±1. For our LLTP we took ±2milliseconds as our error range. After adding the type the entry will look like <entry – 2, -1> for the beginning of the range and <entry+2, +1> for the end of the range [entry±2].

[[File:IntersectionMarzullasalg.jpg]]

The description of the algorithm uses the following variables: best (largest number of overlapping intervals found), cnt (current number of overlapping intervals), best_start and

best_end (the beginning and end of best interval found so far). We used a two dimensional array to keep the second, millisecond and type of the range. The structure of the array is as follows:

time_entry [i][0] = second part of the time
entry time_entry [i][1] = millisecond part of the time
entry time_entry [i][2] = type of the range (i.e. ±1)

Here ‘Time_entry is the name of the array and ‘i’ is used as an index to traverse the array. After creating the array it follows the following step to determine the accurate time. Then it sends back the best chosen time to the level2 client program.

[[File:section3last.jpg]]

=4. List of files=

Level1 server:

DieWithError.c
get-time.c
L1Server.c

Level2 server:

daemon_server.c
DieWithError.c
get-time.c
L2Server.c

Level2 client:

daemon_client.c
DieWithError.c
get-time.c
L2Client.c
marzullo.c
print-time.c
selection_sort.c
set-time.c

Level3 client:

DieWithError.c
get-time.c
print-time.c
set-time.c
UDPTimeClient.c

=Discussion/Output=

The most intresting part of testing this was to see how distrubted architect of NTP client-server works and how could it be improved. I learnt alot by setting up and running them and seeing the results. It was fun improving little bit on exsisting implementation of NTP.

Output form different levels of clients and servers are given in this section. Output from level1 server:

[[File:output1.jpg]]

[[File:output2.jpg]]

[[File:output3.jpg]]

[[File:output4.jpg]]

=Conclusion=

Overall, there has been lots of work going on NTP , due to high demand of distributed clinet-server architect based application. This was carried on a very small linux based lab , in future i guess there will be need to test on large industry based scale , where many systems can be connected and than test to see how the delay from clients to server causes disturbance in algorithm. There will be issues for sure since as the information travels it takes time and that delayed time will need to be adjusted more and will be focus of future work.

=References=

*Ntp home page: http://ntp.org Acessed on 13th Feb 2011

*Wikipedia for network time protocol: http://en.wikipedia.org/wiki/Network_Time_Protocol Accessed on 13th Feb 2011

*Wikipedia for Marzullo’s algorithm: http://en.wikipedia.org/wiki/Marzullo's_algorithm Accessed on 13th Feb 2011

File:Section3last.jpg

2011-02-21T20:59:25Z

Wahmed2:

File:Level3clientcode.jpg

2011-02-21T20:58:45Z

Wahmed2:

File:Level2servercode.jpg

2011-02-21T20:58:35Z

Wahmed2:

File:Level2clientcode2.jpg

2011-02-21T20:58:22Z

Wahmed2:

File:Level2clientcode.jpg

2011-02-21T20:58:13Z

Wahmed2:

File:Level1servercode.jpg

2011-02-21T20:58:04Z

Wahmed2: