https://homeostasis.scs.carleton.ca/wiki/index.php?action=history&feed=atom&title=Mobile_Apps_2023W_Lecture_10 2026-04-28T09:29:17Z Revision history for this page on the wiki MediaWiki 1.42.1 https://homeostasis.scs.carleton.ca/wiki/index.php?title=Mobile_Apps_2023W_Lecture_10&diff=24352&oldid=prev 2023-02-10T21:36:13Z

<p>Created page with &quot;==Notes== &lt;pre&gt; February 10 ----------- Web browser with history visualization/query - understand what pages I&#039;ve been visiting Potential screens: - basic web browser (where user will spend most of their time) - history viewer (text) - history viewer (graphical) How do we display web history? - basic chronological - time, web page, title Potential tasks - what websites do I visit the most? - what topics do I read about? - what authors do I read? - conn...&quot;</p> <p><b>New page</b></p><div>==Notes==<br /> <br /> &lt;pre&gt;<br /> February 10<br /> -----------<br /> <br /> Web browser with history visualization/query<br /> - understand what pages I&#039;ve been visiting<br /> <br /> Potential screens:<br /> - basic web browser (where user will spend most of their time)<br /> - history viewer (text)<br /> - history viewer (graphical)<br /> <br /> How do we display web history?<br /> - basic chronological<br /> - time, web page, title<br /> <br /> <br /> Potential tasks<br /> - what websites do I visit the most?<br /> - what topics do I read about?<br /> - what authors do I read?<br /> - connect the above (topic &amp; author, topic &amp; website)<br /> <br /> Web URLs are easy, get that from user visiting page<br /> Time is easy, we know when things happen<br /> But the other things...they require analyzing the page<br /> <br /> Authorship is actually hard, no standard way of representing it<br /> <br /> The &quot;Semantic web&quot; was an idea for representing the web for automatic analysis rather than human consumption<br /> - all information would be represented in schemas<br /> - so queries would be very easy<br /> <br /> The semantic web never really took off, isn&#039;t what we get generally<br /> <br /> What we *can* get is the plain text of a page<br /> - we have what is between body and headline tags<br /> - if we look at the code for &quot;reader&quot; modes in browsers, they show how to do this<br /> <br /> Let&#039;s assume we have some way to convert a web page into text<br /> <br /> We have a bunch of words<br /> - can run sort &amp; uniq, get a list of words in a page<br /> - could even get frequency, but that typically isn&#039;t so useful<br /> <br /> Given a list of words, what is the &quot;topic&quot; of a page?<br /> - to do this properly you&#039;d need NLP of some kind (AI)<br /> <br /> I could just remember which words are on which pages<br /> - remove frequently-occuring words automatically by deleting words that<br /> get above a certain frequency<br /> <br /> Zipf&#039;s law<br /> - frequency of words in natural languase falls off as 1/n<br /> - so most words are very infrequent, but a few are very frequent<br /> <br /> Instead of topics, we are just remembering the words that are present on different pages<br /> - so in addition to the URL, we need to store which words each page has<br /> - but I don&#039;t think that will be that much storage if we eliminate<br /> frequent words<br /> <br /> So then, how do we structure a database schema to fit all this?<br /> <br /> Potential tables<br /> - page IDs<br /> - URLs -&gt; UIDs<br /> - use UIDs everywhere instead of URLs to save space<br /> - can trim URLs to make sure they work consistently<br /> - raw history<br /> - page URL, time<br /> - page-&gt;words (maybe not needed?)<br /> - only stores words that haven&#039;t reached max frequency<br /> - words-&gt;pages<br /> - again only words that haven&#039;t reached max frequency<br /> - represent pages using a UID, not the URL<br /> - word frequency<br /> - every word we&#039;ve seen<br /> - for each word, its frequency<br /> - max frequency, once above this we stop counting<br /> - ignored URLs<br /> - pages that change frequently<br /> <br /> we can assume that pages don&#039;t change, i.e., URL-&gt;contents mapping stays consistent<br /> <br /> Note that if we view too many pages with a given topic, that topic will turn into a frequent word and won&#039;t show up in our analysis<br /> <br /> To visualize, we don&#039;t want pages-&gt;words, but words-&gt;pages<br /> <br /> One problem with SQL, it doesn&#039;t like variable-sized records<br /> <br /> So if we have a page with 1000 words, 20 unique non-frequent words<br /> - so need to represent the mapping of the page&#039;s URL UID to each of these words<br /> - but another page could have 2, another could have 2000<br /> <br /> Can just have a table of word, UID with duplicates<br /> - then later can index to get stats<br /> <br /> (Need a cap on the maximum length of a word to exclude bizzare text strings)<br /> <br /> Assume that we&#039;ve solved this, what operations do we want to support?<br /> <br /> - get topics (&quot;interesting&quot; words)<br /> - for a topic, get the URLs with them<br /> <br /> But how do we treat news sites?<br /> - visiting the same URL produces completely different content?<br /> - I think it should automatically be excluded<br /> - so cbc.ca/news should automatically be ignored because its contents keep changing<br /> - but pages it points to will be kept because they are consistent<br /> <br /> So every time we visit a page we&#039;ll have to see if we&#039;ve visited it before<br /> - calculate word frequency, if it is very different from a past visit then we can add it to a list of ignored web pages<br /> <br /> When the user wants to see their topic-based history, we would show them the list of words, perhaps using a word cloud<br /> - select a word, then you can see the pages associated with it<br /> - maybe get another word cloud that is associated with that word<br /> (to show co-occurances)<br /> <br /> Do you all like word clouds?<br /> &lt;/pre&gt;</div>

Soma