tag:blogger.com,1999:blog-58879850820541473812024-03-28T02:22:37.932-07:00Probably Done BeforeDaniel Eklundhttp://www.blogger.com/profile/11570452431861145598noreply@blogger.comBlogger20125tag:blogger.com,1999:blog-5887985082054147381.post-63269554976480550082015-10-26T15:32:00.000-07:002015-10-27T04:19:06.957-07:00Visualizing Docker Containers and Images <div dir="ltr" style="text-align: left;" trbidi="on">
<br />
<div style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;">
<a href="https://lh6.googleusercontent.com/W20yWdISt7-emSCAPYqTeuVIhA5vzrgNhkPTtlVMNULelNwYdk8MDSVKPCgXqBDhPVJX1JJAP5NqKduzB6k1JAnFPe9zAAfZe2IP0yjCgPnJ52gJBlIAJUoldz4EmweRM8qftz6I" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="103px;" src="https://lh6.googleusercontent.com/W20yWdISt7-emSCAPYqTeuVIhA5vzrgNhkPTtlVMNULelNwYdk8MDSVKPCgXqBDhPVJX1JJAP5NqKduzB6k1JAnFPe9zAAfZe2IP0yjCgPnJ52gJBlIAJUoldz4EmweRM8qftz6I" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="124px;" /></a>This post is meant as a Docker <b><u>102</u></b>-level post. If you are unaware of what Docker is, or don't know how it compares to virtual machines or to configuration management tools, then this post might be a bit too advanced at this time.<br />
<br />
This post hopes to aid those struggling to internalize the docker command-line, specifically with knowing the exact difference between a <b>container</b> and an <b>image. </b>More specifically, this post shall differentiate a simple <b>container</b> from a <b>running container</b>.</div>
<br />
<br />
<span id="docs-internal-guid-57d74231-a4e4-3b7c-e4bc-e90e6f36f11e"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="185px;" src="https://lh5.googleusercontent.com/CD4S-htuy3O_76Gpw8AvMi1ZG93WFO75-BS3BcDScRjlUSKkjwNhcX-_rs6nCUJcQPW7M_TcEUVNKtnm--G9oP5YXlRp-UE76pSj5hDhaiuPVqGwwI1rLCe3IH4b1QfvN4232sWl" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="547px;" /></span></span><br />
I do this by taking a look at some of the underlying details, namely the layers of the union file system. This was a process I undertook for myself in the past few weeks, as I am relatively new to the docker technology and have found the docker command-lines difficult to internalize.<br />
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #c9daf8; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><b>Tangent</b>: In my opinion, understanding how a technology works under the hood is the best way to achieve learning speed and to build confidence that you are using the tool in the correct way. Often a technology is released with a certain breathless and hype that make it difficult to really understand appropriate usage patterns. More specifically, technology releases often develop an abstraction model that can invent new terminologies and metaphors that might be useful at first, but make it harder to develop mastery in latter stages.<br />
<br />
A good example of this is Git. I could not gain traction with Git until I understood its underlying model, including trees, blobs, commits,tags, tree-ish, etc. I had written about this before in a <a href="http://merrigrove.blogspot.com/2014/02/why-heck-is-git-so-hard-places-model-ok.html" target="_blank">previous post,</a> and still remain convinced that people who don't understand the internals of Git cannot have true mastery of the tool.</td></tr>
</tbody></table>
</div>
<h3 style="text-align: left;">
<span style="font-size: large;"><br /></span></h3>
<h3 style="text-align: left;">
<span style="font-size: large;">Image Definition</span></h3>
<div>
The first visual I present is that of an <b>image</b>, shown below with two different visuals. It is defined as the "union view" of a stack of <b>read-only layers</b>.</div>
<div>
<span id="docs-internal-guid-57d74231-a4e8-b09e-aaaa-af65642a59d8"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="187px;" src="https://lh5.googleusercontent.com/ssll_pqj1x654pX7kqoLmt6ZBYwCj9JsDrbAfi6_fxcxvmqgjEdrnDmDixU83t1xTiCfHI4sGxWsAwIQ27yu14dkMtoxNmVRqk7263m5kCy9Hs2GRgAd4uWprS4r3_G637VCt6Db" style="border: none; transform: rotate(0rad);" width="624px;" /></span></span></div>
<div>
<span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;">On the left we see a stack of read-layers. These layers are internal implementation details only, and are accessible outside of running containers in the host's file system. Importantly, they are read-only (or immutable) but capture the changes (deltas) made to the layers below. Each layer may have one parent, which itself may have a parent, etc. The top-level layer may be read by a union-ing file system (AUFS on my docker implementation) to present a single cohesive view of all the changes as <i>one</i> read-only file system. We see this "union view" on the right.</span></span></div>
<div>
<div dir="ltr" style="margin-left: 0pt;">
<br class="Apple-interchange-newline" />
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #c9daf8; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;">If you want to see these layers in their glory, you might find them in different locations on your host's files system. These layers will not be viewable from within a running container directly. In my docker's host system I can see them at <span style="font-family: Courier New, Courier, monospace;">/var/lib/docker</span> in a subdirectory called <span style="font-family: Courier New, Courier, monospace;">aufs</span>.<br />
<br />
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><span class="s1"># </span><span class="s3">sudo</span><span class="s1"> </span><span class="s4">tree</span><span class="s1"> </span><span class="s4">-L</span><span class="s1"> </span><span class="s4">1</span><span class="s1"> </span><span class="s5">/var/lib/docker/</span></span></div>
<div class="p2">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">/var/lib/docker/</span></span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><span class="s1">├── </span><span class="s6">aufs</span></span></div>
<div class="p2">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><span class="s7">├── </span><span class="s1">containers</span></span></div>
<div class="p2">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><span class="s7">├── </span><span class="s1">graph</span></span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><span class="s1">├── </span><span class="s6">init</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">├── linkgraph.db</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">├── repositories-aufs</span></span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><span class="s1">├── </span><span class="s6">tmp</span></span></div>
<div class="p2">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><span class="s7">├── </span><span class="s1">trust</span></span></div>
<div class="p2">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><span class="s7">└── </span><span class="s1">volumes</span></span></div>
<div class="p3">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><span class="s1"></span><br /></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">7 directories, 2 files</span></span></div>
</td></tr>
</tbody></table>
</div>
</div>
<div style="text-align: left;">
<span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;"><br /></span></span></div>
<h3 style="text-align: left;">
<span style="font-family: Arial; font-size: large;"><span style="white-space: pre-wrap;">Container Definition</span></span></h3>
<div>
<span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;">A container is defined as a "union view" of a stack of layers the top of which is a read-write layer.</span></span></div>
<div>
<span id="docs-internal-guid-57d74231-a50a-b0bf-743c-38c86f35ea6e"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="167px;" src="https://lh6.googleusercontent.com/ZXiLWSSWvtBQUbfe7ShTdKNkrIw2of4ZY_kWjQfxouMx_iSNJcIWT577ueHuDsbdxCUjj9kaEkL-shdxtauRTwyOqEp7jP7Io_vVps7eS6NUV76eH0pDOBKw9VNKCeUIZUkbZzva" style="border: none; transform: rotate(0rad);" width="624px;" /></span></span></div>
<div>
<span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">I show this visual above, and you will note it is nearly the same thing as an image, except that the top layer is read-write. At this point, some of you might notice that this definition says nothing about whether this container is running, and this is on purpose. It was this discovery in particular that cleared up a lot of confusion I had up to this point.</span></div>
<div>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #d9ead3; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><b>Takeaway</b>: A container is defined <i>only</i> as a read-write layer atop an image (of read-only layers itself). <b>It does not have to be running</b>.</td></tr>
</tbody></table>
</div>
<span id="docs-internal-guid-57d74231-a515-76ca-362b-174f93acd967">
</span></div>
<div>
<br />
So if we want to discuss containers running, we need to define a <b>running container</b>.<br />
<br /></div>
<h3 style="text-align: left;">
<span style="font-size: large;">Running Container Definition</span></h3>
<div>
A <b><i>running</i> container </b>is defined as a read-write "union view" and the the isolated process-space and processes within. The below visual shows the read-write container surrounded by this process-space.</div>
<div>
<br /></div>
<div>
<span id="docs-internal-guid-57d74231-a51d-fc82-6081-c725e2ca72ba"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="198px;" src="https://lh5.googleusercontent.com/JbDBazwnRm8XLspqvgu9qVENGbSlhhh25tRN43csuQAndzhq-GQPCJfVVj1EfmG2mEGbzrSuItwIcwgKm1kp-4K_YcdvRsX2BS8BJ7Kz_p6KhV6RrjWcEZ0_ZeFoaNncinGfR_si" style="border: none; transform: rotate(0rad);" width="622px;" /></span></span></div>
<div>
<span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">It is this act of isolation atop the file system effected by kernel-level technologies like cgroups, namespaces, etc that have made docker such a promising technology. The processes within this process-space may change, delete or create files within the "union view" file that will be captured in the read-write layer. I show this in the visual below:</span></div>
<div>
<span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div>
<span id="docs-internal-guid-57d74231-a53b-3210-2640-b88d70e51bb6"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="198px;" src="https://lh3.googleusercontent.com/5X0siIasoQhxO9cLXb39E7EkFRqrjeqwZ2hHaI30FZnmqFanGTgCwClIbn9R3ejzfCn-ulOQxOPErpTInpHHHVZ0iad51TTij-_5Bz48Lb9l4kz1LFdf9WPqO0mRukctqjOYP5L9" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="551px;" /></span></span></div>
<div>
<div dir="ltr" style="margin-left: 0pt;">
<br class="Apple-interchange-newline" />
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #c9daf8; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;">To see this at work run the following command: <span style="font-family: Courier New, Courier, monospace;"><b><span class="s1">docker</span><span class="s2"> </span><span class="s3">run</span><span class="s2"> </span><span class="s3">ubuntu</span><span class="s2"> </span><span class="s3">touch</span><span class="s2"> </span><span class="s3">happiness.txt</span></b></span>. You will then be able to see the new file in the read-write layer of the host system, even though there is no longer a running container (note, run this in your host system, not a container):<br />
<br />
<div class="p1">
</div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;"># find / -name happiness.txt</span></span></div>
<div class="p1">
</div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;"><b>/var/lib/docker/aufs/diff/860a7b...889/happiness.txt</b></span></span></div>
</td></tr>
</tbody></table>
</div>
</div>
<div>
<span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<h3 style="text-align: left;">
<span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"><span style="font-size: large;">Image Layer Definition</span></span></h3>
<div>
<span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">Finally, to tie up some loose ends, we should define an image layer. The below image shows an image layer and makes us realize that <i>a layer is not just the changes to the file system</i>.</span></div>
<div>
<span id="docs-internal-guid-57d74231-a55f-eb97-892d-042ca1f8a5a4"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="71px;" src="https://lh3.googleusercontent.com/gPpVRlJuR2clsgIgCpRk2RD4cO6D5soiC1BNITrFSUnVVyBu1TCLY4iN2xm3pvTcPRD3sJTccZQkUrcZK5Or-qV1aeyj2ueecjg9mcF0TTpjpKB2jutfK3hXIvsSw24eL46752pf" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="624px;" /></span></span></div>
<div>
<span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">The metadata is additional information about the layer that allows docker to capture runtime and build-time information, but also hierarchical information on a layer's parent. <i>Both</i> read and read-write layers contain this metadata.</span></div>
<div>
<span id="docs-internal-guid-57d74231-a553-fae1-7112-537dbc2cd1a9"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="181px;" src="https://lh6.googleusercontent.com/_Ekn74V7FdH-b7gw1ZvK-cpgdxIzhDJE4LAMu_Z3tPhvd9Nu9zT-q5wG04PK49Z51KlSRx4nj6fQs7_02zhkRGyxAPRrc8C1lB_QvvmCrSohcGDafZGpQh9-ARBYdm1Iq-gH7fv4" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="287px;" /></span></span></div>
<div>
<span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">Additionally, as we have mentioned before, each layer contains a pointer to a parent layer using the Id (here, the parent layers are below). If a layer does not point to a parent layer, then it is at the bottom of the stack.</span></div>
<div>
<span id="docs-internal-guid-57d74231-a56b-d758-5da5-83dccdea4be1"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="261px;" src="https://lh4.googleusercontent.com/yVFc2d9VYlD4QGHGqbhT3QYesbJ51Vt4NfbnHuXQiYRuSc4pw6Eyf3WQN-rXgvCF8Mc-asnIGg5bdqg0iWa7KLAYfPsM4yPPLSPoXB-kfC0B8s6oUxSmKvK_FMFmGoSPyF6RYT20" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="359px;" /></span></span></div>
<div>
<div dir="ltr" style="margin-left: 0pt;">
<br class="Apple-interchange-newline" />
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #c9daf8; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><b>Metadata Location:</b><br />
At this time (and I'm fully aware that the docker developers could change the implementation), the metadata for an <b>image</b> (read-only) layer can be found in a file called "<span style="font-family: Courier New, Courier, monospace;">json</span>" within <span style="font-family: 'Courier New', Courier, monospace;">/var/lib/docker/graph </span>at the id of the particular layer:<br />
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">/var/lib/docker/graph/e809f156dc985.../json</span></span></div>
where "<span style="font-family: 'Courier New', Courier, monospace;">e809f156dc985...</span>" is the elided id of the layer.<br />
<br />
The metadata for a <b>container</b> seems to be broken into many files, but more or less is found in <span style="font-family: Courier New, Courier, monospace;">/var/lib/docker/containers/<id></span> where <span style="font-family: 'Courier New', Courier, monospace;"><id></span>is the id of the read-write layer. The files in this directory contain more of the run-time metadata needed to expose a container to the outside world: networking, naming, logs, etc.</td></tr>
</tbody></table>
</div>
</div>
<div>
<span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<a name='more'></a></div>
<h2 style="text-align: left;">
<span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"><span style="font-size: large;">Tying It All Together</span></span></h2>
<div>
<span id="docs-internal-guid-57d74231-a577-c46f-5281-b3750430f89f">Now, let's look at the commands in the light of these visual metaphors and implementation details.</span><br />
<br />
<b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">docker create </b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;"><</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">image-id</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">></b><br />
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #fce5cd; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Input</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td><td style="background-color: #fce5cd; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Output</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a581-bba4-964e-ebeaaf83e574"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">
<span id="docs-internal-guid-57d74231-a581-dc57-8b4d-0d5fb7df2577"><span style="font-size: 14.6667px; vertical-align: baseline;"><img height="63px;" src="https://lh5.googleusercontent.com/4lnWJEIcb2h-u5A1wyEqGbxonn9GUPIk0dMI1t0iWqqCPyt7ohZFtFiH1YjpgBvtV3vLc81_sbRDCWKFlHkry9K3Pwbm7y0PbrEdG8ujrsYmtVavp9k-pzlDgcdtnWUBuFdYfDtc" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="269px;" /></span></span>
</span></span></td><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a582-0c17-7034-a489f847bcd3"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">
<span id="docs-internal-guid-57d74231-a583-950b-e899-e89e4d67e198"><span style="font-size: 14.6667px; vertical-align: baseline;"><img height="63px;" src="https://lh4.googleusercontent.com/wbSPu1jHwsU3aRLJiELa0rmq5InoFhwkWwrSkSVvj4AVdAjDC4up6rPELqfRh8UNRVYdUGvVpvLaw9ZuFxrSO_P4vEepEEZGaRVvRj7MYJ8c_hPNoEzZxXUKUOJk95CTIIRbBXgF" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="269px;" /></span></span>
</span></span></td></tr>
</tbody></table>
</div>
<div dir="ltr" style="margin-left: 0pt;">
The 'docker create' command adds a read-write layer to the top stack based on the image id. It <i>does not</i> run this container.</div>
<div dir="ltr" style="margin-left: 0pt;">
<span id="docs-internal-guid-57d74231-a601-bf76-0b10-1b0ea7770fed"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="121px;" src="https://lh3.googleusercontent.com/oUNBwoB0bXw7SDKa1U9juQ2tgdBPmgqAp32ghI4dsi6r6Y27ZEnk4YUStZImqf1ahfvl-ayZX4Rb4mGrTnPUUBLJMqOQ-zIHWbZxuMliwpvoCbRF4uywNS4jKsGMR4GQgb56naQl" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="603px;" /></span></span></div>
<div dir="ltr" style="margin-left: 0pt;">
<span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<br />
<b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">docker start </b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;"><</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">container-id</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">></b><br />
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #fce5cd; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Input</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td><td style="background-color: #fce5cd; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Output</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a583-950b-e899-e89e4d67e198"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="63px;" src="https://lh4.googleusercontent.com/wbSPu1jHwsU3aRLJiELa0rmq5InoFhwkWwrSkSVvj4AVdAjDC4up6rPELqfRh8UNRVYdUGvVpvLaw9ZuFxrSO_P4vEepEEZGaRVvRj7MYJ8c_hPNoEzZxXUKUOJk95CTIIRbBXgF" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="269px;" /></span></span></td><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a584-5a6c-f0b0-64f4b65b0f1b"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="198px;" src="https://lh4.googleusercontent.com/kNSgOj4ZgnSCnbhaPa8k7wYqo_FQINIn0LEBh7Am7OGGNg28WzoVamfEUChluX3A7QQ4mSWMYW39lFEjM1QKhsVYVjflGMl6ZnABZ7ysFdKyY6J5L0yRvI5kj9e1m6V2fP1zoUX1" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="294px;" /></span></span></td></tr>
</tbody></table>
The command 'docker start' creates a process space around the union view of the container's layers. There can only be one process space per container.</div>
<br />
<b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">docker run </b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;"><</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">image-id</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">></b><br />
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #fce5cd; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin: 0pt 0px;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Input</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td><td style="background-color: #fce5cd; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin: 0pt 0px;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Output</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a5c9-5df8-ab65-52916a4f75b9"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="63px;" src="https://lh6.googleusercontent.com/sqSxEut2WvWc6g6NXIfYiEL_afAJN_CChuHI_8_OUi1Th6xiL4qoio3O4v0cxvdl_5bXW9Ap2TyC8cwVSADocIV6IAlnXVZeDIF4I0cmJqwS8SEDQkNo7XouOwo_amiUDxXTNL5T" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="269px;" /></span></span></td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a5ca-63df-3edf-dc2519c73170"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="198px;" src="https://lh6.googleusercontent.com/jFnrB7tc94exRzat_Y8AYy0W6pWRsefNj1e2mbbiE_vp8aJD9igHOGm_3CQ0KRHVMQZnGv9t3TLEqderwSoifJ3AEfwUZ92B-351sXhbPKz82yympOu4WMMjkoc55-00vxuqIhtq" style="border: none; transform: rotate(0rad);" width="294px;" /></span></span></td></tr>
</tbody></table>
One of the first questions people ask (myself included) is "What is the difference between 'docker start' and 'docker run'. You might argue that the entire point of this post is to explain the subtleties in this distinction.</div>
<div dir="ltr" style="margin-left: 0pt;">
<span id="docs-internal-guid-57d74231-a63e-da8d-4671-182900aaff5a"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="189px;" src="https://lh6.googleusercontent.com/QkPJAtqPBVgM5zYPBP1R08xf4VzBWFX6XKmD5A9FA7kFXoyY_p0srZBxMgLnr4DG3EH86psP4GHxxwM4w4EcxSlJIc2uvqKw84HLCjh4VoRSE9hY8KhQRxTblUJTJW8MGwMymfuN" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="624px;" /></span></span></div>
<div dir="ltr" style="margin-left: 0pt;">
<span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;">As we can see, the docker run command starts with an <b>image</b>, creates a <b>container</b>, and starts the container (turning it into a <b>running container</b>). It is very much a convenience, and hides the details of two commands.</span></span></div>
<div dir="ltr" style="margin-left: 0pt;">
<span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;"><br /></span></span></div>
<div dir="ltr" style="margin-left: 0pt;">
<br class="Apple-interchange-newline" />
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #c9daf8; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><b>Tangent</b>: Continuing with the aforementioned similarity to understanding the Git system, I consider the 'docker run' command to be similar to the 'git pull'. Like 'git pull' (which is a combination of 'git fetch' and 'git merge') the 'docker run' is a combination of two underlying commands that have meaning and power on their own.<br />
<br />
In this sense it is certainly convenient, but potentially apt to create misunderstandings.</td></tr>
</tbody></table>
</div>
<div dir="ltr" style="margin-left: 0pt;">
<span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;"><br /></span></span></div>
<div dir="ltr" style="margin-left: 0pt;">
<span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;"><br /></span></span></div>
<b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">docker ps</b><br />
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #fce5cd; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Input</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td><td style="background-color: #fce5cd; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Output</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;">your host system</td><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a585-a512-a51d-afa374377617"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="197px;" src="https://lh6.googleusercontent.com/Jf8qFmbYmRpDRsK68Gj2GC2b4MOnBx-6o3Vsxr4stsSlMICjwqDg0LlN0k604GEXIPIb1JwcxHKdQHiSt7ap3oYze5eV1_rjSnLaau9EnZAuI4KToOsrwSyXb_Qm_Dc5IUDyt0bf" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="99px;" /></span></span></td></tr>
</tbody></table>
The command 'docker ps' lists out the inventory of <b>running containers</b> on your system. This is a very important filter that hides the fact that containers exist in a non-running state. To see non-running containers too, we need to use the next command.<br />
<br />
<b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">docker ps -a</b><br />
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #fce5cd; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Input</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td><td style="background-color: #fce5cd; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Output</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;">your host system</td><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a587-d1f5-e65b-9cd986fa9d4c"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="299px;" src="https://lh4.googleusercontent.com/9DAK16NqKViF7hL0EIaPC251unKJ27iu5xUXzUAs-hvSyHssi7YYCXkJfMW1m-Ejok7BEBnRasQiH34gkNgyu7hTx9FztYeoB2reGxXs3v8PsqRgRrJslTFhyXv_ZiHBOFJ8UTCq" style="border: none; transform: rotate(0rad);" width="99px;" /></span></span></td></tr>
</tbody></table>
The command 'docker ps -a' where the 'a' is short for 'all' lists out all the containers on your system, whether stopped or running. </div>
<br />
<b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">docker images</b><br />
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #fce5cd; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Input</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td><td style="background-color: #fce5cd; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Output</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;">you host system</td><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a5a3-8757-f3c5-5fbc23cdd874"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="123px;" src="https://lh6.googleusercontent.com/oxiD6NJmtDrxmWzmT4Lryg-XKVkzGC6NVZHFkXssOdYgpdJUb7Su_ejKbN7bLo7L8lWvecoLTpQByuT-H8IJKyiW6iNHdDXvrfnGMxhvO6A05ADYtzhuVHjep6wQi4jqGSdvw0MM" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="113px;" /></span></span></td></tr>
</tbody></table>
The 'docker images' command lists out the inventor of top-level images on your system. <b>Effectively there is nothing to distinguish an image from a read-only layer</b>. Only those images that have containers attached to them or that have been pulled are considered top-level. This distinction is for convenience as there are may be many hidden layers beneath each top-level read-only layer. </div>
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c">
</span>
<br />
<div>
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c"><br /></span></div>
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c">
<b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">docker images -a</b><br />
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c"></span></span><br />
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #fce5cd; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Input</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td><td style="background-color: #fce5cd; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Output</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;">you host system</td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a593-bb76-5de8-d17d114b8bd1"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><span id="docs-internal-guid-57d74231-a5a2-80ed-dab2-ba9eaba00e14"><span style="font-size: 14.6667px; vertical-align: baseline;"><img height="179px;" src="https://lh5.googleusercontent.com/wnBYSzJhbP0U0WEx5BFEeyI024yDecPqisMUpmNt4_dMDRef2s4rVI7S0e2kIzk4xDQ7li5TyAA3s1v_sUxjXidqUGNqkRaTIbG39DhLETmHWAAQhDL0q2zfByk8_ild0HrQ-dUk" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="156px;" /></span></span>
</span></span></td></tr>
</tbody></table>
This command 'docker images -a' shows all the images on your system. This is exactly the same as showing all the read-only layers on the system. If you want to see the layers below one image-id, you should use the 'docker history' command discussed below.</div>
<div dir="ltr" style="margin-left: 0pt;">
<br /></div>
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c"><span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c">
</span></span> <b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">docker stop </b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;"><</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">container-id</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">></b><br />
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #fce5cd; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Input</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td><td style="background-color: #fce5cd; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Output</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a5df-02c7-9202-09dfdf5ffd33"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="265px;" src="https://lh5.googleusercontent.com/2bk2xb6M4BqhezorrgugB5-PzD7WjMx7CEtq_Sx5evGHEYniPx0Zcl15m9Lvxip6fywo4nTvWPyEJHkynd3ksZ66eDVz7NmiVrA3H24gpAWNx7RdFY4ydNMo0svPU5j0CjO4_Rbk" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="294px;" /> </span></span></td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a5df-9b68-8c69-ce80e6e1a034"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="63px;" src="https://lh5.googleusercontent.com/U7jHOVkgsBMN5SmE7hxjrPkVffx-WnmqwIRENiac1-uph-s0xcU5hNgsFx_ufD4xofh_XjBYeIPNPUdNaauhioJgac76r8FIXGXQgnVEeqJD3C7iM3-HzVglB-XtGD5yeqYMk7ia" style="border: none; transform: rotate(0rad);" width="269px;" /></span></span></td></tr>
</tbody></table>
The command 'docker stop' issues a SIGTERM to a running container which politely stops all the processes in that process-space. What results is a normal, but non-running, container.</div>
<div dir="ltr" style="margin-left: 0pt;">
<br /></div>
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c">
</span> <b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">docker kill </b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;"><</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">container-id</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">></b><br />
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #fce5cd; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Input</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td><td style="background-color: #fce5cd; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Output</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a5e0-c11d-c091-369743f3dc79"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="265px;" src="https://lh5.googleusercontent.com/EB7P46KVLLBbFSZzsgIyUBTLK4ySEYav3XsOeY0MC7lnVtt5IcXfgps8K_c6KBcuSHkErtdeqw3tYfjqUbJGaxVgaIPeN6joEfvT1hpi6HBNDYEF-7jMMFIW2PEfVp0SHu4dplLB" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="294px;" /></span></span></td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><img height="63px;" src="https://lh5.googleusercontent.com/U7jHOVkgsBMN5SmE7hxjrPkVffx-WnmqwIRENiac1-uph-s0xcU5hNgsFx_ufD4xofh_XjBYeIPNPUdNaauhioJgac76r8FIXGXQgnVEeqJD3C7iM3-HzVglB-XtGD5yeqYMk7ia" style="border: none; font-family: Arial; font-size: 14.6667px; transform: rotate(0rad); white-space: pre-wrap;" width="269px;" /></td></tr>
</tbody></table>
The command 'docker kill' issues a non-polite SIGKILL command to all the processes in a running container. <strike>This is the same thing as hitting Control-C in your shell</strike>. (EDIT: Control-C sends a SIGINT)</div>
<div dir="ltr" style="margin-left: 0pt;">
<br /></div>
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c">
</span> <b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">docker pause </b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;"><</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">container-id</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">></b><br />
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #fce5cd; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Input</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td><td style="background-color: #fce5cd; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Output</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a5e4-d326-56d8-90657ac14aef"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="265px;" src="https://lh4.googleusercontent.com/nNJ8I2oAfD63z5Rt8BiVxfbGBbNpxlEcd6R6FHzTjPnOZ4xdwBjPLjjNdmB_tLFZvYxokgTrpndBR7KNI8edm9AID38DEUdXPAvydW-zFnulXHmHYY4qXXZyZVwUCiLqAL3YIL-W" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="294px;" /></span></span></td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a5e4-a99d-c207-1b86abacf539"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="212px;" src="https://lh5.googleusercontent.com/KC9FdwTHfAkDX_rOAygWKsZFR830pbA2T_hwNPIOjg2Nt2Xg9J9KDTTZEZbYnxuqnKQDSEZaHxu6TzgMK8nSYH6uWB4wTfN0FTxh2DgwRbKJONBzQtdCwmjPhTeEcBqVXDdXhK3R" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="309px;" /></span></span></td></tr>
</tbody></table>
Unlike 'docker stop' and 'docker kill' which send actual UNIX signals to a running process, the command 'docker pause' uses a special cgroups feature to freeze/pause a running process-space. The rationale can be found here: <a href="https://www.kernel.org/doc/Documentation/cgroups/freezer-subsystem.txt">https://www.kernel.org/doc/Documentation/cgroups/freezer-subsystem.txt</a>, but the short of it is that sending a Control-Z (SIGTSTP) is not transparent enough to the processes within the process-space to truly allow all of them to be frozen.</div>
<div dir="ltr" style="margin-left: 0pt;">
<br /></div>
<div dir="ltr" style="margin-left: 0pt;">
<br /></div>
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c">
</span> <b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">docker rm </b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;"><</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">container-id</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">></b><br />
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #fce5cd; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Input</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td><td style="background-color: #fce5cd; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Output</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a597-e63b-be4c-bef406f932c1"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="96px;" src="https://lh5.googleusercontent.com/kMUMRKEk86RhwsPOyXyOPcTC5A9MyYKovhja8-uYRASW64a6RtPiEPWiKqvPidscBypwdE-Jkjt5eUKL5vqRM-EhRAiRuDotLbVUzo9efcYG3IJ1_a1T5fhZW9RQs2B1G7MP8LVR" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="214px;" /></span></span></td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a590-30e9-5ecb-3cd07e1b7473"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="96px;" src="https://lh3.googleusercontent.com/a0aaZaswAE6X7-pl7inco6Q9KDof0FIYklLCF1sabcB9MGpK_6xNGURijrLWkMXTlErG1hj0kkIAEn4YR3YVz0bEmtdFB-t7rq2Mp6Rbx0iaAZ3P6QLjlUHFj2fkH8HuRy_eeMki" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="109px;" /></span></span></td></tr>
</tbody></table>
</div>
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c">
</span>
<br />
<div>
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c">The command 'docker rm' removes the read-write layer that defines a container from your host system. It must be run on stopped containers. It effectively deletes files.</span><br />
<br /></div>
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c">
<b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">docker rmi </b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;"><</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">image-id</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">></b><br />
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c"></span></span><br />
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #fce5cd; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Input</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td><td style="background-color: #fce5cd; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Output</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a597-4425-9be2-85c6adc9f616"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="98px;" src="https://lh3.googleusercontent.com/7JdT7KTREz2GBzxYujY9-8pW_RgcBDvGgAHOPlGPT_eoZF_IajRmK_fPnyONA7DzUCUOBt8NMU3jgUB8r0zHGv2LfKlm98YSxjUDEwfXNhw0haH2Mc_6VUq64SwsWyc70L054TCy" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="232px;" /></span></span></td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a596-22dc-97d0-df9fb3ba1546"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="98px;" src="https://lh5.googleusercontent.com/PcBNU2ISHRvkON9iFeCQ32rVQPrGo7mt0WlRgmytwDv5dhXJR5ZHbBwc0pWN7eFihzwlE0fqzS_4mEemddc6vWcUMqxNOK9xuRoC3uj56OdH5Jr4esvnGboHgoz8cw3O1Gq5X7yJ" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="99px;" /></span></span></td></tr>
</tbody></table>
The command 'docker rmi' removes the read-layer that defines a "union view" of an image. It removes this image from your host, though the image may still be found from the repository from which you issued a 'docker pull'. You can only use 'docker rmi' on top-level layers (or images), and not on intermediate read-only layers (unless you use -f for 'force').</div>
<div dir="ltr" style="margin-left: 0pt;">
<br /></div>
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c"><span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c">
</span></span>
<br />
<div dir="ltr" style="margin-left: 0pt;">
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c"><span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c"><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">docker commit </b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;"><</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">container-id</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">></b></span></span><br />
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #fce5cd; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Input</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td><td style="background-color: #fce5cd; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Output</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a5bd-e0d3-0ffa-10e31f57ef8d"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="72px;" src="https://lh6.googleusercontent.com/Rjb8_oTDpOE8gg2NPm812pnK0f6UJ7yIqkLLvDQ254QVcpDRrlF64i0TJchrXLcEsIaj6iQfJiNgBOi52BtPGZtW6aeW3vOPiGAcbnySq76d4Xt5pZCef49imm019bOIC24aF9nT" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="99px;" /> or </span></span><span id="docs-internal-guid-57d74231-a5be-159f-2757-c99349d9fcec"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="34px;" src="https://lh5.googleusercontent.com/uOLYoMVGGYofYxtSps2T0gti4496PSyW-U_aR2ohuA3Dy5mg53cupLmatSaTubLBPLw7j6y-EU7rCjqUEW9aPJaKfj9BNhbOoFJPkQXf2OqbS0acri05LexqOO85OR3dRrUxadHm" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="89px;" /></span></span></td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a5be-6be3-5630-2bc8dd97a6d0"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="34px;" src="https://lh5.googleusercontent.com/5MewjGV5xRycHlCnaYjvsU6LoUltw-Kd5QuKPIoPjT4h6YHMSmZEmUIKT2MW_yN4vlejHjf729dfsHZgR92cFoZisZ8FHfcgMRs1ZLaUH0QzYgdO6UhmT7ClpKMIuVlPVcBc0mNs" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="94px;" /></span></span></td></tr>
</tbody></table>
The command 'docker commit' takes a container's top-level read-write layer and burns it into a read-only layer. This effectively turns a container (whether running or stopped) into an immutable image.</div>
<div dir="ltr" style="margin-left: 0pt;">
<span id="docs-internal-guid-57d74231-a5f6-8a44-2eee-14eab605758f"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="142px;" src="https://lh5.googleusercontent.com/bIC8GEJXEzAX9waPTCdQhk_VfTwPhBndhlVs9QXjqgtPeGhI3HrS8ZSm7acbxfM-AGYdKiW9NeixKhEZ5v_1GEnaWUuZYb2lfMkJJwQXzGXcFB6U1rMj5-fQRsuLwpK1N4Sy5t53" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="581px;" /></span></span></div>
</div>
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c"><span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c">
</span></span>
<br />
<div>
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c"><span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c"><br /></span></span></div>
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c"><span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c">
<b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">docker build</b><br />
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c"></span></span></span><br />
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #fce5cd; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Input</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td><td style="background-color: #fce5cd; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Output</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;">Dockerfile: <span id="docs-internal-guid-57d74231-a5c3-1903-8291-fa75cbc031a4"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="53px;" src="https://lh5.googleusercontent.com/lh2RU-vE81gJERjUJYrrKgJb4feMdZNWB_uCsUZW6O655kBkwBJk1nZy2LgJSpC4vMo0Pg67GGOfQVosnpwW4A-3dnRsEAGHAN5e7jP1-kOaiu6QxntETlrsaxwrQ2GfBbTcnqA7" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="41px;" />
and a <img height="63px;" src="https://lh6.googleusercontent.com/sqSxEut2WvWc6g6NXIfYiEL_afAJN_CChuHI_8_OUi1Th6xiL4qoio3O4v0cxvdl_5bXW9Ap2TyC8cwVSADocIV6IAlnXVZeDIF4I0cmJqwS8SEDQkNo7XouOwo_amiUDxXTNL5T" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="269px;" />
</span></span></td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><img height="63px;" src="https://lh6.googleusercontent.com/sqSxEut2WvWc6g6NXIfYiEL_afAJN_CChuHI_8_OUi1Th6xiL4qoio3O4v0cxvdl_5bXW9Ap2TyC8cwVSADocIV6IAlnXVZeDIF4I0cmJqwS8SEDQkNo7XouOwo_amiUDxXTNL5T" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="269px;" /><br />
with many more layers added atop.</td></tr>
</tbody></table>
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c"><span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c"><span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c">The 'docker build' command is an interesting one as it iteratively runs multiple commands at once. </span></span></span><br />
<span id="docs-internal-guid-57d74231-a70d-13ff-6a1a-952cc93e189b"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="173px;" src="https://lh6.googleusercontent.com/s5QlqMAqbr9QBBEk5QvCMPrMl6Q4lnoaNwQyfc8YtF72iJWpIObUuO6Bk_gJbfRUekPzpFxoHVrc3brNHAslPI6aC-uqBWZFgcyVElRrtjP6rNF2pLvbtgQYGxCYOu1Buy2Mv85h" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="624px;" /></span></span><br />
<span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;">We see this in the above visual which shows how the build command uses the FROM directive in the Dockerfile file as the starting image and iteratively 1) runs (create and start) 2) modifies and 3) commits. At each step in the iteration a new layer is created. Many new layers may be created from running a 'docker build'.</span></div>
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c"><span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c"><span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c">
</span></span><br />
</span><br />
<div style="margin: 0px;">
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c"> <b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">docker exec </b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;"><</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">running-</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">container-id</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">></b></span></div>
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c">
</span>
<br />
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #fce5cd; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin: 0pt 0px;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Input</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td><td style="background-color: #fce5cd; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin: 0pt 0px;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Output</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a5bf-daa4-995f-0a8a176d7e9c"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="72px;" src="https://lh6.googleusercontent.com/Rjb8_oTDpOE8gg2NPm812pnK0f6UJ7yIqkLLvDQ254QVcpDRrlF64i0TJchrXLcEsIaj6iQfJiNgBOi52BtPGZtW6aeW3vOPiGAcbnySq76d4Xt5pZCef49imm019bOIC24aF9nT" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="99px;" /></span></span></td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a5c1-1278-756c-38ddbbfd448e"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="72px;" src="https://lh3.googleusercontent.com/8HYkg4IPuryVgNTqnkGOma4OjKm6zRd_i1hFNhOgmIQH0jrhR1Sl3kkSZ7VczKzWSKuXLCgsLRlcnSpTTM6zmS7SGOVwT0pO4nMoJTtPqQV3BLZMJvTri94SphVoe7AU5Qq8qqGZ" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="192px;" /></span></span></td></tr>
</tbody></table>
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c">The 'docker exec' command runs on a running container and executes a process in that running container's process space.</span></div>
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c">
<br />
<b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">docker inspect </b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;"><</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">container-id</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">> or </b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;"><image</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">-id</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">></b><br />
</span><br />
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #fce5cd; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Input</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td><td style="background-color: #fce5cd; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Output</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a60c-faf9-6554-8f5343638a76"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="69px;" src="https://lh6.googleusercontent.com/yyU5ZNqnvqzQaWSNQrwpM04X1KkPr00MsOp3SEnYTpsnicNfEGA6tb6ssYh7xLMFl8kk3PNRTiDvvBwkUNc9i5E1ZTVRLiqwHeoVVdkEwHJed2MstwES3ojKwwpcRBoe_cn68mE_" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="269px;" /></span></span><br />
<span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">or</span><br />
<span id="docs-internal-guid-57d74231-a60e-441f-d72b-fcd951366f08"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="71px;" src="https://lh5.googleusercontent.com/tCBsp5fYjJ7QcL4GhBAABWTWQlu9c-y7UYZrWdr62dpIKawsHxwRJZ_P4tdP2xueC7Vts1GXnCk7MZdTyCtKPVVPdxI_lZXzFaBEzllDr6lSqZAwet_XLdMobH2pPsq8SB2htqkZ" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="269px;" /></span></span></td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a60e-fad7-f118-00a56ca73a26"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="35px;" src="https://lh6.googleusercontent.com/9CUAUKcKF1FD8GjoxlFGe_dSoN53FToBoxkkPfrJFCb5gJIpZVCcpQLYfNtgmKsHlB3FXBKbx_UUUoo1rb17xMbkS-98Bh5zaeGdoFQlcUGe44IUBN_-IcV-pgOqdD4M6LUtGz-r" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="23px;" /></span></span></td></tr>
</tbody></table>
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c">The command 'docker inspect' fetches the metadata that has been associated with the top-layer of the container or image.</span></div>
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c">
</span>
<br />
<div dir="ltr" style="margin-left: 0pt;">
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c"><br /></span></div>
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c">
</span> <b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">docker save </b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;"><</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">image-id</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">></b><br />
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #fce5cd; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Input</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td><td style="background-color: #fce5cd; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Output</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a619-7bb5-152b-a47c429b1ad6"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="85px;" src="https://lh4.googleusercontent.com/dYatT6CYn0TQv6TRhhz5wmv6n0kpH9zYQ18f0kXpZBeTTyHU0VvuqF4hVJ-DEkn2m04g0kgmbdKa39o1aRKSiHSmtfEauZV-cP7tTwPy3DQKFCHU4olsn0Vhsp09YgJWaZnVnRR2" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="173px;" /></span></span></td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a61a-0ca8-0dde-7a2a29fddc8c"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="234px;" src="https://lh6.googleusercontent.com/-FtqgE3Si5dpKhUxNuhVq18Gh_onv2dILPhKqAmJYbyVogHVZUKc1oldsOunW1IdcN9HrGLziRSXKIVZL6oG0QHFz48N52YAoxd7nMcVecJ7b93t8zBTvZ8c2wgbFvIclegQbD53" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="287px;" /></span></span></td></tr>
</tbody></table>
The command 'docker save' creates a single tar file that can be used to import on a different host system. Unlike the 'export' command, it saves the individual layers with all their metadata. This command can only be run on an image.</div>
<div dir="ltr" style="margin-left: 0pt;">
<br /></div>
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c">
</span>
<b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">docker export </b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;"><</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">container-id</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">></b><br />
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #fce5cd; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin: 0pt 0px;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Input</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td><td style="background-color: #fce5cd; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin: 0pt 0px;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Output</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a61d-1734-0920-db82b8e05efd"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="103px;" src="https://lh3.googleusercontent.com/qZ8iVnQg4mwgf4y0eBefMz3a23UJZ4oHe0p9ff2APGJaTnoQyGlxHJ_pzys13zt6-txEk1MeHkPPpIiiIr7vzFU-JTRXbCCNXAa5-k-IaE3T79pQzm2tpoe--tC4yyy0fnMd2F6h" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="283px;" /></span></span></td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a61d-355c-58c7-cd3cc9b0344c"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="234px;" src="https://lh6.googleusercontent.com/Xd0BNMHB-tjJLGGAA5VFmAm8RBA05_qYoJmzSIiICCZRyYz1Haqu3HpY4k4jUrIG8T4b7s1IUslypApoVDNIRv7gdRttm5TqMmM5x0a5BRo8K_dP84SrWMi4wdcOihwj6oFFZWRI" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="287px;" /></span></span></td></tr>
</tbody></table>
The 'docker export' command creates a tar file of the contents of the "union view" and flattens it for consumption for non-Docker usages. This command removes the metadata and the layers. This command can only be run on containers.</div>
</div>
<div dir="ltr" style="margin-left: 0pt;">
<br /></div>
<div dir="ltr" style="margin-left: 0pt;">
<b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">docker history </b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;"><</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">image-id</b><b style="font-family: 'Courier New', Courier, monospace; font-size: x-large;">></b><br />
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #fce5cd; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Input</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td><td style="background-color: #fce5cd; border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Output</span><span style="background-color: transparent; font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> (if applicable)</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a637-3626-9390-c5eeaf48dcd5"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="119px;" src="https://lh3.googleusercontent.com/ow_wnmrkLzG1WS91VFoymb7O06Eb6glrtDAtps875arvBBooBHcZY_bfOU63vxlc4wnIvQCYhB80y7cZVFiThUeasFY0zJIq_H4zkzXZRCRWCJHFxmMW6-721SaF2S3Dib9PRVi8" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="232px;" /></span></span></td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span id="docs-internal-guid-57d74231-a637-e6e7-4ed4-f50b84626945"><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"><img height="125px;" src="https://lh5.googleusercontent.com/Vd8l-41r89rvS65tvDS44YIS6w6z1tqIPIvIByWAGBaHgpGlX8iqQO8OcqCbvgDdVnf49YDNHtvAs8XciyjwPKB67vlB6AdMoYcX7xCObjpZcVoRSqXzQX6HMdCng_h8U4Oi6lpa" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="269px;" /></span></span></td></tr>
</tbody></table>
The 'docker history' command takes an image-id and recursively prints out the read-only layers (which are themselves images) that are ancestors of the input image-id.<br />
<br /></div>
<h3 style="margin-left: 0pt; text-align: left;">
<span style="font-size: large;">Conclusion</span></h3>
<div dir="ltr" style="margin-left: 0pt;">
I hope you enjoyed this visualization of containers and images. There are many other commands (pull, search, restart, attach, etc) which may or may not relate to these metaphors. I believe though that the great majority of docker's primary commands can be easier understood with this effort. I am only two weeks into learning docker, so if I missed a point or something can be better explained, please drop a comment.</div>
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c">
</span>
<br />
<div dir="ltr" style="margin-left: 0pt;">
</div>
<span id="docs-internal-guid-57d74231-a588-9468-c89c-1ffaba509d6c">
</span></div>
</div>
Daniel Eklundhttp://www.blogger.com/profile/11570452431861145598noreply@blogger.com466tag:blogger.com,1999:blog-5887985082054147381.post-45119143880140803732014-04-27T16:32:00.005-07:002014-07-09T06:06:42.247-07:00What the Heck is Base64 Encoding really?<div dir="ltr" style="text-align: left;" trbidi="on">
<h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;">
<span style="color: #666666; font-family: 'Trebuchet MS'; font-size: 17px; font-style: italic; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;">Also, Why most implementations of </span><a href="http://www.crockford.com/wrmg/base32.html" style="line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; font-family: 'Trebuchet MS'; font-size: 17px; font-style: italic; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Crockford-base32</span></a><span style="color: #666666; font-family: 'Trebuchet MS'; font-size: 17px; font-style: italic; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;"> encoding are probably incorrect</span></h1>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #c9daf8; border-bottom: solid #b7b7b7 1px; border-left: solid #b7b7b7 1px; border-right: solid #b7b7b7 1px; border-top: solid #b7b7b7 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">TLDR</span><span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">: </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;">The term </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;">base64 encoding</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;"> is an overloaded term which promotes ambiguous usage. It can mean either a place-based 64-symbol encoding of a single number, or a streaming algorithm for encoding an array of octets. In common usage, it is the latter -- the </span><a href="http://tools.ietf.org/html/rfc4648" style="line-height: 1.15; text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">same technology and IETF RFC specification</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;"> that allows us to encode arbitrary binary files and place them in USENET posts and emails. </span></h3>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Examining the differences between the two meanings and two implementations will teach us something about ambiguity, online/stream algorithms, and that specifications are really really very important.</span></div>
</td></tr>
</tbody></table>
</div>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #c9daf8; border-bottom: solid #b7b7b7 1px; border-left: solid #b7b7b7 1px; border-right: solid #b7b7b7 1px; border-top: solid #b7b7b7 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">TLDR</span><span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">: </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;">The Crockford-base32 encoding proposal, as written here</span><a href="http://www.crockford.com/wrmg/base32.html" style="line-height: 1.15; text-decoration: none;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://www.crockford.com/wrmg/base32.html</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;">, is not as clear as it could be and lacks a canonical implementation and/or test-cases. </span></h3>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This lack of clarity has lead most people to interpret it as a place-based 32-symbol encoding of a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">single number</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. However, the particular wording and context of his proposal implies that the Crockford-base32 encoding was meant to adhere closer to the existing base64 and base32 octet binary encodings (including padding of binary data up to a multiple of the 5-bit space). </span></div>
</td></tr>
</tbody></table>
</div>
<h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The Presence of An Ambiguity</span></h2>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">I want to use this post as a way of clearing up a confusion that seems to exist in many people's understanding of base64 (and also crockford-base32). </span><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Because on ambiguity exists, I am going to invent my own clarifying terminology (</span><span style="color: blue; font-family: 'Droid Sans'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Place-Based Single Number Encoding</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> versus </span><span style="color: blue; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Concatenative Iterative Encoding</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">) to attempt to disambiguate. Along the way, we will get into the nitty gritty details of binary base64-encoding (as commonly implemented in a thousand different systems) and of the radix-placed system that most people are taught as binary, octal, decimal and hexadecimal numbers.</span><br />
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;"><br /></span>
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">But before I do that, let me explain what the Crockford-base32 encoding is meant to do, for those of you who have never heard of it.</span></div>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #c9daf8; border-bottom: solid #b7b7b7 1px; border-left: solid #b7b7b7 1px; border-right: solid #b7b7b7 1px; border-top: solid #b7b7b7 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Human Readable URLs</span></h2>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Producing URL shortening and youtube-like resource strings seems to be the raison d'être for the Crockford and other such encodings. Many famous web applications have something like this: a unique string of letters and numbers that when used within a URL/URI, provide a unique and identifying name for that resource. Resource examples range from videos, to shortened URLS, to collabedit pages, to flickr accounts. </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img alt="Screen shot 2014-04-25 at 5.04.00 PM.png" height="59px;" src="https://lh5.googleusercontent.com/x9BJtl7vJy-yLByhmfV-zxANP4Q_wFttGZqIGAt9Vb_kAePW674OV4jPWr8JOVvnwlZAcG9g3c-2h4b5_ctX1rC5Cg3ZopnTQDHqQsllh2VTjYxUJWy7lwHE3GX-EHQzSQ" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="364px;" /></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">With all these services, we desire to provide a unique identity to a resource whose cardinality may grow into the millions or billions. Certainly, the identity could be something as simple as an integer increasing in value (what the database guys call a sequence). The drawback with a sequence is that it provides people a way to guess resource identities. To counter this, we could generate a random number with insignificant chances of repeating (and of guessing) and encoding it in a high-radix encoding to shorten its string-length.</span></div>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Let's examine two ways to do this.</span></div>
</td></tr>
</tbody></table>
</div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<hr />
<span style="font-family: 'Trebuchet MS'; font-size: 17px; font-weight: bold; line-height: 1.15; white-space: pre-wrap;">Radix Place-Based Single Number Encoding System</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This system is the easier of the two to explain, as it is covered in most high-schools, and CompSci 101. Nevertheless, we will go through it again.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="103px;" src="https://lh3.googleusercontent.com/ZcsIfyJrVs2Ux4QLJMo6KwFA0ERVDZ6k1mWBwwhz27cgXWU1aVRNyQECe3hzhV7d6aBAdzqJOOMzbe5AD6n-lmQpCCPT-6jBBNVkij1aHMYTUv_qrAPlSTbA94OQuCHjww" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="280px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">We start with the number </span><span style="background-color: transparent; color: black; font-family: Impact; font-size: 24px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">101</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. </span><br />
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;"><br /></span>
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">What is this number? Is it the number of dalmatians that should be brutally murdered, skinned, and fabricated into a woman's coat? Or is the cost of a 5 dollar foot long?</span><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">The answer is that it could be either. The number is the ultimate abstraction, that which your brain </span><span style="color: black; font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">interprets</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">, and the glyphs and the symbols are the permanent memory of that number on paper or on disk. But in-between, we need to apply a decoding to turn interpret the glyphs from symbols through numerals to a number. In </span><span style="color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">binary</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">, the glyph/symbol/numeral string of "101" can be interpreted as "five" -- just enough dollars to buy a Subway sandwich. In </span><span style="color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">decimal</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">, which most humans (without compsci/math background) take for granted, the string "101" is interpreted as "one hundred and one."</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="158px;" src="https://lh4.googleusercontent.com/_6ZpjRiswxYAMxW0uIRp3_VVJdrv9RFEiMsIBZsX5YknqaH7iBShjO1zom-7Rewj_rj_s_HZ9mpdxTLORO9xiGZMRs04M6KLzcupWQghmh-isSqSAGc2vhuS2VwxdkS2uQ" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="240px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">The term radix, which is synonymous with the term </span><span style="color: black; font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">base</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">, is defined as </span><span style="color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">the number of unique symbols in a place-based numeric system</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">. The radix-system that most humans use is a base-10 or radix-10 system. The symbols we have today for our numbers came to us during the Renaissance from the Arab mathematicians. We called them arabic numerals, despite the fact that the Arab mathematicians imported them from the Indians who invented the numeral 0. This special numeral 0, along with the other nine symbols, make up the most famous and consistent symbols that the world has ever seen. We all collectively own these symbols today: '0', '1', '2', '3', '4','5','6','7','8','9', and we assume that they are in this order: </span><span style="color: black; font-family: Impact; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">0123456789</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">.</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="95px;" src="https://lh4.googleusercontent.com/TAWdOpalDfS_kojFrgExClEDfP-TvnsIodtf8ldgi7UQdVGqeyn8P3XrJxq-oDVvmSnNPMxG7Pdkhpsd-Pcr_iDj8dLYHJqOOPDENPX_cQRt_p1QqqG6FWV-hmPrz2fonw" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="624px;" /></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To restate, the reason that we call this a radix-10 system is </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">because</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> we have 10 unique symbols. However to be a base-10 system, we do not have to use these exact symbols or in this traditional order. How weird a parallel universe there must be where, due to a transposition error, our system could have been </span><span style="background-color: transparent; color: black; font-family: Impact; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">012346</span><span style="background-color: transparent; color: black; font-family: Impact; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">5</span><span style="background-color: transparent; color: black; font-family: Impact; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">789</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">? Fifty would be "60." </span></div>
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Place-Based Systems</span></h3>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Why are the numbers made from these symbols (of any radix) called </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">place-based systems</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">? The answer comes from the fact that every numeric representation is a string where the place represents a power in that radix from right to left. </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="135px;" src="https://lh3.googleusercontent.com/dA3j8raIOYPY4R6KglkfvIp-9fCbUVYc5FD1xvTa1y-1DPTsnJDgayGHW2kr15yEn2kADZZPIleL9P50RM8xN2J0Ceq8_7aRNsKxZsRf_hBoZDT9FxWUmyWBrBaYlatvMw" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="259px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As you can see above, the simple decimal number 1853 has four </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">places</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">:</span></div>
<ol style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">farthest to the right, the ones </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">place</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 10</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 9px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: super; white-space: pre-wrap;">0</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">second to the right, the tens </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">place</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 10</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 9px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: super; white-space: pre-wrap;">1</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">third from the right, the hundreds </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">place</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 10</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 9px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: super; white-space: pre-wrap;">2</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">and fourth from the right, the thousands </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">place</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 10</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 9px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: super; white-space: pre-wrap;">3</span></div>
</li>
</ol>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">and the string ‘1853’ is thus a representation of a number using places to indicate powers.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
</div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">A famous example of a numeral system that is </span><span style="color: black; font-family: Arial; font-size: 15px; font-style: italic; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">not</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> place-based is the </span><a href="http://en.wikipedia.org/wiki/Roman_numerals" style="line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Roman Numeral</span></a><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> system. A Place-Based System applies not just with decimal, but with all the radix-based systems we see below.</span><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #c9daf8; border-bottom: solid #b7b7b7 1px; border-left: solid #b7b7b7 1px; border-right: solid #b7b7b7 1px; border-top: solid #b7b7b7 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Binary Place-Based Single Number Encoding System: </span><span style="background-color: transparent; color: #666666; font-family: Impact; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">01</span></h3>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Standard coursework tells us of binary, a radix-2 system using (by definition) two symbols: '0' and '1'. In this base, the number 1 is '1', the number 2 is '10', the number 3 is '11', etc. This is the numeral system of computers and disks that provides the unifying clutch between voltage and symbolic logic, truth, turing machines, and computation. </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="220px;" src="https://lh4.googleusercontent.com/aPxoho-kXHHRbYHhq_h-c3lEtpt_nrJBPC0K7xqfr300CqrX2qagbLu_lLZltMFMuYGe5l7rwyE_4jzt-2jYotmYOH7pdGBvctv4ktWa72mcFVNyLOj35JWhIln_R31AUQ" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="312px;" /></div>
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Octal Place-Based Single Number Encoding System: 01234567</span></h3>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If we have a base-8 system, we can only use eight symbols. Reusing our first eight symbols from the base-10 system we get: </span><span style="background-color: transparent; color: black; font-family: Impact; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">01234567</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. In this base, the number seven is 7, but the number eight is 10 because we have run out of symbols. We call these octal numbers. We </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">should</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> call them octal </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">numerals</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.</span><br />
<img height="43px;" src="https://lh6.googleusercontent.com/bpJIFkjop1lIX1ZJo7Z2jCix2Hga7Tdnhq9T2KImCt_Z4inmss_nKiimqnJ_e9X-KYAa4uh1__hpToICAC0X91MlLuYZ3nSx-mdH82VYfc3gXXldeXTv6FuXb9BqXrHixQ" style="-webkit-transform: rotate(0rad); border: none; line-height: 1.15;" width="126px;" /></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Hexadecimal Place-Based Single Number Encoding System: 0123456789ABCDEF</span></h3>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">And what if we have more than 10 symbols? Hexadecimal, a base 16 system, is a good example of one. We commonly choose to use some letters from the latin alphabet (whether capitalized or not) as the 6 remaining symbols. Thus our alphabet of symbols in order is : </span><span style="background-color: transparent; color: black; font-family: Impact; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0123456789</span><span style="background-color: transparent; color: black; font-family: Impact; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">ABCDEF</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. In this base, the number 15 is F, and the number 16 is 10. Most people are familiar with hexadecimal numerals usage in RGB values. </span><img height="199px;" src="https://lh4.googleusercontent.com/n9s7GtmIPHYd-hRuNDOtj2m5vA6wIL_wg0jGV-o1JvtmMsUEN3S_VHTwOLnAVELujqUsPCE0N3Gdeb_GLbCAvttxzTu3tuqdWnsiluVgqaZd3kzzbFVTs_wzdl3OIfVCtw" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="390px;" /></div>
</td></tr>
</tbody></table>
</div>
<b style="font-weight: normal;"><br /></b>
<br />
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Tetrasexagesimal Place-Based Single Number Encoding System</span></h3>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">For base64, we can use the lower-case alphabet </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">and</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> the upper-case alphabet: </span><span style="background-color: transparent; color: black; font-family: Impact; font-size: 15px; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">abcdefghijklmonpqrstuvwxyz</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and </span><span style="background-color: transparent; color: black; font-family: Impact; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">ABCDEFGHIJKLMNOPQRSTUVWXYZ</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. But this only gives us 26*2= 52 symbols, and we need 64 symbols. The symbol alphabet of numerals </span><span style="background-color: transparent; color: black; font-family: Impact; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0123456789</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> gives us ten more for a grand total of 62 symbols. We need to add two more. </span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">For now, let's just choose the plus symbol '</span><span style="background-color: transparent; color: black; font-family: Impact; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">+</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">' and the slash symbol '</span><span style="background-color: transparent; color: black; font-family: Impact; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">/</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">'. Because we can order these symbols in any order we want, how about the following ordering?</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> Value Encoding Value Encoding Value Encoding Value Encoding</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 0 A 17 R 34 i 51 z</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 1 B 18 S 35 j 52 0</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 2 C 19 T 36 k 53 1</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 3 D 20 U 37 l 54 2</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 4 E 21 V 38 m 55 3</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 5 F 22 W 39 n 56 4</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 6 G 23 X 40 o 57 5</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 7 H 24 Y 41 p 58 6</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 8 I 25 Z 42 q 59 7</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 9 J 26 a 43 r 60 8</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 10 K 27 b 44 s 61 9</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 11 L 28 c 45 t 62 +</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 12 M 29 d 46 u 63 /</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 13 N 30 e 47 v</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 14 O 31 f 48 w </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 15 P 32 g 49 x</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 16 Q 33 h 50 y</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In this base, the number seven is 'H', the number 50 is 'y', the number 62 is '+', and the number 64 is 'BA'.</span><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">But now, here is the big question: Is this the same base64 encoding that was used to pass binaries and dirty images on USENET posts? Is this the base64 that most languages, browsers, and email-clients support with library functions? Is this the base64 of </span><a href="http://tools.ietf.org/html/rfc4648" style="line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">RFC 4648</span></a><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">?</span><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">The answer is a big fat </span><span style="color: black; font-family: Arial; font-size: 19px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">NO</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">. To explain why is not the same thing, we will make a digression and consider the notions of functions and types.</span></div>
<h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Functions and Types, Domains and Ranges</span></h2>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Types and functions are utterly fundamental to the art, science, and craft of programming. </span><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">One might say that </span><span style="color: black; font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">types</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> allow us to think about data, while </span><span style="color: black; font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">functions</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> allow us think about computation. Functions and types come clothed in different syntactical and run-time forms in any of your favorite language, but they are omnipresent and never to be discounted.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="73px;" src="https://lh4.googleusercontent.com/7_BHgDuOM_qJACsMwSVZaNI8no0ciklEt8EGGKUHd9f_AF2KfjXZQe4NjgL8XMNRKvAejgX4rhBxHIFWSmg9SQc_NXUElwlicQDOgitmwcjr8QVoU0HMHH71rwTKqjT9HQ" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="236px;" /><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">So why are we bringing them up? The answer is because we they will be helpful in explaining the difference between </span><span style="color: blue; font-family: 'Droid Sans'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Place-Based Single Number Encoding</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> and </span><span style="color: blue; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Concatenative Iterative Encoding</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">.</span><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Up to now, we have been describing the notion of radix and base to represent a number, a system that we called </span><span style="color: blue; font-family: 'Droid Sans'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Place-Based Single Number Encoding</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">. To help us understand this encoding, let us describe it as a function with an input and an output.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="49px;" src="https://lh4.googleusercontent.com/VyJ9pTyMPaG4g-YQSFpt8rbvbv03hdYxC2hr7an5S1lIY6pX9QEglEAyaiV4NF9KzJz4NJbLB_h0ElyAomK5Do-LtqP-c77wyuhRkZUXzJiOB07TRDPVhiuMvFchH-_a3Q" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="624px;" /><br />
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">A representation is nothing more than a string of symbols in some alphabet.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="49px;" src="https://lh3.googleusercontent.com/NFeEVwz0fV37_ewmBeQKOvltRCmORViRKad1SMmBlUtOWLffCLyqtGvRS1LbsqJ8WvbIrOUW196S7A_iD6sS5Rw_5eYPtl9wi3Ky0M4FylekPM6Y5I65jS74gPicEzd1tA" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="624px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Thus, the number 101 is the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">decimal</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> representation of the number “one hundred and one” while 65 is its </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">hexadecimal</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> representation. Both 101 and 65 are strings of symbols.</span><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">This is simplistic, but really exposes the essence of what we have been doing thus far. When we wanted to convert a single number into a binary representation we implicitly knew that the </span><span style="color: #e06666; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Domain</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> of our function (our </span><span style="color: #e06666; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">input type</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">) was that of a number, and that the </span><span style="color: #6aa84f; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Range</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> (the </span><span style="color: #6aa84f; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">return type</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">) was that of a string of zeros and ones. We can denote this function signature like this:</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="28px;" src="https://lh3.googleusercontent.com/X65p65yhf7g-_SBpC2iD9Bu9WsNohaJBhfBAO81zD9zTOW5QUxCcHY5kEQIqdYnl5WuHz2gjswVKQaxjllZlPH9dTt6xAHrFebr_AAVXbOo8OSlKAp66qTCLrgIztfKIWQ" style="-webkit-transform: rotate(0rad); border: none;" width="624px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Ok. It feels like we burdened the obvious with theory. How does this help explain what email-based base64 encoding is? Well, the answer is to show the difference in their function signature, types and all.</span><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">What the base64 encoding of </span><a href="http://tools.ietf.org/html/rfc4648" style="line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">RFC 4648</span></a><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> does, and why it is different, is this:</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="40px;" src="https://lh4.googleusercontent.com/d3BAVFG4RZfZqQGUU8n0IJJfHLX7jbqwM81-pS7Ueq5YP9VDgRll8fnAyKJ2S4dsLW1rPJeG33xSJuqD90wjjMZ3I7wJZ5rUwJn3JRGYq7iUCrP57oxn2O4tcOOMjEUKZg" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="624px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It iteratively consumes an array of octets, transforms them into Base64 representations, and concatenates each representation into an output array.</span><br />
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;"><br /></span>
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">In conclusion, we are saying one way of seeing the difference between these systems is by comparing their signatures</span></div>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none;"><colgroup><col width="102"></col><col width="243"></col><col width="279"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><br /></td><td style="background-color: #d9ead3; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: blue; font-family: 'Droid Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Place-Based Single Number Encoding</span></div>
</td><td style="background-color: #d9ead3; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: blue; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Concatenative Iterative Encoding System</span></div>
</td></tr>
<tr style="height: 0px;"><td style="background-color: #b6d7a8; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Function </span></div>
<div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Signature</span></div>
</td><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<img height="50px;" src="https://lh4.googleusercontent.com/i0vxy8wf4Pk-cuyEPx_f2DIQWPfW6ZnLemJNJ9ccT1o5JPgptD-mHSr5qHz3d2-06jeq6jseldXGZ1w-grFAq-cKSGeCAM_186pnNN4wzv7p7Q8vo2YxmksPDh8TLE5WxQ" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="189px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
</td><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<img height="48px;" src="https://lh4.googleusercontent.com/pU7CgjDyBLE8bu4AglQfKqjWq4jVyVrazHnQmHA4lgFbPPHS3sRCG3ngsNMJgFH_VhiDA81KB1HTH4HiSkpxA5ilPoZILPvDeRvMYK29FPCgXDBr4VB0V_9XZF87CbnZjQ" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="262px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
</td></tr>
</tbody></table>
</div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Does that clear it up?</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="168px;" src="https://lh3.googleusercontent.com/7N8bAQRiDMPliv_zKe6bw0PHmSKO8OCwyh7W3zZ1FN0G35FrElqonKKSYz-YXpm0wXWxunAxn27haT5u70mJLLa5O16Db5JxAgbEYMYbJPr0UmP13aePILxw5jk4oltLkw" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="300px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<hr />
<span style="font-family: 'Trebuchet MS'; font-size: 17px; font-weight: bold; line-height: 1.15; white-space: pre-wrap;">Concatenative Iterative Encoding System</span><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Yup, the term </span><span style="color: blue; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Concatenative Iterative Encoding</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> sounds like a lot. But if we look at each word and examine its function signature, we will find it to be a description that is succinct and accurate.</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="40px;" src="https://lh4.googleusercontent.com/d3BAVFG4RZfZqQGUU8n0IJJfHLX7jbqwM81-pS7Ueq5YP9VDgRll8fnAyKJ2S4dsLW1rPJeG33xSJuqD90wjjMZ3I7wJZ5rUwJn3JRGYq7iUCrP57oxn2O4tcOOMjEUKZg" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="624px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Let's dissect both the term and the function signature, using the following visual of a file to motivate. </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="242px;" src="https://lh6.googleusercontent.com/EwGqqZYhrQkjg3-o9O9f2IEVwy1kEbQjG1pKvvNNUVbVr_bmMPQWYxQB1cP8ycM_D_7nvjzXs8UjXflqqJSSfczfDhYy6WEI0Z5rO3kX3q3MvEgAOWutft4WqyE8r9n5_g" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="326px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">What we see above is a file (any file) represented as a long tape of zeros and ones. I have purposely colored four different groups of octets. Grouping bits into 8-bit chunks, called octets or bytes, makes it easier for us to think about the sheer number of zeros and ones. All files, whether text or not, are ultimately a concatenation of bits, so this is a valid way of understanding a file. </span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="285px;" src="https://lh4.googleusercontent.com/OcTM6NGjubEuaihVLTVNG1d02D9MuJpInppQQm-i7CvAIGEuqzBwnSfcpiN-sxz5hSAkAnjXHE72vEpoLKF1VvEgEfzSooRjWeONOvvFErt1TNlF0esX_bPz4GYb7gDWYw" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="493px;" /><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">We can imagine the encoding function as a box that iteratively works its way down the tape, turning octets into base64 atoms. As the box/function moves its way down the tape, it slowly builds up the output tape by concatenating newly encoded base64 atoms. We are iteratively consuming octets from an array </span><span style="color: #e06666; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">[OCTET]</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> and concatenating them onto an output array of base64 representations </span><span style="color: #6aa84f; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">[BASE64_REPRESENTATION]</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="197px;" src="https://lh5.googleusercontent.com/tHyXx-QhXYxpNNZD0INPaVK2PhZ9pSnPkLp7k0rFoKNA8lYtfYrxRphvrzOnrRE9kICEKCyhqZyJY4Gnqm0H_NPwPgvr0RmmLECP6PSaIMr71-XHLIN1FOHzc0rV6R70rg" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="514px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">The implementation details of how we transform octets into numerals from a base64 are actually quite simple. I will go through them now, but the following sites provide more exhaustive descriptions:</span><br />
<ol style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<a href="http://en.wikipedia.org/wiki/Base64" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">The wikipedia page</span></a></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<a href="http://www.nczonline.net/blog/2009/12/08/computer-science-in-javascript-base64-encoding/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Computer science in JavaScript: Base64 encoding</span></a></div>
</li>
</ol>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">We start with a stream of octets which means that we are grouping a stream of binary bits by 8 bits each.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="103px;" src="https://lh6.googleusercontent.com/FXDHxcG3lBuH37SuWnq2BL4NB-GJOl5PSXp6SRg_WsR4dfwTG7uCLmeSeX6KRF6iZpyjS_Kse3WS6PRd-LHP4P5Z-KtViU3Rqj6lPEPOCBDvTLq3g4EaZxbOhUEOj_TwOA" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="478px;" /><br />
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">As you can see above, by merely choosing to put a grouping marker between every 8-bits, we can interpret this stream of 24 bits as 3 octets. Furthermore, we can choose to interpret each octet as the decimal numbers shown above, thus we see the numbers 77, 97, and 110. This example comes from the wikipedia article, showing how a text input could have been encoded, where the ASCII values for the word ‘Man’ are the input octets -- in ASCII ‘M’=77, ‘a’=97, and ‘n’=110.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So what does base64 encoding do? Does it transform the input at all? No. It merely adjusts the grouping markers. Where before we grouped by 8 bits, now we will group by 6-bits.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="105px;" src="https://lh6.googleusercontent.com/B36sVeN-IBChYkX0bEV810elJIFwjm9UQ2_OEuslv6LVv3BvD32oK0rBSuMLCX0ClMVTIzi7G91TwlRrm67OlioqRiqoxcIG0nkDrFSb33MNCbscWimlK-LKvsGwMbMs3A" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="478px;" /><br />
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">As you see above, we have reinterpreted 24-bits of data as 4 groups of 6-bits (sextets) rather than 3 groups of octets. Again, putting a decimal number for these new groups, we see the numbers 19, 22, 5, 46. But remember that base64 symbol table we had above? I will copy it down here again:</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> Value Encoding Value Encoding Value Encoding Value Encoding</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 0 A 17 R 34 i 51 z</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 1 B 18 S 35 j 52 0</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 2 C 19 T 36 k 53 1</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 3 D 20 U 37 l 54 2</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 4 E 21 V 38 m 55 3</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 5 F 22 W 39 n 56 4</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 6 G 23 X 40 o 57 5</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 7 H 24 Y 41 p 58 6</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 8 I 25 Z 42 q 59 7</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 9 J 26 a 43 r 60 8</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 10 K 27 b 44 s 61 9</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 11 L 28 c 45 t 62 +</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 12 M 29 d 46 u 63 /</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 13 N 30 e 47 v</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 14 O 31 f 48 w </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 15 P 32 g 49 x</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 16 Q 33 h 50 y</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As we look at it, we can assign the number 19 to the base64 encoded numeral ‘T’, ‘W’=22, ‘F’=5, and ‘u’=46. Thus,</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="240px;" src="https://lh6.googleusercontent.com/LFSWeH4C5jYsLhFAUFQTYQ5J05WmX3jy3p9JGOY4FmqsXAz3iget-jUY1bp6R0fKHySAP3zsIA_iA3k8liZxj1XB38x-Y1VEt7dSyUAq-jsvPg0JBFu67Eq-KUYXY4CPyA" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="478px;" /><br />
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">a text file with the ASCII contents of ‘Man’ will result in a base64 encoding of ‘TWFu’.</span></div>
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Summing it all up</span></h3>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The </span><span style="background-color: transparent; color: blue; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Concatenative Iterative Encoding System</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> has the following signature, </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="40px;" src="https://lh4.googleusercontent.com/d3BAVFG4RZfZqQGUU8n0IJJfHLX7jbqwM81-pS7Ueq5YP9VDgRll8fnAyKJ2S4dsLW1rPJeG33xSJuqD90wjjMZ3I7wJZ5rUwJn3JRGYq7iUCrP57oxn2O4tcOOMjEUKZg" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="624px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">which in our particular example looks like:</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="49px;" src="https://lh3.googleusercontent.com/SLKzQ6XiLVgTuQIg_nUNrqah_Eh-mYLFWXLNgAS8yciD1-m1Gr3EckgCW0ItV947tw3kwMeY0Uoa7uX7byN8BbDBVCxoLIF3_lSV41W8tlIna4ofnQzc-V89KSYlRjP97g" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="624px;" /><br />
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">or equivalently</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="48px;" src="https://lh6.googleusercontent.com/EeCIhIL6v-2iB8jYNxj8gC-XKfwg2r6b7xj6VyNmJl7_3hnLd13BMReRWdCfoLQKM609uzARPgAcnhRfz8-ebqEygyLH2Gw3ecX-BriFGsljO9lgPD_O1v2Tug8eq-1i9A" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="624px;" /><br />
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">And that is it. That is what base64 encoding does. It groups a stream of binary bits (grouped in 8-bit chunks) into a stream of output 6-bit sextets. Each sextet is interpreted as a single symbol from the radix-64 alphabet, and re-emitted as the ASCII representation of that symbol. </span></div>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #c9daf8; border-bottom: solid #b7b7b7 1px; border-left: solid #b7b7b7 1px; border-right: solid #b7b7b7 1px; border-top: solid #b7b7b7 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Is it Compressions? No.</span></h3>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">There always seems to be a major misapprehension of base64’s purpose, and one that we should try to clear up right now. </span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Base64 Encoding as a </span><span style="background-color: transparent; color: blue; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Concatenative Iterative Encoding System</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> permits an arbitrary binary file to be translated into a string of text and copied in an email, or posted in a Usenet post. We have protected higher-level applications (like MIME) from arbitrary binary by choosing only letters that can exist within that application’s specification. </span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">We have </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">compressed</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> the string length of </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">symbols</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> but we have not compressed the overall data size. In fact, we have actually </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">inflated</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> the size of the overall data payload by re-embedding the string ‘TWFu’ using ASCII. </span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Let us repeat the last statement: </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">base64 does not compress the input data</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">. In fact it inflates the size. It only compresses the length of the symbol string, by using an alphabet with more symbols.</span></div>
</td></tr>
</tbody></table>
</div>
<span style="font-family: 'Trebuchet MS'; font-size: 21px; line-height: 1.15; white-space: pre-wrap;"><br /></span>
<span style="font-family: 'Trebuchet MS'; font-size: 21px; line-height: 1.15; white-space: pre-wrap;">A Final Comparison</span><br />
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none;"><colgroup><col width="102"></col><col width="262"></col><col width="260"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><br /></td><td style="background-color: #d9ead3; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: blue; font-family: 'Droid Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Place-Based Single Number Encoding</span></div>
</td><td style="background-color: #d9ead3; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: blue; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Concatenative Iterative Encoding System</span></div>
</td></tr>
<tr style="height: 0px;"><td style="background-color: #b6d7a8; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Function </span></div>
<div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Signature</span></div>
</td><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<img height="53px;" src="https://lh4.googleusercontent.com/i0vxy8wf4Pk-cuyEPx_f2DIQWPfW6ZnLemJNJ9ccT1o5JPgptD-mHSr5qHz3d2-06jeq6jseldXGZ1w-grFAq-cKSGeCAM_186pnNN4wzv7p7Q8vo2YxmksPDh8TLE5WxQ" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="199px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
</td><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<img height="43px;" src="https://lh4.googleusercontent.com/pU7CgjDyBLE8bu4AglQfKqjWq4jVyVrazHnQmHA4lgFbPPHS3sRCG3ngsNMJgFH_VhiDA81KB1HTH4HiSkpxA5ilPoZILPvDeRvMYK29FPCgXDBr4VB0V_9XZF87CbnZjQ" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="237px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
</td></tr>
<tr style="height: 0px;"><td style="background-color: #b6d7a8; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Input Type</span></div>
</td><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Single Number</span></div>
</td><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Array of Octets</span></div>
</td></tr>
<tr style="height: 0px;"><td style="background-color: #b6d7a8; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Output Type</span></div>
</td><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">representation of number in different radix</span></div>
</td><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Array of Base64 representations</span></div>
</td></tr>
<tr style="height: 0px;"><td style="background-color: #b6d7a8; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Consumption Direction</span></div>
</td><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Right to Left</span></div>
<div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<img height="39px;" src="https://lh5.googleusercontent.com/puLNYLxkvXo2UWTjVm7njvgyNvtOudJO3CsNXn8SLeu5-BeYTTekVkXueqwL9zNuyWEQWrWAo4sHPj8LOjtKsKrxh1ELL-a8vbELYTi3k_L3nmKJpO1WJARXp62jYHrloQ" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="252px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
</td><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Left to Right</span></div>
<div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<img height="59" src="https://lh5.googleusercontent.com/R_6gr3CC0qdkuyRJT-j_NuQGxDiMTETOGsmM5wUO6HHXgpvf_Z2WP1zkbTzTFMqbWb7gpTyk7ZdXiTJrW-JVar2lF60qkHG3DzAn-2auA3hqXLaai-pOGxzxwmL8vk75HQ" style="-webkit-transform: rotate(0rad); border: none;" width="320" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
</td></tr>
<tr style="height: 0px;"><td style="background-color: #b6d7a8; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Used For</span></div>
</td><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Representing a single number in a different radix, using a different alphabet consisting of different symbols.</span></div>
</td><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Turning arbitrary binary into a left-to-right array of encoded characters that may be embedded and transferred within the context of an applications.</span></div>
</td></tr>
</tbody></table>
</div>
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">In the above chart there is one term that we have not yet discussed called “</span><span style="color: black; font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">Consumption Direction</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">”. The point of this is to show that </span><span style="color: blue; font-family: 'Droid Sans'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Place-Based Single Number Encoding </span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">works on an entire number whose place-based format requires knowledge of the rightmost digit. To put it another way, you need to know the farthest right-most bits before you know what the number is. </span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">On the other hand base64 as a </span><span style="color: blue; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Concatenative Iterative Encoding System</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> consumes binary data from left to right. The reason for this is that base64 encoding usually operates on files which cannot always be loaded into memory. This is why we have subtly been referring to this algorithm as a streaming algorithm -- it does not have access to the entire input, but may start generating the output with only the first three octets. The actual physical implementation of reading bits from disk means that we always have access to the most significant bits (leftmost) before the least-significant bits (rightmost) are event loaded from disk.</span><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">The </span><span style="color: blue; font-family: 'Droid Sans'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Place-Based Single Number Encoding</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> is not appropriate for encoding a file because you would</span></div>
<ol style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">need to load the file into memory to know its numeric representation</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">the number would be HUGE</span></div>
</li>
</ol>
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">On occasion, the two systems will produce the same answer. These occur when the number is bit-represented in multiples of 3-octets. If the number is slightly more than 3-octets, or only 2 or 1 octet, then the base64 specification calls for something called padding which adds zeros to the rightmost side. The effect of this is that base64 respects the leftmost/prefix data, while allowing the rightmost to be vastly different.</span><br />
<div>
<span style="color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">Padding</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> is one of our hints that we are dealing with a streaming algorithm.</span><br />
<h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Finally, An Analysis of Crockford-base32 and its Implementations</span></h1>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The Crockford-base32 implementation has a few desirable properties that I will copy directly from the </span><a href="http://www.crockford.com/wrmg/base32.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">website</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">:</span></div>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #efefef; border-bottom: solid #000000 0px; border-left: solid #000000 0px; border-right: solid #000000 0px; border-top: solid #000000 0px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 8pt; margin-left: 8pt; margin-right: 8pt; margin-top: 8pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The encoding scheme is required to</span></div>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Be human readable and machine readable.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Be compact. Humans have difficulty in manipulating long strings of arbitrary symbols.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Be error resistant. Entering the symbols must not require keyboarding gymnastics.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Be pronounceable. Humans should be able to accurately transmit the symbols to other humans using a telephone.</span></div>
</li>
</ul>
</td></tr>
</tbody></table>
</div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Additionally, it tries to remove accidental obscenities and ambiguous looking symbols (like 0 and O).</span><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">I needed a Crockford32 implementation for Erlang but could not find one, so I looked up various implementations in other languages to try to understand what they did. I also looked closely at the base64 implementation that comes with the Erlang runtime. It was then that I figured out that the Crockford32 implementation implies </span><span style="color: blue; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Concatenative Iterative Encoding System</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> but that all the implementations I could find are </span><span style="color: blue; font-family: 'Droid Sans'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Place-Based Single Number Encoding</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">.</span></div>
<h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The Conflicting Evidence</span></h2>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #d9ead3; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Evidence for thinking Crockford32 was meant to be a</span></div>
<div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: blue; font-family: 'Droid Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Place-Based Single Number Encoding</span></div>
</td><td style="background-color: #d9ead3; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Evidence for thinking Crockford32 was meant to be a</span></div>
<div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: blue; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Concatenative Iterative Encoding System</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">All</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> implementations use this assumption.</span></div>
</td><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Zero</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> implementations use this assumption.</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The quote from the website saying,</span><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">“</span><span style="background-color: linen; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">This document describes a 32-symbol notation for expressing </span><span style="background-color: linen; color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">numbers</span><span style="background-color: linen; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> in a form that can be conveniently and accurately transmitted between humans and computer systems.”</span></div>
<div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; line-height: 1; white-space: pre-wrap;"><br /></span>
<span style="font-family: Arial; font-size: 15px; line-height: 1; white-space: pre-wrap;">Emphasis mine.</span></div>
</td><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Quotes from the page that compare it in context with other </span><span style="background-color: transparent; color: blue; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Concatenative Iterative Encoding Systems</span><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">"</span><span style="background-color: linen; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Base 64 encoding uses a large symbol set containing both upper and lower case letters and many special characters. It is more compact than Base 16, but it is difficult to type and difficult to pronounce.”</span><br />
<span style="background-color: linen; font-family: Arial; font-size: 15px; line-height: 1; white-space: pre-wrap;"><br /></span>
<span style="background-color: linen; font-family: Arial; font-size: 15px; line-height: 1; white-space: pre-wrap;">“Base 32 seems the best balance between compactness and error resistance. Each symbol carries 5 bits.”</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Language that implies that the primary use-case for this encoding is Human consumption which means short strings. </span><br />
<span style="font-family: Arial; font-size: 15px; line-height: 1; white-space: pre-wrap;"><br /></span>
<span style="font-family: Arial; font-size: 15px; line-height: 1; white-space: pre-wrap;">Short strings do not require streaming algorithms, as an entire number can absolutely be loaded into memory:</span><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">“</span><span style="background-color: linen; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Be pronounceable. Humans should be able to accurately transmit the symbols to other humans using a telephone.”</span></div>
</td><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The most damning quote of all:</span></div>
<br />
<div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: linen; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">“If the bit-length of the number to be encoded is not a multiple of 5 bits, then zero-extend the number to make its bit-length a multiple of 5.”</span></div>
<br />
<div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This quote is absolutely about padding enough bits to the rightmost digits.</span></div>
</td></tr>
</tbody></table>
</div>
<b style="font-weight: normal;"><br /></b>
<br />
<h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Conclusion</span></h1>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In the end, I think this has been a nice learning experience for me, and hopefully something worth sharing. Importantly, it drives home a few points:</span></div>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">most base32/16/64 encodings were designed to be streaming algorithms and operate only on octets and usually on a dataset that would not fit into memory</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">the recent explosion in services that want short-URLs while having unique identities for resources have increased the need for non-streaming radix-encoding algorithms</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">specifications are very important. The </span><a href="http://www.crockford.com/wrmg/base32.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Crockford32 page</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is ambiguous, especially in comparison with the </span><a href="http://tools.ietf.org/html/rfc4648" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">base64 specification</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> which even includes test cases</span></div>
</li>
</ul>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span id="docs-internal-guid-eaa5bc7f-a587-4ea7-a91e-a447181f0e77"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Finally, </span><a href="https://gist.github.com/reverendpaco/e07f6311e1b34bac70da" style="text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">here</span></a><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> is an Erlang implementation I wrote for Crockford-32 as a </span><span style="color: blue; font-family: 'Droid Sans'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Place-Based Single Number Encoding</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">.</span></span></div>
</div>
</div>
Daniel Eklundhttp://www.blogger.com/profile/11570452431861145598noreply@blogger.com172tag:blogger.com,1999:blog-5887985082054147381.post-24431500203885818422014-02-04T18:31:00.003-08:002014-02-05T06:36:30.608-08:00Animating the Git Branching Model: Part 1<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
</div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I’ve been trying to learn Apple Motion 5. In my day job, I draw a lot of architecture diagrams to communicate with clients and with my technical teams. I find the use of the visual to be clarifying, therapeutic, and highly-effective. Up until this point, I’ve focused on static diagrams that I can whip up using OmniGraffle. But the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">notion of motion</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> has always enticed me.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="270px;" src="https://lh3.googleusercontent.com/4XBrQxYa22F4aNpfJBxIK-gqmItv6-Ybm-hwOUcdfJpwXcmro9jDqQdjfJkoHTlrTCE5OEIkdNSY9D1nQOfUAdtah4qt5AqwoJvmVSuARFG3I_V7NvgXJyRiIA" style="border: 0px solid transparent;" width="480px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This post does two things. First, it brags about my first attempt at producing an animated gif using a combination of Apple Motion 5, and </span><a href="https://gist.github.com/dergachev/4627207" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">ffmpeg and gifsicle</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. Second, it thematically advances a framework I had introduced in my </span><a href="http://merrigrove.blogspot.com/2014/02/why-heck-is-git-so-hard-places-model-ok.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">last post</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> which argued that to understand Git, you need to understand its underlying </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">models</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. </span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">In that last post, I focused on the </span><a href="http://merrigrove.blogspot.com/2014/02/why-heck-is-git-so-hard-places-model-ok.html" style="line-height: 1.15; text-decoration: none;"><span style="background-color: white; color: #1155cc; font-family: 'Droid Serif'; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Places Model</span></a><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">. Today, I want to start illustrating the </span><span style="background-color: white; color: black; font-family: 'Droid Serif'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Branching Model</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">. </span><br />
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;"><br /></span>
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">I am under no illusions that what I am providing is the best approach to understanding this model. However, if even a few people find this animated gif useful, then it will have been worth it. (Personally, I wish I had something like this when I was first learning Git.)</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial;"><span style="font-size: 15px; line-height: 17.25px; white-space: pre-wrap;">Anticlimactically:</span></span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvCq36-T04l3qJiWqq-n7ZRZE_iScamTF1PIvfwFG-tlChvVKrVkkkWqpsdfS94DuX2GL1vQbdZbvhi-yj8IKlpI7OBO3O5dUVTXMkhp4UPRzR4DOAoc9pdliFnF6H2hAc4cJDdLc07MI/s1600/out.gif" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvCq36-T04l3qJiWqq-n7ZRZE_iScamTF1PIvfwFG-tlChvVKrVkkkWqpsdfS94DuX2GL1vQbdZbvhi-yj8IKlpI7OBO3O5dUVTXMkhp4UPRzR4DOAoc9pdliFnF6H2hAc4cJDdLc07MI/s1600/out.gif" /></a></div>
<span style="font-family: Arial;"><span style="font-size: 15px; line-height: 17.25px; white-space: pre-wrap;"><br /></span></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">You will find at the bottom of this post the steps you can copy-and-paste to follow along with what is going on in this animation. This is my first animation and I decided to keep it simple. Initially I wanted to show all the trickery that occurred with all the forms of </span><span style="font-family: Consolas; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">git reset</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> (from soft to hard), to the graph-rewriting </span><span style="font-family: Consolas; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">git rebase</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">, to the confusions of a detached HEAD. For now, I find this most basic explanation to be a necessary first step. Those other animations will follow.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Here are some things that are worth noting about what’s going on in the animation:</span></div>
<ol style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I am NOT showing what’s going on in the </span><span style="background-color: white; color: black; font-family: 'Droid Serif'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Places Model</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and the </span><span style="background-color: white; color: black; font-family: 'Droid Serif'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Content Model.</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> You are </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">not</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> seeing how content is being added to the index from the working directory, or how content is being added as blobs. All we are seeing is the temporal evolution of a DAG of commits.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">A </span><span style="background-color: transparent; color: black; font-family: Consolas; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">git checkout</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is literally changing a pointer to a pointer, moving HEAD to point to a branch.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">A branch is nothing more than a pointer in a DAG of commits.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The </span><span style="background-color: white; color: black; font-family: 'Droid Serif'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Branch Model</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> uses </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">commits</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> as a proxy for the full DAG of the </span><span style="background-color: white; color: black; font-family: 'Droid Serif'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Content Model</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> which includes graph objects of type </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">commit</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">blob</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">tag</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, and </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">tree</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, etc. Each commit object is the starting point to a DAG that represents the state of the code from that point. To repeat: we are showing the DAG for the </span><span style="background-color: white; color: black; font-family: 'Droid Serif'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Branch Model</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, and not the full DAG of the </span><span style="background-color: white; color: black; font-family: 'Droid Serif'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Content Model.</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
</li>
</ol>
<h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Steps to Follow Along with Animation </span></h1>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 10pt; margin-top: 0pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 17px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (copy and paste these into a command-line prompt on your favorite *nix)</span></div>
<ol style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Consolas; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; margin-left: -54px; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Consolas; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">mkdir ordoTempliOrientis; cd ordoTempliOrientis</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Consolas; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; margin-left: -54px; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Consolas; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">git init</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Consolas; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; margin-left: -54px; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Consolas; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">touch fileNum{1..5}.txt; git add fileNum*</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Consolas; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; margin-left: -54px; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Consolas; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">git commit -a -m "adding five empty files"</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Consolas; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; margin-left: -54px; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Consolas; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">echo "first line" >> fileNum1.txt; git commit -a -m "adding a line… praise LBJ"</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Consolas; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; margin-left: -54px; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Consolas; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">echo "another line" >> fileNum1.txt; git commit -a -m "adding a 2nd line"</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Consolas; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; margin-left: -54px; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Consolas; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">git branch fixXSSBug</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Consolas; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; margin-left: -54px; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Consolas; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">git checkout fixXSSBug</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Consolas; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; margin-left: -54px; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Consolas; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">echo "over here" >> fileNum2.txt; git commit -a -m "modifying different file -- 1st time"</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Consolas; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; margin-left: -54px; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Consolas; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">echo "adding a 2nd line in new branch" >> fileNum2.txt; git commit -a -m "This is the 2nd commit on branch"</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Consolas; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; margin-left: -54px; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Consolas; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">echo "adding a 2nd line in new branch" >> fileNum2.txt; git commit -a -m "3rd commit on branch"</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Consolas; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; margin-left: -54px; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Consolas; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">git checkout master</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Consolas; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; margin-left: -54px; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Consolas; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">echo "When in disgrace" >> fileNum1.txt; git commit -a -m "1st Sonnet Commit"</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Consolas; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; margin-left: -54px; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Consolas; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">echo "with FORTUNE and MENs eyes" >> fileNum1.txt; git commit -a -m "Bill would be proud"</span></div>
</li>
</ol>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
</div>
<hr />
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Now, if you want to verify that these steps worked, feel free to run ‘</span><span style="background-color: transparent; color: black; font-family: Consolas; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">gitk --all</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">‘ from the command line and you should see this:</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="193px;" src="https://lh4.googleusercontent.com/VwADPRrC3MVw8OyGzTiGhFFXsokJa3TkLH5MLnFIA3m-VEEg6CWKqhsgZUH_oV5vDlOPV0PF2SFC97jw7_qARN2e7Qc6Zdsqz-Cu48M2yaZ7kk_2gAHyHGGVEA" style="border: 0px solid transparent;" width="331px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">More interesting and complicated </span><span style="background-color: white; color: black; font-family: 'Droid Serif'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Branching Model</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> animations to follow. Feel free to steal.</span></div>
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span>
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></div>
Daniel Eklundhttp://www.blogger.com/profile/11570452431861145598noreply@blogger.com36tag:blogger.com,1999:blog-5887985082054147381.post-92037966082977155302014-02-03T15:17:00.000-08:002014-02-06T11:15:11.181-08:00Why the Heck is Git so Hard?<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 28px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Why the Heck is Git so Hard? The Places Model™</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 10pt; margin-top: 0pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 17px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> … ok maybe not </span><span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 17px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">hard</span><span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 17px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, but complicated (</span><span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">which is not a bad </span><span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 12px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">thing</span><span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 17px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">)</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This post is aimed at those programmers having experience with SVN (or CVS) and who are about to embark upon learning Git. Your road ahead </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">will</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> be complicated.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><img height="358px;" src="https://lh5.googleusercontent.com/vaW-mnwkfz_ll2mIR5P9ODF9oBQOBUptx8fee65ygDfBqOtlH-hxbnc0ejFhTSkWELQSDZGp_bAr071lRwLLx0Hcb2mgCF-viPyXpP7QyjpAWAXRhioysnZBxA" style="border: 0px solid transparent;" width="500px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The post will show that one of the reasons (</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">there are other</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">) why going from the SVN model of source-control to Git can be so complicated comes from the fact that there are many more places for source to exist. Rather than focusing on the warts, I hope this post will explain to newcomers the </span><span style="background-color: transparent; color: black; font-family: 'Droid Serif'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Places Model</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> that will help motivate </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">why</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> Git is complicated (and yet amazing).</span><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">This model is one of the most important thing to understand about Git -- and </span><span style="color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">memorize</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">. The actual command-lines are the arcana of accretion and arbitrary conflation, and are one of the reasons people get so frustrated with Git. Just use stack-overflow. Or even better, use this </span><a href="http://www.ndpsoftware.com/git-cheatsheet.html#loc=workspace;" style="line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">beautiful site</span></a><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
</div>
<hr />
<span style="font-family: 'Trebuchet MS'; font-size: 21px; line-height: 1.15; white-space: pre-wrap;">SVN is Relatively Simple</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In SVN there are only two different places source can exist: your local directory and the remote repository. All our interactions back and forth (checking out, committing, reverting, status) are between these two places. This is not too difficult to understand. If we have N different interactions, then we only have to understand these N-interactions.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="313px;" src="https://lh6.googleusercontent.com/meqsqjFKFMUprt_ZwrphW6DRdWpbNS4gleaBef1wgreXuqxsQnSbCOiXFijpsc2NrHvETk_881NdMfgx9V_sQgHV7edXwi_80lVGrvbXp269xm_fZHmN0wiCCw" style="border: 0px solid transparent;" width="623px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In the above diagram, I show three types of interactions for a SVN system. </span><span style="background-color: transparent; color: #980000; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Red</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> interactions move source from the working directory place to the repository place. </span><span style="background-color: transparent; color: #38761d; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Green</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> interactions move source in the other direction (from the repository to your working directory). And </span><span style="background-color: transparent; color: #9900ff; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">purple</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> interactions are useful for getting information about both places. In this post we are going to ignore branching. </span></div>
<h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Git: Increasing Places and Increasing Interactions</span></h1>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In Git, there are five places your source can exist: </span></div>
<ol style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">a stash, </span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">your local working directory, </span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">an index (or staging area), </span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">a local repository, </span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">and a remote repository. </span></div>
</li>
</ol>
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Knowing that these places </span><span style="font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">even exist</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> is often the first conceptual impediment for someone coming over to Git from SVN. </span><br />
<b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><img height="305px;" src="https://lh3.googleusercontent.com/XhubaWjP6jDWxcijtU_I6jLYxfHPBqx_1ecbHtBPs6fDXU9FAvw9G0twa7bZx8isgTaGKNFCI6QREiRLw-xle6hhpUFwnVGk1zUFlr2ZyaeZ14XRzpYIrcL9ig" style="border: 0px solid transparent;" width="624px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br />
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #c9daf8; border: 1px solid #b7b7b7; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Why do these Places Exist?</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">These places are part of the power and flexibility of Git over other systems. </span></div>
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">With an index, we can choose how to commit a multitude of changes, without having to commit to all our changes at once. This is a great flexibility. With the local repository versus the remote repository, we turn Git into a distributed version control system, where each repository is conceptually just as equal to the next one. With</span><a href="http://optiflag.rubyforge.org/" style="text-decoration: none;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> stash, we </span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">take advantage of the underlying content DAG to provide us an amazing convenience.</span></div>
</td></tr>
</tbody></table>
</div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In between all of these places, we can (loosely) have the same types and number of interactions. If we assume a well-orderedness to these places then we have to minimally concern ourselves with N*4 interactions. </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="305px;" src="https://lh5.googleusercontent.com/cOhWwRSoOYvgIwXN5dm55DFOTQ3occg1mPuK77kEoVOUiPpnvGpByfklVWgYoxn1PByb4FLMsW9cFh35M1FyZetnv48Wr2T108Uw6UPwNc1kGR46GD9mDhxY5g" style="border: 0px solid transparent;" width="624px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It’s not actually as many as N*4, as a lot of these operations don’t really occur that often (or at all), and because the ‘stash’ is not used that often (or, more accurately, is optional). On the other hand, to increase our numbers, we also acknowledge that many interactions interact with multiple places at the same time, or between places that are not next to each other (consider </span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">git pull</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, for instance). In short, there are many more interactions that can be occurring. A lot more than N.</span><br />
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;"><br /></span>
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">This is the first great step to attaining enlightenment in Git:</span></div>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">knowing</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">that there are five places</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span></div>
</li>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: circle; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">3 that we use all the time, (working directory, index, and local repo)</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: circle; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1 we use when we interact with teams or want to back-up (remote repo)</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: circle; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1 that we can use optionally as much as we like (stash)</span></div>
</li>
</ul>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">and </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">knowing</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">that there are</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">many interactions</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> that can occur between these places and sometimes between more than two places simultaneously.</span></div>
</li>
</ul>
<span style="color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">Knowing that there are many more interactions</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">, gives you a framework to understand the bad parts of Git, the </span><span style="color: black; font-family: Arial; font-size: 15px; font-style: italic; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">names</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> of these interactions. I highly suggest you use </span><a href="http://www.ndpsoftware.com/git-cheatsheet.html#loc=workspace;" style="line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">cheat-sheets like this</span></a><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> while you are learning. I am not going to debate anyone about this. It is a personal opinion that I hold, and I am comforted in knowing that I <a href="http://steveko.wordpress.com/2012/02/24/10-things-i-hate-about-git/" target="_blank">share</a> this opinion with many others: </span><span style="color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">the command-line usage is a pain in the butt</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">.</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><img height="170px;" src="https://lh4.googleusercontent.com/r3_s0efjwGl-twVDrH8Orb6nxbiEBbBpwc5JO4UHzzhLlqZEw-NhZO9FpiSTQ1jbZuaOG12NPiz-Cx1rw7s5b27RZAUrAU3CGLl1jRUzvTR2pDMdZT3ckfY92Q" style="border: 0px solid transparent;" width="344px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If anything, the </span><span style="background-color: transparent; color: black; font-family: 'Droid Serif'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Places Model</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is not the reason Git is so hard. Git is so hard because of the lack of clear mapping between the command-line and Git’s underlying models: places, content, branching, and remotes.</span><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">You have been introduced to the first step in Git enlightenment: the </span><span style="color: black; font-family: 'Droid Serif'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Places Model™</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">.</span></div>
<h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">You are Not Done!</span></h2>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The second step in attaining enlightenment in Git is learning the </span><span style="background-color: transparent; color: black; font-family: 'Droid Serif'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Content Model</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. Therein contained is knowledge of the objects of commits, blobs, trees, tags and the pseudo-objects of everything else: </span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">refs and refspecs, SHA’s, etc.</span></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><img height="212px;" src="https://lh4.googleusercontent.com/NnPJZvFOjGDsGwFrXIHjBocgagDKbhXbdI-tmsvx1GYhN75HLDkhJM4z8S_m_tzNAfSYHLBwD8DU3h9syF8rOsNeuz1WiU6Sg7gwZFDs4BfZXeSQwJjGMiEmRg" style="border: 0px solid transparent; line-height: 1.15; margin-left: auto; margin-right: auto;" width="296px;" /></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The Content Model</td></tr>
</tbody></table>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The third step is learning the </span><span style="background-color: transparent; color: black; font-family: 'Droid Serif'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Branch Model.</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> The </span><span style="background-color: transparent; color: black; font-family: 'Droid Serif'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Branch Model </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">is built atop of the </span><span style="background-color: transparent; color: black; font-family: 'Droid Serif'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Content Model</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> but follows the evolution of commits as they provide temporal DAG-like capabilities (aka ‘versioning’).</span></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><img height="247px;" src="https://lh3.googleusercontent.com/pBdELGlA7vOMhyBBXFTBYNc1pIGoDMHT0J86qx8UJWYX5kIVO_7Bv5HgtofBHxj6G81ujkT_VqqflZDfZ28bBPfRtoEx5_F8CcHzJtMlgtu7Gq1FDdgfLSNIug" style="border: 0px solid transparent; margin-left: auto; margin-right: auto;" width="354px;" /></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The Branch Model</td></tr>
</tbody></table>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This post will not seek to explain these next steps. There are many other online books and posts which cover the fact that Git is essentially a </span><a href="http://en.wikipedia.org/wiki/Directed_acyclic_graph" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">DAG</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> of content-hashed pointers to file contents (a content-addressable versioned file system all hidden in your </span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.git</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> directory). I suggest the following to get you started:</span></div>
<ol style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<a href="http://schacon.github.io/gitbook/1_the_git_object_model.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://schacon.github.io/gitbook/1_the_git_object_model.html</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<a href="http://eagain.net/articles/git-for-computer-scientists/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://eagain.net/articles/git-for-computer-scientists/</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<a href="http://marklodato.github.io/visual-git-guide/index-en.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://marklodato.github.io/visual-git-guide/index-en.html</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<a href="https://people.gnome.org/~federico/news-2008-12.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">https://people.gnome.org/~federico/news-2008-12.html</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
</li>
</ol>
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;"><br /></span>
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">Once you have a true understanding of these models, working with Git becomes easier. Your day to day activities can now be understood within the context of manipulating your source through the temporal and spatial landscape that these models provide.</span><br />
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #c9daf8; border: 1px solid #b7b7b7; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<a href="https://lh3.googleusercontent.com/f8bTS8VPZeGV0QB-fH3AxUi1Nt_kINZNJPPRe83cNnfILbMnNwtuqLS3WpX3WNc40X8Oz5aKSPCBRLk_DnTSsIIstwd5_LqG-NGHDF2kWxRk3BKQQjVB7lz6pQ" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="147px;" src="https://lh3.googleusercontent.com/f8bTS8VPZeGV0QB-fH3AxUi1Nt_kINZNJPPRe83cNnfILbMnNwtuqLS3WpX3WNc40X8Oz5aKSPCBRLk_DnTSsIIstwd5_LqG-NGHDF2kWxRk3BKQQjVB7lz6pQ" style="border: 0px solid transparent;" width="194px;" /></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">One could argue that there is actually a fourth step, which is a mixture of the </span><span style="background-color: transparent; color: black; font-family: 'Droid Serif'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Places Model</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and the </span><span style="background-color: transparent; color: black; font-family: 'Droid Serif'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Branch Model</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, in which we concern ourselves with the lifecycle and availability of branches in different places: the </span><span style="background-color: transparent; color: black; font-family: 'Droid Serif'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Remote Model</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. </span></div>
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Understanding remote refs and how tracking branches work is a first step in the </span><span style="background-color: transparent; color: black; font-family: 'Droid Serif'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Remote Model</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. This model is also the source of a lot of opinion, best-practices, and team-building. If you have ever heard of git-flow, the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">git rebase versus git merge debate</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, and private versus public branches, then this is the realm of the </span><span style="background-color: transparent; color: black; font-family: 'Droid Serif'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Remote Model</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. It is also here that you will have enter the hosted world of GitHub and its competitors.</span></div>
</td></tr>
</tbody></table>
</div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><img height="299px;" src="https://lh3.googleusercontent.com/CiNvk6tc_tfNfPIKeTx8-2KeOZD50ssXS4CwJIU-gZx9JIF8tLnq8Q8FMTAqtf-wXZSZYmPnZVl29AGh1O9_Vyc4wwMuFYztBumt7NE3yZS6jbyrqjwvKc2WhQ" style="border: 0px solid transparent;" width="471px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">Good luck on the winding road ahead. Maybe one day you can be involved in something <a href="http://vimeo.com/60788996">like this</a>.</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
</div>
<hr />
<br />
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #c9daf8; border: 1px solid #b7b7b7; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Personal Context For Wanting to Write this Post</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I have been using Git since about 2007 (6 years now). I am not great at it, mainly because I have not used it constantly. I often go months without programming or am forced to use a client’s source control system.</span></div>
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<a href="https://lh5.googleusercontent.com/adcS_lU38YgqzEDw3YhbVg0MaoQgE9w1dbb5wLqZdZIiPLJylsMXaxZjVVG8CW-wo0l_NV-lhntbWGoIddiwIytMqLTiPXn1rPU2gYeu6Gi_pUUFQo6f3te8Cg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="110px;" src="https://lh5.googleusercontent.com/adcS_lU38YgqzEDw3YhbVg0MaoQgE9w1dbb5wLqZdZIiPLJylsMXaxZjVVG8CW-wo0l_NV-lhntbWGoIddiwIytMqLTiPXn1rPU2gYeu6Gi_pUUFQo6f3te8Cg" style="border: 0px solid transparent;" width="105px;" /></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Previous to Git, I had researched other distributed VCSs like Monotone and Darcs. I even released my only-ever </span><a href="http://optiflag.rubyforge.org/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">open-source</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> project, back in 2006, from a Darcs repository (I still love the elegance of the Darcs model </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">and</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> command-line). In the end Git and Mercurial are the ones that have flourished. (Yes. I know bazaar and monotone are out there).</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Having to go months without using Git and then coming back to it, I am constantly reminded of how complicated it is. Inevitably I have to review </span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">git reset</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> and tend to forget when to use </span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">git checkout</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> versus </span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">git branch</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">. If my memory were better perhaps I wouldn’t be writing this post.</span><br />
<span style="background-color: transparent; font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;"><br /></span>
<span style="background-color: transparent; font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">I am in awe of Git’s promiscuous branching, speed, flexibility, and distributed nature. The versioned content-addressable file-system and DAG that underpins the repository was a genius choice, and one that was even controversial in the beginning. Previous systems stored and calculated deltas.</span><br />
<span style="background-color: transparent; font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;"><br /></span>
<span style="background-color: transparent; font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">With all that there is to like about Git, I have always been so frustrated at the lack of clarity in the command-lines for doing anything beyond the simplest action. Heck, even the simplest actions are harder than they need to be.</span><br />
<span style="background-color: transparent; font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;"><br /></span>
<span style="background-color: transparent; font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">The history of Git shows that Mr. Torvalds set about building the underlying file-system content model with the assumption that eventually a good front-end would be built. In the end, the front-end was accreted into existence and is the one everyone uses today. I often wonder how tolerant defenders of the Git command-line would be if they didn’t have StackOverflow when they were learning.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Perhaps somebody will provide an alternate front-end to the Git places model that fully acknowledges the places</span><span style="background-color: transparent; color: #351c75; font-family: 'Droid Sans'; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">and the interactions as first-class citizens. It might be a naive concept, but having a command-line that did something like</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">git --from </span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">INDEX</span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> --to </span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">REPO</span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> --interaction </span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">COMMIT</span></div>
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">would at least match the underlying places model. Obviously there are hundreds of subtleties I am overlooking, and I am sure such an effort has probably been done before™.</span></div>
</td></tr>
</tbody></table>
</div>
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><span id="docs-internal-guid-2ad61f2e-03d0-1ed5-1b0c-c38ca4f3963b"><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></span>
<br />
<div dir="ltr" style="margin-left: 3.75pt;">
<table style="border-collapse: collapse; border: none;"><colgroup><col width="610"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #c9daf8; border: 1px solid #b7b7b7; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">UPDATE !</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As someone in the comments below mentioned there is another beautiful site out there that focuses on the </span><span style="background-color: transparent; color: black; font-family: 'Droid Serif'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Branch Model</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. I highly suggest you </span><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;"><a href="http://pcottle.github.io/learnGitBranching/" style="text-decoration: none;">go to this link http://pcottle.github.io/learnGitBranching/</a></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and see how utterly skilled people can be with javascript. You can interactively explore the Commit DAG and see how rebases, merges, checkouts and resets work.</span><a href="http://pcottle.github.io/learnGitBranching/?demo" target="_blank"><img height="261px;" src="https://lh3.googleusercontent.com/q2qOVKs2fhbKGedm6DNdE6YflXn-F_tEEFOscIatN8-q3V3Sdp1sVummZcyiyNonnAS2lHO8ZIvD50lupYtxo3tRj_U6tthc8jurAIcSaHh8v1fxzZxr9QmnOw" style="border: 0px solid transparent;" width="446px;" /></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">In </span><a href="http://merrigrove.blogspot.com/2014/02/animating-git-branching-model-part-1.html" style="line-height: 1.15; text-decoration: none;" target="_blank"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">my following post</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">, I had put together a mini-animated gif to show something similar, but I think this site does a much better job. </span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">However, what neither my </span><a href="http://merrigrove.blogspot.com/2014/02/animating-git-branching-model-part-1.html" style="line-height: 1.15; text-decoration: none;" target="_blank"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">animated gif</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> nor </span><a href="http://pcottle.github.io/learnGitBranching" style="line-height: 1.15; text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">this site</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> show is how most commands modify the models </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">at the same time</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">, e.g. how the </span><span style="background-color: transparent; color: black; font-family: 'Droid Serif'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Places Model</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> is affected by such things as </span><span style="background-color: transparent; color: black; font-family: Consolas; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">git reset</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> with options like </span><span style="background-color: transparent; color: black; font-family: Consolas; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">--soft, --hard, --merge, --mixed</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> modifying your index and/or working directory. In fact, the subtleties involved in knowing that different types of </span><span style="background-color: transparent; color: black; font-family: Consolas; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">git reset</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">s are not only modifying your </span><span style="background-color: transparent; color: black; font-family: 'Droid Serif'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Branch Model</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> (where your HEAD-branch pointer resides) but simultaneously your </span><span style="background-color: transparent; color: black; font-family: 'Droid Serif'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Places Model</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">, lead to a lot of confusion. This </span><a href="http://stackoverflow.com/a/2530073" style="line-height: 1.15; text-decoration: none;" target="_blank"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">StackOverflow answer</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> is the one that I still refer to.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">I think there is a place for a good visualization (animated or not) that shows how </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">both</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> models are affected by each type of git command at the same time</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td></tr>
</tbody></table>
</div>
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></div>
Daniel Eklundhttp://www.blogger.com/profile/11570452431861145598noreply@blogger.com259tag:blogger.com,1999:blog-5887985082054147381.post-88966315524094172982014-02-01T17:24:00.001-08:002014-02-01T18:10:06.244-08:00Erlang Radix Syntax, and Lambda as the Ultimate Meaning<div dir="ltr" style="text-align: left;" trbidi="on">
<h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Follow Up to “Gotcha with Bitstrings” </span></h1>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 10pt; margin-top: 0pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 17px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> How to ACTUALLY denote binary numbers in Erlang</span><br />
<img height="174px;" src="https://lh6.googleusercontent.com/rc5B-gWF6d4WSrPBW_6KBVGNBFe3XfxInW5AkgzF2srrPw1EHOrD3xvAYO3GxLU1G40MSssCfdQoOD34VV2OZG9HcvGu2YhJvj31lnNzNBepqJqcysLln3RgWQ" style="border: 0px solid transparent;" width="310px;" /><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">After I put the </span><a href="http://merrigrove.blogspot.com/2014/01/gotcha-with-bitstrings-in-erlang-naked.html" style="line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">previous post</span></a><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> together, I went looking if there </span><span style="color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">was</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> a way for the Erlang lexer to tell if a number is binary, or octal, or hex. Well it turns out: </span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><b>yes</b></span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">, and it is a beautiful bit of syntax. </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">All you do is put the radix (a.k.a. the base) in front of the number and then use the pound-sign ‘<b>#</b>’ to separate them:</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="106px;" src="https://lh4.googleusercontent.com/MvjLZaeFqtSA_z1ofzCBuYLrAkzzxvkfszVKtiwSh9Xc6DjZ1CGlFXvWXcskleEWXiwYo9M0M4N1UmL6w9fE5t2KD0F1YaFy7tsqaWB7IpoK9Mf3TqE8aJ3dYQ" style="border: 0px solid transparent;" width="296px;" /><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">As you can see, the literal </span><span style="background-color: #b6d7a8; color: black; font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">2#01000001</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> is stating: “</span><span style="color: black; font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">use base-2, and treat the numeral after the pound-sign as binary</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">”. The syntax </span><span style="background-color: #b6d7a8; color: black; font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><<2#01000001>></span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;"> on line 128</span><span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;"> is bitstring syntax and was the subject of the </span><a href="http://merrigrove.blogspot.com/2014/01/gotcha-with-bitstrings-in-erlang-naked.html" style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">last post</a><span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">. The evaluation of a naked number inside the chevrons shoves that number into a 8-bit least-significant segment which, in this case, does not truncate any bits. The 65 decimal number is interpreted as the literal UTF-8/ASCII value of "A". They are the same.</span><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">If we try to use a non-binary glyph/symbol (like a ‘2’) for instance, we will get a problem:</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="119px;" src="https://lh4.googleusercontent.com/mis4h9uc84vAIUJUhnhlfX_efqhXRFgzBTFXL7FxAEJAZlMw7KBTs4lg9KAaDJyr1npJ30jJ0Cu_JWHfTjPNEuj0o8Qz3b-SnoPoSBcCRksbKr-ft5WWWgLUcQ" style="border: 0px solid transparent;" width="432px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">We can do hex too, by using 16 as the radix:</span><br />
<img height="58px;" src="https://lh5.googleusercontent.com/39yx_xeqydzIjCHUbKj8Ox2eCgtbXjtUvApTV15kSbERuDTVjzcDWNhVPnnfeJR5X99pKfK4VaLqtx6rMzROxRVH4gZe6Y8kfejkJjKoaI4OmAIKR735EaGBWQ" style="border: 0px solid transparent; line-height: 1.15;" width="213px;" /><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">This little bit of </span><span style="color: black; font-family: Arial; font-size: 15px; font-style: italic; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">syntax</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> is allowed for radices up to 36 (presumably because nobody would agree as to what new glyphs/symbols we should use after we have exhausted the </span><a href="http://en.wikipedia.org/wiki/Indian_numerals" style="line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Indian Numerals</span></a><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> and the </span><a href="http://en.wikipedia.org/wiki/ISO_basic_Latin_alphabet" style="line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Latin Basic Alphabet</span></a><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">).</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="106px;" src="https://lh4.googleusercontent.com/BxaF1VtbScwCAr9z6a5Dl9rm-fhJoZDnfJAZAYBY5_U_MBNO6QS7VHr8KSIhnGtHT_RXP6YcloxQflQV3OmvEhF9O8k4frXJzstzY6AALnnTIhlHjmSZ-t6Y6g" style="border: 0px solid transparent;" width="203px;" /></div>
<b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #c9daf8; border: 1px solid #b7b7b7; padding: 7px 7px 7px 7px; vertical-align: top;"><img height="47px;" src="https://lh3.googleusercontent.com/E_kZm_2_ToI4lvXMnEp3UCrD_GRxgmfO2lc8bWx81DLLfXz6FPQM1WOeuACjxv4QwC3e-04sKNK2gwwPDQSQO-0bfKT8Om7GqNtXJjMJW09soV6pQ3KELa0hng" style="border: 0px solid transparent;" width="609px;" /><span style="background-color: transparent; font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">If we ever do try to adopt extra glyphs for a > base-36 system. I suggest the one above (hereafter known as the Eklund Encoding), as it makes lambda the answer to life, the universe, and everything.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Although for generating tiny URLs from UUIDs without accidentally offending someone, I suggest the </span><a href="http://www.crockford.com/wrmg/base32.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">crockford encoding</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.</span></div>
</td></tr>
</tbody></table>
</div>
<br />
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></div>Daniel Eklundhttp://www.blogger.com/profile/11570452431861145598noreply@blogger.com23tag:blogger.com,1999:blog-5887985082054147381.post-46505476332460434922014-01-31T13:07:00.000-08:002014-02-01T06:17:41.670-08:00Gotcha with Bitstrings in Erlang: Naked Chevron Numerals are Decimal <div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 28px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Some Confusing Erlang Bitstring Syntax </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 10pt; margin-top: 0pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 17px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> or how "A" is </span><span style="background-color: #f9f9f9; color: #666666; font-family: Verdana; font-size: 13px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">01000001 </span><span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 17px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">in decimal </span><span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 17px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">and</span><span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 17px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> binary</span></div>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #c9daf8; border: 1px solid #b7b7b7; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">TLDR Summary</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: #b6d7a8; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><<01000001>></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is not bit syntax, but a decimal number that </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">looks</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> like it’s a binary. It literally is the decimal number “</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">one million and one</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">”.</span></div>
</td></tr>
</tbody></table>
</div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b id="docs-internal-guid-10dd1832-ea1a-7fa3-4e0d-4d6fa235d25c" style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Here's a gotcha that took me ten minutes to figure out (I also almost slapped my head once I figured it out. </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Almost</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">). And now I’ll take 60 minutes to write out a blog post in the hope that more than 6 people world-wide are helped.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I have been writing an online word-game using Erlang and was becoming fairly proud of my Erlang-fu, but I realized I was not quite truly understanding the entire world of how Erlang handles binary (some libraries I use rely heavily on binary: both <a href="https://github.com/extend/cowboy">Cowboy</a> and a <a href="https://github.com/tonyg/erlang-rfc4627">JSON encoder</a>). Unfortunately, Erlang’s use of binary is semi-documented (or non-newbie-friendly (NNF) documented) and there are a lot of terms floating out there with subtle distinctions:</span></div>
<ol style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">binary</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">bitstring</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">iolists</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">iodata</span></div>
</li>
</ol>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This post won’t make explain all these terms, but hopes to explain a simple syntactical aspect of the binary syntax that didn’t quite make it through my thick-skull. As usual, I suggest people read </span><a href="http://learnyousomeerlang.com/starting-out-for-real#bit-syntax" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">LYSE</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to get started.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Here was the source of my confusion: </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">What are the literal integer values that can go within the chevrons <<?>></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">My confusion started when I went to </span><a href="http://en.wikibooks.org/wiki/Erlang_Programming/Bitsyntax" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://en.wikibooks.org/wiki/Erlang_Programming/Bitsyntax</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to read about the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">syntax</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and gotten taken down a long strange trip.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Here is the confusing part, (I took a screenshot in case someone changes it):</span><img height="432px;" src="https://lh5.googleusercontent.com/4SmRmMuRlAuGmrbpVuUh-XN2eVV7Ad3LMZ48UcyM25eNq4pd4r1chfe9r2iHkV9GaWXYAckbScgxCn_LWmt6aDtOnVvrmNljSg2OFvxLtGjw7CvJbttwG6rn_Q" style="border: 0px solid transparent;" width="582px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">You will see that in the </span><span style="background-color: transparent; color: #85200c; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">red</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> oval the literal bitstring syntax </span><span style="background-color: #b6d7a8; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><<01000001>></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is the same thing as the literal bitstring syntax </span><span style="background-color: #b6d7a8; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><<”A”>></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #c9daf8; border: 1px solid #b7b7b7; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The second syntax is all the rage for improving string performance in Erlang libraries. The reason for favoring a bitstring representation </span><span style="background-color: #b6d7a8; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><<”A”>></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> over regular Erlang strings </span><span style="background-color: #b6d7a8; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">“A”</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> has to do with the the fact that regular strings are implemented as a linked-list of 32-bit integers. Using the second syntax makes the letter “A” use 1 byte of storage versus 8-bytes. I’m not great with math, but that sounds like an 8-times decrease.</span></div>
<br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If you are going to be encoding strings that are within the standard ASCII set (like the string representations of HTTP-compliant headers), which never change, then using a bitstring makes a lot more sense. </span></div>
</td></tr>
</tbody></table>
</div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">implication</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> of the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">true</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> evaluation of the statement is </span><span style="background-color: #b6d7a8; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><<”A”>> == <<01000001>></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is that </span><span style="background-color: #b6d7a8; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><<01000001>></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is the syntax for bits and binaries. </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> Look at it → </span><span style="background-color: #b6d7a8; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><<01000001>></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It has 8 places and all the symbols are either </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">’s or </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">’s. It </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">must</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> be a byte, and those </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">’s and </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">’s </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">must</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> be the bits. </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Wrong</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Before, explaining why, let me explain why I thought so (and why this wiki page should be ashamed of itself for using this example), by showing those of you who have never converted a binary to a decimal what the binary number 01000001 is in decimal.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><img height="201px;" src="https://lh3.googleusercontent.com/miVaC0o-jp_Fmrd4Pgs3mlYtR2QvSiAHN3nxV3DfvwxWoD4iFRkH66DJpFnU4X84hWQySJPzfK6NFuvTNfaMhiHNwJXIYvD8rPNY1ADittfY6nFp37w_RJDZCQ" style="border: 0px solid transparent;" width="588px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In the above diagram, we have put the binary number in boxes to show how in a base-2 place-based system (or </span><a href="http://en.wikipedia.org/wiki/Positional_notation" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">positional-notation</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">) the 1’s are counted as that many-times the places-box. We are so used to doing it in base-10, that we forget we are doing it. But with base-2 you need to remember that 2 to the 6th power is the same thing as 64 (in decimal). </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Seeing this algorithm for transforming a base-2 numeral to a base-10 numeral, we see obviously that 01000001 is 64 + 1 = 65. </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">And what is the significance of the decimal number ‘65’?</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Well if you open up your </span><a href="http://www.asciitable.com/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">ASCII</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> table you will see that 65 is the letter “A” (it’s also 65 in UTF-8).</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="198px;" src="https://lh5.googleusercontent.com/kC1AADh7i8ijUW12rJHC0CURhR39suFjWPZp8MoZBdi5z07caXtYcR37QCkfNSTC2jsj3WWO2SiVQX-2pQIi70IUxk0rw7IPwxCXI4l7yhPFEATX-oMubiSAGw" style="border: 0px solid transparent;" width="315px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Ok, well now it seems obvious that </span><span style="background-color: #b6d7a8; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><<”A”>> == <<01000001>></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> evaluates to </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">true</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> because the </span><span style="background-color: #b6d7a8; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><<01000001>></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> syntax is the same thing as 65 which is the letter “A”. Again, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">WRONG</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Back to the screenshot I took on the offending site. You will note the blue ellipse circles another true statement </span><span style="background-color: #b6d7a8; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><<65>> == <<”A”>></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> . Is that contradictory? If they are both equal to </span><span style="background-color: #b6d7a8; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><<”A”>></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, then they should be equal to each other. (Equality is, after all, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">transitive</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. I hope).</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Fire up </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">erl</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and you will see that they </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">are</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> equal to each other:</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="77px;" src="https://lh3.googleusercontent.com/y-lI9uT1YHFuxFGTrTEp3qnd_QkJKl5iYtqTbO5VC7ZTOjfc4OnBMtBh-w5AQN2t8hMIR7rW7yLYqLaW0yL3KbmmyklWwpgf3pTdmR5SUc6duJYPpSNRoYMZIg" style="border: 0px solid transparent;" width="360px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So what gives? Remember, my quest was about </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">syntax</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. So now my first reaction is to say, “</span><span style="background-color: transparent; color: black; font-family: 'Droid Serif'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">ahh, the Erlang lexer must have noticed the 0’s padding within the chevrons and properly guessed that it was a syntactical binary number</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">”. A tiny shiver, as if a warning, passed through me, but I dismissed it and wasted 10 minutes of my life.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So, continuing down this assumption, were we to increase our binary number by 1, we should get 010000</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">10</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> == 66. Right?</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="54px;" src="https://lh6.googleusercontent.com/WNsn4joDiXNnAwEfQZSlcfAYIAmwQKGZIkuRED0K6RVLmvzAlke8nWQmNN_vFfknx_a_BDHhwqDolr7UuzFHH5GPtJ53H-TjBkeZpH79t-DvJf6xqVe-LF6TIA" style="border: 0px solid transparent;" width="385px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Well, it says </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">false</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. So what is the problem?</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Let’s use </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">erl</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to evaluate </span><span style="background-color: #b6d7a8; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><<01000010>></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> directly:</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="53px;" src="https://lh4.googleusercontent.com/OkNs-VoZ0eIgIkfntulRDB8tPw8aEe3Qh8jdCjITJyosEBLphXB3u99xstcPuQPA5UeXE0Hpe2onGfZX9bjCU_LPxewGnCgO9dl9DA9yHVJWWBLAagN3hO7WgA" style="border: 0px solid transparent;" width="254px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Holey Smokes! That makes no sense. Are we saying that the letter after “A” is “J”? Weird.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Let’s Get to the Point</span></h1>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The reason that none of this makes sense, and the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">start</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> of our explanation is that </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">naked </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">numerals</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> within the erlang chevron syntax are treated as </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">decimal</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">numerals</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. I purposefully used the term </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">numerals</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to differentiate the symbol itself from the abstract number. This means that the naked chevron-sandwhiched token </span><span style="background-color: #b6d7a8; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><<01000001>></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is actually a pair of chevrons surrounding the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">number</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> “one million and one” and encoded as such.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Why I made a Mistake</span></h2>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Earlier, I had assumed that the Erlang lexer was clever and noticed the prefixed zero padding, in order to lex </span><span style="background-color: #b6d7a8; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><<01000001>></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> as a binary number. That was my mistake. I suppose I could make an argument that any number preceded by 0 and having only 1’s and 0’s could be interpreted as binary. But it’s really quite a pain to implement.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">But We are Not Done</span></h2>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So, if the syntax of naked numerals within chevrons treats the numerals as a decimal encoding, then why does </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">this</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> evaluate to true?</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="77px;" src="https://lh3.googleusercontent.com/y-lI9uT1YHFuxFGTrTEp3qnd_QkJKl5iYtqTbO5VC7ZTOjfc4OnBMtBh-w5AQN2t8hMIR7rW7yLYqLaW0yL3KbmmyklWwpgf3pTdmR5SUc6duJYPpSNRoYMZIg" style="border: 0px solid transparent;" width="360px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I know, for a fact, that the decimal </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">number</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 65 is not equal to the decimal </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">number</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 1,000,001. We still have some weirdness to account for.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">And here is the final trick, the syntax for naked numerals within chevrons is short-hand for numerals with some bit-length adornment, specifically 8.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="57px;" src="https://lh4.googleusercontent.com/FS6MvPLkDbP91543swgCR9-ffko5podLzIDUVPO-VGq7PCm9qj8yZfKM4jJlP7CO_NYCu4fxaS_lDa8VFqlWzlRLxkNhLZQ-H18XGbLsdGGKurk2-xrG3FJzNA" style="border: 0px solid transparent;" width="518px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">You see that colon, followed by the number </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">8</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">? Well that is syntax for bitstring length. It is stating that the decimal numeral 1,000,010 will be crammed into 8-bits. And if the binary number is greater than 8 bits?? </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Ah hah! That is the trick. If the binary number is greater than 8-bits, then we truncate the most-significant bits and leave the 8-least-significant bits as the number.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So let’s return to the naked syntax </span><span style="background-color: #b6d7a8; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><<01000001>></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. Remember, this is shorthand for </span><span style="background-color: #b6d7a8; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><<01000001:8>></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> which is saying “Convert the decimal numeral ‘one million and one’ into binary, and then truncate everything but the 8 least-significant bits.”</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So what is “one million and one” in binary?</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Well, the answer is </span><span style="background-color: white; color: #212121; font-family: Arial; font-size: 32px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">11110100001001000001</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Ok, so here we go:</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (0100001)</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 9px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: sub; white-space: pre-wrap;">base-</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 9px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: sub; white-space: pre-wrap;">10</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is equal to (11110100001001000001)</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 9px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: sub; white-space: pre-wrap;">base-</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 9px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: sub; white-space: pre-wrap;">2</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Let’s examine the binary numeral again:</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: white; color: #212121; font-family: Arial; font-size: 32px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">111101000010</span><span style="background-color: yellow; color: #212121; font-family: Arial; font-size: 32px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">01000001</span><span style="background-color: yellow; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As you can see I highlighted the last 8 places (or least significant places) and we can see that our decimal numeral is found in the last eight places of the binary numeral.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (</span><span style="background-color: yellow; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0100001</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">)</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 9px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: sub; white-space: pre-wrap;">base-</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 9px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: sub; white-space: pre-wrap;">10</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is equal to (111101000010</span><span style="background-color: yellow; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">01000001</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">)</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 9px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: sub; white-space: pre-wrap;">base-</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 9px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: sub; white-space: pre-wrap;">2</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Again, this is why the wiki page should be ashamed of itself. It used a decimal numeral that accidentally truncates to its binary representation. So very confusing.</span></div>
<h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Finally</span></h1>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If you’re interested in not losing those more significant bits, then just change the bit-length adornment.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><img height="56px;" src="https://lh4.googleusercontent.com/0B3og2oxYhVvlMfqpmiRIJ2jM1UgKyPp6VCjmxTk4nWaw9jlw1j1xL48vuBXcPTvLTECxqLolzUxFYO3-n_ybGghIvqqlMku-ExF1J4GN8w2JjayV-RkFEfMXw" style="border: 0px solid transparent;" width="463px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Here, you see that we took our decimal numeral “one million and ten” and said “cram this into a bitstring of length 96. You can see that we need 3-bytes worth of encoding, but the remaining 9-bytes are just zero padding. (that 74, by the way, is the letter “J” -- and so we come full circle).</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In the end,</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><img height="77px;" src="https://lh3.googleusercontent.com/y-lI9uT1YHFuxFGTrTEp3qnd_QkJKl5iYtqTbO5VC7ZTOjfc4OnBMtBh-w5AQN2t8hMIR7rW7yLYqLaW0yL3KbmmyklWwpgf3pTdmR5SUc6duJYPpSNRoYMZIg" style="border: 0px solid transparent;" width="360px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">is true, but only because you are losing significant precision because of the syntax on the left. Beware.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Enjoy.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></div>
</div>
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></div>Daniel Eklundhttp://www.blogger.com/profile/11570452431861145598noreply@blogger.com11tag:blogger.com,1999:blog-5887985082054147381.post-670950160635762532013-12-22T18:17:00.000-08:002013-12-22T20:33:01.574-08:00The Occultation of Relations and Logic: Exposing the Hidden Meaning from within Shadows and Unix Command Lines<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
</div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 19px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Here is a question and answer stolen from another website</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">: </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> “</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I have [an] nginx log file, and I want to find out market share for each major version of browsers. I am not interested in minor versions and operating systems.</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">”(</span><a href="http://serverfault.com/questions/89773/get-list-of-user-agents-from-nginx-log" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://serverfault.com/questions/89773/get-list-of-user-agents-from-nginx-log</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">).</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b id="docs-internal-guid-4b35e811-1cb9-d490-9cd7-501046944961" style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The answer that got accepted was one of those Jedi like answers that showed the power of true Unix mastery: </span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> cat /var/log/nginx-access.log | grep "GET" |</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> awk -F'"' '{print $6}' | cut -d' ' -f1 | </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> grep -E '^[[:alnum:]]' | sort | uniq -c | sort -rn </span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It is interesting to look at these individual commands as part and parcel of the Unix philosophy. But it’s also interesting to look at these as manifestations of relational operators, as originally described by Codd.</span></div>
<hr />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><a href="https://lh4.googleusercontent.com/B0RAUj3-2RQKUBRxM1RvMfn7wLwAJHy4LPlhpA3h2d48zwSYbxEgygJdqh3xrhINgDwJKd117nfniciyBgoRrjg4dQHj9cE2WI-YqP51GqJJAr_ir1tTbGmTSw" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="251px;" src="https://lh4.googleusercontent.com/B0RAUj3-2RQKUBRxM1RvMfn7wLwAJHy4LPlhpA3h2d48zwSYbxEgygJdqh3xrhINgDwJKd117nfniciyBgoRrjg4dQHj9cE2WI-YqP51GqJJAr_ir1tTbGmTSw" width="154px;" /></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<a href="https://lh6.googleusercontent.com/EtfrbhZx28yBFDZDwtC4baalrBeSStwvcEasQ3F9laeRSw8MGwJdF0gr9lRM4ivV_eWBIyKpyukJwJAdfwX9S-rbW_6vaqQMd514Y333zN2Mlwobs1e3MGAtuQ" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="196px;" src="https://lh6.googleusercontent.com/EtfrbhZx28yBFDZDwtC4baalrBeSStwvcEasQ3F9laeRSw8MGwJdF0gr9lRM4ivV_eWBIyKpyukJwJAdfwX9S-rbW_6vaqQMd514Y333zN2Mlwobs1e3MGAtuQ" width="245px;" /></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<span id="docs-internal-guid-4b35e811-1cc1-1841-cc70-9ea1c3fe1985"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">The purpose of this post is to explore two concrete examples of a <b>relation</b> outside of a database: 1) an NGINX logfile and 2) a pile of trash casting shadows on a wall</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">.</span></span><br />
<div>
<br /></div>
<div>
In our <a href="http://merrigrove.blogspot.com/2013/12/what-heck-is-relation-from-tables-to.html">previous post</a>, we explored different ways to interpret a <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">relation</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. We showed how a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">relation</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> could be viewed as a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">table</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, as a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">set of facts</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, as a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">predicate function</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, as the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">generalization of a function</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, and as </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">subset of a cartesian product</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. </span><br />
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">These interpretations provide us the ability to rise above marketing and technologies and see the underlying abstraction for what it is. There are important lessons that can be derived from following this path, of separating abstraction from implementation:</span></div>
<b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<ol style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">that </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">relations</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">/</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">tables</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> are everywhere and not just within a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">database</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">that in fact a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">database</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> can be understood on a sliding scale from </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">in situ</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">curated</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> ( “schema on read” versus “schema on write”), and just like relations a database can be ‘everywhere’ (this will be the subject of a future post)</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">that relations are fundamental, abstract, and omnipresent (like types)</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">that relations have many different pedagogical manifestations, (as we listed above and covered in the <a href="http://merrigrove.blogspot.com/2013/12/what-heck-is-relation-from-tables-to.html">previous post</a>)</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">that relations provide an interesting way for </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">understanding</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> many very different technologies, from logic programming languages, to semantic databases, to graph databases, to various different NoSQL-ish things, to SQL, Hive, and Pig.</span></div>
</li>
</ol>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">These themes (more </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">opinions</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">) evolved during my years of exposure to the Big Data world as a way to contextualize the exploding menagerie of technologies. I have been grateful to the Big Data world for showing me that the world of databases is neither a dry, nor solved, problem and that databases, knowledge-bases, logic, etc., are all interrelated in ways deeper than previously expected (or at least deeper than</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">previously expected). </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><img height="201px;" src="https://lh4.googleusercontent.com/qfFnJV3jRJqICN-f6NaKG3lJg9toZa1O2FsTfMbdWS5AGL-olLDAQIHwgWtAgSwiIVAFAjWS1sY0ZmawEqvsTSr7nRdImCfpk-JVkq4zyqvAp185vjFeQlx_wg" width="419px;" /><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br />
<h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Let’s Get Practical</span></h1>
<h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;">
<img height="320" src="https://lh5.googleusercontent.com/2Fcgw-X9aOxzeBD8hAQFgPKVCh8LJ8cigrDgGF6okSIG0HxT9natplhC9ngqMdssJXMfvC3hWzB3udBcVGceYveQ5HvX1jCkXICnbSADuCwF4i6DPYo71a97qg" style="font-weight: normal; line-height: 1.15;" width="212" /></h1>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="line-height: 18.399999618530273px;">There is an unfortunate tendency in the IT community to belittle something by calling it non-practical when it tends to be 1) not popular and 2) theoretical. To counter this charge, and to appeal to as large an audience as possible, let’s immediately dive into answering a very practical problem. As we go through this example, we will adopt the interpretation that a relation is a table. This is a very practical manifestation of our hidden relation, but will also be a long discussion.</span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">The Problem Statement</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">: Let us count the number of different browser types that have visited a website by scanning an Nginx log file. </span><br />
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">As we tackle this problem, we will show this answer as a series of relational operators manipulating relations using Unix command lines.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="119px;" src="https://lh3.googleusercontent.com/VET38K_EyZOZPlAcUjX6vwJU2KnLRT8jpDzzEtEveaxmnzmnrt_9zH84CUC-NUbTsocveID6xVGU0TrRJ1hKmwBzaBc_qMtednLKCj_qpERp9zLsS2nCPLoIBw" width="864px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In the above snippet of configuration, somebody has set up their NGINX (or maybe Apache) webserver to log to disk each HTTP interaction with the website. The log file looks like this:</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">127.0.0.1 - - [19/Jun/2012:09:16:22 +0100] "GET /GO.jpg HTTP/1.1" 499 0 "</span><a href="http://domain.com/htm_data/7/1206/758536.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://domain.com/htm_data/7/1206/758536.html</span></a><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; SE 2.X MetaSr 1.0)"</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">127.0.0.1 - - [19/Jun/2012:09:16:25 +0100] "GET /Zyb.gif HTTP/1.1" 499 0 "</span><a href="http://domain.com/htm_data/7/1206/758536.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://domain.com/htm_data/7/1206/758536.html</span></a><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; QQDownload 711; SV1; .NET4.0C; .NET4.0E; 360SE)"</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">127.0.0.1 - - [19/Jun/2012:09:16:25 +0100] "GET /Yyb.gif HTTP/1.1" 499 0 "</span><a href="http://domain.com/htm_data/7/1206/758536.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://domain.com/htm_data/7/1206/758536.html</span></a><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; QQDownload 711; SV1; .NET4.0C; .NET4.0E; 360SE)"</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">127.0.0.1 - - [19/Jun/2012:09:16:25 +0100] "GET /GO.jpg HTTP/1.1" 499 0 "</span><a href="http://domain.com/htm_data/7/1206/758536.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://domain.com/htm_data/7/1206/758536.html</span></a><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; QQDownload 711; SV1; .NET4.0C; .NET4.0E; 360SE)"</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">127.0.0.1 - - [19/Jun/2012:09:16:30 +0100] "GET /Zyb.gif HTTP/1.1" 499 0 "</span><a href="http://domain.com/htm_data/7/1206/758536.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://domain.com/htm_data/7/1206/758536.html</span></a><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">" "Opera/9.80 (Android; Opera Mini/6.7.29878/27.1667; U; zh) Presto/2.8.119 Version/11.10"</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">127.0.0.1 - - [19/Jun/2012:09:16:30 +0100] "GET /GO.jpg HTTP/1.1" 499 0 "</span><a href="http://domain.com/htm_data/7/1206/758536.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://domain.com/htm_data/7/1206/758536.html</span></a><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">" "Opera/9.80 (Android; Opera Mini/6.7.29878/27.1667; U; zh) Presto/2.8.119 Version/11.10"</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The practicality just oozes from the page.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The actual question is (completely stolen from another website): </span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">“</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I have [an] nginx log file, and I want to find out market share for each major version of browsers. I am not interested in minor versions and operating systems.</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">”(</span><a href="http://serverfault.com/questions/89773/get-list-of-user-agents-from-nginx-log" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://serverfault.com/questions/89773/get-list-of-user-agents-from-nginx-log</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">).</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The answer that got accepted was one of those Jedi like answers that showed the power of true Unix mastery: </span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> cat /var/log/nginx-access.log | grep "GET" |</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> awk -F'"' '{print $6}' | </span><span style="font-family: 'Courier New'; font-size: 19px; vertical-align: baseline; white-space: pre-wrap;">cut -d' ' -f1 | </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: 'Courier New'; font-size: 19px; vertical-align: baseline; white-space: pre-wrap;"> grep -E '^[[:alnum:]]' | sort | uniq -c | sort -rn </span><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It is interesting to look at these individual commands as part and parcel of the Unix philosophy. But it’s also interesting to look at these as manifestations of relational operators, as originally described by Codd.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="111px;" src="https://lh3.googleusercontent.com/XnNXMNXNGCi6UnmjaHhwY5MQtUaG-6DTCiaeUY79XjL7tJv4MFy2KCCWu4p0lXEQL-bMVXWBYkX7CT2qB-3v-e2we6MZ4R12kIj1xux7fJJ-yKCp7rLXr9hEyA" width="202px;" /><img height="113px;" src="https://lh5.googleusercontent.com/OISKd8MkTfuhjxXG5yXe4iZKGgTpIXaJMfS0y7A8fYm8LtR1PUXlB9ZDkFcKiZj_eFsf8UfghOiZi_zQsyggnaCBiuXzGwxBdw0jzuUhp0bdvrzaQMfNvzv71w" width="184px;" /><img height="166px;" src="https://lh5.googleusercontent.com/4JVShuII7rcGUI9oV9N2_n2RAI52tRkW-FtW3qInfz2JbeVFczblDnf3lRJchqXdUWBSCQ5ZqQs9nygzAbGvI4O5KhxfUEx7AiA_E9JaQQvtIPjQwIBLwxmxsg" width="226px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The Unix Pipe World According to Relational Algebra</span></h2>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The discussion on the accepted answer at </span><a href="http://serverfault.com/questions/89773/get-list-of-user-agents-from-nginx-log" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://serverfault.com/questions/89773/get-list-of-user-agents-from-nginx-log</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> shows the reasons why this snippet of chained higher-order UNIX command lines do what they do. Below is a recontextualization, with an eye on a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">relational</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> explanation.</span><img height="221px;" src="https://lh3.googleusercontent.com/RBkGLkt0UtiiO3ghmlG6Iu2DAm_5rZRcb1j563b_hjBXC8ejPodE0bh5qQTCILJ0AWcOV5lLQOFkSXCeE1U8Pa2nJATCOpq0fYCFeMclZGZBs5WAfTX-gBtvjg" width="864px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Diving straight in, we can enumerate what is happening as a chained composition of the following </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">relational operations</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">:</span></div>
<ol style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: #c9ccfc; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: #c9ccfc; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">first we create an un-named, one column relation by invoking </span><span style="background-color: #c9ccfc; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">cat</span><span style="background-color: #c9ccfc; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">,</span></div>
</li>
<li dir="ltr" style="background-color: #fffddc; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: #fffddc; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">next we horizontally subset (</span><span style="background-color: #fffddc; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">select</span><span style="background-color: #fffddc; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> operator) some of the tuples and produce a new relation</span></div>
</li>
<li dir="ltr" style="background-color: #f7b0ae; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: #f7b0ae; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">next we vertically subset (</span><span style="background-color: #f7b0ae; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">project</span><span style="background-color: #f7b0ae; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> operator) the sixth column, producing a new relation again</span></div>
</li>
<li dir="ltr" style="background-color: #a8fdcb; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: #a8fdcb; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">next we vertically subset (</span><span style="background-color: #a8fdcb; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">project</span><span style="background-color: #a8fdcb; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> operator) the first column. New relation again</span></div>
</li>
<li dir="ltr" style="background-color: #ffb8dc; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: #ffb8dc; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">we </span><span style="background-color: #ffb8dc; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">select</span><span style="background-color: #ffb8dc; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> again. New relation again</span></div>
</li>
<li dir="ltr" style="background-color: #ffd9c7; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: #ffd9c7; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">then we </span><span style="background-color: #ffd9c7; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">sort</span><span style="background-color: #ffd9c7; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (strictly speaking this shouldn’t matter in the idealized tuple world that Codd first described)</span></div>
</li>
<li dir="ltr" style="background-color: #a5fafe; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: #a5fafe; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">next we do a </span><span style="background-color: #a5fafe; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">group-by</span><span style="background-color: #a5fafe; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and a count, new relation again</span></div>
</li>
<li dir="ltr" style="background-color: #aaabb6; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: #aaabb6; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">finally we </span><span style="background-color: #aaabb6; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">sort</span><span style="background-color: #aaabb6; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> again, final relation</span></div>
</li>
</ol>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">For those interested, let’s go through each one of these steps in more detail.</span></div>
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">First: Create a boring nameless Relation...</span><img height="281px;" src="https://lh4.googleusercontent.com/y7RNqM5EtwA1_DBsibVFzk_9dvVyN0Jxm4SVrIRcrIo90ZBpf4uv61xHbD5JvnnPO5R1Ak3RRJacezY37ApoOgF8KGXM0OG3mDeInIkcFfo-_TYdMrnP1iDBYQ" width="864px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></h3>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">cat</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is the Unix command line to output the contents of a file to standard out. By doing this, we have implicitly created our first relation, a table with a single column. In reality, neither the relation nor the column have explicit names, (but nothing prevents us from giving them automatic names based on their position from the ‘left’ or the ‘right’ -- naming, while an important step, can be automated). </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<a href="https://lh4.googleusercontent.com/15ZrzfLkOHJlA5xAoqlmb5Gko8uoG0kp92V-YzRV-CBo9N8le1dX-A5xOcbNO0HpvpbJOrPJGvpwqyAW2Bf50zX55Q1EHjUK3XEsG4MoqGMp0Gn9B82Y51XIwQ" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="56px;" src="https://lh4.googleusercontent.com/15ZrzfLkOHJlA5xAoqlmb5Gko8uoG0kp92V-YzRV-CBo9N8le1dX-A5xOcbNO0HpvpbJOrPJGvpwqyAW2Bf50zX55Q1EHjUK3XEsG4MoqGMp0Gn9B82Y51XIwQ" width="56px;" /></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This first relation consists of tuples that represent each line in the file, and the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">type</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> of this singular column is a string (or varchar or chararray, etc). Not to belabor a point, but</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">this is absolutely a relation</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, no matter how primitive, un-named and un-indexed it might seem. </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Do not confuse abstractions with implementations</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (this might, in fact, be the overarching theme for this entire series of posts). The relation exists in our minds as a means of understanding something physical and tangible. The relation is a product of our </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">interpretation</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. In the Big Data world, we call this late-binding of interpretation “</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Schema on Read</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">” indicating that our </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">schema</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (another word for </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">interpretation</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">) is decided when we read it from disk, and not when we write it to a system.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Second: Generate a new Relation using the Selection Operator (i.e. grep for matching lines)</span><img height="289px;" src="https://lh6.googleusercontent.com/-1QmMccvuqHFz9q3SBiU0GG-3s6eduVCjex9SFRKrszFW8dyGojiqUlbQ3X__lrnGkCPWifeZf486PRGIVky7nqcEH-T5PDnjNVndiJdnbgiGLdhNDzRyvqV5w" width="864px;" /></h3>
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="font-family: 'Courier New'; font-size: 15px; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;">grep</span><span style="font-family: Arial; font-size: 15px; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;"> is the unix command line that takes a line oriented data and removes (or keeps) those lines that match the given regular-expression predicate. In this example, our regular expression matches the exact string “GET” and keeps those tuples that match, thus providing a horizontal subsetting operator.</span></h3>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Let’s repeat what we have done here: we have transformed one relation (the original contents of the file as produced by </span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">cat</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">) into another relation by using an </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">operator</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. This is why Codd called one half of his study of the relational model ‘an </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">algebra’</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> in his 1972 paper (</span><span style="background-color: white; color: black; font-family: Arial; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Codd, E.F. (1970). "Relational Completeness of Data Base Sublanguages". </span><span style="background-color: white; color: black; font-family: Arial; font-size: 11px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Database Systems</span><span style="background-color: white; color: black; font-family: Arial; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">: 65–98.</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">).</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><img height="111px;" src="https://lh3.googleusercontent.com/XnNXMNXNGCi6UnmjaHhwY5MQtUaG-6DTCiaeUY79XjL7tJv4MFy2KCCWu4p0lXEQL-bMVXWBYkX7CT2qB-3v-e2we6MZ4R12kIj1xux7fJJ-yKCp7rLXr9hEyA" width="202px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Specifically, we have used what is now called the unary </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">selection</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">operator</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to create one relation from another. For those of you following along with a crush on SQL, this is exactly the same as doing something like : </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> select * from FILE </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: lime; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">where contents = “%GET%”</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(the WHERE clause is highlighted in green and shows the syntactic manifestation of the selection operator </span><span style="background-color: white; color: black; font-family: Arial; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">σ</span><span style="background-color: white; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">in the SQL language)</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Third: Project out only the Data we are Interested in</span><img height="375px;" src="https://lh3.googleusercontent.com/Pb9TiUrgWyDhZlUhhIvPD-hHfko4f71xOTSkP5BJaJGp-vj_cjbUvZb2EQQ6TN-R6eddW9KoMs1mViOleJPv1miYf6KulXJr_zdPrjWqJuV30gjzDV5ACDEh-Q" width="864px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></h3>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Even people familiar with Unix will bow down before the impressive power of </span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">‘awk’</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. In the following snippet of </span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">awk</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, we are using the command line switch ‘-F’ to indicate what is our delimiter between fields. Specifically, we have asked </span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">awk</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to use the double-quotes character, and to make as many fields as there are delimiters </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">+1</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. It turns out, in the case of </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">this</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> NGINX log file, that there will be six fields separated by five double-quote characters. Our </span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">awk</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> command then prints out the sixth column. You can see that the result should be a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">vertical</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> subset of our original line.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">At this point, you can see that we have really done two things:</span></div>
<ol style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">we have re-interpreted the original relation by applying a new schema and in doing so have generated a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">new</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> table/relation of six columns</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">we have asked </span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">awk</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to only pass through the sixth column, and thus have generate a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">new</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> table/relation</span></div>
</li>
</ol>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This second point is the application of the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">projection operator</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, fondly known as pi (</span><span style="background-color: white; color: black; font-family: Arial; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">π</span><span style="background-color: white; color: black; font-family: Arial; font-size: 16px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">)</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. The vertical sub-setting of a relation, to create a new relation, will change the schema and </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">remove </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">dimensions</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. This is shown in the following graphic where a table of two columns: x and y, has the ‘y’ </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">dimension</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> projected out.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="113px;" src="https://lh5.googleusercontent.com/OISKd8MkTfuhjxXG5yXe4iZKGgTpIXaJMfS0y7A8fYm8LtR1PUXlB9ZDkFcKiZj_eFsf8UfghOiZi_zQsyggnaCBiuXzGwxBdw0jzuUhp0bdvrzaQMfNvzv71w" style="line-height: 1.15;" width="184px;" /></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> For those of you following along with a crush on SQL, this is exactly the same as doing something like: </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> select </span><span style="background-color: lime; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">user_agent</span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> from GET_RECORDS</span><span style="background-color: lime; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(the highlighted green shows the syntactic manifestation of the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">projection</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> operator </span><span style="background-color: white; color: black; font-family: Arial; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">π</span><span style="background-color: white; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">in the SQL language)</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Let’s Take a Break</span></h3>
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<img height="206px;" src="https://lh6.googleusercontent.com/4i77W-PgcxlF6LKwBa91Kn0OJEIN7z9YNX6QyZFhDZp-Ozpo6te5pg477iMVpO0HD2QpQMHxgWoYM6XUKg54YAvBSQ-UP8s1PvoPN3Eo3lleDiLkCBJLXGEX9w" width="151px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></h3>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">We said we would veer into the realm of practicality. However, even this discussion was rife with all sorts of mathematical concepts of </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">dimensions</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, and </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">algebras</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, and </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">operators…</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> Let’s sum up the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">practical</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> points then</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<ol style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">relations provide us a way to </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">think</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> about our data</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">grep</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is like the “where” clause in a SQL statement… it subsets horizontally, and keeps the schema the same</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">awk</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (with a print $6 command) can be like a projection operator, or like selecting a column from a table, and it changes the dimensionality of our data</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">UNIX pipes are awesome </span></div>
</li>
</ol>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">On the Road Again</span></h3>
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<img height="183px;" src="https://lh3.googleusercontent.com/Ix1HG7ZVtYV-E5Fk2WAHavTfN-HbtJ0ftZIJjotka6N7NNLjStTOxrPO8nY4si1Bs3O6t1Ggk2uS9DOFRU_Pdpyvhf4-0iGInqWinPvngKNzZwnyQL8v9xQ6Ew" width="163px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></h3>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So far, in this example, we have been treating a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">relation as if it were a table</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> with rows/tuples and columns/dimensions. We might as well finish up the last few steps, but for those of you who want to get onto the next example, you can always skip past these last sections and see how </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">relations</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> can be found in </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">shadows</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, when </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">relations are a subset of a cartesian product</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. To review these concepts, please see the previous post: “<a href="http://merrigrove.blogspot.com/2013/12/what-heck-is-relation-from-tables-to.html">What the Heck are Relations?</a>” </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Four: Get the Browser Type from the User Agent using Spaces</span><img height="265px;" src="https://lh3.googleusercontent.com/cooiIq7GSpTmTxrkxb6TNAm_cOOMKQkW6J5yOweFIG4_Q9ELN6X-MhABVOQcwlus9wEf14TwtdhNzZHmHDDA0IKgxSwn-TP3MdFEz2H5qJpIsnappZdkXVKD_A" width="800px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></h3>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">cut</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is another unix command that does what awk does. It has different syntax and the choice of awk over cut may be one of convenience and/or familiarity. At this point, we have used the </span><span style="background-color: transparent; color: black; font-family: Courier New, Courier, monospace; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">-d</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> switch to say that the space character has become our new delimiter. In the above graphic, we can see how the User Agent text (a payload from the HTTP header) has been split, and we have isolated our first column, which is the actual browser and version (see Section 14.43 of RFC 2616 (HTTP/1.1)).</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In the above graphic, the result of this command for this row will be the text “Mozilla/4.0”. We are getting close to what we need.</span></div>
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Five: Remove those Garbage Rows/Tuples that are clearly not real Browsers</span><img height="297px;" src="https://lh3.googleusercontent.com/x3jIhnKAm43VIqXY1Ahf_CFzfb7M5XNWVa2OWDsuoWVnAcUwJlK-fzn1-fIaxcF3-ea-8Abgyo6RMeBRco9Z7Yzy5gRHI7ar-7IHd482jQGRiT48lI1Xs36cmg" width="847px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></h3>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This is a maintenance step to get rid of all those garbage lines that come from bad user agents. When you are exposed to the wild wild world of the open internet, there’s nothing stopping somebody from sending weird values for the User Agent. At this point, we implement a heuristic that says that only browser versions beginning with alphanumeric characters are allowed.</span></div>
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Six: Sort so that the following </span><span style="background-color: transparent; color: #666666; font-family: 'Courier New'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">uniq</span><span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> operator will work </span><img height="243px;" src="https://lh6.googleusercontent.com/-4a1e_ooXPn4Vj7HGK3KWaL5mi-MItHIAXHSvKWo8ZckocfCKSvLdVd7tgtm_GUlbF0hSFpGofnQeNmqv5PG_t2bsm4LOKNo7nsWIVw8Xh306KuU1I0i2b3FaA" width="864px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></h3>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Sort is an interesting “</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">operator”</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. I have purposely put the term operator in quotes because the original mathematics behind Codd’s Relational Model made no claim about </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">ordering</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. In fact, ordering a relation does not change the equality semantics of that relation (according to the model) as ordering is irrelevant in set-based models. One may even posit that the only need for ordering is to provide an easy to read listing for human consumption.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr">
<table style="border-collapse: collapse; border: none; width: 100%;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #c9daf8; border: 1px solid #b7b7b7; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Tangent</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">: The other interesting thing to note here is that we have to do this because we know a priori that the following UNIX command </span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">uniq</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> will only group and coalesce contiguous rows. So we are doing a necessary step based on the “limitations” of uniq. And while this practical bit of hackery breaks our beautiful model, it does have a nice implementation analog to the MapReduce machinery of Hadoop which needs to provide a shuffle/sort before giving data to a reducer. </span></div>
</td></tr>
</tbody></table>
</div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Seven: Run the uniq command to do a GroupBy and Count</span><img height="276px;" src="https://lh6.googleusercontent.com/0vLp97JKE8j-0ZBuCNOAfGrP8PAo6c0xBVJmJzm1gpsAMJZCSlEwJ_Ix2S746SvIp8kCaqmP4ud-xP6aBJw6fKiSUP9bjMJL8mdekkpq7x6uzP_JrOYd-MeqnA" width="864px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></h3>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Hit the man page for </span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">uniq</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and you will see that using the -c switch does a group-by and a count. For those of you who love SQL, this is like</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">select </span><span style="background-color: lime; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">count(browser_type)</span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> from browser_records</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: lime; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">group by browser_type</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> ;</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr">
<table style="border-collapse: collapse; border: none; width: 100%;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #c9daf8; border: 1px solid #b7b7b7; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Tangent</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">: To be honest, I am not sure how Codd’s original work treated GroupBys as operators. However, in Pig, which permits Zeroth normal form data, this is essentially two operations: an initial GroupBy operator that turns the original relation into a hard-coded two-column relation with the group key and a bag of the partitioned rows nested within the cell. Then a count UDF on a FOREACH operator, which is essentially an extended projection semantic.</span></div>
</td></tr>
</tbody></table>
</div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Eight: A final sort for the benefit of us Humans</span><img height="281px;" src="https://lh4.googleusercontent.com/6UCrC_GH1yCFLysgOZha-9kP3Z9S6liIVKL-67tBHI0H3gKJPNUuOrppE0UECRcw9r6wl9ur1s4XPBlvKcNtSgMszY3ZLpNBT6nDXW2o0N4qbferRmsKlzZ22Q" width="864px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Finally, and to wrap this up, we provide a final sort operator which allows us to see the browsers with the most hits at the top.</span></h3>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">And the analogous SQL:</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">select </span><span style="background-color: lime; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">bcount, browser_type</span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> from browser_counts</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: lime; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">order by bcount desc</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> ;</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br />
<div dir="ltr">
<table style="border-collapse: collapse; border: none; width: 100%;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #c9daf8; border: 1px solid #b7b7b7; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Tangent: Horizontally Scaled Implementation -- a Practical Pig Program </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><img height="473px;" src="https://lh4.googleusercontent.com/1gz0wkbZbU4D9RtHZ0Fh3ImcK3rkxafOsMNcRP9ezLBQHlyjwZaF7S07TS3A72bSPgGL-q2oS82PKrGIui9hlsX0yXFVChS9aBsJBc1iZ8YtFQjedeunM0CHrA" width="844px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In the above graphic, I am showing an alternate and </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">real</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> implementation of our UNIX operators using Apache Pig (a relational language built atop Hadoop). This is an actual Pig program that does the same thing as these UNIX commands. The difference, of course is obvious: while the UNIX command lines are run on a single CPU and might run for days if this NGINX file is terabytes large, the Pig program compiles down to MapReduce tasks which spawn mappers and reducers on cluster nodes to implement the abstract relational operators and may effectively parallelize this embarrassingly parallel problem -- i.e. horizontal scalability.</span></div>
<br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The nearly one-to-one mapping between the Unix command line and the Pig code is not accidental. I purposely chose the UNIX example as a way to explain how Pig works. Most people who have never seen Pig but only worked with SQL find that the syntax is difficult to swallow. The entire UNIX command line example was addressed to gently introduce the idea of a step-wise transformation operator syntax. Again, I don’t think Pig gets enough love.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="147px;" src="https://lh4.googleusercontent.com/8k8xxoxr8CR8lgI5-Gi6ouBLplYKVorqEWHq8K21kXeLJZmSP3FqjlHm3pSCrWz-03GVxB_MYI1Ck9s3wIxpUsexayj2m8BS3Bqjx4Q_WZl_YgK5-QP4DL_VNw" width="98px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
</td></tr>
</tbody></table>
</div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Finishing Up: How far we have come. Where we can go.</span></h2>
<h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;">
<img height="206px;" src="https://lh6.googleusercontent.com/4i77W-PgcxlF6LKwBa91Kn0OJEIN7z9YNX6QyZFhDZp-Ozpo6te5pg477iMVpO0HD2QpQMHxgWoYM6XUKg54YAvBSQ-UP8s1PvoPN3Eo3lleDiLkCBJLXGEX9w" width="151px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></h2>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">At this point, we have provided a relational explanation for this hard example of parsing an NGINX log file. It was a practical example, and showed us that there is some applicability to the abstract terms we have developed thus far. </span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">For a more theoretic view of what’s going on, please see the previous </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">post</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> where considered the many different manifestations (or interpretations) of the word </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">relation</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. This will be a fun discussion and will give us an inkling into how so many different systems could provide their own twist around storing data. If we do our job correct, then we will have a holistic view where we understand data as knowledge, rows/tuples as facts, and NoSQL technologies as access-oriented implementations that have chosen a particular access-vector over another. RDF triples, graph databases, datalog, document databases, key-value databases: no matter how inverted or how materialized for other access-patterns, the underlying relation is present. </span></div>
<b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
</div>
<hr />
<span style="font-family: 'Trebuchet MS'; font-size: 21px; line-height: 1.15; white-space: pre-wrap;">Another Concrete Example: Steganographical Shadows</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Given the historical journey of abstraction we just surveyed, it would be great fun to show one more last concrete example of a relation. One might say that the overriding theme of this and previous posts is not to confuse abstraction with implementation. To drive home this last point, we take the notion of a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">relation as a subset of a cartesian product</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to a very very concrete place, the real world that you and I live in, the world of three </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">actual </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><cough></cough></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">debatable</span><span style="font-family: Arial; font-size: 15px; white-space: pre-wrap;">) spatial dimensions, in which x, y, and z mean width, height, and depth. The world in which my body occupies a subset of cartesian points, and has extent and boundaries. A world in which every object is a subset of the entire universe.</span></div>
</div>
</div>
<b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Consider a collection of trash on the ground in an art studio.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="459px;" src="https://lh4.googleusercontent.com/cP3cqsqbAGH3gsjflXzPvmrwWvH997RyO256fVZ4KiYU2oBsmiNQEXPWK36SONk8U9890HBqsD_1rXrhHr4wspD-DQv6Hfe11WQs56Rn6y6qmFhXv0x6p5sJoQ" width="613px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I think you might see where I am going with this. </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Our 3-dimensional collection of trash is a subset of our 3-D universe, and each point may be represented as a tuple:</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="191px;" src="https://lh6.googleusercontent.com/iP8u9LJLtGiSGKprEFoyZjXhpbVEqT483sZuJzt1AZBR8gdn83NJ1q3QJGtp8ohQFc64KMBYCvsNUtTA4fB7-_aYo62K14MKIAHEpUKQsBy-AsL2QbxIV_hItA" width="85px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Our trash is a relation. We shall name him </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">TRASHY</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><img height="279px;" src="https://lh3.googleusercontent.com/mIoPsaOxddAGgcCmguEhtrD60b_HdtppJhSnd7k7-c8Sdly5_xeSmSuIHcWxiMs3X22OO5lQZYhJoQLRsC44mGjcpAfFBWrgWZkfPmORyTIDpOLtdespojbGGA" width="372px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">But as you can see in the picture, if we shine a light just right on our trash then we get an interesting shadow on the wall. Actually, let’s say this differently: if we </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">project</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> a light on our trash then we get an interesting shadow on the wall. </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The use of the word </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">project</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is not an accident. We have not only provided a concrete real-world example for a relation (a pile of trash we call </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">TRASHY</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">), but we have also provided a concrete real-world example of an </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">relational algebra operator</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, what Codd called pi (</span><span style="background-color: white; color: black; font-family: Arial; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">π</span><span style="background-color: white; color: black; font-family: Arial; font-size: 16px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">)</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> . We have used light waves to project out </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">depth </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(the z dimension) from our 3-dimensional relation to produce a new 2-dimensional relation. What’s more, the act of projection has revealed hidden meaning (* time to check your dollar bill for the all seeing-eye!! It casts light in all directions!!! It projects new meaning in all shadows!!! ).</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><img height="191px;" src="https://lh6.googleusercontent.com/tV_vbxlYhLBtnGnL7AaGYHeHwroQ5aotiJ00M8Kwj4A67vyEbgCM16ntNVfjp2PXASXA1M8_QW3OKB04ANQqhK9Q2wkKPrZlSWouR1tqxx4mjohneoe6bWiiaA" width="303px;" /><img height="305px;" src="https://lh4.googleusercontent.com/o2AaIzm95dhoLOd2j-TNT5AXqunF4opK0oSyrhPBua68MRdEGrIEl_RQQeuPDAuYP2bzjEPxaadlReEWOOVun8xuGxNpoZo7MOQ4_KWhR0KTJhgrw577q1g4jw" width="408px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">For those of you who love SQL-ish:</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (</span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">select </span><span style="background-color: lime; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">x, y</span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> from TRASHY) as RANDOLPH_AND_MORTIMER</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Conclusion and Further Posts</span></h1>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As mentioned in the <a href="http://merrigrove.blogspot.com/2013/12/what-heck-is-relation-from-tables-to.html">previous post</a>, there is a huge difference between </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">abstraction</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">implementation</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. We have gone through these various examples of concrete relational manifestations not just to reveal what has been hidden, but to face the future with eyes wide open, to recognize the space where technologies occupy and to place new arrivals by coordinates affected by </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">data access patterns</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Abstraction</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> has shown that we can think of </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">relations</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">facts</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">databases</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> as omnipresent. But </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">implementation</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> will tell us why we should choose one technology over another and for what reasons, despite the unifying </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">relational</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> construct. Perhaps this is the promise of the NoSQL movement, which wisely re-marketed the prefix “No” to be an acronym for “Not Only” instead of a negation. It is now more widely accepted than before that SQL is not the only way to do relational programming (a fact long known by the originators of the database movement, Codd and Date). Soon, the next step will be taken and people will see how databases themselves have many different manifestations, from single-server curated systems, to shared-nothing columnar architectures, to distributed file stores with high-latency MapReduce engines, to internet-wide Open World statements.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Prolog/kanren/datalog, rules-engines, knowledge-bases, traditional databases (Oracle, MySQL, Postgress, Vertica, Greenplum, Sybase, Informix, SQLServer, Asterdata, etc etc etc), Big Data technologies (Pig, Hive, Dremel, Stinger, Tez, Hawq, Impala, Cascading, Cascalog, Scrunch, Trident), graph-databases, RDF-Triplestores, document-databases (from the Mongos of today to the eXist and Taminos of a decade ago). </span></div>
<br />
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Most of these serve very different purposes. All of these have a relation embedded somewhere in their DNA. </span></div>
Daniel Eklundhttp://www.blogger.com/profile/11570452431861145598noreply@blogger.com19tag:blogger.com,1999:blog-5887985082054147381.post-59453226512675708352013-12-22T07:14:00.001-08:002013-12-25T19:14:45.424-08:00What the Heck is a Relation? From Tables to Cartesian Products to Logic<div dir="ltr" style="text-align: left;" trbidi="on">
<b id="docs-internal-guid-2cbe1272-1ad0-28da-4ac7-fe339eb954bc" style="font-weight: normal;"><br /></b>
<b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://lh4.googleusercontent.com/Nn-vO3Q6nnp5LJK1ljeIPo2KRQ0-eAsImI02GMWXONb7tEPpaxvq69MCYpz3QB1cCB9Sb26tNELXSqpudJ1nCa0g-A4sW2sWwBpXo0AlszzhOU8B_qecr8vnTg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="147px;" src="https://lh4.googleusercontent.com/Nn-vO3Q6nnp5LJK1ljeIPo2KRQ0-eAsImI02GMWXONb7tEPpaxvq69MCYpz3QB1cCB9Sb26tNELXSqpudJ1nCa0g-A4sW2sWwBpXo0AlszzhOU8B_qecr8vnTg" width="98px;" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Awww...</td></tr>
</tbody></table>
<div style="text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgenPu6ZxylxL8gRHG8Ba2i5VEbmW8ExrypQpdpplmr1ArwsgYws6-zNH8FReIaP61Kz09qoAFSnQTZbg0YWO0GHtGoVxID8Fyf7cviTLVqn0I2bYZbGtlEZL0yRV8e_3LLRbx4_nbQCJY/s1600/relation.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgenPu6ZxylxL8gRHG8Ba2i5VEbmW8ExrypQpdpplmr1ArwsgYws6-zNH8FReIaP61Kz09qoAFSnQTZbg0YWO0GHtGoVxID8Fyf7cviTLVqn0I2bYZbGtlEZL0yRV8e_3LLRbx4_nbQCJY/s200/relation.png" width="200" /></a></div>
<br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br />I have given several talks and courses on Apache Hive and Pig and other new-ish ‘Big Data’ languages built upon Hadoop and MapReduce. These technologies implement an alternate approach to relational from SQL while still retaining the character that makes them DSLs within the domain of relations/tables. In the case of Hive, the syntax is nearly the same as SQL. In the case of Pig, the syntax has inspired much confusion, and </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">not enough love</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. In both cases, the semantics are nearly the same with SQL.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">During these talks, I have stressed several points about the nature of database technology, unix command lines, data serialization and higher-order functional programming that I will now try to boil down to a set of coherent posts. In attempting this, I hope to anchor these themes around the unity of a mathematical concept, the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">relation</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. Ideally this (and following) posts will be both a comment on the mathematical underpinnings that have driven multiple multi-billion dollar industries (the 70’s relational boom, and the current Big Data fervor), and on the social/psychological/marketing forces that drive us to differentiate technologies from one another. </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">But to start this, we need to understand the foundational notion of the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">relation</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. </span></div>
<h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">What the heck is a Relation?</span></h1>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Attempt 1: A relation is a subset of a cartesian product</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><img height="171px;" src="https://lh4.googleusercontent.com/2qNPbYN4O0nK2MFKIp4bbenIzwewOzw7AM8fSpmFdFUS4p5DfBqSJkvOswG2WfE6AqGyFgl7tMxDrz0XgqQ4q-jMYVX8XIWu3n9IRe3hrJux40baw777KteQdA" width="192px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">Let us stipulate that while this answer might be sufficient and accurate, it suffers from an explanatory paucity. Ok. So now what?</span><br />
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Attempt 2: A relation is a table.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="183px;" src="https://lh3.googleusercontent.com/SS3rB7ueBwh5hdHE579a8Sl4y16eYR4ISqSaDrUJfmw7nzHBgIP-cpvX4IZimYGjZiqgq0JAPvur9o0xXO_SZ6YZsOGHBJVSMOnX4yCWppv5H2Y6wjFUrDgl9g" width="310px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Yes. This is correct too. It provides an alternate explanation and one that everyone who has ever worked in the IT field should be familiar with, </span><span style="font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">ad nauseum</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">. But how can this be the same as the first definition, that “</span><span style="font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">a relation is a subset of a cartesian product</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">”? Actually, what the heck is a “cartesian product”, and what does this have to do with a table again?</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">We are going to need to spend some time reconciling these two different manifestations. We will probably need to learn some new terminology, like </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">set</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">domain</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, and </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">cartesian product</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. But before we do that, let’s try a couple more explanations.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Attempt 3: A relation is a generalization of a function.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="330px;" src="https://lh4.googleusercontent.com/7ZML7t-N4rlUI2G9clwtP4QKeL9an0l2s-zBzZ4CVcaZZOH1s-58yyEDBi9w991DJ6IkTLghIT334DGnJJbSBOaMfq68BZbIDSNc9hu7pGhSxCVGJuP1WhW09A" width="574px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">This too, provides us a concise answer, and is yet another explanation that we will need to reconcile. Very mathematical. Very true. But dry, and seemingly unrelated. (For what it’s worth, the grey arrow is the famous </span><span style="font-family: Arial; font-size: 15px; font-style: italic; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">vertical line test</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">).</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Attempt 4: </span><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">A relation is a property that assigns </span><a href="http://en.wikipedia.org/wiki/Truth_values" style="text-decoration: none;"><span style="background-color: white; color: #0b0080; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">truth values</span></a><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;"> to k-</span><a href="http://en.wikipedia.org/wiki/Tuple" style="text-decoration: none;"><span style="background-color: white; color: #0b0080; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">tuples</span></a><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;"> of individuals</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><img height="191px;" src="https://lh5.googleusercontent.com/wMjSvH0XtpSevg7iVhBe0p42QTlncCNh6RDWc6_zktdBHEZjrik6tKIRsbgDkmmow8zPYVPwXwqXf3LsoUNAJAWqp7-OZplpzE_h6ltr2xoUBv4vYqYU3NUaSw" width="288px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I’m afraid that if you found Attempt 1 and 3 off putting, this one will bumfuzzle you with its terseness. It has something to do with another term that you might have heard: a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">predicate</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, which is a function that assigns </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">truth-values</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Are all these things the same thing? And what does this have to do with databases? with Prolog/Datalog? with Hive, with Pig, with SQL? with logic? with predicate calculus? with relational algebra? with inference? with knowledge? with truth?</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It seems we have have started with something simple, and gotten ourselves into quite a mess. We claimed that we needed to reconcile these notions, (these manifestations), of this abstraction that we call a relation. But in order to do this we will need to discuss things like </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">sets</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">predicates</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">domains</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">ordering</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">functions</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">operators</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">tuples</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">individuals</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">facts</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">ground terms</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">truth-values</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> etc. We are entering the realm of meta-mathematics.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="153px;" src="https://lh5.googleusercontent.com/3geRhEodw8xE4gZsiiZ2dVkiLxIkEBJNOZ64cpP-l09OKto_Xxy4w0YyHMvJCLttFxyhzyCbwk4QnZ6HIOqpr3d3xBj1Qbv0i2BC2O2gm52UDqxni9SFYGK-4w" width="322px;" /></div>
<b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The Many Faces of the Relation</span></h2>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Let us list out the ways we have just described a relation:</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<ol style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">as an </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">arbitrary subset of an arbitrary cartesian product</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">as the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">generalization of a function</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">as a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">table</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">as a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">predicate function</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (‘a property that assigns to every k-tuple a truth-value’)</span></div>
</li>
</ol>
<div>
<span style="font-family: Arial;"><span style="font-size: 15px; white-space: pre-wrap;"><br /></span></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To reconcile all these notions we can do a pairwise match and compare each interpretation against the other. Let’s start with the two interpretations that will give us the most traction for the terminology that they introduce: a relation as a table versus a relation as an arbitrary cartesian product subset.</span></div>
<b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The Table Versus the Cartesian Subset</span></h3>
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<img height="255px;" src="https://lh3.googleusercontent.com/lhIShxq7I1uL6cwFvvRRkKr8j4Ds7vzlmjSrd9afAfY2mhIVtFWTaR2aJCrC31MqPdRTpE9PT7CukVA8twoqbJEihi9W3NdNsAxlRHhkWCG7GJKC53AUq8PMtA" width="518px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></h3>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To get this started on the right foot we need to nail down the terms </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">dimension</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">cartesian product</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, and </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">domain</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. This is not daunting, and most everyone intuitively knows what they are, they have just never used the terms in a different context or realized the connection. Rather than belabor the point, we are going to show the following equalities:</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="179px;" src="https://lh3.googleusercontent.com/FzTAtQJuEIaYeMEo4AuFi6uiG7WwqUdyOVb1iZfY1mM3IBLe5GhN-p7oyr-YB7IxADhOEnO9wI7bfgz50nCMAlFA7kwGcW3EqFx1BheItaSijoshhU6DXHa1qw" width="287px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It’s helpful to start with something simple, like plain old vanilla </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">x</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">y</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> values on a normal x-y plane. This was the original Cartesian plane, as willed into existence by Descartes as he lay in bed and stared at a bug crawling up and down a wall.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="202px;" src="https://lh5.googleusercontent.com/4n_dWFzuDC3hCLlHMDRiohcsgIRAfR6nI6JEVnJUbyOEdRpg8BrBsMV13UlkulC-ruPw8RiFMx9SZkVxlHVh6pYrQ7N71E_f8AlrrfkyoIbceaKsHudKEIJmoQ" width="188px;" /></div>
<b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In this case, our cartesian product is a multiplication of two dimensions x, and y. If we assume that </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">x</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is an integer, then we are saying its domain is an integer. If we assume that </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">y</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is an integer, then we are saying its domain is an integer. Together, these two domain multiplied give us the cartesian plane of (x,y). An </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">individual</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> on this plane is also called a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">tuple</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, or a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">point</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> -- all three words are the same.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As we see above, we have selected an arbitrary subset of this cartesian product. This subset has three individual/tuples/points: {(0,4), (1,4), (1,5)}. We could list these point out in a table if we wanted:</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="164px;" src="https://lh3.googleusercontent.com/wqh_eg7XsPcnWrUpu_YUbb-GfPWKJjr17UU7N3I03j-qUUnFU6RJEVUSl7WE8VVWMJUuwQcGkh06g1s6q7NatyHOZYNGnZLYQ9x7Hk0EN3fvCCRvKPgwoMfjJg" width="97px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">and just like that we have shown the equivalence between a table and a subset.</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> We see that a row is the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">individual</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, or the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">tuple</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, or the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">point</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. They are the same. We see that a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">column</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">dimension</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. We see that the column’s </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">type</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is the same thing as a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">domain</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. </span><br />
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;"><br /></span>
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">Let’s do one more thing. Let’s give the relation a name. Let’s call it “Bob”. Finally, we have a formal relation as described by Codd.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="272px;" src="https://lh6.googleusercontent.com/7ZORYfmZX0jQ5NfcML2GwUAGy8DoUVyXQphiGkADkk4hzRNFS_T7MHLhn3UODwAGRnlJaQaHxOeA166L0RlW2pWnu_BSdk_PKhFSTikxXYhNsS51U85zeTrgkQ" width="299px;" /></div>
<b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Abstracting from Numbers</span></h3>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">While Descartes thought about numbers, nothing prevents us from using the idea of </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">dimensions</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> in a non-numeric way. We can make our </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">dimension</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> anything we want. First off, rather than </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">infinite</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> numbers, we can have </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">finite</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> colours, or </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">finite</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> vegetables. </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><img height="246px;" src="https://lh4.googleusercontent.com/9XOAURGgoOP-lTdX6EP4xQsMSguzKMe0lOE8I46YF2f8csSqyXAGj_i5ZEXFZs6iS4pbGFAejDV8U0-gmYZLS8O9lsvHLzugNjU0EK3KAfGgt-o4dLb5GDZ9uA" width="454px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In the above diagram, we see a green rectangle that represents the entire universe of possible coloured vegetables if and only if we assume that there are only 4 colours {</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">red</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">,</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">green</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">,</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">yellow</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">,</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">blue</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">} and 3 vegetables {</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">peppers</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">,</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">berries</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">,</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">avocado</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">}. By multiplying these out, we see that the entire universe of individuals possible (called a domain of discourse) is 12. This is a finite cartesian product. Nevertheless, we can take any arbitrary subset of this cartesian product, name it, and we have ourselves a relation.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="273px;" src="https://lh6.googleusercontent.com/HVh-dTsAAHspQTTwbYPvlG-9JIUehdWPP24uLykLeP1V0JCKDSM8ZILgWpzcW5v6yG8dk_DEzXgcsA5ox7Cujavl7JdpKAVJjUtxHKxaZHzgeMc-fqABpUAoHQ" width="582px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Here we can see that we have selected a subset of five </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">tuples/individuals/points</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, and simultaneously shown this subset as a listing in a table. We have named this relation </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">TrueFood</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, presumably under the assumption that these particular </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">individuals</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> represent a listing of those things that are actual foods.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<span style="color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-weight: bold; line-height: 1.15; white-space: pre-wrap;">Moving to Higher Dimensions</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Finally, let us consider the classic table example, the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Employee</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> table, in which the FirstName, LastName, and Department may capture a listing of everyone in a company. If we draw an analogy with spatial dimensions, then we can imagine last names running across the X-coordinate, first names running on the Y-coordinate and departments running on the Z-coordinate.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="199px;" src="https://lh4.googleusercontent.com/_hZWGL8d153ihSriZ-kDGPPOwcSuaByKFe1O9nq8VK6iWaHgOJigczUP7Oz89SAQvKC-_aBs65ASSmVsNCl8w_f4ipQwVnmAYYYVeawU_crGkfeORAs9HhIrkQ" width="184px;" /><img height="171px;" src="https://lh6.googleusercontent.com/ECJF0Yv9acpcaWufjjxzwGZVv_bPmQQizhAiDh4ArT4M6lX9qeCoTvN2Q2rqx8FnrCbf01j1d-NIbe4gCCleAJKIaWKHeeBgZWKg9uCelzhR78-5W-HK1_pJfA" width="289px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As shown above, this is merely a spatial representation of three different dimensions. There is no inherent ordering, much like the example with veggies and colouring. Nevertheless, this is absolutely a valid visual metaphor. The entire space of possible cartesian products would range across individuals of 3-tuples. </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="525px;" src="https://lh5.googleusercontent.com/yp512NT1MLdHLjS7MghkVj93QXgrFEnnu3SwaLT-4SkuhWNsn_m3M-v99kiwEefeGLy4HANnu5RUgbb569aI1VaC1bNLXovtV7DSX1ATPQr-fJMka6pSti3wYA" width="510px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Just to make things simple, imagine a universe in which there were only 7 first names, 7 last names, and 3 departments. In this finite 3-dimensional space, there are 63 possible individuals/points/tuples. Now imagine an arbitrary subset of this universe of individuals.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="378px;" src="https://lh5.googleusercontent.com/Ms4PGC1kG7A4gty2_wuyEN275zS-v9Cn-Joazt1Zg5agIB3PmIjymmrmj97ZHVm1swU7svociXuQaZQqXUBPzT0XzMG4NS9H3F8ittOwgz0ccipn7Q2E0zXM1Q" width="326px;" /><img height="301px;" src="https://lh6.googleusercontent.com/xsjzJH9sajYPc5vu5_OfLNVhGeHKUfvMzNQqz2GqvzG1Fr9q7UNRGI2iYZF7JtGkHXib40ep5DY_Y2M7yFPt8m-346bgtmO55EpAc-3RJ7t2cV8zJ0_1S0nfSw" width="278px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">We can name this subset </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Employees</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, and we have a relation that represents our listing of all the people in our universe who we believe to be employees within our company. It is this final point that provides us a hint on how we can reconcile our metaphor further, by drawing out the analogy to statements of </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">facts</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. This allows us to see our table not only as a subset of the entire universe of people, but as our collected knowledge about who is an employee and who is not (* we will return to this when we examine our last statement that a “</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">relation is a property that assigns to every k-tuple a truth-value</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">”).</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><img height="368px;" src="https://lh3.googleusercontent.com/GXp4gpCSuTjnzrUpK_b2G-Wn3FmS9w4F5JYy9UXk8yM_NiYSxJXi6jCsATBsjrRplpqsb78JvBI8JkiCLG4EVrKYK9-dvJS1mSoZ9zZevve26uBbBkLcOynhvQ" width="864px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">We are tantalizingly close to seeing how relations may be manifested within the world of logic -- mainly in understanding that almost all logical systems start with statements of </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">fact</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> -- “</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Socrates is a man</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">”. But before we continue down that path, let’s talk about our understanding of relations as generalizations of a function.</span></div>
<b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Cartesian Subsets Versus Relation-is-a-generalization-of-a-Function </span></h3>
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<img height="291px;" src="https://lh3.googleusercontent.com/6_bF1FaPxpZZgWNkpZA6kgZj-40vq7h0VriU7eT7NZQb3aVxE3m0FjhC3zEutiV0qdybqJLhsTm6j0INE3ETs6OZjeQo47AZJFla-L_y8U0IjEBCsWQrBjuXDA" width="516px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></h3>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Now that we understand how a relation may be viewed as an arbitrary subset in cartesian space, it makes sense to push through and understand the statement “</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">a relation is the generalization of a function</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.” This particular statement can be seen visually when we see that a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">function</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is itself a subset of cartesian space.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><img height="295px;" src="https://lh3.googleusercontent.com/1IGYOYJKVxGULNj-w2CUD4hvwY_Ar0-8fnn198b6sO7dgoqR9HFbiivntu-ke2ne60NeKf_2WVRaojZ_TzIjg1-pEm2mOXEkCP9on8z-aKr5W-0y7j5iyyELgA" width="354px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Above we see the classic example of a numeric function in cartesian space. Our function squares the input, and we may say that our function </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">maps</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> values of </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">x</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to values of </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">y</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. Clearly we have a subset in the thick black line that visually represents all the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">tuples</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">/</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">rows</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">/</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">individuals</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> that belong to this 2-dimensional cartesian space. This function is a relation. </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">All functions are subsets of cartesian products, and therefore all functions are relations</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. </span><br />
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;"><br /></span>
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">But not all relations are functions! Clearly, functions are subsets that meet a particular criteria. </span><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">We used the term </span><span style="color: black; font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">map</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> because that is the standard nomenclature that is associated with the set-theoretic description of functions. Intensional or extensional debates flourish around the true nature of what a function is, but it is probably best not to get too religious about it at this point. The </span><a href="http://en.wikipedia.org/wiki/History_of_the_function_concept" style="line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">history</span></a><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> of the notion of a function is a fascinating read.</span><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Most people who remember their high school algebra remember the statement that a function passes a “vertical line test” in demanding that an input value (a domain) be </span><span style="color: black; font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">mapped</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> on only one output value (the range). This is the simple explanation of the criteria that separates the nature of a function as a subset from an arbitrary subset. The vertical line test is clearly only applicable for two-dimensional visual representations -- more generally, one might say that at least one of the dimensions represents an </span><span style="color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">output</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> or </span><span style="color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">range</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">, and that the cardinality of its projection equals the cardinality prior to projection.</span><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">We could equally have said that the function y=x</span><span style="color: black; font-family: Arial; font-size: 9px; vertical-align: super; white-space: pre-wrap;">2</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="color: black; font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">relates</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> the values of </span><span style="color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">x</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> and </span><span style="color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">y</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">. This is absolutely correct too. However, when we use the word </span><span style="color: black; font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">maps</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> we are adding the further constraint of the vertical line test.</span><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Just to provide one more interesting explanation on how relations are generalization of functions. Let us compare a </span><span style="color: black; font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">relational programming language</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">, versus a </span><span style="color: black; font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">functional programming language</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">. Now, if you saw the phrase</span><span style="color: black; font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;"> relational programming language</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> and immediately thought SQL, then this is not quite what we mean. Instead, we are talking about general purpose programming languages alternatively known as </span><span style="color: black; font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">logic programming languages</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">. Let us compare how the relational/logic programming language </span><span style="color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">Prolog</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> would syntactically represent the </span><span style="color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">addition operation</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> from how a functional language would represent the </span><span style="color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">addition operation</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">.</span><br />
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;"><br /></span>
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">The following lines of codes are valid snippets of Prolog code:</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> plus(6,2,4). % Here we are saying that 6 is related to 2</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> % and 4 by being the result of adding 2 and 4.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> % but just as equally, we can say that 4 is related</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> % to 6 and 2 by being that which added to 2</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> % gives 6.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> plus(200,199,1).</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> plus(X,2,4)? % Under this query, we will get X=6</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> plus(6,Y,4)? % Here we will get Y=2</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">And the following line of psuedo-code is valid in any language that has functions (no need to demand the use of a full blown functional language like Haskell):</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> x = plus(199,1); // x will be 200 if we implemented this correctly :)</span><br />
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;"><br /></span>
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">The difference in them is interesting. </span><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">While the relational language has the word </span><span style="color: black; font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">plus</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> taking 3 arguments and has no output variable, the function-like languages use 2 arguments and assigns/binds the output to a variable. The difference is one of generality. </span><span style="color: black; font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">Functions</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> promote one of the dimensions to an output, (they </span><span style="color: black; font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">map</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> all inputs/domains to an output/range). But </span><span style="color: black; font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">relations</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> assume no special direction: 6 is related to 2 and 4 by being the result of adding 2 and 4, </span><span style="color: black; font-family: Arial; font-size: 15px; font-style: italic; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">but just as trutfully</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">, we can say that 4 is related to 6 and 2 by being that which added to 2 gives 6.</span></div>
<b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Finally: Comparing A Relation as a Subset versus A Relation as a Predicate Function, or A Relation is a Property that Assigns to each K-Tuple a Truth Value</span></h3>
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<img height="272px;" src="https://lh4.googleusercontent.com/iYgUqfbg4-OMXfnCeBmIOVDlx3Nugwd_xVQ9cEJJ0s3rUmLL7-AjdF-7jgPbRbgqiW2N3YWKN4JUp1YG8TdaX6KzJomESrDoQB_iiL2sh-E8ZL3wwEU7osVyLg" width="599px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></h3>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Let’s keep going. </span><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">This reconciliation is very interesting, and one that confused me for a long time. Here we have gone to the trouble of saying that a relation is a generalization of a function, and now we are going to turn around and call it a function. This seemingly contradictory statement is not really contradictory, but is a result of the fact that we are at the lowest level. Things like </span><span style="color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">sets</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">, and </span><span style="color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">relations</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> and </span><span style="color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">functions</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> are the fundamental building blocks of logic, meta-mathematics and mathematics. When you are this low down then we cannot help but re-use our low-level notions to describe things in terms of another. So let’s explore this weirdness.</span><br />
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;"><br /></span>
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">Here are the two equivalent statements:</span></div>
<div dir="ltr">
<table style="border-collapse: collapse; border: none; width: 100%;"><colgroup><col width="*"></col><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">A relation is a property that assigns to every k-tuple a truth-value</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
</td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">A relation is a predicate function</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
</td></tr>
</tbody></table>
</div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So, let’s first define a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">predicate function</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">: a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">predicate function</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is a function that gives a truth-value about an element. Another way of saying this is that a predicate is a function that returns an element of the set { </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">true</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">false </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">}. </span><br />
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">The phrase “k-tuple” is mathy for an </span><span style="color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">individual</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> or </span><span style="color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">element</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> or </span><span style="color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">point</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> or </span><span style="color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">row</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> with k-dimensions. “Truth-value” is mathy for something that is either </span><span style="color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">true</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> or </span><span style="color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">false</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">. “Predicate Function” is mathy for a </span><span style="color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">function</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> that returns a </span><span style="color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">boolean.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><img height="311px;" src="https://lh3.googleusercontent.com/zTi5_8emjlVLkcJFzLMq_QFGivj5DkgxiNR7DvsHtG90BF2QRmJcJgbwddnPZjSyJhKFWOFSw40EdR6g0Jw2f0hgrcP2E8brahQyaQnqCsYZQ7HKYrTor1ZEQQ" width="375px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">We are still not there. How can a relation, which we know is a subset of a cartesian product, also be a predicate function? Well, look at the graphic above, where we show the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Employee</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> table/relation/subset. As you can see, the entire universe of possible people is a larger space of </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">individuals</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. However, only the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">individuals</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">/</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">tuples</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">/</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">rows</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">/</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">elements</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> of our subset are what we call </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">employees</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. So, why not make our predicate function something simple like: “</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Is the individual X an element of the subset Employees?</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">”. </span><br />
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;"><br /></span>
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">And just like that, we have unified the notions. Our function signature looks something like this: </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> EMPLOYEE(X) -> BOOLEAN</span><img height="368px;" src="https://lh3.googleusercontent.com/GXp4gpCSuTjnzrUpK_b2G-Wn3FmS9w4F5JYy9UXk8yM_NiYSxJXi6jCsATBsjrRplpqsb78JvBI8JkiCLG4EVrKYK9-dvJS1mSoZ9zZevve26uBbBkLcOynhvQ" width="864px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The implementation of the function would merely check to see if </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">X</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> was an element of the subset . This is called an </span><a href="http://en.wikipedia.org/wiki/Indicator_function" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">indicator function</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. So if we were to call our function on a couple of possible </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">individuals</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">/</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">tuples</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">/</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">rows</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">/</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">elements</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> we would get</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">EMPLOYEE(Daniel Smith, Marketing) → TRUE</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">EMPLOYEE(Jebediah Vargas, Marketing) → FALSE</span><br />
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;"><br /></span>
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">This is equivalent to asking, is there a row in the table with this value? If there is, then the person is an employee. If there’s not then the person is not an employee.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr">
<table style="border-collapse: collapse; border: none; width: 100%;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #c9daf8; border: 1px solid #b7b7b7; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Tangent</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">: This last point is a fundamental one that has different interpretations. In the Closed World Assumption, a database is complete and every tuple represents a fact. If a fact is not present, then it is false. In the Open World Assumption, a database is a collection of known facts, and the lack of a fact is not equivalent to saying that the predicate is false. See Reiter.</span></div>
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This is a fascinating place where philosophy meets the technical. For a concrete example of this, look to the Semantic Web effort.</span></div>
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The Semantic Web is a system of binary relations manifested in something called RDF triples. An RDF triple is nothing more than a single floating tuple of a binary relation, and the entire internet is a database of disperse binary tuples, representing currently-known facts. The philosophy of the Semantic Web is very much around the Open World Assumption.</span></div>
</td></tr>
</tbody></table>
</div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr">
<table style="border-collapse: collapse; border: none; width: 100%;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="background-color: #c9daf8; border: 1px solid #b7b7b7; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">SUPER IMPORTANT Tangent</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">: Given everything we’ve said about these different manifestations, what does having a NULL value for one of the dimensions imply? Can you see why allowing NULLS in a database has historically really bothered a lot of people?</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><b id="docs-internal-guid-6b043bab-1c66-32b3-ea86-ba5b08537d4c" style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></span>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">We can easily add a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">null</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;"><u>table interpretation</u></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, as shown below:</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<img height="171px;" src="https://lh3.googleusercontent.com/6rzZ9Aqvf91AHcU-XtqV7c2k-XGHMJKljjk7BbNK9dyH5R0zDXh4ObPZE4M1mSU80tUJlza3OVTzBZBHjFwRyW2GJeLNYOF99jy3aomSTdsXqecUXyT0iPaGng" width="216px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<span style="background-color: transparent; font-family: Arial; font-size: 15px; line-height: 1.15; vertical-align: baseline; white-space: pre-wrap;">but what are the possible interpretations when we try to think of this table under the </span><span style="background-color: transparent; font-family: Arial; font-size: 15px; line-height: 1.15; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">cartesian subset interpretation</span><span style="background-color: transparent; font-family: Arial; font-size: 15px; line-height: 1.15; vertical-align: baseline; white-space: pre-wrap;">? How do we represent the missing dimension? As a line? As a <i>possible</i> line? </span><br />
<span style="background-color: transparent; font-family: Arial; font-size: 15px; line-height: 1.15; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="background-color: transparent; font-family: Arial; font-size: 15px; line-height: 1.15; vertical-align: baseline; white-space: pre-wrap;">What about the <u>predicate function interpretation</u>, where the function needs to return either <b>TRUE</b> of <b>FALSE</b>? </span><span style="font-family: Arial; font-size: 15px; line-height: 17.25px; white-space: pre-wrap;">Are all green veggies true food? Is there one green veggie that is a true food and we just don't know which one it is? </span><span style="background-color: transparent; font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">What happens if someone asks "</span><span style="background-color: transparent; font-size: 15px; line-height: 1.15; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">TRUE_FOOD( "green avocado" )?</span></span><span style="background-color: transparent; font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">" Should we return </span><b style="background-color: transparent; font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">TRUE</b><span style="background-color: transparent; font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;"> or </span><b style="background-color: transparent; font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">FALSE</b><span style="background-color: transparent; font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">? If we don't, and return something else, we have exited standard logic with Boolean values and entered <a href="http://en.wikipedia.org/wiki/Three-valued_logic">more esoteric realms</a>.</span></div>
</td></tr>
</tbody></table>
</div>
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
</div>
<hr />
<br />
<h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Conclusion</span></h3>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://lh5.googleusercontent.com/BRn8YDcGUOnTqba2UahLBLsHFtJkU7j29e4PobhRKciGEpVnUrTNDJHL7pS1WVIUyxnTq3DcNuRop1utRMaWU9iLJ-D8UP-Nmbmcm4uD7HSLcSO4FP2W4IMvjA" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="211px;" src="https://lh5.googleusercontent.com/BRn8YDcGUOnTqba2UahLBLsHFtJkU7j29e4PobhRKciGEpVnUrTNDJHL7pS1WVIUyxnTq3DcNuRop1utRMaWU9iLJ-D8UP-Nmbmcm4uD7HSLcSO4FP2W4IMvjA" width="129px;" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Tree of Life</td></tr>
</tbody></table>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The overriding theme for this little post, was that the notion of a relation as an abstraction provides a unifying means of thinking about data. </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I have never claimed that the implementation details are irrelevant</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. There is a reason that OLAP/MPP databases are separate products from OLTP databases -- they materialize and store their relations differently to account for different access patterns. There is a reason that graph databases are a separate product (index-free adjacency inverts the primacy in typical relational stores of the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">relation</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> over the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">individual</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to be the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">individual</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> over the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">relation</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">). There is a reason why Prolog is a language for storing a small number of mostly-static facts of atomic tuples, while a database is a technology for storing a huge number of mostly-dynamic facts. There is a reason why database management systems exist in the first place (curated schema with </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">a priori</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> schema knowledge provides the management system up-front knowledge on types and indexes that make relational queries real-time).</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<span id="docs-internal-guid-6b043bab-1c4f-ebdf-d5ff-e3df3a2f3feb"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">I have also not claimed that any of these ideas are new. It is always useful, to go back and read the original source material, especially Codd’s 1972 paper where he proved that relational algebra and relational calculus (itself a modified version of predicate calculus -- a fundamental logic) were one and the same. Go read the papers of Gallaire, Minker, Reiter, Nicolas, Kowalski, whose 30 year old premises recognize the logic programming and databases are the same and propose the notion of deductive databases. (* there are dozens of source material papers I should eventually collect together into a bibliography, if I were so inclined).</span></span><br />
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;"><br /></span>
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">In the following post: “<a href="http://merrigrove.blogspot.com/2013/12/the-occultation-of-relations-and-logic_22.html">The Occultation of Relations and Logic</a>” we will drive home the point that <b>relations are abstractions</b>. Too often, people equate relations </span><span style="font-family: Arial; font-size: 15px; line-height: 17.25px; white-space: pre-wrap;">with SQL, or (worse) with </span><span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">database management systems (Oracle, MySQL). We will reveal the opposite, and show how relations may be used to <i>think</i> about the world, and how even things like shadows and log files may be interpreted as <i>facts</i> and <i>tables</i> and <i>cartesian products, </i>i.e. as <b>relations</b>.</span><br />
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;"><br /></span>
<span style="font-family: Arial; font-size: x-small; line-height: 1.15; white-space: pre-wrap;">(thanks to Dean Wampler, and Bhavana C for feedback)</span><br />
<br />
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></div>
Daniel Eklundhttp://www.blogger.com/profile/11570452431861145598noreply@blogger.com22tag:blogger.com,1999:blog-5887985082054147381.post-32806527646724163192011-12-28T10:20:00.000-08:002013-12-09T18:08:42.764-08:00What the Heck are Algebraic Data Types? ( for Programmers )<div dir="ltr" style="text-align: left;" trbidi="on">
This post is meant to be a gentle introduction to Algebraic Data Types.<br />
<br />
Some of you may be asking why you should learn Algebraic Data Types and how will they change your world? I am not going to answer that, but suffice it to say that Algebraic Data Types are the underpinning of the type systems to the ML derived languages, Haskell and OCaml included, and their construction and properties allow for the power (and inference) that accompanies these type systems. They are cropping up in other languages, like Scala, F#, and Clojure. Well, here are my 2 cents.<br />
<br />
I wrote this blog post because I had trouble learning what an Algebraic Data Type was, and could not find an introduction that did not either 1) leap immediately to copy and pasted phrases, 2) mislead, like make the claim that "<b>algebraic data types are the same thing as discriminated unions</b>" or 3) delve into graduate level formalism. This blog post represents the results of what I have teased from numerous sources.<br />
<br />
There are now tons of us programmers with sadly insufficient mathematical grounding, who are attempting to branch out and learn something new. The confusion that comes from a sea of new words and overloaded acronyms can be very frustrating (e.g. don't use the acronym ADT for algebraic data types, because that generally means <i><b>abstract</b> data type</i> -- a completely orthogonal concept). I wanted to be able to put a check mark next to my internal to-do list item for "What the heck are Algebraic Data Types?" This blog post is meant to help people do the same.<br />
<br />
<span style="font-size: large;">Quick Start </span><br />
To start, let us attempt a definition of <i>algebraic</i> because for some weird reason, no introduction to Algebraic Data Types ever tries to explain this etymology. For purposes of simplicity, let us define algebra to be two things: 1) a <b>set</b> of <span style="color: #990000;">objects</span> (<i>do not think objects as in object oriented</i>) and 2) the <span style="color: #38761d;">operations</span> used on those objects to create new objects from that same set. An introduction to <i>algebraic data types</i> is therefore an introduction to types and the operations used to create new types.<br />
<br />
<span style="font-size: large;">Examples of some other Algebras</span><br />
In the case of <i>numeric</i> algebra (informally known as high-school algebra), the set is the set of <b><span style="color: #990000;">numbers</span></b> (whether they be natural, rational, real, or complex) and the operations used on these objects can be (but definitely not limited to be) <b style="color: #38761d;">addition</b>, or <b style="color: #38761d;">multiplication</b>. The algebra of numbers is therefore the study of this set, and the laws by which these operators generate (or don't generate) new members from this set.<br />
<br />
Another example: the <a href="http://en.wikipedia.org/wiki/Relational_algebra"><i>relational</i> algebra</a>. In this algebra, the set of objects are the set of <b><span style="color: #990000;">relations</span></b>. Unlike numbers which we tend to (erroneously) view as atomic, a relation is a <i>thicker</i> kind of object containing a number of named columns, their associated domains, and a number of rows. Each '<b><span style="color: #990000;">relation</span></b>' (otherwise known as '<b><span style="color: #990000;">table</span></b>') is therefore the object unto which operators may operate. These operations include, amongst others,<br />
<ul>
<li><b style="color: #38761d;">projection</b> (π) create a new relation from the existing one by removing columns, </li>
<li><b style="color: #38761d;">selection</b>(<i>σ</i>) create a new relation from an existing one by removing rows that satisfy a predicate function, <b style="color: #38761d;"> </b></li>
<li><b style="color: #38761d;">joining</b> (⋈) create a new relation from <i>two</i> existing ones by matching up the domains and values of a chosen column,</li>
<li><b style="color: #38761d;">renaming</b> (ρ) create a new relation from an existing one by renaming a column </li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXErtf2VgUNM1EsIxKBdgqdkg-HsC8Y4pD5ugUwHvZJEX1Kpx_YEVHb-qILmlwcJHVf1ftAhxYfq2VywrXTEAY0dQzPXzHtzmVbirS5EN1GfHHf8OSgLU_OLqEMI__KyWTVhiDRFnkYx8/s1600/relationalOperators.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXErtf2VgUNM1EsIxKBdgqdkg-HsC8Y4pD5ugUwHvZJEX1Kpx_YEVHb-qILmlwcJHVf1ftAhxYfq2VywrXTEAY0dQzPXzHtzmVbirS5EN1GfHHf8OSgLU_OLqEMI__KyWTVhiDRFnkYx8/s640/relationalOperators.png" width="193" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
The key thing here to realize is that an algebra allows us to talk about the objects and the operations abstractly, and to consider the laws that these operations obey as they operate on the underlying set.<br />
<br />
So, again, what is an algebraic data type? Let's repeat these questions. What is the set? What are the operators? First, the underlying set is the set of <b><span style="color: #990000;">types</span></b>. For you programmers out there, types are so familiar that we often overlook how fundamental they are. If we were to start with a set of base types, we could use operators to create new ones. Our first operator, the <b><span style="color: #38761d;">product</span></b> operator, will create what is known as a product type, or a record type, or a tuple-type. We pair, or treble, or quadruple, or quintuple, or n-tuple existing types together, to create an aggregate type. A classic example is the <b><span style="color: #990000;">Pair</span></b> type. As a hard example, consider the Pair as implemented in a bevy of object oriented languages:<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">class Pair{</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> float x;</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> float y;</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">}</span><br />
<br />
<i>change the above code's syntax to your favorite python, ruby, scala, etc. It doesn't matter.</i> <b>The act of writing 'class' and defining this type IS our product operator</b>. Object oriented programmers tend to call this operation <b><i style="color: #38761d;">composing</i></b> -- but whatever you call it, it's the same thing. From the pre-existing type <b><span style="color: #990000;">float</span></b> we have created a new type called <b><span style="color: #990000;">Pair</span></b>. Does our operator allow us to keep on going with this? Let's try it:<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">class PairOPairs{</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> Pair start;</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> Pair end;</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> int time;</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">}</span><br />
<br />
Well, there you go. From two existing types (<b><span style="color: #990000;">Pair</span></b> and <b style="color: #990000;">int</b>), we have created a third. This is what operators <b>do</b> in an algebra.<br />
<br />
Our class definition as a product operator has spun a new type into existence from other ones, just like<br />
<ul>
<li>'<b><span style="color: #38761d;">+</span></b>' allows us to combine <b><span style="color: #990000;">3</span></b> and <b><span style="color: #990000;">4</span></b> to get <b><span style="color: #990000;">7</span></b>, </li>
<li>'<b style="color: #38761d;">*</b>' gives us <b style="color: #990000;">12</b> from <b><span style="color: #990000;">3</span></b> and <b><span style="color: #990000;">4</span></b>, </li>
<li>'<b><span style="color: #38761d;">join</span></b>' allows to create a new relation (which we generally don't name) from '<b><span style="color: #990000;">Employee</span></b>' and '<b><span style="color: #990000;">Salary</span></b>',</li>
<li>'<b><span style="color: #38761d;">or</span></b>' allows us to get '<b><span style="color: #990000;">true</span></b>' from '<b><span style="color: #990000;">true</span></b>' and '<b style="color: #990000;">false</b>' together,</li>
<li>'<b><span style="color: #38761d;">+</span></b>' allows us to get <b><span style="color: #990000;">"hotdamn"</span></b> from <b style="color: #990000;">"hot"</b> and <b style="color: #990000;">"damn"</b> (commutativity matters though!),</li>
<li>'<b><span style="color: #38761d;">not</span></b>' allows to turn '<b style="color: #990000;">true</b>' into '<b style="color: #990000;">false</b>' (the only non-binary operator in this list).</li>
</ul>
We have used an operator on an underlying set to create/synthesize a new member of that set.<br />
<br />
Now, some languages allow us to create these types anonymously, what we call tuples. For theoretic purposes, the tuple containing real numbers can be much the same (isomorphic) as a Pair class containing real numbers. The differences? Well those are practical and vary from programming language to programming language, but are about ease-of-use and the languages implementation of type checking. Practical matters are of great importance to programmers.<br />
<br />
The Haskell syntax for product types is:<br />
<span style="font-family: 'Courier New', Courier, monospace;"> data </span><b style="background-color: white; color: #990000; font-family: "Courier New",Courier,monospace;">Pair</b><span style="font-family: 'Courier New', Courier, monospace;"> = </span><b style="font-family: "Courier New",Courier,monospace;"><span style="color: #38761d;">Pair</span></b><span style="font-family: 'Courier New', Courier, monospace;"> </span><b style="font-family: "Courier New",Courier,monospace;"><span style="color: #990000;">Real</span></b><span style="font-family: 'Courier New', Courier, monospace;"> </span><b style="color: #990000; font-family: "Courier New",Courier,monospace;">Real</b>.<br />
<br />
Note how the '<b style="color: #990000;">Pair</b>' on the left is different from the '<b><span style="color: #38761d;">Pair</span></b>' on the right. The one on the left is the type. The one on the right is the particular named product operator, what Haskell calls a data constructor.<br />
<br />
All this time we have used the <b style="color: #38761d;">product</b> operator to create new types from existing ones. And we called it a product type. Why? Well. The answer is that we are producing a Cartesian product (or multiplication) on members of several underlying sets. Again, consider the Pair. Let's rename it to Point to drive home a point:<br />
<br />
class Point{<br />
float x_coordinate;<br />
float y_coordinate;<br />
}<br />
<br />
In the act of creating the class '<b><span style="color: #990000;">Point</span></b>' we have created a new type of x's and y's together. The x's have values in the real domain. The y's have values in the real domain too. The <b><span style="color: #990000;">Point</span></b> is a Cartesian product from values of type of Reals against values of type Reals or <b style="color: #990000;">(R X R)</b>. As another example, we could have done this:<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">class Person{</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> String firstName;</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> String lastName;</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> int ssn;</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">}</span><br />
<br />
Here we are using Cartesian multiplication to produce a Cartesian product of two strings and an integer ("firstname","lastname",123456714). The three constituent elements may range throughout the space of all possible <b><span style="color: #990000;">strings</span></b> and <b><span style="color: #990000;">integers</span></b> (t<i>his is not yet ideal, as we would love to restrict the length of our integer type -- see <a href="http://en.wikipedia.org/wiki/Dependent_type">dependent types</a></i>) to create a value that is a triple of strings and an integer. Thus our new type is of <b style="color: #990000;">(String X String X Int)</b>. Are there any differences? Of course. I don't know of a way to implement a comparison operator between the <b style="color: #990000;">Person</b>, modulo subjective judgements on character. With <b style="color: #990000;">Point</b>, I can implement a 'distance' from the origin metric for some ordering.<br />
<br />
How you use the types is 100% up to you. Do you want to use 'Pair' to capture a point in x/y space, or to represent the length of your left foot and right foot? Well that's up to you. The algebra of data types is abstract. The choice of names and how we compose these types from other types is the act of domain modelling and is something for which programmers get paid.<br />
<br />
So far we have looked at one operator. A 'product' operator. Are there others? Sure. The object oriented guys have something called <b><span style="color: #38761d;">sub-typing</span></b> which allows you to generate a new type from another via 'extension'. Anything else? Well, there is an interesting one that the typical Java enterprise developer rarely sees. It is called a variant type or a <b style="color: #38761d;">sum type</b> or a <span style="color: black;">discriminated union</span>, and allows you again to compose a new aggregate type from existing types. But what is the difference between this and a product type? Well, consider again a product type:<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">class Point{</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> float x</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> float y</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">}</span><br />
<br />
and now consider a completely different example as a sum type:<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">union Identity{</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> String name</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> UUID uuid</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">}</span><br />
<br />
In the first case, a value of type 'Point' has constituent values of an 'x' <b>AND</b> a 'y'. In the case of the discriminated or disjoint union (<b style="color: #38761d;">sum type</b>), values of the type Identity have constituent values of a 'name' <b>OR</b> a 'uuid' -- <u>but not both at the same time</u>. The difference is fundamental. If Java had this then something like the following would be possible, AND typecheck!<br />
<br />
<div style="font-family: "Courier New",Courier,monospace;">
Identity myIdentity = new Identity();</div>
<div style="font-family: "Courier New",Courier,monospace;">
if( db.hasOnlyName() ){</div>
<div style="font-family: "Courier New",Courier,monospace;">
myIdentity = "Daniel Eklund"; // could be a string</div>
<div style="font-family: "Courier New",Courier,monospace;">
}else{</div>
<div style="font-family: "Courier New",Courier,monospace;">
myIdentity = java.util.UUID.randomUUID(); // OR a UUID</div>
<span style="font-family: 'Courier New', Courier, monospace;">}</span><br />
<br />
The Haksell notation for the sum type is:<br />
<span style="font-family: 'Courier New', Courier, monospace;"> data <b style="color: #990000;">Identity</b> = S <b><span style="color: #990000;">String</span></b> <b><span style="color: #38761d;">|</span></b> U <b style="color: #990000;">UUID</b></span><br />
<br />
<span style="font-size: small;"><i>(note how the bar symbol '|' is the sum operator)</i></span><br />
<br />
Mathematicians like to overload symbols and words. We all subconsciously respect that '<b><span style="color: #38761d;">+</span></b>' means something different when it is used on two numbers versus when it used with two strings versus when it used on two vectors. Why do we call this new operator a 'sum' and its result a 'sum type' then? Well I could answer that, or let you figure that out on your own. The important thing to do is to let go of concrete biases about what a 'sum' is but also realize that there must be a reason that we analogize with the pre-existing operator 'sum' from a different algebra. (Hint: think about the algebra of boolean values and what AND and OR mean. Now think about 0s and 1s.) <br />
<br />
What programming languages choose to expose as mechanisms for operating on types tells us a lot about what the language wants us to respect. Some languages are concerned about the mathematical tractability of what can be expressed or inferred about these types from <b>limiting</b> the operators. When we have this tractability, we can write <a href="http://en.wikipedia.org/wiki/Hindley%E2%80%93Milner">algorithms</a> in our type checker that do wondrous things for us. Some languages are not so concerned about what can be proven mathematically about these types, but want us to have notions that make sense to us. These languages often require us to constantly remind ourselves what types we have by annotating the values when they are used as parameters or return values. Some languages ignore types. <br />
<br />
It is important to realize that an algebra of data types exists in all of these languages. But what kind of algebra is a different question. <br />
<br />
Thus when you hear about algebraic data types, you will most often be referring to a Haskell-like system -- something called an <a href="http://en.wikipedia.org/wiki/Initial_algebra">initial algebra</a>. Definitely, the <a href="http://en.wikipedia.org/wiki/Algebraic_data_type">wikipedia</a> article presumes this. The wikipedia article is a hodge-podge mess.<br />
<br />
In this case we mean algebraic data type to mean one which has <b>both</b> a '<b style="color: #38761d;">product</b>' and '<b><span style="color: #38761d;">sum</span></b>' operator, which is closed, which has a unit-type, which has associative properties, a fixed-point operator, and a bunch of other stuff which someone <a href="http://blog.lab49.com/archives/3011">else's blog post</a> and a <a href="http://stackoverflow.com/questions/16770/haskells-algebraic-data-types">stackoverflow</a> discussion do a better job explaining. Please <a href="http://blog.lab49.com/archives/3011">do read this other post</a> as it is both more rigorous and continues exploring how the analogy (morphism) of arithmetic algebra with type algebra can give us beautiful things like derivative types. My gosh. It's cool.<br />
<br />
The purpose of this posting is merely an introduction, and, as such, will not be rigorous. I have erred on the side of hand-wavey explanation and can even now see things which are technically suspect. But whatever. Study. Figure it out.</div>
Daniel Eklundhttp://www.blogger.com/profile/11570452431861145598noreply@blogger.com33tag:blogger.com,1999:blog-5887985082054147381.post-14564263039690674402011-08-26T08:06:00.000-07:002011-08-26T08:06:05.832-07:00Diving Deep into the Erlang Ecosystem<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWdZERvEHugRFoJJkxBGnJo1HUegYyiG2zaKj-a6yuin-UP26jVY8cnL5ux2yupYSZ9qV12DJXnkAaGQYcbaLlVSOmG56BkUQQnusXGcPWJidc1kc_cL4TeE1MbNtNj5zLCpAQRJSKoOk/s1600/ErlangMindMap.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWdZERvEHugRFoJJkxBGnJo1HUegYyiG2zaKj-a6yuin-UP26jVY8cnL5ux2yupYSZ9qV12DJXnkAaGQYcbaLlVSOmG56BkUQQnusXGcPWJidc1kc_cL4TeE1MbNtNj5zLCpAQRJSKoOk/s640/ErlangMindMap.png" width="488" /></a></div><br />
Daniel Eklundhttp://www.blogger.com/profile/11570452431861145598noreply@blogger.com3tag:blogger.com,1999:blog-5887985082054147381.post-37833060550599953372011-01-18T10:42:00.000-08:002011-01-18T10:44:40.958-08:00Simple Boggle Solver in HaskellYears ago, I used the idea of solving a boggle board as my way of learning common lisp. This past week, I have done the same with Haskell.<br />
<br />
<a href="https://github.com/reverendpaco/Haskell-Dictionary-Trie">https://github.com/reverendpaco/Haskell-Dictionary-Trie</a><br />
<br />
A Simple Boggle Solver<br />
<ul><li>cabal install binary</li>
<li>git clone git@github.com:reverendpaco/Haskell-Dictionary-Trie.git</li>
<li>cd ./Haskell-Dictionary-Trie/src</li>
<li>ghc --make WordServer.hs</li>
<li>./WordServer (listens on 9900)</li>
<li>(on another terminal) telnet localhost 9900</li>
<li>type in letters "SERSPATGLINESERSOO" which will be wrapped into the closest approximate square, for example</li>
</ul><pre>SERSPATGLINESERSOO ->
S E R S
P A T G
L I N E
S E R S
O O</pre><ul><li>hit return</li>
<li>watch as the words in this board are returned to you</li>
</ul>Daniel Eklundhttp://www.blogger.com/profile/11570452431861145598noreply@blogger.com6tag:blogger.com,1999:blog-5887985082054147381.post-2932467674361245702010-12-30T08:21:00.000-08:002010-12-30T08:21:45.666-08:00I think I finally grok it<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjW-CEpznuF2Lkp-hRXIC4uQVB2WtG5OWQOB1dumUSQiKGdNL6BPQfPGMr0bp0CugrzuJ8g7vm-2gE3JTATyaKCgLljBtfuvfDg4C_m7I_gMDciFfbqD6jX_et8-MYpTNfs0At_Cb05WFM/s1600/morphismfamily.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="596" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjW-CEpznuF2Lkp-hRXIC4uQVB2WtG5OWQOB1dumUSQiKGdNL6BPQfPGMr0bp0CugrzuJ8g7vm-2gE3JTATyaKCgLljBtfuvfDg4C_m7I_gMDciFfbqD6jX_et8-MYpTNfs0At_Cb05WFM/s640/morphismfamily.png" width="640" /></a></div><div class="separator" style="clear: both; text-align: center;"><br />
</div><div class="separator" style="clear: both; text-align: left;">Sometime it's so much easier to make your own diagrams. The arrow represents the "is a" relationship with labels to specify.</div><div class="separator" style="clear: both; text-align: left;"><br />
</div><div class="separator" style="clear: both; text-align: left;">I left the crazy cousin "homeo" out of the family tree, cause he's just too topological.</div>Daniel Eklundhttp://www.blogger.com/profile/11570452431861145598noreply@blogger.com3tag:blogger.com,1999:blog-5887985082054147381.post-24190110384813344552010-07-26T10:30:00.000-07:002010-07-26T15:25:23.032-07:00Clojure Implementation of Dictionary Trie<span class="Apple-style-span" style="font-size: x-large;">I finally have Clojure</span><br />
This is the <a href="http://gist.github.com/490814">Clojure implementation of the Dictionary Trie</a> previously implemented in Haskell. You can copy and paste it into a repl, or load it into a file. A basic usage goes:<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="background-color: #cfe2f3;">(generate-trie </span></span></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="background-color: #cfe2f3;"> ["rebate" "reborn" "realize" "real"])</span></span></span><br />
or for fun<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="background-color: #cfe2f3;">(println (generate-pretty</span></span></span><br />
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="background-color: #cfe2f3;"> (generate-trie </span></span></span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="background-color: #cfe2f3;"> ["rebate" "reborn" "realize" "real" "relied"])))</span></span></span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">to get the Dot</div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: sans-serif; font-size: 12px;"></span></div><pre id="source" style="padding-bottom: 8px; padding-left: 16px; padding-right: 16px; padding-top: 8px;"><span class="Apple-style-span" style="background-color: #ffe599;">"_"[label="∅"]
"_"->"r_"[label=" r"]
"r_"[label="re"]
"r_"->"lr_"[label=" l"]
"r_"->"ar_"[label=" a"]
"r_"->"br_"[label=" b"]
"lr_"[label="lied(*)"]
"ar_"[label="al(*)"]
"ar_"->"iar_"[label=" i"]
"iar_"[label="ize(*)"]
"br_"[label="b"]
"br_"->"obr_"[label=" o"]
"br_"->"abr_"[label=" a"]
"obr_"[label="orn(*)"]
"abr_"[label="ate(*)"]</span></pre><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">and copy those results <a href="http://graph.gafol.net/">here </a>to see the image:</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqRiNkvSojMMgUKGqdzQnnABBfPVX4_HxgwVEQxcK69vnvA9nNY7xHP_FnmyO6L1kjs6SY-6S8VKeYUgVJyjE_H1fW1esjJpJHsAchtNv4i4RkHIpHLuM3FozQyzxjRV5kflXAWls72Rw/s1600/clojureTrie.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqRiNkvSojMMgUKGqdzQnnABBfPVX4_HxgwVEQxcK69vnvA9nNY7xHP_FnmyO6L1kjs6SY-6S8VKeYUgVJyjE_H1fW1esjJpJHsAchtNv4i4RkHIpHLuM3FozQyzxjRV5kflXAWls72Rw/s320/clojureTrie.png" /></a></div><br />
<span class="Apple-style-span" style="font-size: x-large;">Discussion</span><br />
<ol><li><b>defmulti</b> can be used to mimic Haskell's constructor/structure based dispatch</li>
<li>Haskell has trained me to favor "higher-order" over macros. Higher-orders like <b>partial</b>, <b>comp</b>, etc. can do a lot of what you need.</li>
<li>Looking forward to Clojure 1.2's <b>defrecord</b> capabilities as I have become a fan of abstract/algebraic data types</li>
<li>Clojure has gotten a lot of mileage with the 80% of data structures we all use (in their immutable form): vector, hashmap, list. The upcoming <b>defprotocol</b> will allow the hard work for that last 20%</li>
<li>as much as I love Haskell, you can't beat a quick (println) for debug.</li>
<li>It was easier to mentally map the Haskell code to the Clojure code rather than re-reason the problem through. </li>
<li>I like <b>recur</b> over hoping for a compiler TCO. </li>
<li>Types do help debug. I found a lot of quirky edge-cases in testing Clojure run-time. Hard to put my finger on individual anecdotal evidence... more of a "feel thing".</li>
<li>Just as in the Haskell implementation, I don't think there's any great advantage to be gained in using a zipper data structure. I will try though, mainly to familiarize myself with the zipper.</li>
</ol><div><span class="Apple-style-span" style="font-size: x-large;">API Methods</span></div><div><ul><li>generate-trie</li>
<li>query-node</li>
<li>add-word-to-trie</li>
</ul></div>Daniel Eklundhttp://www.blogger.com/profile/11570452431861145598noreply@blogger.com2tag:blogger.com,1999:blog-5887985082054147381.post-26365602422298426302010-06-21T09:04:00.000-07:002010-07-26T15:43:25.761-07:00Implementing the Dictionary Trie: Insertion and Search Algorithms: HaskellIn a <a href="http://merrigrove.blogspot.com/2010/06/implementing-dictionary-trie-refining.html">previous post</a>, I settled on a data structure for the Dictionary Trie. In this post, I present the Haskell code I used to implement the search and insertion algorithms. To understand what motivates the dictionary trie, make sure to read the following posts (including an informal discussion on the algorithms used in the insertion):<br />
<ol><li><a href="http://merrigrove.blogspot.com/2010/05/dictionary-trie.html">What is a dictionary trie</a>?</li>
<li><a href="http://merrigrove.blogspot.com/2010/05/dictionary-trie-manipulation-insertion.html">How we would go about creating a dictionary trie from inserts</a>.</li>
<li><a href="http://merrigrove.blogspot.com/2010/06/implementing-dictionary-trie-refining.html">What is a basic Haskell data structure for a dictionary trie</a>.</li>
</ol><span class="Apple-style-span" style="font-size: x-large;">The Data Structure</span><br />
<script src="http://gist.github.com/491354.js"> </script><br />
Highlights:<br />
<ol><li>Algebraic</li>
<li>Parametrically Polymorphic</li>
</ol><div>As I had mentioned on my previous post about refining this data structure, the TrieNode is the essential component to the tree. I have used an enumeration type (NodeType) to indicate the node type rather than a set of booleans, in the belief that Haskell's pattern matching could make the code that much easier to understand.</div><span class="Apple-style-span" style="font-size: x-large;">The Insertion Algorithm</span><br />
The insertion algorithm exposes one public API function:<br />
<br />
<ul><li>addWordToTrie</li>
</ul><div>which takes a string (actually an array of the parametrically typed value) and the dictionary trie. </div><div>There are several utility functions that I do not export out of the module (none of the module code at the top is shown here). These utility functions use internal utility data structures that are concerned with the type of match that two strings have towards each other -- the source string is always compared against a target string and there's always one of five different matches: </div><div><ul><li>exact match</li>
<li>target is prefix of source</li>
<li>source is prefix of target</li>
<li>shared prefix</li>
<li>nothing in common</li>
</ul></div><div>In the below code, I have inserted a graphic image as a visual description of the algorithm for the four different cases that you would actually run into during the insertion. The utility function <b>insertNode</b> is a direct mapping (and formal encoding) of the informal scenarios discussed in this previus <a href="http://merrigrove.blogspot.com/2010/05/dictionary-trie-manipulation-insertion.html">post</a>. The correspondence to the informally developed algorithms and the actual functions was quite pleasing, and was made possible by the pattern-matching capabilities of the language</div><br />
<script src="http://gist.github.com/491358.js"></script><br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_j-73oUAsYJhc_TJBQazT_mi088Frt-cv07w2kHzdeFe10dbq-qmQV4EFD5B3Cw7SOXOHsGW9hBKBhbkOkkyYY6Jjljcz5wKyd0aGn2pgp_kSjey3BdY2U2er3Bir389eArsMfFytT_g/" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="110" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_j-73oUAsYJhc_TJBQazT_mi088Frt-cv07w2kHzdeFe10dbq-qmQV4EFD5B3Cw7SOXOHsGW9hBKBhbkOkkyYY6Jjljcz5wKyd0aGn2pgp_kSjey3BdY2U2er3Bir389eArsMfFytT_g/" width="200" /></a> <b>Exact Match</b><br />
<script src="http://gist.github.com/491361.js"> </script><br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgWUqYloQ0FUOFC7a5XAZBhEmi3KIvgfjYLxxVZPGKC5H7pv-r_OFDzm_nEwUBWi-MpLTrjRFM87bgVKQujoWG8MEylRjpOYjbB_J5M2F_2Ms1jaS73Nc4jVWdYMzoVfXBCJ7InADrd_r4/" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span class="Apple-style-span" style="-webkit-text-decorations-in-effect: none; color: black;"></span></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEN6lMXf7VFWcCi8jlM_GC0vmgfFPC5ZfVrj1SxDvjgQxAol1_I1IM0RDRM_xbmusb7tOO0HsEJ28xobNC5N3eE0_zImCxlFC7mYzBIhuBqcWXpIpvuZYl053B9FU1H9mN9zF2iUfP2hI/" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEN6lMXf7VFWcCi8jlM_GC0vmgfFPC5ZfVrj1SxDvjgQxAol1_I1IM0RDRM_xbmusb7tOO0HsEJ28xobNC5N3eE0_zImCxlFC7mYzBIhuBqcWXpIpvuZYl053B9FU1H9mN9zF2iUfP2hI/" width="104" /></a><b>Target Is Smaller Than Source</b><br />
<script src="http://gist.github.com/491362.js"> </script><br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgWUqYloQ0FUOFC7a5XAZBhEmi3KIvgfjYLxxVZPGKC5H7pv-r_OFDzm_nEwUBWi-MpLTrjRFM87bgVKQujoWG8MEylRjpOYjbB_J5M2F_2Ms1jaS73Nc4jVWdYMzoVfXBCJ7InADrd_r4/" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgWUqYloQ0FUOFC7a5XAZBhEmi3KIvgfjYLxxVZPGKC5H7pv-r_OFDzm_nEwUBWi-MpLTrjRFM87bgVKQujoWG8MEylRjpOYjbB_J5M2F_2Ms1jaS73Nc4jVWdYMzoVfXBCJ7InADrd_r4/" width="62" /></a><b>Source Is Smaller Than Target</b><br />
<script src="http://gist.github.com/491368.js"> </script><br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEioiYNZHp3cAY_g2jOWoFPQsGp4Gm3gOi2tWWGt1C8uro-bvYagLtLJ-5d_E6mKsckKaGUi2mNtfu7i1gI1e6nZdPFcJXHoiVp0uCFLJYmGwxY29CYh-PwGiegRRBGil-ShGtlJ5D8WUtA/" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEioiYNZHp3cAY_g2jOWoFPQsGp4Gm3gOi2tWWGt1C8uro-bvYagLtLJ-5d_E6mKsckKaGUi2mNtfu7i1gI1e6nZdPFcJXHoiVp0uCFLJYmGwxY29CYh-PwGiegRRBGil-ShGtlJ5D8WUtA/" width="142" /></a><b>Source and Target Share Same Prefix</b><br />
<script src="http://gist.github.com/491373.js"> </script><br />
<span class="Apple-style-span" style="font-size: x-large;">The Querying Algorithms</span><span class="Apple-style-span" style="font-size: x-small;"></span><br />
<script src="http://gist.github.com/491376.js"> </script><br />
<span class="Apple-style-span" style="font-size: x-large;">Discussion</span><br />
These are the subjects I want to touch on in my creation of the above code:<br />
<ol><li>Keeping things functional with record syntax</li>
<li>Just Maybe stuff</li>
<li>Potentially too redundant in spots </li>
<li>my first use of home-grown operators</li>
<li>does pattern matching on record syntax get me that much?</li>
<li>no magic</li>
<li>functionally passing back self</li>
</ol><div><span class="Apple-style-span" style="font-size: x-large;">Keeping Thing Functional with Record Syntax</span></div><div>In order to make a 'change' in the functional world, what you have to do is actually make a new value. For values that are aggregates of others, like our data structure, you need to make sure that the old values for the other stuff remains the same, so essentially you 'copy' the other values. I was extremely pleased to see that with the record syntax, you could code such that a 'modification' to one field in the record affected the overall change leaving the rest of the fields the same as previous. This required the use of Haskell's pattern-matching to name the overall structure/node. </div><div><span class="Apple-style-span" style="font-size: x-large;">Just Maybe Stuff</span></div><div>I know that I am not properly understanding the way for dealing with Maybe values. I feel that there has to be a better way for extracting the underlying value than the where-clause utility functions I use. Does this reflect an underlying misapprehension of the pholosophy of Maybe values? Maybe.</div><div><span class="Apple-style-span" style="font-size: x-large;">Potentially Too Redundant</span></div><div>I had two internal data structures to represent the matching of the source and target strings. One was purely a description of the matching that occurred (the type MatchType) and consisted of five nullary constructor data types. The other redundantly captured the state of the match but also encapsulated the prefix-suffix pair that resulted. I have a strong suspicion I could have done this with one pass. I will revisit.</div><div><span class="Apple-style-span" style="font-size: x-large;">First Use of Homegrown Operators</span></div><div>This is an issue of terseness and psychology. Once I had the algorithms developed and everything working according to plan, I refactored some prefix-oriented functions into operators. I turned:<br />
<br />
<ul><li>insertNewChildNode into *-></li>
<li>takeTogetherWhile into =><=</li>
</ul><div>In the end, this reduced the <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">camelCaseClutterAndExcessinveJavaLikeNaming</span><span class="Apple-style-span" style="border-collapse: collapse; font-size: small;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">™</span></span>, but as these functions were internal there probably was no real gain.</div></div><div><span class="Apple-style-span" style="font-size: x-large;">Pattern Matching Cost Benefit Analysis</span></div><div>The record syntax (as a data constructor) permits pattern matching on the left-hand side of function definitions or on case expressions. There were several times I wondered whether the use of the pattern-matching was really worth it. While it did lower the terseness on the right-hand side (no need to fetch the data out of the record with 'accessor' functions), it did raise the level of noise on the left-hand side.</div><div><span class="Apple-style-span" style="font-size: x-large;">No Magic</span></div><div>As I've been diving deep into Haskell, I've been itching to try out the magic that provides the allure to this language. I have been attracted to function composition and currying. I've desired to chain together functions in a <a href="http://www.haskell.org/haskellwiki/Pointfree">point-free</a> style or employ functors or applicative style idioms. I found that my code showed little of this magic. The reasons could be (and/or):<br />
<br />
<ul><li>I am new to a different paradigm and still thinking idiomatically imperative</li>
<li>I am dealing with programming in the large which is really just massaging, manipulation, and transformation of record syntaxes.</li>
</ul><div><span class="Apple-style-span" style="font-size: x-large;">Functionally Passing Back Self</span></div><div>The utility function <b>insertNode</b> is the heart of the insertion algorithm and is tree-recursive. As mentioned above, the functional nature of Haskell requires that in tree updates, we are actually doing tree 'copies' (the underlying implementation may share state to increase performance, but as far as the language semantics go, the values should all be new and copied). To accomplish this, functions which are tree recursive have to always do the same thing: </div><div><ul><li>they need to pass back the node they are currently operating on as the result of the function</li>
<li>and assume that the invocations on themselves (the recursive call) are 'modifications' to the child nodes</li>
</ul><div>This grew tedious after a while, and felt like unnecessary boilerplate. It was stuff like this that drew the development for higher-order functions like fold and map, and I considered exploring the benefit of casting the Trie as a <a href="http://learnyouahaskell.com/functors-applicative-functors-and-monoids#functors-redux">functor</a>, or of implementing a <a href="http://en.wikipedia.org/wiki/Zipper_(data_structure)">zipper data structure</a>. </div></div><div>I will revisit this.</div></div>Daniel Eklundhttp://www.blogger.com/profile/11570452431861145598noreply@blogger.com1tag:blogger.com,1999:blog-5887985082054147381.post-65712583281836682402010-06-10T11:54:00.000-07:002010-06-10T16:58:15.126-07:00Another Digression: Tries for Capturing Pronunciation<object height="385" width="480"><param name="movie" value="http://www.youtube.com/v/uVmU3iANbgk&hl=en_US&fs=1&"></param><param name="allowFullScreen" value="true"></param><param name="allowscriptaccess" value="always"></param><embed src="http://www.youtube.com/v/uVmU3iANbgk&hl=en_US&fs=1&" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="480" height="385"></embed></object><br />
<br />
<br />
Those linguist guys came up with an alphabet of symbols to try to capture the totality of human sounds as a sequence of characters. They called this alphabet <a href="http://en.wikipedia.org/wiki/International_Phonetic_Alphabet">IPA</a>. Here is the IPA for the word '<a href="http://www.merriam-webster.com/dictionary/antediluvian">antediluvian</a>'<br />
<br />
<span class="Apple-style-span" style="font-family: 'Times New Roman', 'Times Serif', serif; font-size: 16px; line-height: 20px;"> <span class="pr" style="font-family: 'Times New Roman', 'Times Serif', serif; font-size: inherit; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">\<span class="unicode" style="font-family: 'lucida sans unicode'; font-size: 0.9em; font-style: normal; font-weight: normal; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;"><span class="Apple-style-span" style="background-color: #cfe2f3;">ˌ</span></span><span class="Apple-style-span" style="background-color: #cfe2f3;">an-ti-də-</span><span class="unicode" style="font-family: 'lucida sans unicode'; font-size: 0.9em; font-style: normal; font-weight: normal; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;"><span class="Apple-style-span" style="background-color: #cfe2f3;">ˈ</span></span><span class="Apple-style-span" style="background-color: #cfe2f3;">lü-vē-ən, -(</span><span class="unicode" style="font-family: 'lucida sans unicode'; font-size: 0.9em; font-style: normal; font-weight: normal; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;"><span class="Apple-style-span" style="background-color: #cfe2f3;">ˌ</span></span><span class="Apple-style-span" style="background-color: #cfe2f3;">)dī-</span>\</span></span><br />
<br />
I just had an idea that maybe we can use words of IPA pronunciation in a trie and it might give us some useful stuff. For instance, 'does one word sound like another?', both at the front and at the end -- 'friendship' and 'phrenology' sound alike near the front (at least, according to me), and 'cone' and 'loan' have a similar ending sound (i.e. they rhyme -- anytime).<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://en.wikipedia.org/wiki/Phrenology"><img border="0" height="195" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgamDXOSBh1wGya6aBdMqqldpTuy3cqozPJIAZV68IeSIHMJh8pKfcO5Ho_eiY6OYOkv9-CyC6SQp9Zu1ZfL1MZQmALZt6Elq0p6TwW7GsRIV3YfwE58l1b8rFlQk_HFf6RfVie7q3_FfQ/s200/phrenology.png" width="200" /></a></div><br />
I am going to make a lot of assumptions, but as I am not attempting to be a linguist (not today), I can get away with doing some cool stuff while waving my hand with the pronouncement tha<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">t this is just an approximation</span>. The title of this blog is <b><i>probably done before</i></b>. I am sure that someone has done this before, but I am not going to google it -- I want to have fun exploring this on my own.<br />
<br />
<span class="Apple-style-span" style="font-size: x-large;">Plan</span><br />
<ol><li>Find a dictionary of IPA words. (Strictly speaking I need to find two parallel dictionaries: an IPA dictionary along with the script word: a mapping from words of an IPA language (<span class="Apple-style-span" style="font-family: 'Times New Roman', 'Times Serif', serif; font-size: 16px; line-height: 20px;"><span class="unicode" style="font-family: 'lucida sans unicode'; font-size: 0.9em; font-style: normal; font-weight: normal; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;"><span class="Apple-style-span" style="background-color: #cfe2f3;">ˌ</span></span></span><span class="Apple-style-span" style="font-family: 'Times New Roman', 'Times Serif', serif; font-size: 16px; line-height: 20px;"><span class="Apple-style-span" style="background-color: #cfe2f3;">an-ti-də-</span></span><span class="Apple-style-span" style="font-family: 'Times New Roman', 'Times Serif', serif; font-size: 16px; line-height: 20px;"><span class="unicode" style="font-family: 'lucida sans unicode'; font-size: 0.9em; font-style: normal; font-weight: normal; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;"><span class="Apple-style-span" style="background-color: #cfe2f3;">ˈ</span></span></span><span class="Apple-style-span" style="font-family: 'Times New Roman', 'Times Serif', serif; font-size: 16px; line-height: 20px;"><span class="Apple-style-span" style="background-color: #cfe2f3;">lü-vē-ən, -(</span></span><span class="Apple-style-span" style="font-family: 'Times New Roman', 'Times Serif', serif; font-size: 16px; line-height: 20px;"><span class="unicode" style="font-family: 'lucida sans unicode'; font-size: 0.9em; font-style: normal; font-weight: normal; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;"><span class="Apple-style-span" style="background-color: #cfe2f3;">ˌ</span></span></span><span class="Apple-style-span" style="font-family: 'Times New Roman', 'Times Serif', serif; font-size: 16px; line-height: 20px;"><span class="Apple-style-span" style="background-color: #cfe2f3;">)dī-</span></span>) to words of a written language (<span class="Apple-style-span" style="color: blue;">antediluvian)</span>.</li>
<li>Insert these IPA words into two tries: one with the constituent IPA characters from left to right, and one right to left. The first tree should tell me 'friendship' and 'phrenology' share a pronunciation prefix -- even though they don't share the same written-word prefix. The second trie should tell me that 'loan' and 'cone' rhyme (despite having different suffixes). Here is where my hand-waving should come in handy.</li>
</ol>This should be fun, and I am sure I am going to run into a LOT of problems.<br />
<ol></ol>Daniel Eklundhttp://www.blogger.com/profile/11570452431861145598noreply@blogger.com1tag:blogger.com,1999:blog-5887985082054147381.post-52024710900480692352010-06-10T10:15:00.000-07:002010-06-17T13:49:49.181-07:00Dictionary Trie Digression: Auto-Generating Those Pretty Graphs<span class="Apple-style-span" style="font-size: x-large;">I am Lazy (with a capital L)</span><br />
Some say laziness is <i>a</i> hallmark of a programmer.<br />
<br />
Did you think I hand drew this tree with Visio or Office Draw ?<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhex5ncdx8KWYo_-vonNkU08s-ULU2NAPI_ZYJ8jcuTNVrtJO2V-OVW3G_SF-NKKIh3ipBSwECMly9KBEtWKaI9rgHcl5hGyioTO05My2GZ7oCDhJR2laAtw8H6W122rlLm7INHtOMJw2Y/s1600/lazy.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="312" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhex5ncdx8KWYo_-vonNkU08s-ULU2NAPI_ZYJ8jcuTNVrtJO2V-OVW3G_SF-NKKIh3ipBSwECMly9KBEtWKaI9rgHcl5hGyioTO05My2GZ7oCDhJR2laAtw8H6W122rlLm7INHtOMJw2Y/s400/lazy.png" width="400" /></a></div> Well, no.<br />
<br />
As I started to post a couple of weeks ago, I realized that I needed to show tree structures early and often. Originally, I was doing something silly like this:<br />
<pre>---"r"
|
|
---"ea"
| |
| |
| ---"l"
| |
| |
| ---"r"
|
|
---"ock"
</pre>but then blogspot mangled it, and I realized it wasn't as pretty as a graphic like the one above. I figured to myself that there MUST be a simple domain specific language for rendering simple graphs/trees based on a text. Ideally, something like this:<br />
<pre><span class="Apple-style-span" style="background-color: #cccccc;">a -> b -> c
b -> d</span>
</pre>should generate something like this:<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhb35JQDG_u4SgMXXAWoGtZpnhHS-OiE-EkUimcE_7ddLL25CMPSHW0nPCVfZlh9OF6bhWghl3RPrt93UvaWpDyg_RDwODy0y1bc5MGIRpHljnqD3tvUBkylifo6XK9drGlUVQgB5_cnuU/s1600/simple.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhb35JQDG_u4SgMXXAWoGtZpnhHS-OiE-EkUimcE_7ddLL25CMPSHW0nPCVfZlh9OF6bhWghl3RPrt93UvaWpDyg_RDwODy0y1bc5MGIRpHljnqD3tvUBkylifo6XK9drGlUVQgB5_cnuU/s200/simple.png" width="142" /></a></div>Well, guess what? This is <b>exactly</b> a language, and the above snippet was used to generate this image. The language is called <a href="http://en.wikipedia.org/wiki/DOT_language">Dot</a> and is declarative and simple. A package called <a href="http://www.graphviz.org/">Graphviz</a> provides a nice set of free tools for generating images from a Dot file.<br />
<br />
<span class="Apple-style-span" style="font-size: x-large;">I Continued Being Lazy</span><br />
I did not want to install these free tools. I figured, rightly so, that some<i>where</i>, some<i>body</i> had probably wrapped the Graphviz tool-set with HTTP. O brave new world of web-services, ajax and SOA! A little googling and I found my savior: <a href="http://graph.gafol.net/">http://graph.gafol.net/</a>. And so now, I had everything I needed. Time to stop being lazy.<br />
<br />
<span class="Apple-style-span" style="font-size: x-large;">Laziness Tabled: Utility Functions for Rendering Tries to Dot</span><br />
So back in a previous post I had come up with my final (for now) data structure for a Trie (as an inductively generated set of nodes).<br />
<br />
Here it is:<br />
<pre><code>
<span class="Apple-style-span" style="background-color: #eeeeee;">data NodeType = Top|Word|NonWord
deriving (Show,Eq)
data TrieNode a = TrieNode { wordFragment :: [a],
children :: (Map.Map a (TrieNode a)),
nodeType:: NodeType } deriving (Show,Eq)</span>
</code>
</pre><br />
With only this structure in mind, it is possible to write the code to generate the dot<br />
language. Here is the code:<br />
<br />
<pre><code>
<span class="Apple-style-span" style="background-color: #f3f3f3;">data Dot = DotPath {fid::String, tid::String} |
DotNode{dotId::String,label::String}
instance Show Dot where
show DotPath{fid=from,tid=to@(x:_)} = "\"" ++ from ++ "\""
++ "->"
++ "\"" ++ to ++ "\""
++ "[label=\" " ++ ( x : "\"]")
show DotNode{dotId=nodeId,label=outputLabel} = "\"" ++ nodeId ++ "\""
++ "[label= \"" ++ outputLabel ++ "\"]"
generateDotLabel nodeDatum = case (nodeType nodeDatum) of
Top -> "∅"
Word -> wf ++ "(*)"
NonWord -> wf
where wf = (wordFragment nodeDatum)
generateDot :: TrieNode Char -> String -> [Dot]
generateDot nodeDatum path = myDotNode : myPaths ++ myChildrensDots
where myDotNode = DotNode{dotId= path,label=(generateDotLabel nodeDatum)}
myPaths = map generatePathNode $ (Map.keys (children nodeDatum))
myChildrensDots = foldr (++) [] $</span></code></pre><pre><code><span class="Apple-style-span" style="background-color: #f3f3f3;"> map generateChildrenDots $ (Map.assocs (children nodeDatum))
generatePathNode x = DotPath{fid=path,tid=x:path}
generateChildrenDots (k,v) = generateDot v (k:path)
generateDotPretty tree = L.intercalate "\n" $ map show $ generateDot tree "_"</span>
</code>
</pre><br />
When I ran it on the Trie generated by the following words:<br />
<pre>testWords = ["rebate",
"reborn",
"realize",
"real",
"relied",
"rebates",
"reb",
"relief",
"realizes",
"redder",
"red"]
</pre><br />
I get this:<br />
<pre><span class="Apple-style-span" style="background-color: #fff2cc;">"_"[label= "∅"]
"_"->"r_"[label=" r"]
"r_"[label= "re"]
"r_"->"ar_"[label=" a"]
"r_"->"br_"[label=" b"]
"r_"->"dr_"[label=" d"]
"r_"->"lr_"[label=" l"]
"ar_"[label= "al(*)"]
"ar_"->"iar_"[label=" i"]
"iar_"[label= "ize(*)"]
"iar_"->"siar_"[label=" s"]
"siar_"[label= "s(*)"]
"br_"[label= "b(*)"]
"br_"->"abr_"[label=" a"]
"br_"->"obr_"[label=" o"]
"abr_"[label= "ate(*)"]
"abr_"->"sabr_"[label=" s"]
"sabr_"[label= "s(*)"]
"obr_"[label= "orn(*)"]
"dr_"[label= "d(*)"]
"dr_"->"ddr_"[label=" d"]
"ddr_"[label= "der(*)"]
"lr_"[label= "lie"]
"lr_"->"dlr_"[label=" d"]
"lr_"->"flr_"[label=" f"]
"dlr_"[label= "d(*)"]
"flr_"[label= "f(*)"]</span>
</pre><br />
You can copy this into <a href="http://graph.gafol.net/">http://graph.gafol.net/</a> and see that it generates the image at the top of this post.<br />
<br />
<span class="Apple-style-span" style="font-size: x-large;">Some Observations</span><br />
<br />
<ol><li>Dot Id's</li>
<li>instances of type classes</li>
<li>pattern matching</li>
<li>generate dot can't be used on any types (note the type) -- or 'does not play well with arbitrarily contained types'</li>
<li>probably not efficient</li>
<li>use of a wrapper data structure</li>
</ol><div><i><b>Dot Id's</b></i></div><div>I could not use the word fragment itself as the id, otherwise we would introduce nodes with multiple parents (an acyclic graph). The Dot language allows me to use any arbitrary sequence of characters as the id, so long as they are surrounded by quotations. So, I generated the ids from the path down to the node, pushing the character key onto the path as we descended recursively -- initializing the path with a bogus start character of an underscore '_'. You can verify that the id's are in the reverse order of the descending path. The word fragments are used, of course, to render the label of the node.</div><div><br />
</div><div><i><b>Instances of Type Classes</b></i></div><div>This was the first time I overrode show and purposely put my data structure (the TrieNode) into a type class (namely Show). As I have read the literature, type classes are Haskell's unique contribution to the functional community, (aside from assembling everything together in one nice package). Type classes allow for a disciplined framework for <a href="http://en.wikipedia.org/wiki/Type_polymorphism#Ad-hoc_polymorphism_for_early_bound_languages">ad-hoc polymorphism</a>. Another disciplined framework for ad-hoc polymorphism is subclassing in Java and overriding a method name. I might revisit overriding show ( using the (++) concatenation operator), but, for now, this is how I pretty printed the Dot commands.</div><div><br />
</div><div><i><b>Pattern Matching</b></i></div><div>For those following along (not just me), that here is the justification in my earlier post for pattern matching with a contained data type kicks in. The function <span class="Apple-style-span" style="font-family: monospace; white-space: pre;">generateDotLabel </span>uses a case expression to match against the nullary data constructors of NodeType. In essence, this is a nice symbolic switch statement. There are plenty of other pattern matches going on -- I particularly like the deconstruction of a List as (x:xs). You can see this at work in my definition of show for the DotPath. Here is where understanding left from right comes in handy. The (x:_) on the left hand side of the equation is a pattern match, matching the x to the first element of the list and using the underscore to disregard the tail. On the right hand side of the equation, as I am assembling the string, we see (x: path). This is <b>not</b> a pattern match. This is an insertion operation. They look alike. They are not. <b>Pay attention to what side of the equation you are on</b>.</div><div><br />
</div><div><b><i>Does Not Play Well with Arbitrary Contained Types</i></b></div><div>In an earlier post I stated the desire to make the TrieNode parametric. I have made it parameteric. See the type parameter 'a' in the data declaration below.</div><div><pre><code>
<span class="Apple-style-span" style="background-color: #eeeeee;">data NodeType = Top|Word|NonWord
deriving (Show,Eq)
data TrieNode a = TrieNode { wordFragment :: [a],
children :: (Map.Map a (TrieNode a)),
nodeType:: NodeType } deriving (Show,Eq)</span>
</code>
</pre></div><div>However when writing the code for generating the Dot, it became clear to me that unless I added type constraints ( Show a => ) to my data declaration, then I could not guarantee that my generating functions could render to Dot. I enforced this by explicitly declaring the type for the main function for rendering.</div><span class="Apple-style-span" style="font-family: monospace;"><span class="Apple-style-span" style="white-space: pre;"><span class="Apple-style-span" style="font-family: 'Times New Roman';"><span class="Apple-style-span" style="white-space: normal;"><br />
</span></span></span></span><br />
<pre><code><span class="Apple-style-span" style="background-color: #eeeeee;">generateDot :: TrieNode Char -> String -> [Dot]
</span></code>
</pre>This is something I plan on revisiting as it is possible to render the 'structure' but auto-generate the labels. Gotta think about that.<br />
<br />
<b><i>Probably Not Efficient</i></b><br />
Yup. I am generating an array of Dot data types from a recursive operation. More efficiently I should have had an accumulator that enters each sub-frame of the call-stack. I am not sure I am inefficient. I just suspect it. The main function is (the where clause is left off):<br />
<br />
<br />
<pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><code><span class="Apple-style-span" style="background-color: #f3f3f3;">generateDot nodeDatum path = myDotNode : myPaths ++ myChildrensDots</span></code></pre><br />
<br />
which just looks like it might not be <a href="http://en.wikipedia.org/wiki/Tail_recursion">TCO</a>, what with the concatenation operations and <span class="Apple-style-span" style="font-family: monospace; white-space: pre;">myChildrensDots </span>representing the results of the recursive calls. On the other hand I am not sure I understand enough of Haskell's laziness to say whether this is really that that bad.<br />
<br />
<b><i>New Wrapper Data Type</i></b><br />
Originally I had written this code as a series of string concatenation operations as I traversed the tree. This grew ugly. I introduced the Dot data structure with its two data constructors DotNode and DotPath to capture everything I need (for my cases) in a single Dot output line. The process of rendering these data structure as strings is left for the overridden show instance and for a series of cool chained list functions:<br />
<br />
<pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><code><span class="Apple-style-span" style="background-color: #f3f3f3;">generateDotPretty tree = L.intercalate "\n" $ map show $ generateDot tree "_"</span></code></pre><br />
Luckily the Dot language is declarative and requires very little sequencing guarantees (i.e. you can put these commands in whatever order).Daniel Eklundhttp://www.blogger.com/profile/11570452431861145598noreply@blogger.com0tag:blogger.com,1999:blog-5887985082054147381.post-52730162038617690502010-06-09T09:43:00.000-07:002010-06-09T09:48:56.514-07:00Implementing the Dictionary Trie: Refining (Redefining) the Data StructureWhen learning a new programming language, sometimes analysis paralysis occurs. Having read through many Haskell tutorials, I have become reacquainted with the novice problem (and the psychological confusion) of how to model. I have learned a new set of features. Now, should I use type classes, parametric polymorphism, sum algebraic data types? (Mind you, I am only just learning the theory behind some of these terms).<br />
<br />
I am reminded of learning OO with Java, and of not knowing when to use interfaces or abstract classes, or when to favor inheritance over composition. Familiarity and experience eventually affords you an artistic eye (and a mental model of the constraints of that language) to see just exactly what features of the language should be used. Obviously (being new to Haskell), I have not yet developed this sense.<br />
<br />
<b>Tangent</b>: <i>I am finding it even more difficult to come up with a mental model in <b>Scala</b>, which combines OO with functional programming. Scala has a lot of tools: traits, absract classes, case classes, abstract type members, generic type parameters, singleton objects. With <b>Clojure, </b>on the other hand, it seems I can only use hashmaps -- (perhaps this is an advantage?) Although defprotocol and deftype might be a response for named models.</i><br />
<br />
<span class="Apple-style-span" style="font-size: x-large;">I Have Cheated</span><br />
So the purpose of this post was to show a refinement of the data structure. I tried to do it through intuition alone (posting simultaneously as I figured things out), but in the end I went ahead and implemented the search and insertion algorithms in order to produce the following refinement.<br />
<br />
Let's review. From the previous post:<br />
<pre><code>
data TrieNode = TrieNode { wordFragment :: String, children :: Map.Map Char TrieNode
isTop :: Bool, isWord :: Bool}
</code></pre>Refinement:<br />
<pre><code>
data NodeType = Top|Word|NonWord
deriving (Show,Eq)
data TrieNode a = TrieNode { wordFragment :: [a],
children :: (Map.Map a (TrieNode a)),
nodeType:: NodeType } deriving (Show,Eq)
</code></pre><br />
<span class="Apple-style-span" style="font-size: x-large;">Balancing Flexibility and Over-Engineering</span><br />
Classic problem. Time, elegance, maintainability, desire to learn -- all can make something too flexible (over-engineered), or too rigid (under-engineered).<br />
<br />
If we are designing a dictionary trie, why not try to make it abstract and reusable? Why not spend the extra effort to generify the data and the functions so that it can be used in multiple scenarios?<br />
<br />
If the above paragraph was too abstract itself, consider the following questions. Should our dictionary trie only be used for prefixes of actual human language words? Or should we use the word <i>word</i>, abstractly, and permit an arbitrary alphabet, from bits to anything else? Could we use our dictionary trie to store <a href="http://en.wikipedia.org/wiki/Longest_prefix_match">IP routing information</a>, for instance? <br />
<br />
Here is where I find Haskell's <a href="http://en.wikipedia.org/wiki/Parametric_polymorphism#Parametric_polymorphism">parametric polymorphism</a> very compelling: I had already written the insertion, and search algorithms with the String-typed data declaration. I allowed Haskell's type inference to take care of most of my functions inasmuch as I did not declare function types. Upon deciding to try to be a bit more abstract, I changed all references to <i>String</i> to <i>[a]</i> and all references of <i>Char</i> to <i>a</i>, and it almost worked immediately -- (<i>almost</i>, because I had some debug functions which invoked the functions with strings and therefore allowed the compiler's inference algorithm to cascade the String assumption).<br />
<br />
Horse before the cart. Let's review the new code<br />
<pre><code>
data NodeType = Top|Word|NonWord
deriving (Show,Eq)
data TrieNode a = TrieNode { wordFragment :: [a],
children :: (Map.Map a (TrieNode a)),
nodeType:: NodeType } deriving (Show,Eq)
</code></pre><br />
So what's new here?<br />
<br />
<ol><li>Generic data types (parametric polymorphism applied to the type constructor) -- i.e. the use of a type parameter 'a' </li>
<li>The use of an enumeration type NodeType (here's where I get to apply my jargon : it's a sum type of nullary data constructors) </li>
</ol><div>The first point, (the use of a type parameter 'a'), has given us the advantage of genericity. The second point, (using a new data type to enumerate states of the node), is mainly for clarity but it also gives us a slightly better ability to <a href="http://en.wikipedia.org/wiki/Pattern_matching">pattern match</a> in our insert and search code.</div><div><br />
</div><div>To prove this last point, we will have to start developing the algorithms we discussed in previous posts for search and insertion.</div>Daniel Eklundhttp://www.blogger.com/profile/11570452431861145598noreply@blogger.com1tag:blogger.com,1999:blog-5887985082054147381.post-7804792075248163542010-06-08T14:15:00.000-07:002010-06-08T16:19:25.413-07:00Implementing the Dictionary Trie: Defining the Data Structures<span class="Apple-style-span" style="font-size: x-large;">Purpose</span><br />
<br />
In this post, we are going to start implementing the dictionary trie. To start this we will design the data structure. Below is a UML-ish representation of a TrieNode<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3splRWrYRL6zfVO-xRJ7ZtIY6zDZVbyzB0Ggte1C1dF7Imr1Hz1e7WlgWQjE59OhOKctDZPqwALB7rqWrOfr8XATzVXLedSUo2b2hAMVEvcMxupOuQHZ2jGcY4LHjHlxQwZ3dcv3tjA8/s1600/trieNodeThird.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="120" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3splRWrYRL6zfVO-xRJ7ZtIY6zDZVbyzB0Ggte1C1dF7Imr1Hz1e7WlgWQjE59OhOKctDZPqwALB7rqWrOfr8XATzVXLedSUo2b2hAMVEvcMxupOuQHZ2jGcY4LHjHlxQwZ3dcv3tjA8/s200/trieNodeThird.png" width="200" /></a></div><br />
Because I want to learn some new (<i>to me</i>) functional languages (<b>Haskell</b>, <b>Clojure</b>, and <b>Scala<span class="Apple-style-span" style="font-weight: normal;">)</span><span class="Apple-style-span" style="font-weight: normal;">, and this blog is more for my benefit than anything else, we will use Haskell in this post. </span></b><br />
<br />
I won't go over the benefit of functional languages here. There are many other posts/articles that do a better job explaining/selling functional languages over other paradigms. <br />
<br />
<span class="Apple-style-span" style="font-size: x-large;">Quick Review of Trie Node</span><br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3splRWrYRL6zfVO-xRJ7ZtIY6zDZVbyzB0Ggte1C1dF7Imr1Hz1e7WlgWQjE59OhOKctDZPqwALB7rqWrOfr8XATzVXLedSUo2b2hAMVEvcMxupOuQHZ2jGcY4LHjHlxQwZ3dcv3tjA8/s1600/trieNodeThird.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="120" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3splRWrYRL6zfVO-xRJ7ZtIY6zDZVbyzB0Ggte1C1dF7Imr1Hz1e7WlgWQjE59OhOKctDZPqwALB7rqWrOfr8XATzVXLedSUo2b2hAMVEvcMxupOuQHZ2jGcY4LHjHlxQwZ3dcv3tjA8/s200/trieNodeThird.png" width="200" /></a><br />
<br />
We came up with this UML'is representation of a single node of our data structure back in the very <a href="http://merrigrove.blogspot.com/2010/05/dictionary-trie.html">first post</a>.<br />
<ol><li>The "wordFragment" stores the substring of the word at that point in the trie. For leaf nodes, these word fragments are guaranteed to be suffixes to a word in the dictionary.</li>
<li>The "children" is a set of pointers to other TrieNodes. It is this aspect that allows this single node to become a tree. Do not read too much into the array notation.</li>
<li>The "isWord" indicator is needed to indicate if a non-leaf tree is a word end (although we will use it in leaf nodes as well). All leaf nodes are guaranteed to be word ends. </li>
<li>The "isTop" indicator is used only for the special degenerate top-level of the trie. Please review the <a href="http://merrigrove.blogspot.com/2010/05/dictionary-trie-manipulation-insertion.html">second post Dictionary Trie Insertion</a> for the explanation of why this trie is needed.</li>
</ol><div><span class="Apple-style-span" style="font-size: x-large;">Haskell Implementation First Draft</span></div><pre class="brush: haskell;"><code>
data TrieNode = TrieNode { wordFragment :: String, children :: [TrieNode]
isTop :: Bool, isWord :: Bool}
</code></pre><br />
This is absolutely sufficient for implementing everything required. We have here a Haskell <a href="http://book.realworldhaskell.org/read/defining-types-streamlining-functions.html#deftypes.record">record</a> syntax which provides us some syntactic sugar and some autogenerated functions (loosely akin to "getters" for people coming from an OO background).<br />
<br />
In order to initialize a top node, we would:<br />
<pre class="brush: haskell;"><code>
topNode = TrieNode { wordFragment ="" , children = [] , isTop = True, isWord = False}
</code></pre><br />
But let's consider some details.<br />
<br />
<span class="Apple-style-span" style="font-size: x-large;">Walking Down the Tree</span><br />
Remember this tree (from <a href="http://merrigrove.blogspot.com/2010/05/dictionary-trie-manipulation-insertion.html">the second post</a>)?<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCBOGMPZG0-6TyhJ_LZjZPH7s1YvYXAiz55-TxUJ_8q_dZ0Wurubl1XCKWNyRzWwQZhRDmDf4W-N8hp9CaXyP8XvOdrvUJxs4dDH9ELHL5iNo9atH1lT8GPgftW9DLIl6Noiextb3JlXc/s1600/allAddedAndCooledWithDegenerate.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="292" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCBOGMPZG0-6TyhJ_LZjZPH7s1YvYXAiz55-TxUJ_8q_dZ0Wurubl1XCKWNyRzWwQZhRDmDf4W-N8hp9CaXyP8XvOdrvUJxs4dDH9ELHL5iNo9atH1lT8GPgftW9DLIl6Noiextb3JlXc/s400/allAddedAndCooledWithDegenerate.png" width="400" /></a></div><br />
We can see the arrows leading down from the prefixes to the end nodes. Walking down on the left hand side, we get: "<b>re</b>" + "<b>al</b>" + "<b>ize</b>" + "<b>s</b>" = "<b>realizes</b>". But there is something missing (in both this diagram and in our Haskell record syntax) in describing how we represent which arrow points to which node.<br />
<br />
We need something more like this:<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgz3JFoRdo51jLKB0ggaWTCiOABRJ4Fbg8_zSGNwG0T682cR1HDR-Kt6N0ke5x-oq023f5hLCj2V3V7PZswRO0H5ZSs5CAwd6eHjARtKamw6z0O-vnVOHV8Qf9SeRS74RcHKmEkV8mmBcY/s1600/trieNodeWithNamedEdges.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="290" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgz3JFoRdo51jLKB0ggaWTCiOABRJ4Fbg8_zSGNwG0T682cR1HDR-Kt6N0ke5x-oq023f5hLCj2V3V7PZswRO0H5ZSs5CAwd6eHjARtKamw6z0O-vnVOHV8Qf9SeRS74RcHKmEkV8mmBcY/s400/trieNodeWithNamedEdges.png" width="400" /></a></div><br />
which shows that we can access our sub-nodes by assigning the first character in the child node's "wordFragment" to the edge. <br />
<br />
So let's change our code a bit, and instead of using a a list to represent the children in our node, we will use a <a href="http://www.haskell.org/ghc/docs/6.10.3/html/libraries/containers/Data-Map.html">Map</a> (with a capital M, to remove ambiguity with the function 'map').<br />
<pre class="brush: haskell;"><code>
import qualified Data.Map as Map
data TrieNode = TrieNode { wordFragment :: String, children :: Map.Map Char TrieNode
isTop :: Bool, isWord :: Bool}
</code></pre><br />
There is nothing wrong with a list. It would be just inelegant to find the appropriate sub-child by searching through the list, when a hash lookup based on the first character would get us better performance.Daniel Eklundhttp://www.blogger.com/profile/11570452431861145598noreply@blogger.com4tag:blogger.com,1999:blog-5887985082054147381.post-45949512179084221782010-05-27T04:54:00.000-07:002010-05-27T06:44:36.642-07:00Dictionary Trie Manipulation (Insertion)<div><span class="Apple-style-span" style="font-size: x-large;">Purpose</span><br />
This post is about an algorithm for creating a trie from a set of words.</div><div><br />
</div><div>To do this, I am proposing instead to come up with an algorithm to add one word to an already existing trie, and to treat the creation of a trie as a series of add operations (with an empty trie as the degenerate case).</div><div><br />
To understand what a dictionary trie <i>is</i> and what its advantages are, see my <a href="http://merrigrove.blogspot.com/2010/05/dictionary-trie.html">previous post (A Dictionary Trie)</a>. Or google <a href="http://lmgtfy.com/?q=trie">it</a>.<br />
<br />
<span class="Apple-style-span" style="font-size: x-large;">Onward</span></div><div>Let us use the trie we've already generated from the <a href="http://merrigrove.blogspot.com/2010/05/dictionary-trie.html">previous post</a>, and consider adding the following new words:</div><div><br />
</div><div><span class="Apple-style-span" style="color: #666666;">real</span></div><div><span class="Apple-style-span" style="color: #666666;">relied</span></div><div><span class="Apple-style-span" style="color: #666666;">rebates</span></div><div><span class="Apple-style-span" style="color: #666666;">reb</span></div><div><br />
</div><div>"<b>reb</b>" is probably not a real word. Just pretend it is and move on.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjc4xGjMe1TjlcAuRcIZ7TOxSwbOjgc3qfHbfWfca55C8a7i7vHvfrVu0iYVle1UQCdHcfKYF-koEGDh7_MRj8vUsFSwKGgVhnfEmE7X5uZqz0VsjED0W4dkJxeYqZhHakDkKTAPas_hsw/s1600/thirdTrie.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjc4xGjMe1TjlcAuRcIZ7TOxSwbOjgc3qfHbfWfca55C8a7i7vHvfrVu0iYVle1UQCdHcfKYF-koEGDh7_MRj8vUsFSwKGgVhnfEmE7X5uZqz0VsjED0W4dkJxeYqZhHakDkKTAPas_hsw/s320/thirdTrie.png" /></a></div><br />
</div><div><br />
</div><div>These words were chosen to highlight all the conditions we could run into when we add a word to an existing trie:</div><div><br />
<ol><li>"<b>real</b>" will split a node and extend the branch (with the growth internal).</li>
<li>"<b>relied</b>" is probably the trickiest. It will extend the branch in two directions.</li>
<li>"<b>rebates</b>" will split a node and extend the branch (with the growth at the leaf node and outwards).</li>
<li>"<b>reb</b>" will switch an indicator to TRUE.</li>
</ol><div><span class="Apple-style-span" style="font-size: x-large;">Adding "real" <b> <span class="Apple-style-span" style="font-size: medium;">(insertion string is prefix of target string)</span></b></span></div><div><br />
</div><div>In this scenario "real" will split a node and extend the branch (with the growth internal).</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjTVuDHTX-plS-IIyttXz4oZEbBlW2RkUttgMg_dvQH86CgQ6KKu1nQd8nNS9gswqr9WcX2QtE2VHVErozIMAhJ8avNQ3ZC79GI6dR7je_VYmwiSKQeCtVNtIdOGpx-ogz50tYo4eSNFJg/s1600/addingREAL.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjTVuDHTX-plS-IIyttXz4oZEbBlW2RkUttgMg_dvQH86CgQ6KKu1nQd8nNS9gswqr9WcX2QtE2VHVErozIMAhJ8avNQ3ZC79GI6dR7je_VYmwiSKQeCtVNtIdOGpx-ogz50tYo4eSNFJg/s320/addingREAL.png" /></a></div><div><br />
</div><div class="separator" style="clear: both; text-align: center;"></div><div>The above image shows that we are inserting a new node internal to the branch and extending the branch. BE CAREFUL -- we are not just moving the old node down, we have also <i>modified </i>the characters it contains within (<i>this distinction is important when doing functional language modifications</i>).<br />
<br />
The above diagram shows my convention of coloring the new nodes grey, and dotting the new edges to the tree graph. A dashed node is an old node with a modified word-fragment. </div><div><br />
<span class="Apple-style-span" style="font-size: x-large;">Algorithm</span><br />
We enter the "<b>alize</b>" node with our insertion string as "<b>al</b>" (<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">to understand this, realize that we had started at the top node with our insertion string as "</span><b><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">real</span></b><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">" but as "</span><b><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">re</span></b><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">" was the word fragment contained in the top node, we removed that prefix from our insertion string and submitted the new insertion string "</span><b><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">al</span></b><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">" into the sub-node</span>).<br />
<br />
Anyways, to repeat, we enter the the "<b>alize</b>" node with our insertion string as "<b>al</b>". We see that our <i>insertion string is the prefix of our target string</i>. At this point we create a new node here. We make "<b>al</b>" the word-fragment of our new node, and we turn its word-indicator to ON. We shift the old node down and change its word-fragment by removing the prefix of our "<b>al</b>" insertion string. Hence, "<b>ize</b>" is the shifted node, still pointing to all its previous children.<br />
<br />
Quick Summary: <br />
<pre><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">if ( insertion string is prefix to target string) then "extend branch internally"</span></span>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;"> where "extend branch internally" = use the target string as </span></span>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;"> a new node's word-fragment, </span></span>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;"> and point this new node to the </span></span>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;"> old node with shortened word-fragment. </span></span>
</pre></div><div><div><span class="Apple-style-span" style="font-size: x-large;">Adding "relied" <b><span class="Apple-style-span" style="font-size: medium;">(insertion string and target string share a prefix)</span></b></span></div><div><br />
</div><div>In this situation "relied" will extend the branch in two directions. This is the trickiest of our scenarios.</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEioiYNZHp3cAY_g2jOWoFPQsGp4Gm3gOi2tWWGt1C8uro-bvYagLtLJ-5d_E6mKsckKaGUi2mNtfu7i1gI1e6nZdPFcJXHoiVp0uCFLJYmGwxY29CYh-PwGiegRRBGil-ShGtlJ5D8WUtA/s1600/addingRELIED.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEioiYNZHp3cAY_g2jOWoFPQsGp4Gm3gOi2tWWGt1C8uro-bvYagLtLJ-5d_E6mKsckKaGUi2mNtfu7i1gI1e6nZdPFcJXHoiVp0uCFLJYmGwxY29CYh-PwGiegRRBGil-ShGtlJ5D8WUtA/s320/addingRELIED.png" /></a></div><div><br />
</div><div><span class="Apple-style-span" style="font-size: x-large;">Algorithm </span><br />
<span class="Apple-style-span" style="font-size: x-large;"></span>Here we enter the node with our insertion string as "<b>lied</b>" and our target string as "<b>lief</b>". Unlike the previous algorithm, the insertion string is not a prefix of the target string. Nor is the converse true, "<b>lief</b>" is not a prefix of "<b>lied</b>". But, we do notice that both <i>the insertion and target string share a prefix</i>.<br />
<br />
So now, we create two new nodes, one with the shared prefix ("<b>lie</b>"), the other with the suffix of the insertion string ("<b>d</b>"). We point the shared prefix node to the insertion suffix node and to the old node, making sure to modify the old node to change its word-fragment to the target string suffix ("<b>f</b>").<br />
<br />
<br />
<div><div style="margin: 0px;">Quick Summary:</div><pre style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">if ( insertion string and target string share prefix) then "create fork"</span></span>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;"> where "create fork" = 1) use the target string shared prefix as </span></span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">a new node's word-fragment,</span></span></pre><pre style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;"> 2) use the target strings suffix as a new node's word-fragment, now point #1 to this one</span></span></pre><pre style="margin: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;"> 3) point this new node to the old node with shortened word-fragment. </span></span></pre></div><div><div></div></div><br />
<br />
</div><div><div><span class="Apple-style-span" style="font-size: x-large;">Adding "rebates" <b><span class="Apple-style-span" style="font-size: medium;">(target string is prefix of insertion string)</span></b></span></div><div>In this situation "rebates" will extend the branch (with the growth outwards at the leaf node).</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEN6lMXf7VFWcCi8jlM_GC0vmgfFPC5ZfVrj1SxDvjgQxAol1_I1IM0RDRM_xbmusb7tOO0HsEJ28xobNC5N3eE0_zImCxlFC7mYzBIhuBqcWXpIpvuZYl053B9FU1H9mN9zF2iUfP2hI/s1600/addingREBATES.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEN6lMXf7VFWcCi8jlM_GC0vmgfFPC5ZfVrj1SxDvjgQxAol1_I1IM0RDRM_xbmusb7tOO0HsEJ28xobNC5N3eE0_zImCxlFC7mYzBIhuBqcWXpIpvuZYl053B9FU1H9mN9zF2iUfP2hI/s320/addingREBATES.png" /></a></div><div><br />
<span class="Apple-style-span" style="font-size: x-large;">Algorithm </span><br />
<span class="Apple-style-span" style="font-size: x-large;"></span>Here we enter the node with "<b>bates</b>" as our insertion string and "<b>bate</b>" as our target string. We notice immediately that the target string is a prefix of the insertion string. <br />
<br />
At this point, we take the subtracted suffix of our insertion string "<b>s</b>" and see if there is a child for this. If not, we add it to the graph. We create a new node with a word-fragment of "<b>s</b>" and point the current node to this new node (make sure to turn on its word-indicator).<br />
<br />
</div><div><div><span class="Apple-style-span" style="font-size: x-large;">Adding "reb" <b><span class="Apple-style-span" style="font-size: medium;">(target string is equal to insertion string)</span></b></span></div><div>In this "<b>reb</b>" will switch an indicator to TRUE. This is probably the easiest of our scenarios, since nothing what we do modifies the structure of the tree. All we are doing is recognizing that a node that was previously an intermediate non-word node, has now become a word node.</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_j-73oUAsYJhc_TJBQazT_mi088Frt-cv07w2kHzdeFe10dbq-qmQV4EFD5B3Cw7SOXOHsGW9hBKBhbkOkkyYY6Jjljcz5wKyd0aGn2pgp_kSjey3BdY2U2er3Bir389eArsMfFytT_g/s1600/REBtransform2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_j-73oUAsYJhc_TJBQazT_mi088Frt-cv07w2kHzdeFe10dbq-qmQV4EFD5B3Cw7SOXOHsGW9hBKBhbkOkkyYY6Jjljcz5wKyd0aGn2pgp_kSjey3BdY2U2er3Bir389eArsMfFytT_g/s320/REBtransform2.png" /></a></div><div><br />
<span class="Apple-style-span" style="font-size: x-large;">Algorithm </span><br />
<span class="Apple-style-span" style="font-size: x-large;"></span>Here we enter the node and see that our target string "<b>b</b>" and our insertion string "<b>b</b>" are the same. All we do is switch the word-indicator to ON if it was not already on. If it was already on, then I think that proves that this word has already been inserted. Right?<br />
<br />
</div><div><span class="Apple-style-span" style="font-size: x-large;">Finally (or is it?)</span></div><div>After all these words have been added, and we have:</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigoEP3Ee8-ulhKfDEHuem8-CyPAiSmLkSpOaWHVzNk7Hl2RFHGjPRXTEXF4lpE0mnGS6OmhwdpwdekQLF6QhO-pZfUsDRyHW9pDUISq7C6FaIDj5x2MrrL50ZMxOCfI5XCLNCi3lpm9EY/s1600/allAdded.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="230" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigoEP3Ee8-ulhKfDEHuem8-CyPAiSmLkSpOaWHVzNk7Hl2RFHGjPRXTEXF4lpE0mnGS6OmhwdpwdekQLF6QhO-pZfUsDRyHW9pDUISq7C6FaIDj5x2MrrL50ZMxOCfI5XCLNCi3lpm9EY/s400/allAdded.png" width="400" /></a></div><div>where this tree shows the transformations that took place. </div><div><br />
</div><div>Remember, my arbitrary legend:</div><div><ul><li>a grey node is brand new with a dotted arrow showing a new edge</li>
<li>a dashed node is a modification to a node (whether changing the word fragment or switching an indicator on)</li>
</ul><div>Let's let this thing cool down, and we get our final tree:</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQkMqIoLLQyJZNTUrACnL2ZzLkAuRFiva9AAs4Rjcnfp9UQzpNO9WQzj7Lg09uJJcUylY4cBDpWW6PLSFg6vpXUT7OydrfMmn8FasjtSi_EqWRzAjsg46bqqA_tKeVR9uZfGvej3NG8o4/s1600/allAddedAndCooled.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="230" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQkMqIoLLQyJZNTUrACnL2ZzLkAuRFiva9AAs4Rjcnfp9UQzpNO9WQzj7Lg09uJJcUylY4cBDpWW6PLSFg6vpXUT7OydrfMmn8FasjtSi_EqWRzAjsg46bqqA_tKeVR9uZfGvej3NG8o4/s400/allAddedAndCooled.png" width="400" /></a></div><div><br />
<br />
<span class="Apple-style-span" style="font-size: x-large;">But Are We Done?</span><br />
<br />
At the top of this post we said that creation of a trie should be treated as repeated applications of insertion, and if we want to use these collected algorithms, we will need to do one more thing and that is define a starting trie.<br />
<br />
So let's define the degenerate case.<br />
<br />
<span class="Apple-style-span"><span class="Apple-style-span" style="font-size: x-large;">Degenerate Trie (aka Empty Trie, aka the Top Node)</span></span><br />
<div class="separator" style="clear: both; text-align: center;"><br />
</div><div class="separator" style="clear: both; text-align: left;">Our degenerate trie should be empty. It should be the prefix of all words but not actually have a word-fragment, (or if you prefer, it's word-fragment should be ""). In any case, this is analogous to how regular expressions have beginning of line characters (^) or beginning of word characters (\b) -- anchors.</div><div class="separator" style="clear: both; text-align: left;"><br />
</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7m4uIWYvS9dep2srjFFiFY0z64v7cG_-YeeJS1UOTmXwbh2VuOr6J5PVHeqslpsLF9H9_xM2sf8n09-s5UIQED86YOTO3URAcyoOaz2iQUk2CZq6QHqLW4nJCu3Ovgb6zi0H536GCnQ8/s1600/degenerateTrie.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7m4uIWYvS9dep2srjFFiFY0z64v7cG_-YeeJS1UOTmXwbh2VuOr6J5PVHeqslpsLF9H9_xM2sf8n09-s5UIQED86YOTO3URAcyoOaz2iQUk2CZq6QHqLW4nJCu3Ovgb6zi0H536GCnQ8/s320/degenerateTrie.png" /></a></div><div class="separator" style="clear: both; text-align: center;"><br />
</div><div class="separator" style="clear: both; text-align: left;">The key to using this degenerate case is to make sure that any `isPrefix` operator in any programming language will treat this as a prefix to all words. If not, we can always add an <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">is-top</span> indicator to our data structure and modify our algorithms to check prefixes <b><i>and</i></b> whether the node is the top node. We'll discuss that later.</div><div class="separator" style="clear: both; text-align: left;"><br />
</div><div class="separator" style="clear: both; text-align: left;">Now that we have defined a degenerate trie, we can see what our first insertion will look like:</div><div class="separator" style="clear: both; text-align: left;"><br />
</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjC5lbc_3U6g74cdbJcK1W7m9ueS4yUXprk54TQ55OURno0WQpKiJV4hw3ePhZKtiLHFNZNr0ii0s3rJcQVOZPkxKovr_kCGxVsmLUDlTGaW6WbnFYq7P_0wW1oBAkECmUzcHMbKQsOcTQ/s1600/firstInsertionIntoDegenerateTrie.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjC5lbc_3U6g74cdbJcK1W7m9ueS4yUXprk54TQ55OURno0WQpKiJV4hw3ePhZKtiLHFNZNr0ii0s3rJcQVOZPkxKovr_kCGxVsmLUDlTGaW6WbnFYq7P_0wW1oBAkECmUzcHMbKQsOcTQ/s320/firstInsertionIntoDegenerateTrie.png" /></a></div>Bully!<br />
<br />
Our final tree upon repeated applications of insertion will be:<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjTQz_aM8_MzTvkC3cEyouvBpZAazrVheQG1CdAo8QLVpMHAwWRUUMiDsgI46hEsBOXXbqgfo3i8k4HCOyGPZvbz8rPErKsstYnIc1l4YvERXMhY0Ph8nrc8PoeYE5qQtEgjjscaO-A1wo/s1600/allAddedAndCooledWithDegenerate.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="292" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjTQz_aM8_MzTvkC3cEyouvBpZAazrVheQG1CdAo8QLVpMHAwWRUUMiDsgI46hEsBOXXbqgfo3i8k4HCOyGPZvbz8rPErKsstYnIc1l4YvERXMhY0Ph8nrc8PoeYE5qQtEgjjscaO-A1wo/s400/allAddedAndCooledWithDegenerate.png" width="400" /></a></div><br />
yay!<br />
<br />
<span class="Apple-style-span" style="font-size: x-large;">Next Steps</span><br />
<br />
In our first <a href="http://merrigrove.blogspot.com/2010/05/dictionary-trie.html">post</a> on dictionary tries, we discussed 1) what a trie was, 2) how to retrieve words from it, and 3) what its advantages were.<br />
<br />
This post focused on how to create that trie from insertions. We got two things at once.<br />
<br />
For our next step we could go many different ways. Perhaps we could write delete code? Maybe we need to start implementing this in some programing language? Is there anyway to create an extremely basic algebra for tries (to add and subtract tries from each other)? </div></div></div></div></div></div>Daniel Eklundhttp://www.blogger.com/profile/11570452431861145598noreply@blogger.com26tag:blogger.com,1999:blog-5887985082054147381.post-30385066505156866222010-05-26T08:03:00.000-07:002010-06-09T07:13:15.997-07:00A Dictionary Trie<div style="text-align: left;">If you were given the following words:</div><div><div><br />
</div><div><span class="Apple-style-span" style="font-family: 'courier new';"><span class="Apple-style-span" style="color: #666666;">rebate</span></span></div><div><span class="Apple-style-span" style="font-family: 'courier new';"><span class="Apple-style-span" style="color: #666666;">reborn</span></span></div><div><span class="Apple-style-span" style="font-family: 'courier new';"><span class="Apple-style-span" style="color: #666666;">realize</span></span></div><div><span class="Apple-style-span" style="font-family: 'courier new';"><span class="Apple-style-span" style="color: #666666;">relief</span></span></div><div><span class="Apple-style-span" style="font-family: 'courier new';"><span class="Apple-style-span" style="color: #666666;">realizes</span></span></div><div><span class="Apple-style-span" style="font-family: 'courier new';"><span class="Apple-style-span" style="color: #666666;">redder</span></span></div><div><span class="Apple-style-span" style="font-family: 'courier new';"><span class="Apple-style-span" style="color: #666666;">red</span></span></div><div><br />
</div><div><div>You will immediately notice that they all share the same prefix.</div><div><br />
</div><div>So now if I asked you to create a data structure to store these words (don't worry about <i>why</i> yet), their shared prefix should get you thinking about trees.</div><div><br />
</div><div>So let us create a tree.<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjO7t_s31hbKpvV6PkYSj9DOGiaq9bBqITe0TisTnCjTfIkabyjU5fukzG0BLKe-NLyNZa4cEuUyZ5UmbpVFDZuuZeQsBHFjPlzdZjVnpdi5jZbOlbBTjNv2R6o83qv-EM2bL99YyCSpLc/s1600/firstTrie.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="131" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjO7t_s31hbKpvV6PkYSj9DOGiaq9bBqITe0TisTnCjTfIkabyjU5fukzG0BLKe-NLyNZa4cEuUyZ5UmbpVFDZuuZeQsBHFjPlzdZjVnpdi5jZbOlbBTjNv2R6o83qv-EM2bL99YyCSpLc/s640/firstTrie.png" width="640" /></a></div><br />
</div><div><br />
</div></div><br />
As you note, the algorithm to retrieve the words from this tree is simple:</div><div><br />
</div><div><span class="Apple-style-span" style="font-size: medium;"><span class="Apple-style-span" style="font-family: 'courier new';">Re</span></span><b><span class="Apple-style-span" style="font-size: medium;"><span class="Apple-style-span" style="font-family: 'courier new';">trie</span></span></b><span class="Apple-style-span" style="font-size: medium;"><span class="Apple-style-span" style="font-family: 'courier new';">val Algorithm</span></span></div><div><ol><li><span class="Apple-style-span" style="font-family: 'courier new';">Start at the root and walk down to each end node. </span></li>
<li><span class="Apple-style-span" style="font-family: 'courier new';">As you walk down, accumulate the string fragments stored in the node by concatenation: "</span><b><span class="Apple-style-span" style="font-family: 'courier new';">re</span></b><span class="Apple-style-span" style="font-family: 'courier new';">" + "</span><b><span class="Apple-style-span" style="font-family: 'courier new';">bate</span></b><span class="Apple-style-span" style="font-family: 'courier new';">" yields "</span><b><span class="Apple-style-span" style="font-family: 'courier new';">rebate</span></b><span class="Apple-style-span" style="font-family: 'courier new';">". </span></li>
<li><span class="Apple-style-span" style="font-family: 'courier new';"> Once you reach a leaf node, you have completed a word.</span></li>
</ol></div><div><br />
</div><div>If we were to program this data structure, it would consist of a single node structure with pointers to other nodes (as children). The following psuedo-code shows this.<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixg0_0wVPsfMSFD0dDBYohg6gaa_Dvhni5riVt8W1r_zeP-MQLXVASsK5VL0KXRhIAq9UQpkqDAE3lPnbelxPFIgXynAabOd9FEpeUm-nU_55eb5o0KfHJAsiFfHIZcADkJOd49Y1V0iM/s1600/trieNodeFirst.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixg0_0wVPsfMSFD0dDBYohg6gaa_Dvhni5riVt8W1r_zeP-MQLXVASsK5VL0KXRhIAq9UQpkqDAE3lPnbelxPFIgXynAabOd9FEpeUm-nU_55eb5o0KfHJAsiFfHIZcADkJOd49Y1V0iM/s320/trieNodeFirst.png" /></a></div><br />
Let's continue refining our tree structure. First thing you will note is that the above tree is only one level deep.</div><div><br />
</div><div>This highlights the obvious need to continue generating prefixes down the tree. For instance, the fragments "<b>bate</b>" and "<b>born</b>" share a prefix of a "<b>b</b>", and "<b>alizes</b>" and "<b>alize</b>" share "<b>alize</b>".</div><div><br />
</div><div>So let's grow branches to our tree:<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEihiI_73xG83DBFgHzDCkqlRvCagoI8pEDbPo2ZLd-OoSjbquRRvjtvhhQDlSLNnG9jv4-Wmn3fZDveKscVIEn2wxBjm1P5L-Tbv2ELNGR9iKTB91zgolWVlOofALI3_hpw7100xrbc-0M/s1600/secondTrie.png"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5475629820073912322" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEihiI_73xG83DBFgHzDCkqlRvCagoI8pEDbPo2ZLd-OoSjbquRRvjtvhhQDlSLNnG9jv4-Wmn3fZDveKscVIEn2wxBjm1P5L-Tbv2ELNGR9iKTB91zgolWVlOofALI3_hpw7100xrbc-0M/s320/secondTrie.png" style="cursor: pointer; display: block; height: 168px; margin-bottom: 10px; margin-left: auto; margin-right: auto; margin-top: 0px; text-align: center; width: 320px;" /></a></div><div><div style="text-align: left;">Now this data structure is more efficiently storing prefixes than the previous. But does our algorithm to re<b>trie</b>ve words still work? (Hint: no. (oops, that wasn't a hint))</div></div><div><br />
</div><div><div><span class="Apple-style-span" style="font-size: medium;"><span class="Apple-style-span" style="font-family: 'courier new';">Flawed Re</span></span><b><span class="Apple-style-span" style="font-size: medium;"><span class="Apple-style-span" style="font-family: 'courier new';">trie</span></span></b><span class="Apple-style-span" style="font-size: medium;"><span class="Apple-style-span" style="font-family: 'courier new';">val Algorithm</span></span></div><div><ol><li><span class="Apple-style-span" style="font-family: 'courier new';">Start at the root and walk down to each end node. </span></li>
<li><span class="Apple-style-span" style="font-family: 'courier new';">As you walk down, accumulate the string fragments stored in the node by concatenation.</span></li>
<li><span class="Apple-style-span" style="font-family: 'courier new';"> Once you reach a leaf node, you have completed a word.</span></li>
</ol><div>So what fails here? </div><div><br />
</div><div>Step 3 does. Our new tree has an entire word as a prefix and we would reach a word before we reached a leaf node.</div><div><br />
</div><div>The word "<b>realize</b>" is a prefix to the word "<b><span class="Apple-style-span" style="color: #3366ff;">realize</span>s</b>" but our third step in the algorithm would skip it. Same with "<b>red</b>" and "<b><span class="Apple-style-span" style="color: #3366ff;">red</span>der</b>". </div><div><br />
</div><div>Solution? Easy. Let's add extra information to each node. We will call this boolean information a word-indicator:<span class="Apple-style-span" style="-webkit-text-decorations-in-effect: underline; color: #0000ee;"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5475629825011021042" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjc4xGjMe1TjlcAuRcIZ7TOxSwbOjgc3qfHbfWfca55C8a7i7vHvfrVu0iYVle1UQCdHcfKYF-koEGDh7_MRj8vUsFSwKGgVhnfEmE7X5uZqz0VsjED0W4dkJxeYqZhHakDkKTAPas_hsw/s320/thirdTrie.png" style="cursor: pointer; display: block; height: 158px; margin-bottom: 10px; margin-left: auto; margin-right: auto; margin-top: 0px; text-align: center; width: 320px;" /></span><br />
<div><span class="Apple-style-span" style="font-size: medium;"><span class="Apple-style-span" style="font-family: 'courier new';">New Re</span></span><b><span class="Apple-style-span" style="font-size: medium;"><span class="Apple-style-span" style="font-family: 'courier new';">trie</span></span></b><span class="Apple-style-span" style="font-size: medium;"><span class="Apple-style-span" style="font-family: 'courier new';">val Algorithm</span></span></div><div><ol><li><span class="Apple-style-span" style="font-family: 'courier new';">Start at the root and walk down to each end node.</span></li>
<li><span class="Apple-style-span" style="font-family: 'courier new';">As you walk down, accumulate the string fragments stored in the node by concatenation.</span></li>
<li><span class="Apple-style-span" style="font-family: 'courier new';">Once you reach a leaf node <b>OR</b> if you hit a word-indicator (represented above as an asterisk), you have completed a word.</span></li>
</ol>Our new node in standard UML:<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrvZLz1q2ckAmXJl7tkd4jpKTAfsa883w31632EUrxVyP3Oh1N3B8Y5oq3wsc0jHkHq9W6cDMa7OHLVgk2X8TeN69VZEJbCU11MpEOMP1apQlBpOKPR1yDJMo24xuCx-s_1QCZ0yq4o-A/s1600/trieNodeSecond.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrvZLz1q2ckAmXJl7tkd4jpKTAfsa883w31632EUrxVyP3Oh1N3B8Y5oq3wsc0jHkHq9W6cDMa7OHLVgk2X8TeN69VZEJbCU11MpEOMP1apQlBpOKPR1yDJMo24xuCx-s_1QCZ0yq4o-A/s320/trieNodeSecond.png" /></a></div><br />
So that's it. With this node definition and the above algorithm, we have accomplished a lot.</div><div><br />
</div><div><span class="Apple-style-span" style="font-size: x-large;">Summary</span></div><div><ol><li><span class="Apple-style-span" style="font-size: medium;">We have explained the Dictionary Trie data structure. </span></li>
<li><span class="Apple-style-span" style="font-size: medium;">We have explained a simple algorithm for re<b>trie</b>ving words from this trie.</span></li>
</ol><div>So what are the advantages of this data structure?</div><div><ol><li><span class="Apple-style-span" style="font-size: medium;">Efficient space storage</span></li>
<li><span class="Apple-style-span" style="font-size: medium;">Word lookup by prefix</span></li>
</ol><div>Examples of this second advantage abound. When you start typing in your browser and the browser gives you auto-complete, something like the dictionary trie is being used. Code editors with auto-complete probably use tries too, though augmented with semantic information.</div><div></div><div><br />
</div><div>I decided to use the idea for a dictionary trie when writing a <a href="http://en.wikipedia.org/wiki/Boggle">boggle</a> solver many years ago trying to learn Common Lisp. My problem was to search a random matrix of characters (the boggle board) for words. A naïve recursive search starting at each character in the matrix and crawling through the possibilities blows up exponentially. A quick solution to this is to stop crawling for possible words after you have hit a prefix string that does not exist in the dictionary. (Why search for words of the form "<b>ggrxz</b>-" if you know "<b>ggrxz</b>" is not a prefix of any word? "<b>Ggrxzanacious</b>" is a great word though.) </div><div><br />
</div><div>We have shown what a trie is, and how to retrieve words from it. Next is to show how to create one, and to start defining its capabilities (how to add words, delete words, etc).<br />
<br />
<br />
</div></div></div></div></div></div><br />
<hr style="color: red; height: 5px;" />Daniel Eklundhttp://www.blogger.com/profile/11570452431861145598noreply@blogger.com0