Karol Kuczmarski's Bloghttp://xion.io/2019-12-24T14:22:00+00:00Taking string arguments in Rust2019-12-24T14:22:00+00:00Karol Kuczmarskitag:xion.io,2019-12-24:post/code/rust-string-args.html<p>Strings of text seem to always be a complicated topic when it comes to programming.
This counts double for low-level languages
which expose the programmer to the full complexity of memory management and allocation.</p>
<p>Rust is, obviously, one of those languages.
Strings in Rust are therefore represented using two distinct types:
<code>str</code> (the string <em>slice</em>) and <code>String</code> (the owned/allocated string).
Learning <a href="https://doc.rust-lang.org/book/second-edition/ch08-02-strings.html">how to juggle those types</a>
is something you need to do very early if you want to be productive in the language.</p>
<p>But even after you’ve programmed in Rust for some time,
you may still trip on some more subtle issues with string handling.
In this post, I will concentrate on just one common task:
writing a function that takes a <em>string argument</em>.
We’ll see that even there, we can encounter a fair number of gotchas.</p>
<h4>Just reading it</h4>
<p>Let’s start with a simple case:
a function which merely <em>reads</em> its string argument:</p>
<div class="highlight"><pre><span class="k">fn</span><span class="w"> </span><span class="n">hello</span><span class="p">(</span><span class="n">name</span><span class="o">:</span><span class="w"> </span><span class="o">&</span><span class="kt">str</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">println</span><span class="o">!</span><span class="p">(</span><span class="s">"Hello, {}!"</span><span class="p">,</span><span class="w"> </span><span class="n">name</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>As you’re probably well aware,
using <code>str</code> rather than <code>String</code> is the idiomatic approach here.
Because a <code>&str</code> reference is essentially an address + length,
it can point to any string wheresoever:
a <code>'static</code> literal, a heap-allocated <code>String</code>, or any portion or substring thereof:</p>
<div class="highlight"><pre><span class="n">hello</span><span class="p">(</span><span class="s">"world"</span><span class="p">);</span><span class="w"></span>
<span class="n">hello</span><span class="p">(</span><span class="o">&</span><span class="n">String</span><span class="o">::</span><span class="n">from</span><span class="p">(</span><span class="s">"Alice"</span><span class="p">));</span><span class="w"></span>
<span class="n">hello</span><span class="p">(</span><span class="o">&</span><span class="s">"Dennis Ritchie"</span><span class="p">[</span><span class="mf">0.</span><span class="p">.</span><span class="mi">6</span><span class="p">]);</span><span class="w"></span>
</pre></div>
<p>Contrast this with an argument of type <code>&String</code>:</p>
<div class="highlight"><pre><span class="k">fn</span><span class="w"> </span><span class="n">hello</span><span class="p">(</span><span class="n">name</span><span class="o">:</span><span class="w"> </span><span class="o">&</span><span class="n">String</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">println</span><span class="o">!</span><span class="p">(</span><span class="s">"Hello, {}!"</span><span class="p">,</span><span class="w"> </span><span class="n">name</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>which mandates an actual, full-blown <code>String</code> object:</p>
<div class="highlight"><pre><span class="n">hello</span><span class="p">(</span><span class="o">&</span><span class="n">String</span><span class="o">::</span><span class="n">from</span><span class="p">(</span><span class="s">"Bob"</span><span class="p">));</span><span class="w"></span>
<span class="c1">// (the other examples won't work)</span>
</pre></div>
<p>There are virtually no circumstances when you would want to do this,
as it potentially forces the caller to needlessly put the string on the heap.
Even if you anticipate all function calls to involve actual <code>String</code> objects,
the automatic <code>Deref</code> coercion from <code>&String</code> to <code>&str</code>
should still allow you to use the more universal, <code>str</code>-based <span class="caps">API</span>.</p>
<h4>Hiding the reference</h4>
<p>If <em>rustc</em> can successfully turn a <code>&String</code> into <code>&str</code>,
then perhaps it should also be possible to simply use <code>String</code> when that’s more convenient?</p>
<p>In general, this kind of “reverse <code>Deref</code>” doesn’t happen in Rust
outside of method calls with <code>&self</code>.
It seems, however, that it would sometimes be desirable;
one reasonable use case involves chains of iterator adapters,
most importantly <code>map</code> and <code>for_each</code>:</p>
<div class="highlight"><pre><span class="kd">let</span><span class="w"> </span><span class="n">strings</span><span class="o">:</span><span class="w"> </span><span class="n">Vec</span><span class="o"><</span><span class="n">String</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">vec</span><span class="o">!</span><span class="p">[</span><span class="s">"Alice"</span><span class="p">.</span><span class="n">into</span><span class="p">(),</span><span class="w"> </span><span class="s">"Bob"</span><span class="p">.</span><span class="n">into</span><span class="p">()];</span><span class="w"></span>
<span class="n">strings</span><span class="p">.</span><span class="n">into_iter</span><span class="p">().</span><span class="n">for_each</span><span class="p">(</span><span class="n">hello</span><span class="p">);</span><span class="w"></span>
</pre></div>
<p>Since the compiler doesn’t take advantage of<code>Deref</code> coercions when inferring closure types,
their argument types have to match exactly.
As a result, we often need explicit <code>|x| foo(x)</code> closures
which suffer from poorer readability in long <code>Iterator</code> or <code>Stream</code>-based expressions.</p>
<p>We can make the above code work
— and also retain the ability to make calls like <code>hello("Charlie");</code> —
by using one of the built-in traits that generalize over the borrowing relationships.
The one that works best for accepting string arguments is called <code>AsRef</code><sup id="fnref:1"><a class="footnote-ref" href="#fn:1" rel="footnote">1</a></sup>:</p>
<div class="highlight"><pre><span class="k">fn</span><span class="w"> </span><span class="n">hello</span><span class="o"><</span><span class="n">N</span><span class="o">:</span><span class="w"> </span><span class="n">AsRef</span><span class="o"><</span><span class="kt">str</span><span class="o">>></span><span class="p">(</span><span class="n">name</span><span class="o">:</span><span class="w"> </span><span class="n">N</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">println</span><span class="o">!</span><span class="p">(</span><span class="s">"Hello, {}!"</span><span class="p">,</span><span class="w"> </span><span class="n">name</span><span class="p">.</span><span class="n">as_ref</span><span class="p">());</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>Its sole method, <code>AsRef::as_ref</code>, returns a reference to the trait’s type parameter.
In the case above, that reference will obviously be of type <code>&str</code>,
which circles back to our initial example, one with a direct <code>&str</code> argument.</p>
<p>The difference is, however, that <code>AsRef<str></code> is implemented
<a href="https://play.rust-lang.org/?gist=91f084a1ae0527f3af1cba3fca421da6&version=stable">for all interesting string types</a>
— both in their owned and borrowed versions.
This obviates the need for <code>Deref</code> coercions and makes the <span class="caps">API</span> more convenient.</p>
<h4>Own it</h4>
<p>Things get a little more complicated when the string parameter is needed
for more than just reading. For storage and potential mutation,
a <code>&str</code> reference is not enough: you need an actual, full-blown <code>String</code> object.</p>
<p>Now, you may think this is not a huge obstacle.
After all, it’s pretty easy to “turn” <code>&str</code> into a <code>String</code>:</p>
<div class="highlight"><pre><span class="k">struct</span><span class="w"> </span><span class="n">Greetings</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">Vec</span><span class="o"><</span><span class="n">String</span><span class="o">></span><span class="w"> </span><span class="n">names</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">impl</span><span class="w"> </span><span class="n">Greetings</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Don't do this!</span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="n">hello</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">name</span><span class="o">:</span><span class="w"> </span><span class="o">&</span><span class="kt">str</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">names</span><span class="p">.</span><span class="n">push</span><span class="p">(</span><span class="n">name</span><span class="p">.</span><span class="n">clone</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>But I strongly advise against this practice, at least in public APIs.
If you expose such function to your users, you are essentially tricking them
into thinking their input will only ever be read, not copied,
which has implications on both performance and memory usage.</p>
<p>Instead, if you need to take ownership of the resulting <code>String</code>,
it is much better to indicate this in the function signature directly:</p>
<div class="highlight"><pre><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="n">hello</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">name</span><span class="o">:</span><span class="w"> </span><span class="n">String</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">names</span><span class="p">.</span><span class="n">push</span><span class="p">(</span><span class="n">name</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>This shifts the burden on creating the <code>String</code> onto the caller,
but that’s not necessarily a bad thing.
On their side, the added boilerplate can pretty minimal:</p>
<div class="highlight"><pre><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">greetings</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Greetings</span><span class="o">::</span><span class="n">new</span><span class="p">();</span><span class="w"></span>
<span class="n">grettings</span><span class="p">.</span><span class="n">hello</span><span class="p">(</span><span class="n">String</span><span class="o">::</span><span class="n">from</span><span class="p">(</span><span class="s">"Dylan"</span><span class="p">));</span><span class="w"> </span><span class="c1">// uhm...</span>
<span class="n">greetings</span><span class="p">.</span><span class="n">hello</span><span class="p">(</span><span class="s">"Eva"</span><span class="p">.</span><span class="n">to_string</span><span class="p">());</span><span class="w"> </span><span class="c1">// somewhat better...</span>
<span class="n">grettings</span><span class="p">.</span><span class="n">hello</span><span class="p">(</span><span class="s">"Frank"</span><span class="p">.</span><span class="n">to_owned</span><span class="p">());</span><span class="w"> </span><span class="c1">// not too bad</span>
<span class="n">greetings</span><span class="p">.</span><span class="n">hello</span><span class="p">(</span><span class="s">"Gene"</span><span class="p">.</span><span class="n">into</span><span class="p">());</span><span class="w"> </span><span class="c1">// good enough</span>
</pre></div>
<p>while clearly indicating where does the memory allocation happen.</p>
<div class="footnote">
<hr />
<ol>
<li id="fn:1">
<p>It is also idiomatically used for functions taking <code>Path</code> parameters,
i.e. <code>AsRef<Path></code>. <a class="footnote-backref" href="#fnref:1" rev="footnote" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>Yes, the npm ecosystem is at fault2018-11-27T19:27:00+01:00Karol Kuczmarskitag:xion.io,2018-11-27:post/programming/another-npm-snafu.html<p>Even if you are like me and don’t use any (server-side) Javascript, or node.js, or npm,
it may feel like every other week there are news about
<a href="https://github.com/dominictarr/event-stream/issues/116">yet another npm snafu</a>.</p>
<p>The latest incident could be the most severe one yet,
at least from the security standpoint.
<a href="https://arstechnica.com/information-technology/2018/11/hacker-backdoors-widely-used-open-source-software-to-steal-bitcoin/">In a nutshell</a>,
the owner of <em>event-stream</em> — a popular package even by npm standards —
had transferred ownership of it to another person.
The new owner would then publish a new version of the package
which contained a targeted backdoor, intended to <em>steal large amounts of Bitcoin</em>
from the users of one particular wallet provider.</p>
<p>Quite a step up from <a href="https://www.theregister.co.uk/2016/03/23/npm_left_pad_chaos/">the <em>left-pad</em> fiasco</a>, isn’t it?…</p>
<p>Like it usually happens with any major incident,
lots of people are eager to put the blame on someone, or something, as quickly as possible.
Rather unsurprisingly, the original owner of <em>event-stream</em> came up as a pretty obvious target.
Without his abdication of ownership rights, the argument goes,
the entire ordeal wouldn’t have happened in the first place.
More broadly, as a maintainer of a popular package,
<a href="https://github.com/dominictarr/event-stream/issues/116#issuecomment-441161123">he owes it to the community</a>
to make sure it remains available and safe to use,
now and for the foreseeable future.</p>
<p>But if those are the responsibilities, then what are the rewards?…
Well, in the author’s own words,
<a href="https://gist.github.com/dominictarr/9fd9c1024c94592bc7268d36b8d83b3a">“you get literally nothing from maintaing a popular package”</a>.
Looks like once the fun of it wears off, it’s just all work and no play,
for little to no tangible benefit.</p>
<p>This problem is of course not specific to Javascript or npm.
Here’s, for example,
<a href="https://gist.github.com/richhickey/1563cddea1002958f96e7ba9519972d9">a good take on the issue</a>
from the creator of <a href="https://clojure.org/">Clojure</a>.</p>
<p>However, this <em>doesn’t mean</em> every other package manager is equally susceptible
to the sort of issues that have shaken npm <a href="https://blog.npmjs.org/post/173526807575/reported-malicious-module-getcookies">time</a>
and
<a href="https://blog.npmjs.org/post/163723642530/crossenv-malware-on-the-npm-registry">time again</a>.
To say the same thing could’ve happened to Python/Ruby/Go/Haskell/etc. is vacuous.
While factually true, it’s an instance of the
<a href="https://www.lesswrong.com/posts/dLJv2CoRCgeC2mPgj/the-fallacy-of-gray">Fallacy of Grey</a>:
a claim that because nothing is <em>ideal</em>, everything must be <em>equally flawed</em>.</p>
<p>In reality, the Javascript ecosystem facilitated by npm is
<a href="https://www.reddit.com/r/programming/comments/a0kxmw/i_dont_know_what_to_say_backdoor_in_popular/eaij0rt/">singularly vulnerable</a>
to problems of this kind.
It follows directly from how granular the npm packages are, and how wide and deep their dependency trees get.</p>
<p>Indeed, it would be quite a fascinating exercise to quantify the difference numerically,
by comparing the average size of a dependency graph between npm, PyPI, Rubygems, Hoogle, and so on.
It’s rather obvious npm would come as a “winner”, likely with Rust’s Cargo not very far behind.</p>
<p>Apparently, this not unintentional either —
the node.js community seems to <a href="https://twitter.com/joepie91/status/1065193459349446656">like it this way</a>.
But as the yesterday’s incident has exposed,
this state of affairs relies on the distributed conscientiousness of <em>very</em> many parties
— in this case, npm package authors.
When one inevitably falters, the impact reaches far and wide,
rippling through the massively intertwined npm registry.</p>
<p>While it may be satisfying, then, to blame the package owner as the immediate culprit,
it’s way more productive to consider how the damage could have been mitigated.</p>
<p>We can, for example, look at the unimaginably dense dependency graph of npm —
if anyone ever dared to visualize it in its entirety —
and realize it shouldn’t really have so many edges.
Had it been little sparser, more similar to the graphs of PyPI or Ruby,
then removing (i.e. compromising) a single leaf node would’ve had proportionally smaller impact.</p>
<p>So yes, it’s true that all package managers are exposed to the risk of compromised packages.
It is, however, something that can be estimated and compared — and unfortunately,
it is clear that for npm, this risk is currently the largest.</p>A Haskell retrospective2018-08-18T13:07:00+02:00Karol Kuczmarskitag:xion.io,2018-08-18:post/programming/haskell-retrospective.html<p>Approximately a year ago,
I had the opportunity to work on <a href="https://code.facebook.com/posts/745068642270222/fighting-spam-with-haskell/">Sigma</a>
— a large, distributed system that protects Facebook users from spam and other kinds of abuse.</p>
<p>One reason it was a pretty unique experience is that Sigma is almost entirely a <em>Haskell</em> codebase.
It was the first time I got to work with the language in a professional setting,
so I was eager to see how it performs in a real-world, production-grade application.</p>
<p>In this (rather long) post, I’ll draw on this experience
and highlight Haskell’s notable features from a practical, engineering standpoint.
In other words, I’ll be interested in how much does it help with solving actual problems
that arise in the field of software development <span class="amp">&</span> maintenance.</p>
<h4>Haskell Who?</h4>
<p>Before we start, however,
it seems necessary to clarify what “Haskell” are we actually talking about.</p>
<p>Granted, this may be a little surprising.
From a far-away vantage point,
Haskell is typically discussed as a rather uniform language,
and it is often treated as synonymous with <em>functional programming</em> in general.</p>
<p>But if you look closer, that turns out to be a bit of a misrepresentation.
In reality, Haskell is a complex manifold of different components,
some of which can be thought as their own sublanguages.
Roughly speaking, Haskell
— as it’s used in the industry and in the <span class="caps">OSS</span> world today —
should be thought of as a cake with at least the following layers:</p>
<ul>
<li>
<p>The base Haskell language, as defined by the Haskell ‘98 and 2010 reports.
At least in theory, this is the <em>portable</em> version of the language
that any conforming compiler is supposed to accept.
In practice, given <a href="http://taylor.fausak.me/2017/11/15/2017-state-of-haskell-survey-results/#question-14">the absolute monopoly of <span class="caps">GHC</span></a>,
it is merely a theoretical base that needs to be significantly augmented
in order to reach some level of practical usability.</p>
</li>
<li>
<p>A bunch of <span class="caps">GHC</span> extensions that are
<a href="http://dev.stephendiehl.com/hask/#the-benign">widely considered</a>
mandatory for any real-world project.
Some, like <code>TupleSections</code> or <code>MultiParamTypeClasses</code>,
are mostly there to fix some surprising feature gaps
that would be more confusing if you had worked around them instead.
Others, like <code>GADTs</code> or <code>DataKinds</code>,
open up completely new avenues for type-level abstractions.</p>
</li>
<li>
<p>A repertoire of common third-party libraries with unique DSLs,
like <a href="https://hackage.haskell.org/package/conduit"><em>conduit</em></a>,
<a href="https://hackage.haskell.org/package/pipes"><em>pipes</em></a>,
or <a href="https://hackage.haskell.org/package/lens-tutorial-1.0.3/docs/Control-Lens-Tutorial.html"><em>lens</em></a>.
Unlike many “regular” packages that merely bring in some domain-specific <span class="caps">API</span>,
these fundamental libraries shape both the deeper architecture
and the surface-level look <span class="amp">&</span> feel of any Haskell codebase that uses them.</p>
</li>
<li>
<p>A selection of <a href="https://ocharles.org.uk/blog/posts/2014-12-01-24-days-of-ghc-extensions.html">less common extensions</a>
which are nevertheless encountered in Haskell code with some regularity.</p>
</li>
<li>
<p><a href="https://wiki.haskell.org/Template_Haskell">Template Haskell</a>,
the language for compile-time metaprogramming whose main application is probably <em>generics</em>.<br>
To be clear, neither “template” nor “generics” have anything to do with the usual meanings
of those terms in C++ and Java/C#/Go<sup id="fnref:1"><a class="footnote-ref" href="#fn:1" rel="footnote">1</a></sup>.
Rather, it refers to a kind of <span class="caps">AST</span>-based “preprocessing”
that allows Haskell code to operate on the generic <em>structure</em> of user-defined types:
their constructors, parameters, and record fields<sup id="fnref:2"><a class="footnote-ref" href="#fn:2" rel="footnote">2</a></sup>.<br>
Direct use of <span class="caps">TH</span> in application code is extremely rare,
but many projects rely on libraries which utilize it behind the scenes.
A great example would be <a href="https://hackage.haskell.org/package/persistent-2.7.1/docs/Database-Persist.html">Persistent</a>,
a database interface library where the <span class="caps">ORM</span>
uses Template Haskell to construct record types from a <span class="caps">DB</span> schema at compile time.</p>
</li>
</ul>
<h4>There is a language in my type system</h4>
<p>What’s striking about this ensemble of features and ideas
is that most of them don’t seem to follow from the ostensible premise of the language:
that it is functional, pure / referentially transparent, and non-strict / lazily evaluated.
Instead, they are mostly a collection of progressively more sophisticated refinements
and applications of Haskell’s <em>type system</em>.</p>
<p>This singular focus on type theory — especially in the recent years<sup id="fnref:3"><a class="footnote-ref" href="#fn:3" rel="footnote">3</a></sup> —
is probably why many people in the wider programming world think
it is necessary to grok advanced type system concepts
if you even want to dabble in functional programming</p>
<p>That is, of course, patently untrue<sup id="fnref:4"><a class="footnote-ref" href="#fn:4" rel="footnote">4</a></sup>.
<em>Some</em> features of a strong static type system are definitely useful to have in a functional language.
You can look at <a href="http://elm-lang.org/">Elm</a>
to see <a href="http://package.elm-lang.org/packages/elm-lang/core/5.1.1/Result#map3">how awkward</a>
things become when you deprive an <span class="caps">FP</span> language of its typeclasses
and <a href="http://package.elm-lang.org/packages/elm-lang/core/5.1.1/Result#andThen">composition sugar</a>.</p>
<p>But when the focus on type systems becomes too heavy,
the concepts <a href="https://wiki.haskell.org/GHC/Type_system">keep piling up</a>
and the language becomes increasingly impenetrable.
Eventually, you may end up with an ecosystem where the recommended way to implement an <span class="caps">HTTP</span> <span class="caps">API</span>
is to call upon
<a href="http://haskell-servant.readthedocs.io/en/stable/tutorial/Server.html#a-first-example">half a dozen compiler extensions</a>
in order to specify it as
<a href="http://haskell-servant.readthedocs.io/en/stable/tutorial/Server.html#from-combinators-to-handler-arguments">one humongous type</a>.</p>
<p>But hey, isn’t it desirable to have this kind of increased type safety?</p>
<p>In principle, the answer would of course be yes.
However, the price we pay here is in the precious currency of <em>complexity</em>,
and it often turns out to be way too high.
When libraries, frameworks, and languages get complicated and abstract,
it’s not just safety and/or productivity that can (hopefully) increase —
it is also the burden on developers’ thought processes.
While the exact threshold of diminishing or even negative returns is hard to pinpoint,
it can definitely be reached even by the smartest and most talented teams.
Add in the usual obstacles of software engineering — shifting requirements,
deadlines, turnover — and you may encounter it much sooner than you think.</p>
<p>For some, this is a sufficient justification to basically
<a href="https://golang.org/">give up on type systems</a> altogether.
And while I’d say such a knee-jerk reaction is rather excessive and unwarranted,
it is at least equally harmful to letting your typing regime grow in boundless complexity.
Both approaches are just too extreme to stand the test of practicality.</p>
<h4>The legacy of bleeding edge</h4>
<p>In other words, Haskell <em>is</em> hard and this <em>does</em> count as one of its serious problems.
This conclusion isn’t exactly novel or surprising,
even if some people would <a href="http://dave.fayr.am/posts/2011-08-19-lets-go-shopping.html">still argue with it</a>.</p>
<p>Suppose, however, that we have somehow caused this issue to disappear completely.
Let’s say that through some kind of divine intervention,
it was made so that the learning curve of Haskell
is no longer a problem for the majority of programmers.
Maybe we found a magic lamp and — for the lack of better ideas —
we wished that everyone be as proficient
in applicative parsers as they are in inheritance hierarchies.</p>
<p>Even in this hypothetical scenario, I posit that
the value proposition of Haskell would <em>still</em> be a tough sell.</p>
<p>There is this old quote from Bjarne Stroustrup (creator of C++)
where he says that programming languages divide into those everyone complains about,
and those that no one uses.<br>
The first group consists of old, established technologies
that managed to accrue significant complexity debt through years and decades of evolution.
All the while, they’ve been adapting to the constantly shifting perspectives
on what are the best industry practices.
Traces of those adaptations can still be found today,
sticking out like a leftover appendix or residual tail bone —
or like the built-in support for <span class="caps">XML</span> in Java.</p>
<p>Languages that “no one uses”, on the other hand,
haven’t yet passed the industry threshold of sufficient maturity and stability.
Their ecosystems are still cutting edge, and their future is uncertain,
but they sometimes champion some really compelling paradigm shifts.
As long as you can bear with things that are rough around the edges,
you can take advantage of their novel ideas.</p>
<p>Unfortunately for Haskell,
it manages to combine the <em>worst parts</em> of both of these worlds.</p>
<p>On one hand, it is a surprisingly old language,
clocking more than two decades of fruitful research around many innovative concepts.
Yet on the other hand, it bears the signs of a fresh new technology,
with relatively few production-grade libraries,
scarce coverage of some domains (e.g. <span class="caps">GUI</span> programming),
and not too many stories of commercial successes.</p>
<h4>There are many ways to do it</h4>
<p>Nothing shows better the problems of Haskell’s evolution over the years
than the various approaches to handling <em>strings</em> and <em>errors</em> that it now has.<sup id="fnref:5"><a class="footnote-ref" href="#fn:5" rel="footnote">5</a></sup></p>
<h5>String theory</h5>
<p>Historically, <code>String</code> has been defined as a list of <code>Char</code>acters,
which is normally denoted as the <code>[Char]</code> type.
The good thing about this representation is that many string-based algorithms
can simply be written using just the list functions.</p>
<p>The bad thing is that Haskell lists are the so-called <em>cons lists</em>.
They consist of the single element (called <code>head</code>),
followed by another list of the remaining elements (called <code>tail</code>).
This makes them roughly equivalent to what the data structures theory
calls a <em>singly-linked list</em>
— a rarely used construct that has a number of undesirable characteristics:</p>
<ul>
<li>linear time (<code>O(n)</code>) for finding a specific element in the list</li>
<li>linear time for finding an element with a specific <em>index</em> in the list</li>
<li>linear time for insertion in the middle of the list</li>
<li>poor cache coherency due to scattered allocations of list nodes<sup id="fnref:6"><a class="footnote-ref" href="#fn:6" rel="footnote">6</a></sup></li>
</ul>
<p>On top of that, keeping only a single character inside each node
results in a significant waste of memory.</p>
<p>Given those downsides,
it isn’t very surprising that virtually no serious Haskell program
uses <code>String</code>s for any meaningful text processing.
The community-accepted replacement is <a href="https://hackage.haskell.org/package/text">the <em>text</em> package</a>,
whose implementation stores strings inside packed arrays, i.e. just as you would expect.
As a result, Haskell has at least two main types of “strings” — or even <em>three</em>,
since <code>Text</code> has both lazy and strict variants.</p>
<p>That’s not all, however:
there is also <a href="https://hackage.haskell.org/package/bytestring">the <em>bytestring</em> package</a>.
Although technically it implements generic byte buffers,
its <span class="caps">API</span> has been <a href="https://hackage.haskell.org/package/bytestring-0.10.8.2/docs/Data-ByteString.html#g:11">pretty rich and enticing</a>.
As a result, many other packages would rather use <code>ByteString</code>s directly in their interfaces
than to incur the conversions to and from <code>Text</code>.<br>
And just like in case of <code>Text</code>,
separate lazy and strict variants of <code>ByteString</code> are also available.
But unlike <code>Text</code>, byte strings also have <code>Word8</code> and <code>Char8</code> versions,
where the latter is designed to handle legacy cases of <span class="caps">ASCII</span>-exclusive text support.</p>
<p>Well, I hope you kept count of all these types!
I also hope you can memorize the correct way of converting between them,
because it’s commonplace to see them used simultaneously.
This may sometimes happen even within the same library,
but it definitely occurs in application code
that utilizes many different dependencies.
What it usually results in are numerous occurrences of something like <code>Text.pack . foo . Text.unpack</code>,
with conversion functions copiously sprinkled in
to help win in the <a href="http://javran.github.io/posts/2014-02-28-type-tetris-and-typeclassopedia.html">Type Tetris</a>.</p>
<h5>Errors and how to handle them</h5>
<p>A somewhat similar issue applies to error handling.
Over the years, Haskell has tried many approaches to this problem,
often mixing techniques that are very rarely found in a single language,
like exceptions combined with result types.</p>
<p>Nowadays, there is some consensus about those
<a href="https://prime.haskell.org/wiki/Libraries/Proposals/MonadFail">mistakes of the past</a>,
but the best we got is their deprecation:
the current version of <span class="caps">GHC</span> still supports them all.</p>
<p>What are all those techniques? Here’s an abridged list:</p>
<ul>
<li>the <code>error</code> function, terminating the program with a message
(which is obviously discouraged)</li>
<li>the <code>fail</code> method of the <code>Monad</code> typeclass
(which is now deprecated and moved to <code>MonadFail</code>)</li>
<li><a href="https://hackage.haskell.org/package/mtl-2.2.1/docs/Control-Monad-Error.html#t:MonadError">the <code>MonadError</code> class</a>
with the associated <code>ErrorT</code> transformer, now deprecated in favor of…</li>
<li>a <em>different</em> <a href="https://hackage.haskell.org/package/mtl-2.2.1/docs/Control-Monad-Except.html#t:MonadError"><code>MonadError</code> class</a>,
with <code>ExceptT</code> as the new transformer</li>
<li><a href="https://hackage.haskell.org/package/base-4.10.0.0/docs/Control-Exception.html">exceptions in the <code>IO</code> monad</a>,
normally raised by the standard I/O calls to signal abnormal conditions and error;
however, libraries and application code are free to <a href="https://hackage.haskell.org/package/base-4.10.0.0/docs/Control-Exception.html#g:2">also throw them</a>
and use for their own error handling</li>
<li>the <code>Either</code> sum type / monad, which is essentially a type-safe version of the venerable return codes</li>
</ul>
<p>If you really stretched the definition of error handling,
I could also imagine counting <code>Maybe</code>/<code>MaybeT</code> as yet another method.
But even without it, that’s half a dozen distinct approaches
which you are likely to encounter in the wild in one form or another.</p>
<h4>Implicit is better than explicit</h4>
<p>The other kind of troublesome legacy of Haskell
relates to the various design choices in the language itself.
They reflect ideas straight from the time they were conceived in,
which doesn’t necessarily agree with the best engineering practices as we understand them <em>now</em>.</p>
<h5>Leaky modules</h5>
<p>Take the <em>module system</em>, for example.</p>
<p>Today, it is rather uncontroversial that the purpose of splitting code into multiple submodules
is to isolate it as much as possible and prevent accidental dependencies.
The benefit of such isolation is better internal cohesion for each module.
This can simplify testing, improve readability, foster simplicity,
and reduce cognitive burden on the maintainers.</p>
<p>Contemporary languages help achieving this goal
by making inter-module dependencies explicit.
If you want to use a symbol (functions, class) from module A inside another module B,
you typically have to both:</p>
<ul>
<li>declare it <em>public</em> in module A</li>
<li>explicitly <em>import</em> its name in module B</li>
</ul>
<p>The first step helps to ensure that the <span class="caps">API</span> of module A is limited and tractable.
The second step does the same to the external dependencies of module B.</p>
<p>Unfortunately, Haskell requires neither of those steps.
In fact, it encourages precisely the <em>opposite</em> of well-defined, self-contained modules,
all by the virtue of its default behaviors:</p>
<ul>
<li>the default module declaration (<code>module Foo where ...</code>)
implicitly declares <em>every symbol</em> defined in the module <code>Foo</code>
as public and importable by others</li>
<li>the default import statement (<code>import Foo</code>) brings in <em>every public symbol</em>
from the module <code>Foo</code> into the global namespace of the current module</li>
</ul>
<p>In essence, this is like putting <code>public</code> on each and every class or method
that you’d define in a Java project,
while simultaneously using nothing but wildcard (star) imports.
In a very short order, you will end up with project
where everything depends on everything else, and nothing can be developed in isolation.</p>
<h5>Namespaces are apparently a bad idea</h5>
<p>Thankfully, it is possible to avoid this pitfall
by explicitly declaring both your exported and imported symbols:</p>
<div class="highlight"><pre><span class="c1">-- Foo.hs --</span>
<span class="kr">module</span> <span class="nn">Foo</span> <span class="p">(</span> <span class="nf">foo</span><span class="p">,</span> <span class="nf">bar</span> <span class="p">)</span> <span class="kr">where</span>
<span class="nf">foo</span> <span class="ow">=</span> <span class="o">...</span>
<span class="nf">bar</span> <span class="ow">=</span> <span class="o">...</span>
<span class="nf">baz</span> <span class="ow">=</span> <span class="o">...</span> <span class="c1">-- not exported</span>
<span class="c1">-- Bar.hs --</span>
<span class="kr">import</span> <span class="nn">Foo</span> <span class="p">(</span><span class="nf">foo</span><span class="p">)</span>
<span class="c1">-- `bar` is inaccessible here, but `foo` is available</span>
</pre></div>
<p>But while this helps fighting the tangle of dependencies,
it still results in cluttering the namespace of any non-trivial module
with a significant number of imported symbols.</p>
<p>In many other languages, you can instead import the module as a whole
and only refer to its members using <em>qualified names</em>.
This is possible in Haskell as well, though it requires yet another variant
of the <code>import</code> statement:</p>
<div class="highlight"><pre><span class="kr">import</span> <span class="k">qualified</span> <span class="nn">Data.Text</span> <span class="k">as</span> <span class="n">Text</span>
<span class="nf">duplicateWords</span> <span class="ow">::</span> <span class="kt">Text</span><span class="o">.</span><span class="kt">Text</span> <span class="ow">-></span> <span class="kt">Text</span><span class="o">.</span><span class="kt">Text</span>
<span class="nf">duplicateWords</span> <span class="ow">=</span> <span class="kt">Text</span><span class="o">.</span><span class="n">unwords</span> <span class="o">.</span> <span class="n">map</span> <span class="p">(</span><span class="kt">Text</span><span class="o">.</span><span class="n">unwords</span> <span class="o">.</span> <span class="n">replicate</span> <span class="mi">2</span><span class="p">)</span> <span class="o">.</span> <span class="kt">Text</span><span class="o">.</span><span class="n">words</span>
</pre></div>
<p>What if you want both, though? In the above code, for example,
the qualified name <code>Text.Text</code> looks a little silly,
especially when it’s such a common type.
It would be nice to import it directly, so that we can use it simply as <code>Text</code>.</p>
<p>Unfortunately, this is only possible when using <em>two</em> <code>import</code> statements:</p>
<div class="highlight"><pre><span class="kr">import</span> <span class="nn">Data.Text</span> <span class="p">(</span><span class="kt">Text</span><span class="p">)</span>
<span class="kr">import</span> <span class="k">qualified</span> <span class="nn">Data.Text</span> <span class="k">as</span> <span class="n">Text</span>
<span class="nf">duplicateWords</span> <span class="ow">::</span> <span class="kt">Text</span> <span class="ow">-></span> <span class="kt">Text</span>
<span class="nf">duplicateWords</span> <span class="ow">=</span> <span class="kt">Text</span><span class="o">.</span><span class="n">unwords</span> <span class="o">.</span> <span class="n">map</span> <span class="p">(</span><span class="kt">Text</span><span class="o">.</span><span class="n">unwords</span> <span class="o">.</span> <span class="n">replicate</span> <span class="mi">2</span><span class="p">)</span> <span class="o">.</span> <span class="kt">Text</span><span class="o">.</span><span class="n">words</span>
</pre></div>
<p>You will find this duplication pervasive throughout Haskell codebases.
Given how it affects the most important third-party packages (like <code>text</code> and <code>bytestring</code>),
there have been a few proposals to improve the situation<sup id="fnref:7"><a class="footnote-ref" href="#fn:7" rel="footnote">7</a></sup>,
but it seems that none can go through the syntax bikeshedding phase.</p>
<p>Contrast this with Rust, for example, where it’s common to see imports such as this:</p>
<div class="highlight"><pre><span class="kn">use</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">io</span><span class="o">::</span><span class="p">{</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">Read</span><span class="p">};</span><span class="w"></span>
<span class="k">fn</span><span class="w"> </span><span class="n">read_first_half</span><span class="p">(</span><span class="n">path</span><span class="o">:</span><span class="w"> </span><span class="o">&</span><span class="nb">Path</span><span class="p">)</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">io</span><span class="o">::</span><span class="nb">Result</span><span class="o"><</span><span class="n">String</span><span class="o">></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// (omitted)</span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>where <code>self</code> conveniently stands for the module as a whole.</p>
<h5>Wild records</h5>
<p>Another aspect of the difficulties with keeping your namespaces in check
relates to Haskell <em>record types</em> — its rough equivalent of <code>struct</code>s from C and others.</p>
<p>When you define a record type:</p>
<div class="highlight"><pre><span class="kr">data</span> <span class="kt">User</span> <span class="ow">=</span> <span class="kt">User</span> <span class="p">{</span> <span class="n">usrFirstName</span> <span class="ow">::</span> <span class="kt">String</span>
<span class="p">,</span> <span class="n">usrLastName</span> <span class="ow">::</span> <span class="kt">String</span>
<span class="p">,</span> <span class="n">usrEmail</span> <span class="ow">::</span> <span class="kt">String</span>
<span class="p">}</span> <span class="kr">deriving</span> <span class="p">(</span><span class="kt">Show</span><span class="p">)</span>
</pre></div>
<p>you are declaring not one but multiple different names,
and dumping them all straight into the global namespace.
These names include:</p>
<ul>
<li>the record <em>type</em> (here, <code>User</code>)</li>
<li>its <em>type constructor</em> (also <code>User</code>, second one above)</li>
<li>all of its fields (<code>usrFirstName</code>, <code>usrLastName</code>, <code>usrEmail</code>)</li>
</ul>
<p>Yep, thats right.
Because Haskell has no special syntax for accessing record fields,
each field declaration creates an unqualified getter function.
Combined with the lack of function overloading,
this creates many opportunities for name collisions.</p>
<p>This is why in the above example,
<a href="https://en.wikipedia.org/wiki/Hungarian_notation">Hungarian notation</a> is used to prevent those clashes.
Despite its age and almost complete disuse in basically every other language,
it is still a widely accepted practice in Haskell<sup id="fnref:8"><a class="footnote-ref" href="#fn:8" rel="footnote">8</a></sup>.</p>
<h4>Purity beats practicality</h4>
<p>We have previously discussed the multiple ways of working with strings
and handling errors in Haskell. While somewhat confusing at times,
there at least appears to be an agreement in the community as to which one
should generally be preferred.</p>
<p>This is not the case for some subtler and more abstract topics.</p>
<p>Haskell is, famously,
a <a href="https://en.wikipedia.org/wiki/Purely_functional_programming"><em>purely functional</em> programming language</a>.
Evaluating functions, in a mathematical sense,
is all a Haskell program is supposed to be doing.
But the obvious problem is
that such a program wouldn’t be able to do anything actually <em>useful</em>;
there needs to be some way for it to <em>effect</em> the environment it runs in,
if only to print the results it computed.</p>
<p>How to reconcile the functional purity with real-world applications
is probably the most important problem
that the Haskell language designers have to contend with.
After a couple of decades of research <em>and</em> industrial use
it still doesn’t have a satisfactory answer.</p>
<p>Yes, there is the <code>IO</code> monad, but it is a very blunt instrument.
It offers a distinction between pure code and “effectful” code,
but allows for no granularity or structure for the latter.
An <code>IO</code>-returning function can do literally anything,
while a pure function can only compute some value based on its arguments.
Most code, however, is best placed somewhere between those two extremes.</p>
<p>How to represent different varieties of <em>effects</em> (filesystem, logging, network, etc.)?<br>
How to express them as function constraints that can be verified by the compiler?<br>
How to compose them? How to extend them?<br></p>
<p>These (and others) are still very much open questions in the Haskell community.
The traditional way of dealing with them are <em>monad transformers</em>,
but they suffer from many shortcomings<sup id="fnref:9"><a class="footnote-ref" href="#fn:9" rel="footnote">9</a></sup>.
More recent solutions like <a href="http://hackage.haskell.org/package/extensible-effects">effects</a>
or <a href="http://www.haskellforall.com/2012/06/you-could-have-invented-free-monads.html">free monads</a> are promising,
but exhibit performance issues that likely won’t be solvable without full compiler support.
And even so, you can <a href="https://markkarpov.com/post/free-monad-considered-harmful.html">convincingly argue against</a>
those new approaches,
which suggests that we may ultimately need something else entirely.</p>
<p>Of course, this state of affairs doesn’t really prevent anyone
from writing useful applications in Haskell.
“Regular” monads are still a fine choice.
Indeed, even if you end up stuffing most of your code inside plain <code>IO</code>,
it will already be a step up compared to most other languages.</p>
<h4>Good Enough™</h4>
<p>Incidentally, something similar could probably be said about the language as a whole.</p>
<p>Yes, it has numerous glaring flaws and some not-so-obvious shortcomings.<br>
Yes, it requires disciplined coding style and attention to readability.<br>
Yes, it will force you to courageously tackle problems
that are completely unknown to programmers using other languages.<br>
In the end, however, you will probably find it better than most alternatives.</p>
<p>Basically, Haskell is like pizza:
even when it’s bad, it is still <em>pretty good</em>.</p>
<p>But what’s possibly the best thing about it
is that you don’t even really need to adopt Haskell in order to benefit from its innovations
(and avoid the legacy warts).</p>
<p>There is already a breed of mainstream languages
that can aptly be characterized as “Haskell-lite”:
heavily influenced by <span class="caps">FP</span> paradigms but without subscribing to them completely.
The closest example in this category is of course Scala,
while the newest one would be Rust.<br>
In many aspects, they offer a great compromise
that provides some important functional features
while sparing you most of the teething issues
that Haskell still has after almost 30 years.
Functional purists may not be completely satisfied,
but at least they’ll get to keep their typeclasses and monoids.</p>
<p>And what if you don’t want to hear about this <span class="caps">FP</span> nonsense at all?…
Well, I’m afraid it will get harder and harder to avoid.
These days, it’s evidently fine for a language to omit generics
but it seems inconceivable to ditch <a href="https://gobyexample.com/closures">first-class functions</a>.
Even the traditional <span class="caps">OOP</span> powerhouse like Java
cannot do without
<a href="https://docs.oracle.com/javase/tutorial/java/javaOO/lambdaexpressions.html">support for anonymous (“lambda”) functions</a>
anymore.
And let’s not forget
<a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Using_promises#Chaining">all</a>
<a href="https://docs.oracle.com/javase/8/docs/api/java/util/Optional.html#flatMap-java.util.function.Function-">the</a>
<a href="http://msdn.microsoft.com/en-us/library/system.linq.enumerable.selectmany.aspx">numerous</a>
<a href="https://docs.oracle.com/javase/8/docs/api/java/util/stream/Stream.html#flatMap-java.util.function.Function-">examples</a>
of monadic constructs
that pervade many of the mature APIs, libraries, and languages.</p>
<p>So even if you, understandably, don’t really want to come to Haskell,
it’s looking more and more likely that Haskell will soon come to you :)</p>
<div class="footnote">
<hr />
<ol>
<li id="fn:1">
<p>In case of Go, I’m of course referring to a feature that’s notoriously <em>missing</em> from the language. <a class="footnote-backref" href="#fnref:1" rev="footnote" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:2">
<p>For a close analogue in languages other than Haskell,
you can look at the
<a href="https://doc.rust-lang.org/book/first-edition/procedural-macros.html">current state of procedural macros</a>
in Rust (commonly known as “custom derives”). <a class="footnote-backref" href="#fnref:2" rev="footnote" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:3">
<p>What seems to excite the Haskell community in 2018, for example,
are things like <a href="https://ghc.haskell.org/trac/ghc/wiki/LinearTypes">linear types</a>
and <a href="https://typesandkinds.wordpress.com/2016/07/24/dependent-types-in-haskell-progress-report/">dependent types</a>. <a class="footnote-backref" href="#fnref:3" rev="footnote" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:4">
<p>The obvious counterexample is <a href="https://en.wikipedia.org/wiki/Clojure">Clojure</a>
and its cousins in the Lisp family of languages. <a class="footnote-backref" href="#fnref:4" rev="footnote" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn:5">
<p>Although
<a href="https://www.reddit.com/r/haskell/comments/8ilw75/there_are_too_many_prettyprinting_libraries/">the abundance of pretty-printing libraries</a>
is high up there, too :) <a class="footnote-backref" href="#fnref:5" rev="footnote" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
<li id="fn:6">
<p>This can be mitigated somewhat by using a contiguous chunk of memory
through a dedicated <em>arena allocator</em>, or implementing the list as an array. <a class="footnote-backref" href="#fnref:6" rev="footnote" title="Jump back to footnote 6 in the text">↩</a></p>
</li>
<li id="fn:7">
<p>See for example <a href="https://theam.github.io/require/">this project</a>. <a class="footnote-backref" href="#fnref:7" rev="footnote" title="Jump back to footnote 7 in the text">↩</a></p>
</li>
<li id="fn:8">
<p>Some <span class="caps">GHC</span> extensions like <code>DisambiguateRecordFields</code> allow for correct type inference
even in case of “overloaded” field names, though. <a class="footnote-backref" href="#fnref:8" rev="footnote" title="Jump back to footnote 8 in the text">↩</a></p>
</li>
<li id="fn:9">
<p>To name a few:
they don’t compose well (e.g. can only have one instance of a particular monad in the stack);
they can cause some <a href="https://www.fpcomplete.com/blog/2017/06/tale-of-two-brackets">extremely tricky bugs</a>;
they don’t really cooperate with the standard library which uses <code>IO</code> everywhere
(often requiring tricks like <a href="https://github.com/fpco/unliftio">this</a>). <a class="footnote-backref" href="#fnref:9" rev="footnote" title="Jump back to footnote 9 in the text">↩</a></p>
</li>
</ol>
</div>Add examples to your Rust libraries2018-02-28T08:37:00+01:00Karol Kuczmarskitag:xion.io,2018-02-28:post/code/rust-examples.html<p>When you’re writing a library for other programs to depend on,
it is paramount to think how the developers are going to use it in their code.</p>
<p>The best way to ensure they have a pleasant experience
is to <em>put yourself in their shoes</em>.
Forget the internal details of your package, and consider only its outward interface.
Then, come up with a realistic use case and just <em>implement it</em>.</p>
<p>In other words, you should create complete, end-to-end,
and (somewhat) usable <em>example applications</em>.</p>
<h4>Examples are trouble</h4>
<p>You may think this is asking a lot, and I wouldn’t really disagree here.</p>
<p>In most languages and programming platforms,
it is indeed quite cumbersome to create example apps.
This happens for at least several different reasons:</p>
<ul>
<li>
<p><strong>It typically requires bootstrapping an entire project from scratch.</strong>
If you are lucky, you will have something like <code>create-react-app</code>
to get you going relatively quickly.
Still, you need to wire up the new project so that it depends on the <em>source code</em> of your library
rather than its published version,
and this tends to be a non-standard option — if it is available at all.</p>
</li>
<li>
<p><strong>It’s unclear where should the example code live.</strong>
Should you just throw it away, once it has served its immediate purpose?
I’m sure this would discourage many people from creating examples in the first place.
It’s certainly better to keep them in the version control,
allowing their code to serve as additional documentation.
<br><br>
But if you intend to do this,
you need to be careful not to deploy the example along with your library
when you upload it to the package registry for your language.
This may require maintaining an explicit blacklist and/or whitelist,
in the vein of <a href="https://docs.python.org/2/distutils/sourcedist.html#the-manifest-in-template"><code>MANIFEST</code> files</a>
in Python.</p>
</li>
<li>
<p><strong>Examples may break as the library changes.</strong>
Although example apps aren’t integration tests that have a clear, expected outcome,
they should at the very least <em>compile correctly</em>.
<br><br>
The only way to ensure that is to include them in the build/test pipeline of your library.
To accomplish this, however, you may need to complicate your <abbr title="continuous integration"><span class="caps">CI</span></abbr> setup,
perhaps by introducing additional languages like Bash or Python.</p>
</li>
<li>
<p><strong>It’s harder to maintain quality of example code.</strong>
Any linters and static analyzers that you’re normally running will likely need to be configured
to also apply to the examples.
On the other hand, however, you probably don’t want those checkers to be <em>too strict</em>
(it’s just example code, after all), so you may want to turn off some of the warnings,
adjust the level of others, and so on.</p>
</li>
</ul>
<p>So essentially, writing examples involves quite a lot of hassle.
It would be great if the default tooling of your language
helped to lessen the burden at least a <em>little</em> bit.</p>
<p>Well, good news! If you’re a <a href="http://rust-lang.org">Rust</a> programmer,
the language has basically got you covered.</p>
<p>Cargo — the standard build tool and package manager for Rust —
has some dedicated features to support <em>examples</em> as a first-class concept.
While it doesn’t completely address all the pain points outlined above,
it goes a long way towards minimizing them.</p>
<h4>What are Cargo examples?</h4>
<p>In Cargo’s parlance, an <em>example</em> is nothing else
but a Rust source code of a standalone executable<sup id="fnref:1"><a class="footnote-ref" href="#fn:1" rel="footnote">1</a></sup>
that typically resides in a single <code>.rs</code> file.
All such files should be places in the <code>examples/</code> directory,
at the same level as <code>src/</code> and the <code>Cargo.toml</code> manifest itself<sup id="fnref:2"><a class="footnote-ref" href="#fn:2" rel="footnote">2</a></sup>.</p>
<p>Here’s the simplest example of, ahem, an <em>example</em>:</p>
<div class="highlight"><pre><span class="c1">// examples/hello.rs</span>
<span class="k">fn</span><span class="w"> </span><span class="n">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">println</span><span class="o">!</span><span class="p">(</span><span class="s">"Hello from an example!"</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>You can run it through the typical <code>cargo run</code> command;
simply pass the example name after the <code>--example</code> flag:</p>
<div class="highlight"><pre>$ cargo run --example hello
Hello from an example!
</pre></div>
<p>It is also possible to run the example with some additional arguments:</p>
<div class="highlight"><pre>$ cargo run --example hello2 -- Alice
Hello, Alice!
</pre></div>
<p>which are relayed directly to the underlying binary:</p>
<div class="highlight"><pre><span class="c1">// examples/hello2.rs</span>
<span class="kn">use</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">env</span><span class="p">;</span><span class="w"></span>
<span class="k">fn</span><span class="w"> </span><span class="n">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">env</span><span class="o">::</span><span class="n">args</span><span class="p">().</span><span class="n">skip</span><span class="p">(</span><span class="mi">1</span><span class="p">).</span><span class="n">next</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="nb">println</span><span class="o">!</span><span class="p">(</span><span class="s">"Hello, {}!"</span><span class="p">,</span><span class="w"> </span><span class="n">name</span><span class="p">.</span><span class="n">unwrap_or</span><span class="p">(</span><span class="s">"world"</span><span class="p">.</span><span class="n">into</span><span class="p">()));</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>As you can see, the way we run examples is very similar to how we’d run
<a href="https://doc.rust-lang.org/cargo/reference/manifest.html#the-project-layout">the <code>src/bin</code> binaries</a>,
which some people use as normal entry points to their Rust programs.</p>
<p>The important thing is that you don’t have to worry what to do with your example code anymore.
All you need to do is drop it in the <code>examples/</code> directory,
and let Cargo do the rest.</p>
<h4>Dependency included</h4>
<p>Of course in reality,
your examples will be at least a <em>little</em> more complicated than that.
For one, they will surely call into your library to use its <span class="caps">API</span>,
which means they need to depend on it <span class="amp">&</span> import its symbols.</p>
<p>Fortunately, this doesn’t complicate things even one bit.</p>
<p>The library crate itself is already an <em>implied dependency</em> of any code
inside the <code>examples/</code> directory.
This is automatically handled by Cargo,
so you don’t have to modify <code>Cargo.toml</code> (or do anything else really)
to make it happen.</p>
<p>So without any additional effort,
you can just to link to your library crate in the usual manner,
i.e. by putting <code>extern crate</code> on top of the Rust file:</p>
<div class="highlight"><pre><span class="c1">// examples/real.rs</span>
<span class="k">extern</span><span class="w"> </span><span class="n">crate</span><span class="w"> </span><span class="n">mylib</span><span class="p">;</span><span class="w"></span>
<span class="k">fn</span><span class="w"> </span><span class="n">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">thing</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">mylib</span><span class="o">::</span><span class="n">make_a_thing</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="nb">println</span><span class="o">!</span><span class="p">(</span><span class="s">"I made a thing: {:?}"</span><span class="p">,</span><span class="w"> </span><span class="n">thing</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>This goes even further, and extends to any dependency <em>of the library itself</em>.
All such third-party crates are automatically available to the example code,
which proves handy in common cases such as <a href="http://tokio.rs">Tokio</a>-based asynchronous APIs:</p>
<div class="highlight"><pre><span class="c1">// example/async.rs</span>
<span class="k">extern</span><span class="w"> </span><span class="n">crate</span><span class="w"> </span><span class="n">mylib</span><span class="p">;</span><span class="w"></span>
<span class="k">extern</span><span class="w"> </span><span class="n">crate</span><span class="w"> </span><span class="n">tokio_core</span><span class="p">;</span><span class="w"> </span><span class="c1">// assuming it's in mylib's [dependencies]</span>
<span class="k">fn</span><span class="w"> </span><span class="n">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">core</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">tokio_core</span><span class="o">::</span><span class="n">reactor</span><span class="o">::</span><span class="n">Core</span><span class="o">::</span><span class="n">new</span><span class="p">().</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">thing</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">core</span><span class="p">.</span><span class="n">run</span><span class="p">(</span><span class="n">mylib</span><span class="o">::</span><span class="n">make_a_thing_asynchronously</span><span class="p">()).</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="nb">println</span><span class="o">!</span><span class="p">(</span><span class="s">"I made a thing: {:?}"</span><span class="p">,</span><span class="w"> </span><span class="n">thing</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<h4>More deps</h4>
<p>Sometimes, however, it is very useful to pull in an additional package or two,
just for the example code.</p>
<p>A typical case may involve <em>logging</em>.</p>
<p>If your library uses the usual <a href="https://docs.rs/log"><code>log</code> crate</a> to output debug messages,
you probably want to see them printed out when you run your examples.
Since the <code>log</code> crate is just a <a href="https://en.wikipedia.org/wiki/Facade_pattern">facade</a>,
it doesn’t offer any built-in way to pipe log messages to standard output.
To handle this part,
you need something like <a href="http://docs.rs/env_logger">the <code>env_logger</code> package</a>:</p>
<div class="highlight"><pre><span class="c1">// example/with_logging.rs</span>
<span class="k">extern</span><span class="w"> </span><span class="n">crate</span><span class="w"> </span><span class="n">env_logger</span><span class="p">;</span><span class="w"></span>
<span class="k">extern</span><span class="w"> </span><span class="n">crate</span><span class="w"> </span><span class="n">mylib</span><span class="p">;</span><span class="w"></span>
<span class="k">fn</span><span class="w"> </span><span class="n">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">env_logger</span><span class="o">::</span><span class="n">init</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="nb">println</span><span class="p">(</span><span class="s">"{:?}"</span><span class="p">,</span><span class="w"> </span><span class="n">mylib</span><span class="o">::</span><span class="n">make_a_thing</span><span class="p">());</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>To be able to import <code>env_logger</code> like this,
it natually has to be declared as a dependency in our <code>Cargo.toml</code>.</p>
<p>We won’t put it in the <code>[dependencies]</code> section of the manifest,
however, as it’s not needed by the library code.
Instead, we should place it in a
<a href="https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#development-dependencies">separate section</a>
called <code>[dev-dependencies]</code>:</p>
<div class="highlight"><pre><span class="k">[dev-dependencies]</span>
<span class="na">env_logger</span> <span class="o">=</span> <span class="s">"0.5"</span>
</pre></div>
<p>Packages listed there are shared by tests, benchmarks, and — yes, examples.
They are not, however, linked into regular builds of your library,
so you don’t have to worry about bloating it with unnecessary code.</p>
<h4>Growing bigger</h4>
<p>So far, we have seen examples that span just a single Rust file.
Practical applications tend to be bigger than that,
so it’d be nice if we could provide some multi-file examples as well.</p>
<p>This is easily done, although for some reason it doesn’t seem to be mentioned
in <a href="https://doc.rust-lang.org/cargo/reference/manifest.html#examples">the official docs</a>.</p>
<p>In any case, the approach is identical to
<a href="https://doc.rust-lang.org/cargo/reference/manifest.html#the-project-layout">executables inside <code>src/bin/</code></a>.
Basically, if we have a single <code>foo.rs</code> file with executable code,
we can expand it to a <code>foo/</code> subdirectory with <code>foo/main.rs</code> as the entry point.
Then, we can add whatever other submodules we want —
just like we would do for a regular Rust binary crate:</p>
<div class="highlight"><pre><span class="c1">// examples/multifile/main.rs</span>
<span class="k">extern</span><span class="w"> </span><span class="n">crate</span><span class="w"> </span><span class="n">env_logger</span><span class="p">;</span><span class="w"></span>
<span class="k">extern</span><span class="w"> </span><span class="n">crate</span><span class="w"> </span><span class="n">mylib</span><span class="p">;</span><span class="w"></span>
<span class="kn">mod</span><span class="w"> </span><span class="n">util</span><span class="p">;</span><span class="w"></span>
<span class="k">fn</span><span class="w"> </span><span class="n">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">env_logger</span><span class="o">::</span><span class="n">init</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">ingredient</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">util</span><span class="o">::</span><span class="n">create_ingredient</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">thing</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">mylib</span><span class="o">::</span><span class="n">make_a_thing_with</span><span class="p">(</span><span class="n">ingredient</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="nb">println</span><span class="p">(</span><span class="s">"{:?}"</span><span class="p">,</span><span class="w"> </span><span class="n">thing</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<!-- -->
<div class="highlight"><pre><span class="c1">// examples/multifile/util.rs</span>
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="n">create_ingredient</span><span class="p">()</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="kt">u64</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="mi">42</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>Of course, it won’t be often that examples this large are necessary.
Showing how a library can scale to bigger applications can, however,
be very encouraging to potential users.</p>
<h4>Maintaining maintainability</h4>
<p>Thus far, we have discussed how to create small and larger examples,
how to use additional third-party crates in example programs,
and how to easily build <span class="amp">&</span> run them using built-in Cargo commands.</p>
<p>All this effort spent on writing examples would be of little use
if we couldn’t ensure that they <em>work</em>.</p>
<p>Like every type of code,
examples are prone to breakage whenever the underlying <span class="caps">API</span> changes.
If the library is actively developed, its interface represents a moving target.
It is quite expected that changes may sometimes cause old examples to stop compiling.</p>
<p>Thankfully, Cargo is very dilligent in reporting such breakages.
Whenever you run:</p>
<div class="highlight"><pre>$ cargo test
</pre></div>
<p><em>all examples are built</em> simultaneously with the execution of your regular test suite<sup id="fnref:3"><a class="footnote-ref" href="#fn:3" rel="footnote">3</a></sup>.
You get the compilation guarantee for your examples essentially for free
— there is no need to even edit your <code>.travis.yml</code>,
or to adjust your continuous integration setup in any other way!</p>
<p>Pretty neat, right?</p>
<p>This saying, you should keep in mind that
simply compiling your examples on a regular basis is not a foolproof guarantee
that their code never becomes outdated.
Examples are <em>not</em> integration tests,
and they won’t catch important changes in your implementation
that aren’t breaking the interface.</p>
<h4>Examples-Driven Development?</h4>
<p>You may be wondering then, what’s exactly the point of writing examples?
If you got tests on one hand to verify correctness,
and documentation on the other hand to inform your users,
then having a bunch of dedicated executable examples may seem superfluous.</p>
<p>To me, however, an impeccable test suite and amazing docs
— which also remain comprehensive and awesome for an entire lifetime of the library! —
sound a bit too much like a perfect world :)
Adding examples to the mix can almost always improve things,
and their maintenance burden should, in most cases, be very minimal.</p>
<p>But I have also found out that starting off with examples early on
is a great way to <em>validate the interface design</em>.</p>
<p>Once the friction of creating small test programs has been eliminated,
they become indispensable for prototyping new features.
Wanna try out that new thing you’ve just added?
Simple: just make a quick example for it, run it, and see what happens!</p>
<p>In many ways, doing this feels similar to trying out things in a
<abbr title="Run Eval Print Loop"><span class="caps">REPL</span></abbr>
— something that’s almost exclusive to dynamic/interpreted languages.
But unlike mucking around in Python shell, examples are <em>not</em> throwaway code:
they become part of your project, and remain useful for both you <span class="amp">&</span> your users.</p>
<div class="footnote">
<hr />
<ol>
<li id="fn:1">
<p>It is also possible to create examples
<a href="https://doc.rust-lang.org/cargo/reference/manifest.html#examples">which are themselves just libraries</a>.
I don’t think this is particularly useful, though,
since all you can do with such examples is build them,
so they don’t provide any additional value over normal tests
(and especially <a href="https://doc.rust-lang.org/beta/rustdoc/documentation-tests.html">doc tests</a>). <a class="footnote-backref" href="#fnref:1" rev="footnote" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:2">
<p>Because they are outside of the <code>src/</code> directory,
examples do not become a part of your library’s code, and are not deployed to <a href="https://crates.io">crates.io</a>. <a class="footnote-backref" href="#fnref:2" rev="footnote" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:3">
<p>You can also run <code>cargo build --examples</code> to only compile the examples,
without running any kind of tests. <a class="footnote-backref" href="#fnref:3" rev="footnote" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
</ol>
</div>Unfolding a Stream of paginated items2018-01-27T18:04:00-08:00Karol Kuczmarskitag:xion.io,2018-01-24:post/code/rust-unfold-pagination.html<p>My <a href="https://github.com/Xion/ezomyte">most recent Rust crate</a>
is an <span class="caps">API</span> client for <a href="https://www.pathofexile.com/developer/docs/api-resource-public-stash-tabs">the Path of Exile’s public stash tabs</a>.
One problem that I had to solve while writing it was to turn a sequence of <em>paginated</em> items
(in this case, player stash tabs) into a single, asynchronous <code>Stream</code>.</p>
<p>In this post, I’ll explain how to use
<a href="https://docs.rs/futures/0.1.17/futures/stream/trait.Stream.html">the <code>Stream</code> interface</a>,
along with functions from <a href="https://docs.rs/futures">the <code>futures</code> crate</a>,
to create a single <code>Stream</code> from multiple batches of entities.</p>
<h4>Pagination 101</h4>
<p>To divide a long list of items into <em>pages</em>
is a very common pattern in many <span class="caps">HTTP</span>-based APIs.</p>
<p>If the client requests a sequence of entities
that would be too large to serve as a single response,
there has to be some way to split it over multiple <span class="caps">HTTP</span> roundtrips.
To accomplish that, <span class="caps">API</span> servers will often return a constant number of items at first (like 50),
followed by some form of <em>continuation token</em>:</p>
<div class="highlight"><pre>$ curl http://api.example.com/items
{
"items": [
{...},
{...},
{...}
],
"continuationToken": "e53c68db0ee412ac239173db147a02a0"
}
</pre></div>
<p>Such token is preferably an opaque sequence of bytes,
though sometimes it can be an explicit offset (index) into the list of results<sup id="fnref:1"><a class="footnote-ref" href="#fn:1" rel="footnote">1</a></sup>.
Regardless of its exact nature, clients need to pass the token with their next request
in order to obtain another batch of results:</p>
<div class="highlight"><pre>$ curl 'http://api.example.com/items?after=e53c68db0ee412ac239173db147a02a0'
{
"items": [
{...},
{...}
],
"continuationToken": "4e3986e4c7f591b8cb17cf14addd40a6"
}
</pre></div>
<p>Repeat this procedure for as long as the response contains a continuation token,
and you will eventually go through the entire sequence.
If it’s really, <em>really</em> long (e.g. it’s a Twitter firehose for a popular hashtag),
then you may of course hit some problems due to the sheer number of requests.
For many datasets, however, this pagination scheme is absolutely sufficient
while remaining relatively simple for clients to implement.</p>
<h4>Stream it in Rust</h4>
<p>What the client code would typically do, however,
is to hide the pagination details completely
and present only the final, unified sequence of items.
Such abstraction is useful even for end-user applications,
but it’s definitely expected from any shared library that wraps the third-party <span class="caps">API</span>.</p>
<p>Depending on your programming language of choice,
this abstraction layer may be very simple to implement.
Here’s how it could be done in Python,
whose concepts of <em>iterables</em> and <em>generators</em> are a perfect fit for this task<sup id="fnref:2"><a class="footnote-ref" href="#fn:2" rel="footnote">2</a></sup>:</p>
<div class="highlight"><pre><span class="kn">import</span> <span class="nn">requests</span>
<span class="k">def</span> <span class="nf">iter_items</span><span class="p">(</span><span class="n">after</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
<span class="sd">"""Yield items from an example API.</span>
<span class="sd"> :param after: Optional continuation token</span>
<span class="sd"> """</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="n">url</span> <span class="o">=</span> <span class="s">"http://api.example.com/items"</span>
<span class="k">if</span> <span class="n">after</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span><span class="p">:</span>
<span class="n">url</span> <span class="o">+=</span> <span class="s">"?after=</span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="n">after</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
<span class="n">response</span><span class="o">.</span><span class="n">raise_for_status</span><span class="p">()</span>
<span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">response</span><span class="o">.</span><span class="n">json</span><span class="p">()[</span><span class="s">'items'</span><span class="p">]:</span>
<span class="k">yield</span> <span class="n">item</span>
<span class="n">after</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">json</span><span class="p">()</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s">"continuationToken"</span><span class="p">)</span>
<span class="k">if</span> <span class="n">after</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
<span class="k">break</span>
<span class="c"># consumer</span>
<span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">iter_items</span><span class="p">():</span>
<span class="k">print</span><span class="p">(</span><span class="n">item</span><span class="p">)</span>
</pre></div>
<p>In Rust, you can find their analogues in the <code>Iterator</code> and <code>Stream</code> traits,
so we’re off to a pretty good start.
What’s missing, however, is the equivalent of <code>yield</code>:
something to tell the consumer “Here, have the next item!”,
and then <em>go back</em> to the exact same place in the producer function.</p>
<p>This ability to jump back and forth between two (or more) functions
involves having a language support for <em>coroutines</em>.
Not many mainstream languages pass this requirement,
although Python and C# would readily come to mind.
In case of Rust, there have been some nightly
<a href="https://github.com/rust-lang/rfcs/blob/master/text/2033-experimental-coroutines.md">proposals</a>
and <a href="https://docs.rs/futures-await">experiments</a>,
but nothing seems to be stabilizing anytime soon.</p>
<h4><span class="caps">DIY</span> streaming</h4>
<p>But of course, if you do want a <code>Stream</code> of paginated items,
there is at least one straightforward solution:
just implement the <code>Stream</code> trait directly.</p>
<p>This is actually quite a viable approach, very similar to rolling out a custom <code>Iterator</code>.
Some minor differences stem mostly from a more complicated state management in <code>Stream::poll</code>
compared to <code>Iterator::next</code>.
While an iterator is either exhausted or not,
a stream can also be <a href="https://docs.rs/futures/0.1.17/futures/enum.Async.html">waiting</a>
for the next item to “arrive” (<code>Ok(Async::NotReady)</code>),
or have errored out permanently (<code>Err(e)</code>).
As a consequence, the return value of <code>Stream::poll</code> is
<a href="https://docs.rs/futures/0.1.17/futures/type.Poll.html">slightly more complex</a>
than just plain <code>Option</code>, but nevertheless quite manageable.</p>
<p>Irrespective of difficulty,
writing a custom <code>Stream</code> from scratch would inevitably involve a lot of boilerplate.
You may find it necessary in more complicated applications, of course,
but for something that’s basically a glorified <code>while</code> loop,
it doesn’t seem like a big ask to have a more concise solution.</p>
<h4>The stream unfolds</h4>
<p>Fortunately there is one!
Its crucial element is the standalone
<a href="https://docs.rs/futures/0.1.17/futures/stream/fn.unfold.html"><code>stream::unfold</code> function</a>
from the <code>futures</code> crate:</p>
<div class="highlight"><pre><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="n">unfold</span><span class="o"><</span><span class="n">T</span><span class="p">,</span><span class="w"> </span><span class="n">F</span><span class="p">,</span><span class="w"> </span><span class="n">Fut</span><span class="p">,</span><span class="w"> </span><span class="n">It</span><span class="o">></span><span class="p">(</span><span class="n">init</span><span class="o">:</span><span class="w"> </span><span class="n">T</span><span class="p">,</span><span class="w"> </span><span class="n">f</span><span class="o">:</span><span class="w"> </span><span class="n">F</span><span class="p">)</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">Unfold</span><span class="o"><</span><span class="n">T</span><span class="p">,</span><span class="w"> </span><span class="n">F</span><span class="p">,</span><span class="w"> </span><span class="n">Fut</span><span class="o">></span><span class="w"> </span><span class="n">where</span><span class="w"></span>
<span class="w"> </span><span class="n">F</span><span class="o">:</span><span class="w"> </span><span class="n">FnMut</span><span class="p">(</span><span class="n">T</span><span class="p">)</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="nb">Option</span><span class="o"><</span><span class="n">Fut</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">Fut</span><span class="o">:</span><span class="w"> </span><span class="n">IntoFuture</span><span class="o"><</span><span class="n">Item</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="n">It</span><span class="p">,</span><span class="w"> </span><span class="n">T</span><span class="p">)</span><span class="o">></span><span class="p">,</span><span class="w"></span>
</pre></div>
<p>Reading through the signature of this function can be a little intimidating at first.
Part of it is Rust’s verbose syntax for anything
that involves both trait bounds and closures<sup id="fnref:3"><a class="footnote-ref" href="#fn:3" rel="footnote">3</a></sup>,
making <code>stream::unfold</code> seem more complicated than it actually is.
Indeed, if you’ve ever used <code>Iterator</code> adapters like <code>.filter_map</code> or <code>.fold</code>,
the <code>unfold</code> function will be pretty easy to understand.
(And if you haven’t, don’t worry! It’s really quite simple :))</p>
<p>If you look closely, you’ll see that <code>stream::unfold</code> takes the following two arguments:</p>
<ul>
<li>first one is essentially an arbitrary initial value, called a <em>seed</em></li>
<li>second one is a <em>closure</em> that receives the seed and returns an optional <em>pair</em> of values</li>
</ul>
<p>What are those values?…
Well, the entire purpose of the <code>unfold</code> function is to create a <code>Stream</code>,
and a stream should inevitably produce some <em>items</em>.
Consequently, the first value in the returned pair will be the next item in the stream.</p>
<p>And what about the second value? That’s just the <em>next state</em> of the seed!
It will be received by the very same closure
when someone asks the <code>Stream</code> to produce its next item.
By passing around a useful value — say, a continuation token —
you can create something that’s effectively a <code>while</code> loop from the Python example above.</p>
<p><div class="graphviz" style="text-align: center;"><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAjQAAACtCAYAAACnfeWhAAAAAXNSR0IArs4c6QAAPAFJREFUeAHtnQn8VcP7x0dJKrJFWVK0oewUP0QLSYsliRCJUEmWUMjeJnuI7GuiQiVLUQmVsqWNEBJZ2hRlm//zefznOvd+73Lu9r3n3O9nXq/v9947Z87MM++Zc85zZp5nZhMrwTCQAAmQAAmQAAmQQIgJlAux7BSdBEiABEiABEiABJQAFRp2BBIgARIgARIggdAToEIT+iZkBUiABEiABEiABDaNRfDkk0+a8ePHx0bzd8AJNG3a1PTq1SvgUpYUb/bs2WbYsGElDzCmzBFo1KiRGTBgQCjrPXToUDNnzpxQyl4oobt06WLatm1bqOJZbhESKDFC88knn5gpU6YUYVWLt0ozZswwUAzCGJYtW2aef/55Q9v0MLZe7mSeO3eumTp1au4yLOWc3nnnHTNr1qxSLjW8xU2YMMEsXrw4vBWg5IEkUGKEBlLWrVvXjB49OpACU6iSBNq3b18yMmQxo0aNMuXLlw+Z1BQ3VwR69OhhFi1alKvsCpIPRkkxws2QmkCtWrVSJ2IKEkiTQIkRmjTPZ3ISIAESIAESIAESKDgBKjQFbwIKQAIkQAIkQAIkkC0BKjTZEuT5JEACJEACJEACBSdAhabgTUABSIAESIAESIAEsiVAhSZbgjyfBEiABEiABEig4ASo0BS8CSgACZAACZAACZBAtgSo0GRLkOeTAAmQAAmQAAkUnAAVmoI3AQUgARIgARIgARLIlgAVmmwJ8nwSIAESIAESIIGCE6BCU/AmoAAkQAIkQAIkQALZEqBCky1Bnk8CJEACJEACJFBwAlRoCt4EFIAESIAESIAESCBbAlRosiXI80mABEiABEiABApOgApNwZuAApAACZAACZAACWRLgApNtgR5PgmQAAmQAAmQQMEJ5ESh+fLLL80555xjli1blnaFbr/9dnPfffelPM9vOpfRn3/+aaZMmWIuueQS88orr7hofpJAmSewdu1aveYuuOACc+WVV5qVK1emZOL3elq3bp0ZP3685psyUyZISMAv74QZ8AAJlEECOVFoPvjgA/Poo4+aefPmpY3wkUceMU888UTK8/ymcxlBltGjR5s777zTLF++3EXzM48E7r33XrLOI99cZY2Xj4YNG5obbrjBPPnkk3qNpMrb7/X06quvmt69e5tRo0alyjKwx3E/Wrx4cUHl88u7oEKycBIIGIGcKDQnn3yy+emnn0zr1q3Trt6sWbPMW2+9FXUe8sKN0RvipfMej/1+wAEHmJ49e8ZG5+y3HyUstrB49YpNE+bfl112mdlll13MEUccYR566CGzatWqglQnk7YpiKAFKHT27NnmpZdeMk2bNjXVq1c3n3zyibn66qtTSuL3esK9oHHjxmbTTTdNmWdQE9xxxx1mjz32MHvvvbe57bbbMhp5zrZufnlnWw7PJ4FiIpAThQZAqlWrlhGXKlWqmEqVKkXO/fvvv03nzp3N0qVLI3H4Epsu6mCCH+6muskmmyRIkVk0FLD+/fundXKieqWVScATW2sN/t555x1z/vnnm+233960bdvWPPfcc+a3334rFekzaZtSESwghcyfP9+UK1fOuGsC123FihV9Sef3ekL++Atr+Oeff1T0Tz/91Fx11VWmZs2a5n//+5954IEHzC+//FJq1fLLu9QEKmBBuH/MnDlTR90LKAaLDjiBnLxG4QYwbdo0s8UWW5iDDz5Yq/ztt9+asWPHmosuusgsWLBA3wp33XVXc/rpp0fd7H788UczYcIEtcHZuHGjHp88ebLZYYcd9Kbbvn17s+OOOxpvOi/Tzz77TDs63jQPO+wwc+KJJ3oPZ/wd5U2cOFHLrVOnjsEb0+67766jSccff7zKhhvcTjvtZNq1a6flJJIlWb1wIuqLEahtttnGdOrUyWy33XYZyx2EE51iA1kw0gYbJjw00TZo/2OOOcZUqFAhY1HTbRuMFD377LOmR48eZtKkSToqgdEkPDAwHQkZYf+F/tOiRYsouX7//XczdepUg2nV8uXLmzPPPNPsvPPOkTQ4jhEP9FPIhbq6PoH0K1asMC+//LL2+Y4dO5qqVatGzh03bpz566+/DOJd8MZ9//33eg3BnuLoo4/WaSIobB9//LEmP+mkkwyuKYRU1xtsW55++mmVBdcr+i4C+i7kRUAd3377bVU+0d/RTk7x0QQJ/sEG54UXXtCXkIMOOkiVWj/nJcguUNFoHwQ8TDG6hVFftAX6Ae4DeNHKJsyZM8dMnz7dbNiwwRx33HFmv/32S5ndr7/+qv1s4cKFqmyhnaB0eUO8awQvjrgn++lPya4ZlJPPexb604cffqh/6JO4N3711Vfar5o1a2ZOOeUUb1X5nQT+IyAPn6hw+eWXWxkyjopL9kPe+KwMM1vJ0d5///2aVG7gVt7ONU6Gb23Xrl2tvKnr74EDB2oauVFYsbuxW265pZWhb41bvXq1HTlypKbr27evlZu3/fnnn0ukc/Ig76OOOsrKDdpKh7e1a9e2YmDsDlvIBrlk+iMS5+eLXMz2wAMPtHLjsJDztNNOs88//7yeKhealQef1g/y4TdCMlni1QtliKJjzz33XCsPW/vRRx8pR3ljVrk1U5//5KFk5QbrM3X+km222WbKG8zj/YkCofHyULcygmNFCbbyINQ4cPYT0m2bxx57zFauXNmi7Hvuucfuu+++Wp4oBfbNN9+05513npWbphV7KysKuRWlJyIG2l+UF+2HkO+mm26ytWrVsvK2qGlE0bH16tXT/GRqwnbv3t2i36K8Dh06aF8WBc6eeuqpVh7wFu3kDchblFdvlJbnjYNcsX1YbF807rXXXtNz/Vxv6Gvvv/++9mXIgu/4EwNhzUOM5608KOwXX3yhPPbZZx+9tnD9uRDvelq0aJGVlxj77rvvWnlQWlGUrCivtn79+u40X58XXnihlYeVr7T5ToS6x+u/Lk4UVW1P1BPMwB/3tzPOOCMt0a655hp7/fXXa3/C9Y8+2qdPn0ge8XgjnUyF2TFjxlhRWuywYcO03z7++OOR85JdI376U7JrJlf3LFHEVfZvvvnGyguBcgDDGjVqRNjLS4+Vkb7Ib3BHP2EggUQEsh6h2WuvvcyAAQP0DU0ueA146+vWrZsZPHiwzkPLRarxoiQYuRBNv3799G337LPPVo8ITFEgbLXVVpERHsxhi7Ki8bHpNFL+wQi1VatW+hYpyoy+3WC0Rzq9S5LR51NPPaWjTRhxQrjlllv0DQ3f8QaFqRS5ECPyIT6ZLInqhfl5vO3LAw9ZGMzd403r0ksvLWFDpAmS/JMbXcHfXNxQfSIxRSnQQ/CygZE3RgnABgFvZHi7TxXSbRu01xtvvKGjE2ANTvIQVlsfjBhhZA9v2fvvv78RBUG9f/D2fcghh+jIC0ZJ9txzT+2v6NfXXnutwVQERiKPPPJI7WtoL4yU4BMBIzPo+5g6hbwIGOWTh48BIzcdg2vBMdFE8i82DtdXbICs3uDnehNlU/nKA0OvFy9r2Bw9/PDD2qdde4gCbxo0aGBw7cJwOFE466yz9Do49NBDNYkoiGbIkCGJkieNF+Wp4H0YAqLNkwVMHyNg5BUjHnA+QJvvtttuOoKLfuHaOFE+OO9RcaRwnqGiaOso34wZMxKdYv744w+9V2CEAqNzCBhpxCgGuKNN0V+SXSN++hPaNN41g3tytvcsUaIN+hZGLmG7JS/QWg/0T4wcyYNKf+MffnsDmIqi7I3idxKIIpC1QoPc4s3BO7sYXAQu4GLCQ8Mb4p2L47FD1vHSYSrADfliWgvD7nhYZhsgM6bQ5I1LlQzcqNywvMs7Vj6/snjPgys6bkJe42U8RPy40To5ytpnJm3j2g5TBAjIQ0YCDaaLrrjiigjCH374QRWPJUuWqEIjI3M61QjjWUwJoE8gfP755xHF2ykAMCB1AW2IgIeUCygTD0BMccFwGqFJkybucOQzXlzkYJIvfq+3eFnAExDyubogDR4c6Pd4OEJZ906VuTxkhEunA6677joXpdctlD0ojgyJCeAlqU2bNlEJMG3nlKWoA///A1OjUMahbHsDXuqeeeYZVUqhcPi5Rrznx/se75pBumzvWZjKxAsBFHlvXaGspQpQcKjQpKJUto/nRKHxixBvMV4NPNl53gd/onR443799dfVBgdvRXgLnjt3bqLkvuObN2+ubw64OcD+4a677jIybRZ1fqx8fmVx58k0lD7cZMopYoMTVUCaPzASUWjvnnhKp7casFnBjQwPR4xKwZ4Gnl/wjIkddfCe5/2eSdu4t2X3ifxkOF9ts/CwThSQHsoMRiA333zziBKTaiQqHgdnM7R+/fpExeU83s/1husRthgweo0N8FaTqVx9iMJzKTY4W55GjRpFHXJ9PCrSxw8YKGO0o9ABiij6ZaIArugDGFWAkox+DAV56623NrDxSBXwIEf/Q7/3BnDDNZIo4KUNwY0cu3RoJwS0I4Kfa0QTJvnnrhX3iaS5uGeBHe6lUIKh3MB+CC+hXuUmiViqQMuUYOSlIFlaHit7BALriuDnpojh/5tvvlmHuMVmQYd9c9GEuIhvvfVWHU2CQTLW7YgdRo+Vz68s7jx3o8hk7Z5c1LG08sANDHXGQx43cEwJYmoB001wHXY8/MqTSdvEyxtyYa2R2GFtb1o8zKFo4WEOrzaxn/EeTvg9WZ2SHUuYYR4PQB4Yo2MqIPahIvZBWjKOxwtuNBRGm7EhaPWMlS/d36gP+gz+WrZsqdNw8HiCBx8Mwt317CdfKJFQiLAAYTph22231eTvvfde1Gnol1CYXTv5uUaiMvD5w9UxF/cs8IQihlF1sSPSkfZkyhxERPkYIXJeZ2KzydFsn21XVpIFTqFxN8LYm2tsg+BhA2UG00JuuD3Vm3NsHol+w54AecGbAbYd8HwRg9JIcsjolc+PLLH1wigFhvRxUWLqwxswzA8bnbAG1BU3H9z8jz32WB0OxzQaPI0wzO5GKzKpX7ptk6gMvIVjtGTEiBFRSfAW6lauxo0WCg9czxFy1b+iCkzxw93kMeWVr4BpLnjOoK97A2wz4G0I7754wU2zYeqpWIPjj2kejObB9gNTPxiVcdPd6dYdecIuC55TWGXdG+CJFns/cMfddCRGNbwBUzjop86OKdk14uqTSX/Kxz0LDMU4Wu93sEHDqJeT0VtHfIedGpRIeJ+iT8L+BlNjsHkTo/TY5PxdBgnkRKGBbQAC3rxdcG9v3rlRHEda77QTfq9ZsyZiHIkREQS8hSAdjDYRYtPBDRUBK5KiLLib4kIXC3+DY7hBI18El1Z/+PgHGwkYxSGIx4o54YQTotbZgYywt8DNSLxC1F0XaZPJEq9e4hGjRoEYIoYNDh4oGIqF3M4dF/mGJUCRwR/cnzECg2F7jMhgegkccxHSbRsoLW6ax7uGCNzj8aaHmyJG4zBcj+kO8VTSGyRkxXkwEIUrNvquU3RgBwPFBwH9DMFdA/ju+pvXFsrJ4H2QwHgdhsPeEBsHm4HaYvCOvvX111/r9A+MKhHQX5yS5fd6++677/Qc75QKDJgxguY1/kW+uAZxDIopQuz1hJEJ2GvgPPeQBRvYGsHYFddurNGzZhTwf24kAlNpgwYN0lEEPDCxtlKullTAdY77G6aoMFWM5QTOFicJxLkXtFjeUMJhsAvW3hceGBJjNA19FyHZNeK3P7n+6r1mkHe+7lkYfcIo+NKlSw2m4dHnYl98YIMJhQfu7Xjpg3I5fPhwXRYE9xwofLBDctcE5GUoYwTkAooK6bpty1tGxG1bbgBWHmBWHs5WNGh1t4NbsjwU1DVZNHyNkzdfdRe9++671W1VkFsxzrTSQVUWGRHRdHDjlGkBmyidTAWpq2PdunWtvGmrC7B0eCsKghXbGivGcpqPTBtYeShF1TPZD7GZsHKjVjdfMbazspS7urK6c+CuLW8RVubMVTbEJ5NFbgp6qrde8nBSd3Px+NK8wAB5ykJeVkZ/XFG+PuEOHAS3bbm5WHlg+pLZJYL7KeouDz4XlfQz3baBy77YN2kZcLGV6ZFI/mKToO7FKB9/6L9w4XYBrshw05aHvRWPKAsXU7jzy9C+FQ8VdVWWh4yeKw8aKwquuniLbYDGyWiUuuAjH3nDj8gg6xVpEXAPFmUlqu7x4lAH9DW4lWMJAbi7i2Gxuvji+vB7vYnxr5UHh8oh+zhZUVhcVa28EKgscBuGG22XLl2sjEhEjoNbvOtJRifVbRv8cM2Lgqbu6Ycffrgu4yCjDZE8kn2BOy6u9yAEuEDDHT2dIMpd2m7bWKIC7Qp2uDfiHuZCIt7gKU4EVrausHCvRt9AP0PfdCHVNZKqPyW7ZkRZsLm4Zzm3bSdz7CeWD0BfkhckvS+KgmPFoys2WeQ3rjHx/FI3bzHKt6LwRF1XkYT8UtQE8EYQFdJVaKJOztEPXDTyhucrN3kzjUonb8BRvzP5IcO3ehoULHkTj5sF4mPLjv0dK0uiemFdExk2tvJWFLesVJFBUWhSyRnveLoKTaZtE69sFydvhRYKZrwA5VJGXCKH0IYyGhP5nc0X9I/YvOLFoQw8yFz/klHPtJVeP3KibniQY32a2L6b6nysieI4Yf2edEOQFJp0ZUf6TBQanIf+hb6X7ksM7j+y3IUVGxRkExX8XCPZ9qds71mpFBpXIbHXUYUNSh/WOEoVoOC7dZ9kWk+V81Tn8HjxEEhsUi89qFAB0xbwGvITZGG+qGTxPExcAsy94i9ZQLlubxvYDyQKXhdXlyaVLInqhSFmbBbIkJqAm19Pt22S5SyjMAkPY/rBayuBNsSwdy5CvL4aLw5lwcsKfwixQ/EamYN/qJtzOU83O6zN5EKsF46L52dJAuhfmUwv4/4TzzMNJfi5RrLtT6V1z8K0H6atYW/kjJ5LUvwvBlNqsNXDlhVw1IAXGoyPMTUF7yiG4iYQSIUmX8hhhIs562QhnqKSLD2PkQAJkAAJ5JdA7No7qUqDvRGW3IAiBGNjLOqKbXiww3zsi2eqvHg8PATKlEIDo7J4K2WGp7koKQmQAAmQgF8CUISwEj08vzBqAzd7rMiO1ZYZio9ATrycig8La0QCJEACJFAMBDCVCs8prDvVunVr9brEdiHwkmIoLgJUaIqrPVkbEiABEiCBOATgci8eXLqsAFZdxjpKL774YpyUjAorASo0YW05yk0CJEACJJA2ARgJY68xjNJgg1pspOzWk0o7M54QKAJUaALVHBSGBEiABEgg3wRgGAy7mnHjxukWFNgLjxuq5pt6/vOnQpN/xiyBBEiABEgggASwCjz2pqotq3HDDV4WVQyglBTJLwEqNH5JMR0JkAAJkEDREahevbqRleXVrRvbT/To0cN4t+wpugoXcYWo0BRx47JqJEACJEACqQlg7yjsJYW9oLBP1JFHHqn7uKU+kymCRIAKTZBag7KQAAmQAAkUjECHDh3M7NmzdZNjbHaJncwZwkOACk142oqSkgAJkAAJ5JkAdpDHTvNYWV42WTVTpkzJc4nMPlcEqNDkiiTzIQESIAESKAoC2DcKdjXHHXecLsZHY+FwNGuZ2vogHE1CKUmABEiABApNABvFPv300wab18JYeNmyZZGNiwstG8uPT4AKTXwujCUBEiABEijjBLBtwqBBg3RH9J49e5r169ebgQMHlnEqwa0+FZrgtg0lIwESIAESCACBCy+80FSpUsWcc8455vfff9cNLgMgFkWIIRBXoVmyZAl3I40BFeSfc+bMMS1btgyyiCllO/XUUw3ehhjKJoG5c+fq0H5Ya491S2A8yl2c/bXgTz/95C9hgFJ16dLFYBrqjDPOMBs2bDD33Xcf71kBah+IUkKh2WeffUyLFi0CJibFSUYAlviNGzdOliSwx3bZZRfTsWPHwMpHwUqHwIEHHmgaNWpUOoXluJS1a9fqarOFXozts88+M3///bfZc889c1zD3GfXtm1b06BBg9xnnOccO3XqZDbffHNVXNHe2OySL2J5hp5G9ptYCWmkZ1ISIAESIAEPgX79+umS+W+//bapU6eO50jpfsUoLQxYsUcRQ34JvPLKK7qx5fnnn2/uvvvu/BbG3H0ToNu2b1RMSAIkQAIlCdx4443m3XffLagyA6mwCFxYR7lKUg12DNy54QGFaadrrrkm2MKWIelKTDmVobqzqiRAAiSQNYEKFSro5oZZZ5RFBr/88otZsWKFadiwYRa58NR0CJx88sk65QRD4a222sr07ds3ndOZNg8EqNDkASqzJAESIIHSJOCW6OcITWlSN7o+DWyoLr74YlVqunfvXroCsLQoAlRoonDwBwmQAAmEjwAUGqxuu9NOO4VP+JBL3Lt3b7N69WrdpXvnnXc2bdq0CXmNwis+FZrwth0lJwESIAElMH/+fE43FbAvDBgwwHz99dcGy0/AOHy//fYroDRlt2gaBZfdtmfNSYAE0iRQaNfsROJSoUlEpvTiR4wYoctnwCX9u+++K72CWVKEABWaCAp+IQESIIHEBKZPn27q1atnfvjhh8SJCnRk4cKFoVh/pkB4SqVYGIePGTPGVK1a1UCpWbduXamUy0L+I0CF5j8W/EYCJEACcQlguftu3boZLDxao0aNuGkKFbly5UqDlXfDsKBeoRiVVrlbb721mThxolm+fLmuKMxl3kqL/L/lUKEpXd4sjQRIIIQEsAIv3rofeOCBwEmP0RkEKjTBaJrddtvNjB071mDxPW5kWbptwpWCS5c3SyMBEiCBnBLA8vuXXHKJgfswl+HPKdqsMrvnnntMnz59zKRJk8wxxxyTVV482R8BjtD448RUJEACJBBIAhihwb5IVGaC1TwXXXSRej117txZPaCCJV1xSkOFpjjblbUiARIoIwRoEBzchh45cqSuDdShQwfdoTu4khaHZFRoiqMdWQsSIIEySgC7bIdx5+qy0FyVK1dWe5olS5Zwa4RSaHAqNKUAmUWQAAmQQD4IYF2cpUuXmvr16+cje+aZAwJ169Y1WKNm+PDh6gGVgyyZRQICVGgSgGE0CZAACQSdwJdffmnggUWFJtgthRWEzzzzTIONLLGJKEN+CFChyQ9X5koCJJAFgSeeeCKLs7M/ddy4cebmm2/OPqM854DpJgQs+McQbAL33nuv2WKLLUzXrl0N16fJT1tRockPV+ZKAiSQIYG33nrL9O/fP8Ozc3Pa9ddfbxYsWJCbzPKYy+LFiw02RKxSpUoeS2HWuSCw5ZZbmqeeesq8/vrrBi7dDLknwM0pc8+UOZIACQiBd99918DGAwu+Pf744+aoo47SvW4AZ/LkyWbWrFm6Q3SnTp3Mdtttp8ygzBx//PHqgoxF7Nzu0V988YW+3Z577rnm119/NRjB+fPPP82OO+5ocL4L8co84IADDPItV66cOfTQQ8348eMNFAFMA8SbqkFZ2IsHMgc9YIQmXh2CLndZlQ/979prrzVXXXWVOe644wzsaxhyR4AL6+WOJXMiARIQAth1uEePHrpSau/evfX3G2+8YVq1amVGjRplevbsaVq0aKGKDqZ1pk6daqZNm2b22msv89FHH5levXoZPKhHjx5tsJQ8di5u1KiRWbNmjfn222+VMZSaXXbZRXeYhhKTqMxmzZoZvBmj3NNPP12H+rfffnv9DQXn008/Ndtuu22JdoMittlmm5WID1oE6geFJogrGAeNVVDkgSJ+0EEHab978803uX5QLhtG5vIYSIAESCCnBD7//HMr9ykroyP2r7/+sj/++KOV/YbssGHD7HXXXRcpSxQUTSfKTiTuhBNOsDVr1oz8xpeTTz7ZigITFYe85Y03EpeoTNmHScuQh7+Vh4mmf/nllzVORmsi54fxy6677mqHDBkSRtHLtMzvv/++LV++vBVFtExzyHXlOeWUS+2QeZEACSgBN1XUpk0bIzdug1ERhNtvv13fTjFK4wLWUMEGi96Qyaq3icpEvsivTp06ZtNN/73lYTQI4ZtvvtHPMP7buHGjWbZsmdYrjPKXZZkxQnPppZeaK664wuAagR0UQ/YEqNBkz5A5kAAJxBDAdA4ClBkXVq9erbsQww6mXbt2LjruZyYKTbwy42YukU4ueUNMlCTw8Vh/5p9//jG777574GWlgCUJ3HDDDQbedJiefemll0omYEzaBP6966R9Gk8gARIggfQIOIVj3rx5KU/MRKFJmWmRJYDxMgJGnhjCR6BSpUrmwQcfNDL9qX/hq0HwJKZCE7w2oUQkUJQEqlatanbbbTdz//33G7Friaoj3Fnd9A+UGSwW5w2YKtqwYYM3qsx/h0JTrVo1A64M4SQAo25422H6CVOIDNkRoEKTHT+eTQIkEIfA+vXrNfbnn3+OOtq3b1+1+2jevLl6N3344YdGjITVg0kMXDUtXLF/+OEHg1Vw8dBGXsccc4xBXo8++qj+xucvv/yiaVatWqXnJSpz3bp16t0EzyUXnFyxipU7HobPr776itNNYWioFDLeeuut5vvvv1f7shRJeTgFASo0KQDxMAmQQHoE4HKNN04EuF7fddddumYMfl9wwQWmX79+Zs6cOQZvp40bN9a1ai688EIc1tCxY0dVQA488EB1/caicYg75JBDdOn4gw8+WN25cRwu3WPGjFE373hlQsm5+uqrNV8saDZhwgS14xk4cKDGYWRo7ty55sknn1QDzX8lCMd/2NDUqlUrHMJSyoQEsPwArolbbrlF1z9KmJAHUhLgOjQpETEBCZBArglgZAQjMJiCwo7EsQFrzsDmBmvIeIO4fkc8pjAFtfnmm3sPZ/wdylX16tV1fZqMMynlE6HQYT2foUOHZlUyFins0qVLVnnw5OwIoC/D8+5///ufriacXW7FeTaufbx8HHvssQkryBGahGh4gARIIF8EYBDZsGHDuMoMytxqq61KKDOId+7f+J4rZQYPE4x2YBQoTCEXIzRB2GYiTMzzJSv68m233WaefvppI2vU5KuY0OYLm7rOnTvrdZqsEuVlz5LrkyXgMRIgARIoZgIwOO7Tp4/ZY489QrNqK+yCBgwYoC6/WCkYqyXD3giK4n333WcqVKgQWdtk+fLl5vnnn9ctH2SRw4jdjdtmArZFWC0ZdhwwMMaIDfLD6NgOO+yg20bArXjmzJm6FQWUTYREZWI158cee0ynE+fPn29GjhypKznvvffeoeFbiP6OLUIwLTpjxgzdmdsrA0Y0sV0IpnDBHSObXmNw2JHBrgzTsZMmTTJjx47VKVqMcqKvPPzwwxoHw3v0DWw1gmPYBgTpsWI2tgjBCtwPPfSQloHVt7FCtwsLFy7UdJjmfOWVV3T6FtNl6A9YPgByP/fcc7okAuK9IVEfRBr0ySlTpqiyUqNGDZUT8sOWDnLCWPq0004zEydO1LJg/4Z1e2JHb7W8XK/Ux/xIgARIgATyS0Bc33Wl49dee83KnkD6XbaZsLIPlpUpPHviiSeqALK0vj3vvPPsBx98YOVhaGW3ZyvrnugxMci2hx12mJVRLyvKjcVvBKSTh4OVB5v+xj9ZMyVSnowMJSwTKzAjP5x/xx13WNlZ2rZt21Z/i91SJD9+iU8A7QB2+HRBlAwrD3CNw6rbN910kxWlwv7222+aRJRHbXNRzK1semn33XdfzePjjz+2smClFYXXTp8+3Ypio/0C+YviY0WJ1/NlpDRqFe61a9daUZYiq3Dj92WXXaZ5nnTSSdp/xObHHnHEEbrasSgaVhQOTYPVvCGHKL9OfJusD0I+8fLSvGVrEiujMPbiiy+2Mv1rRaGxYvhvZf0qK0qxphGnAuUgClwkf+8XGN8xkAAJkAAJhIiAvCHrDV5sjWyiLR/wIJRF9/RB5qrWrVs3Pe+9997TqHjbTMjbuqbxKjRuqwgoUAiJysQx2XhRz5cRBfzUgG0qxObH/eRnEgJHH310RJlAMjFctzKaYsXzT8+S/c6U7+zZsyO5QBmAoiIjGxonoyn6CcUDyo8LYoOi6aBsuuBnWxGklZEYVYScIgVFR0Z7bJMmTSLKlRjhW9kDzcoebZq9nz4oo08qU7KtSVydZaTJiR33kysFSy9gIAESIIEwEcCWBxhyx7QDpswQYreZwFQPpiqwvL4LcIfHQnxLlizRKQnEZ7KIYbJtJjDthYApPBdg8CrKkPvJzyQE4O0E7z945Mnolk63YDoIRuuw98JGrgiiVOoUE7679sBO9QiOPaYhYUzrNluV0RsDr0G3yasm9vkPfQ19x7Uv+h/KrVevXiQOBv6yD5vBkgIIzz77bMo+CPsh9EE/W5Ok6qtUaHw2JpORAAmQQFAIfPfddxEbGdhCILjtHJyMsF+BHcK9997rouJ+pnpIxDspUZnx0iIOsskrdaLDjPcQgB2MTBmaa6+9VhUasIYyA5spPPxxHAF2Ky649nCfLh7ee7C7gX0L1n6CrQ2UGxkFckmy+qxYsWKJ82Gj49aE8tsHYzNxfTm2z6Tqq1RoYknyNwmQAAkEnABGaGINL2NFxkNh8eLFugYQHjKJQqqHRKLzGJ8/AlBe9t9/fx3VgtH3UUcdpYopRmywzpPfgH3TMBqHdZ5kGkgNvAcNGpTU9dlv3kiXqO+4eL990G+ZLt9E6em2nYgM40mABEggoAT8KDSYXsCb8ogRI6JqgU1C4QmFgAdEvG0mcAzTGwyFIYAFI7E69pAhQwwckf/8808drYE03pGZVNJhOhKjdI888ojZZ599jNjOGDHwjToNafLV1n76YJQwCX44RSa2r8Ymp0ITS4S/SYAEygQBMTQ0jz/+eCjr6p1ycsP7bjsHV6FOnTqpPcPll19usLw+3G4x/dC9e/eIWzAedrHbTGBEoHbt2rrIIFx3Fy1apG7fyBdbVeCBmqhMpBFjUXzo1IZ+kX+QDe63sVMI7jg/SxKA7ZN4O+lWIXCph6s0ODplFK7QUE4RXHtgOxBvwL5pL7zwgipEmGqC2zZcs73Bz7YiaDeUEbvfFFzCxVPJm52mcwqSnz6IPJA/5HPB9WXYgCGgnyKIMbum/eSTT/R3iX+SEQMJkAAJlDkCV155pZW1P0JZb1mfww4fPtzKlJKVVX7VS0TWjLF33nmnlQdDpE4LFixQt1258WsaWVdEXbhdArgHw8126623tnfffbeLVpdtxMHNGy65Yoiqrr1w9X311VcTljl16lT1rEJ5Mt1h5UFsxTBU3YARJ6MNVkYbIuXwS3IC8AyD9w88lcReRd2uRSlRj7FtttnGytoz2lZw6wbfU045xc6aNSuS6bhx46wYAesx1wfw2bJlS20bJIQnkmwromlwPcBTCu7ZrVq1UndpeDPBVRznwSV/1KhReo5Mi2mcGAeruzi8nwYPHqxx6DvysqByJOuDcCXHcgPIW9agsbIujhVlPeJeDhd02SZF85FVsTUdeIiirXGx/7j1gZBkIAESKHsEmjZtaho0aKALv4Wp9ph+gDEmFsvr0KGDL9Ex0oJhe7cBqPekRNtM4C0bZcGbBZ+wh4g1OvXmw++5J4ARNSwqh9E1LCYHDyUEeZBrm4iLdNJC33jjDd0f6vDDD9eROFE6dAQFozZY6FBc7CPn52tbEVdAsj7o0iT7RJ0xKgUOiQIVmkRkGE8CJFDUBGSBOH0wYNXVMAVMN8Eg+O233zZ4UDEULwHYjNStW9fAHVtG39KqKPY9at++vU4zOa8hlwGmqtz0o4srhk/a0BRDK7IOJEACaROAnUjYlBlUcsWKFVpXLBPPUNwEoIjAQ0lWAzYYXUknwM4EtjfYygDr0WCLAXg8PfPMM0amhgzsW4otUKEpthZlfUiABIqaAIx4EbA2CUPxEzjnnHPUGBeKSDrh7LPPNsOGDVPjbmwEK3YtagwOI9wbb7xR90VKJ78wpOWUUxhaiTKSAAmQwP8TwNs63tqdBwjBFD+BM844Q0dXsEFoJgE2UMnWIsokzyCewxGaILYKZSIBEiCBBATgJotdiBnKDgHZg8uI95LByruZhLKgzIALFZpMegfPIQESIIECEcBaI1RoCgS/QMVipWDZaNSIm3aBJAhHsVRowtFOlJIESIAElAAVmrLXEeByLztqqz1MOisFlzVSVGjKWouzviRQxgmkWj496Hg45RT0FsqPfJ07d9Y1ZaZPn56fAoogVyo0RdCIrAIJkIB/AtjPBvvahDVAoZFVYsMqPuXOkMAee+xhsMcT1o9hiE+ACk18LowlARIoUgLYrbhatWqhrR32SqpatWpo5afgmROQLQnMiy++qAtCZp5L8Z5JhaZ425Y1IwESiEOgTp06pk2bNnGOhCOKCk042ikfUkKhwWJ58HhiKElg05JRjCEBEiCB4iUgm1KGunLYLZkjNKFuwoyFxwJ5WN0aO2/LhpIZ51OsJ1KhKdaWZb1IgAR0JAZLxmNDO9kp2MiO1JFP73dswBiWAIUmTPKGhWtY5GzdurWRHc91td+wyFxaclKhKS3SLIcESKDUCWC9lkmTJmm5m266qe44DS+nWE8n2NRgN+DKlSuXuozpFIgdh7F0PRWadKgVV9pWrVqZESNGmFWrVtE4PKZpaUMTA4Q/SYAEiodAixYtVImBIoDl3//4448Syky5cuV0j5ugKzNoFciPumy++ebF00isSVoEmjZtqumx2zpDNAEqNNE8+IsESKCICDRv3tykWogMCkLv3r1DUeuNGzeqnBUrVgyFvBQy9wSwyeS+++5rpk2blvvMQ54jFZqQNyDFJwESSEygZs2aBn+JAqah4DlSu3btREkCFb9hwwaVhyM0gWqWUhfm0EMPpadTHOpUaOJAYRQJkEDxEIARZaLN+f766y/Tt2/f0FSWIzShaaq8Ctq4cWPz4Ycflpg+zWuhIcicCk0IGokikgAJZE4A005QXGIDbGfwYGjSpEnsocD+hh0QQiIFLbCCU7CcEjjggAMMvPcWL16c03zDnhkVmrC3IOUnARJISqBZs2ZxV1aFbU3Y1qSBvQ8CNitkKLsEsA0ClNr58+eXXQhxak6FJg4URpEACRQPAaw3U69evRIVgm3NCSecUCKeESQQdAJQZrDi9cKFC4MuaqnKR4WmVHGzMBIggUIQiLWjKV++vLniiisMpp3CGDhCE8ZWy63MUGi++uqr3GYa8tzCeTWHHDrFJwESKF0CsKNx9icouVKlSqZr166lKwRLI4EcEoBn3tKlS3OYY/izokIT/jZkDUiABFIQOPLIIyN2J3DV7tWrl6lSpUqKs4J32BkDY4E9hrJNoEaNGmbFihVlG0JM7anQxADhTxIggeIjgMXI9t57b60YDGsvuuiiUFbSrT/j1qMJZSUodE4IwDbsp59+yklexZIJFZpiaUnWgwRIICmBY489Vo+feuqpZqeddkqaNqgHnULj1qMJqpyUK/8EsJ/X2rVr819QiEqgQhOixqKoJBA2AljjBQasQfgbOnSo4nv66acDIY9jks7Cfk6h8TNC8+WXXwaqnq6+Ze1zl112yctli73HMPWYamuPvBQe0Ey523ZAG4ZikUCxEGjXrp1u/ljo+kAJuP32203//v0LLUqk/HRl2WyzzQz+0nkzHzx4sNl9990jZfJL6RGYOHGimTx5cl4KhGKIAIUmrN56uQZDhSbXRJkfCZBAFIEGDRqYjh07RsUV6geMg3fddddCFV+i3GHDhpWISxWxzTbbmFWrVqVKFjnesmVLc+CBB0Z+80vpEVi2bFneFBq3+jWM3Bn+JcApJ/YEEiCBMkMgSMpMptDTVWgyLYfnBZvAunXrdPmBYEtZutJRoSld3iyNBEiABLIisO2225qVK1dmlQdPDj8B9IHtttsu/BXJYQ2o0OQQJrMiARIggXwTwPojy5cvz3cxzD/gBDCdhb7A8B8BKjT/seA3EiABEgg8AexBhYcZQ9kmAC82GntH9wEqNNE8+IsESIAEAk0ACs23334baBkpXP4JfPLJJwa7bjP8R4AKzX8s+I0ESIAEAk8Ahs3ff/+9rkESeGEpYF4IwH4GIzT0XovGS4Ummgd/kQAJkECgCey5557m77//NosXLw60nBQufwTeeustXTTxsMMOy18hIcyZCk0IG40ikwAJlF0C9evXN1h7ZP78+VEQOA0VhaOof4wfP15HZ+jlFN3MVGiiefAXCZAACQSaAFYKhlIzb968iJxXXnmladu2beQ3vxQvgd9//92MGzfOnHzyycVbyQxrRoUmQ3A8jQRIgAQKReDggw827733ni57f95555lbb73VwEh02rRphRKJ5ZYSAexF9ttvv5nOnTuXUonhKYYKTXjaipKSAAmQgBI44ogjzMyZM02nTp3MI488Yqy1Og2VyVYKRBoeArCdQht36NDB5GvTy/DQKCkpN4EoyYQxJEACJBBoAo0bNzbYeXvs2LGR3Zaxtw82Q/ziiy9MnTp1Ai0/hcuMwEMPPWQ+//xz88ILL2SWQZGfxRGaIm9gVo8ESKC4CGCn7fPPP9/8+uuvEWXG1RDGwnfddZf7yc8iIgBX/auuusp069bNNGrUqIhqlruqUKHJHUvmRAIkQAJ5JfDTTz+Zww8/3Lz//vvG7bbsLfDPP/80I0eONGvWrPFG83vICWCq6fTTTzdVqlQxQ4cODXlt8ic+FZr8sWXOJEACJJAzAtju4JBDDjELFy6Mq8y4gqDUYGqCoXgI9OjRw8yYMcM899xzZuutty6eiuW4JlRocgyU2ZEACeSXAFZIPeecc8rcfkazZs0ymG6CAXCygLf52267TRffS5aOx4JP4J9//jG9evXSUTcoqVxIL3mbUaFJzodHSYAEAkbggw8+MI8++mjUOiwBEzEv4sCz5ZtvvjGDBg0yW265pXo1JSoI9havv/56osOMDwGBVatWmRNPPNGMGDFCFZouXbqEQOrCikiFprD8WToJkECaBLCgGGxJWrduHXXmE088EfW7GH9UqlTJ9O3bV0en+vXrp55OMASODeXKlTMPPvhgbHSgfqMNX3311ZzLlGm+Qeo/EyZMMPvtt5955513DL7DEJghNQEqNKkZMQUJkEDACFSrVi1KIuxt079//6i4Yv5RtWpVc+ONN+qIDaYkKlSoEDVig6kKLLQX1IBpMSwMt3Tp0pyKmGm+Qek/sJM55phjTLt27czee+9tPv74Y3PsscfmlFExZ1ZStS/m2rJuJEACoSeAhzVWxN1iiy0MVszFw+j444/XzfoeeOABs9NOO+kDwVV08uTJBvYn22yzjS5E593/Bga2P/zwgznyyCPNpEmTdMPHjh07mpo1a6pLNN6QsSJv06ZN1SDX5RmUz+23397ccccd5rLLLjPXX3+9TsWVL1/ewDAYn3jApxOwHxTWtrnooovMggULzEsvvWSwuzc8bDDq48Ly5ct1dAWGyrDraNGihR5CeTBc/eOPP/Q3OGIzTbQB2g2jDviN/BC3ww47aLu1b9/e7Ljjji77lJ8//vijrrmDT6y5c8ABB5idd945Yb7YLmDq1KkG05XgcuaZZ2p6FJSs/ySqZ0oB00gAmzBwfuyxx1QJxQ7aGJVp06ZNGrkwKQj810PJgwRIgAQCTgAPWayO27x5czN37lyVForKPvvsYypWrGgaNGigyggO4KGKbQF+/vln3ecID6499thDH9RYw+Xyyy83e+21lxk+fLg+wKG8vPjii2a33XYzr7zyijnjjDP0QXPPPfcYrMwLpSioAavGwmgUO3DD7gLBKTMrVqzwJbbb8LBPnz7m7rvvNrfffruuRgzbjSFDhkTyAEcoT/vvv78qJyeccILp2bOnHoeygNEFKFldu3Y1tWvXVgUTruSYLoNCs2HDhsioA5QQtBmO+Q2rV682xx13nIHiiTaEAgZFJVG+69atM/Xq1dMysI4L3N2hhEHJQUjUf5LV06+sSAcj7vXr1xvYNX300UcGnLHaL5SqunXrqkI2YMAAHZFBmXPmzKEykw5gb1qBzUACJEACeSEgK9paeejkNG+ZSoGbj73//vsj+cpD1cpoQOQ3vshDw1533XWROBl90PNatWoVidtqq62sjPJY2RtH48SLyMr0jW3SpEkkTh5GVjaEtDfffHPkvFx9yQcfyCZTFfaoo47S+p511lm+xZUHvp4joyeRc2T0w8qogf4WRdDuvvvuVpSEyHGx79BzZCQrEic7gVtRMG337t2tPMCtPLAjx/BFHux6zsMPPxwV7+eHKJhWRtQiSWWEwz7zzDP6O16+Tz31lJXRJSsjcVFpZs+eHckjtv/4rWckgwRfrrnmGq0n+qv3T0ajrIxq2auvvtq+9tprduPGjQlyYHQ6BDjl5NXu+J0ESCDwBDASEy9ssskmUdEYYTjooIMiowc4iNGAlStXRtLBFgVTFm6EAN5DmLJyb/RIWLlyZR31+eqrryLnBf0LRqxEWdC6wTMKmxmiHqmC44CRLBcwiiUPXf357LPP6sjGFVdc4Q7rlB0YLlmyJDIth3Mw6iAPdANuGPGKF2LbLF6a2DjIhilHjKBhJAgjamgzb/Dme9ppp+mUVPXq1XUUx23giS0EMGXpgvccv/V05yb6xLTotttua0Sp0ilSTLFBVvQzhtwToEKTe6bMkQRIoAAEvA8kTEvA/uHcc8+NsqfxI1Y8hQlGt5g2CGPATtzx6uS3LphGkrdkTS4jL2rrcu+996Y8HUoPpsFgZ4Ppr3jeWN42S5nh/yfAdCOmmrDWzssvv6xbPWB6yxu8+cL2B8oMFCzsf+WUGNj0eIP3nHTq6c0j9ruM7KmyHOuRF5uOv3NDgDY0ueHIXEiABApMwPtAcgas8+bNS1sqbz7ekxPFe9ME9TuUklwE5AM7HRgdpwowwsVIzaJFi8wNN9wQN3kmTNG2UNIwagRDYiyy6LXxQUHefDFCBHsfbOgJT7hatWqllCWdesbNjJEFIUCFpiDYWSgJkEAuCeAB5oxgkS+mkjAVIXY2EeNPVx6G/zENw5A+gX333VdHqrDYmzdgROy+++6LROE3lIwxY8aYCy+8UBUQZ8SNRE7h8LZZ5OQUXzCVhtGVo48+2nz44YfqYQXDbYR4+cKAGQpY27ZtNU3syIw7zyuL33pqhvwXGAJUaALTFBSEBEjADwExoNRk8F5yAW/qcL+GC+wXX3yhD123AB2mKDBagIefGAnrxo1wRcY0CqaRXH4uL3jFeO1sEI908KIp9oCtFRCc2zW+gzMYgRc8zOCKjSkfjJLA7X306NFGjH/VawfpEeD2jSkeTHUNHjxY7UgwLeQ8i5yLNlzikW86a+bA9uWNN97QcmAXBC8rty5RvHzRdvAwgh0P6uIUL0xJQvFCiO0/UH781FNP5r/gEJDOxEACJEACeSGQay+emTNnWlkpWD1GGjVqZGW9DpVb3F2t2GhY2bjPisuxxsmbuJXVdDVe7rj6CS8eeRO38Ga66aabNB9Zy8WOGjXKwrMF3jhIK0abFt408H6SB7LGIe/HH388p5xyzccrnCh2Kre4AXujE34XpU89mFB/sT2yogRYMY61Mtql+chIh5WRDiuu87Z+/foah7RoB3Gb1nzBVRb6s+KObUXZ0ThxG7fiJq3pZZE4KwqJxsPLB+c3a9bMfv311wnlij2ANhLDYG0feDf17t07Uj7Sxub77rvvWplmUq8rcWm3MjqnXlvirm1lCw3NPl7/SVZPPcnHPzFMVxY+kjJJDghsgjykUzGQAAmQQM4JiPuzLkqHt/l8hzVr1ujib7EeJBgVwMgNpqD8ePrkW05v/vnkgzrD+wjrmmCxtlwHUUJ0igejXZkEPHowSoK1aNIJWEcGBsZYVA8jQOJ6H3V6vHwxzYR+UKVKFU2LNJiGgtGuC4n6Tzb1hBcWjJdhGM2QfwL0cso/Y5ZAAiRQCgRiH2yuSLgiN2zY0P3kZ44IJDKu9Zs97F28ykyPHj1SnoqpLSzOhwAX6HghNl+kgSGxU2bwG2m8ygziEvWfbOuJvBlKhwAVmtLhzFJIgARIgASSEJCppyRH/z2ErR4YSCARASo0icgwngRIgARIoNQIYCsDBhLIhgC9nLKhx3NJgARIgARIgAQCQYAKTSCagUKQAAmQAAmQAAlkQ4AKTTb0eC4JkAAJkAAJkEAgCFChCUQzUAgSIAESIAESIIFsCFChyYYezyUBEiABEiABEggEASo0gWgGCkECJEACJEACJJANASo02dDjuSRAAiRAAiRAAoEgQIUmEM1AIUiABEiABEiABLIhQIUmG3o8lwRIgARIgARIIBAEqNAEohkoBAmQAAmQAAmQQDYEqNBkQ4/nkgAJkAAJkAAJBIIAFZpANAOFIAESIAESIAESyIYAFZps6PFcEiABEiABEiCBQBDgbtuBaAYKQQLFS2DYsGEGfwzxCTRt2jT+gRzFHnTQQTnKidlkQmDnnXfO5DSekwEBKjQZQOMpJEAC/ggMHDjQrFy50l/iMpqqQYMGeal59erVzejRo/OSNzP1T6By5cr+EzNlVgQ2sRKyyoEnkwAJkAAJkAAJkECBCdCGpsANwOJJgARIgARIgASyJ0CFJnuGzIEESIAESIAESKDABKjQFLgBWDwJkAAJkAAJkED2BP4PaDMay9okBEcAAAAASUVORK5CYII="></div></p>
<p>The last important bits about this pair of values is the wrapping.</p>
<p>First, it is actually a <code>Future</code>, allowing your stream to yield objects
that it doesn’t quite have yet —
for example, those which ultimately come from an <span class="caps">HTTP</span> response.</p>
<p>Secondly, its outermost layer is an <code>Option</code>.
This enables you to <em>terminate</em> the stream when the underlying source is exhausted
by simply returning <code>None</code>.
Until then, however, you should return <code>Some</code> with the (future of) aforementioned pair of values.</p>
<h4>Paginate! Paginate!</h4>
<p>If you have doubts about how all those pieces of <code>stream::unfold</code> fit in,
then looking at <a href="https://docs.rs/futures/0.1.17/futures/stream/fn.unfold.html#example">the usage example in the docs</a>
may give you <em>some</em> idea of what it enables you to do.
It’s a very artificial example, though:
the resulting <code>Stream</code> isn’t waiting for any asynchronous <code>Future</code>s,
which is the very reason you’d use a <code>Stream</code> over an <code>Iterator</code> in the first place<sup id="fnref:4"><a class="footnote-ref" href="#fn:4" rel="footnote">4</a></sup>.</p>
<p>We can find a more natural application for <code>unfold</code>
if we go back to our original problem.
To reiterate, we want to repeatedly query an <span class="caps">HTTP</span> <span class="caps">API</span> for a long list of items,
giving our callers a <code>Stream</code> of such items they can process at their leisure.
At the same time, all the details about pagination and handling of continuation tokens or offsets
should be completely hidden from the caller.</p>
<p>To employ <code>stream::unfold</code> for this task,
we need two things: the initial seed, and an appropriate closure.</p>
<p>I have hinted already at using the continuation token as our seed,
or the state that we pass around from one closure invocation to another.
What remains is mostly making the actual <span class="caps">HTTP</span> request and interpreting the <span class="caps">JSON</span> response,
for which we’ll use the <em>defacto</em> standard Rust crates:
<a href="https://docs.rs/hyper"><code>hyper</code></a>, <a href="https://serde.rs">Serde</a>,
and <a href="https://docs.rs/serde_json"><code>serde_json</code></a>:</p>
<div class="highlight"><pre><span class="kn">use</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">error</span><span class="o">::</span><span class="n">Error</span><span class="p">;</span><span class="w"></span>
<span class="kn">use</span><span class="w"> </span><span class="n">futures</span><span class="o">::</span><span class="p">{</span><span class="n">future</span><span class="p">,</span><span class="w"> </span><span class="n">Future</span><span class="p">,</span><span class="w"> </span><span class="n">stream</span><span class="p">,</span><span class="w"> </span><span class="n">Stream</span><span class="p">};</span><span class="w"></span>
<span class="kn">use</span><span class="w"> </span><span class="n">hyper</span><span class="o">::</span><span class="p">{</span><span class="n">Client</span><span class="p">,</span><span class="w"> </span><span class="n">Method</span><span class="p">};</span><span class="w"></span>
<span class="kn">use</span><span class="w"> </span><span class="n">hyper</span><span class="o">::</span><span class="n">client</span><span class="o">::</span><span class="n">Request</span><span class="p">;</span><span class="w"></span>
<span class="kn">use</span><span class="w"> </span><span class="n">serde_json</span><span class="p">;</span><span class="w"></span>
<span class="kn">use</span><span class="w"> </span><span class="n">tokio_core</span><span class="o">::</span><span class="n">reactor</span><span class="o">::</span><span class="n">Handle</span><span class="p">;</span><span class="w"></span>
<span class="kr">const</span><span class="w"> </span><span class="n">URL</span><span class="o">:</span><span class="w"> </span><span class="o">&</span><span class="kt">str</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"http://api.example.com/items"</span><span class="p">;</span><span class="w"></span>
<span class="k">fn</span><span class="w"> </span><span class="n">items</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">handle</span><span class="o">:</span><span class="w"> </span><span class="o">&</span><span class="n">Handle</span><span class="p">,</span><span class="w"> </span><span class="n">after</span><span class="o">:</span><span class="w"> </span><span class="nb">Option</span><span class="o"><</span><span class="n">String</span><span class="o">></span><span class="w"></span>
<span class="p">)</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">Box</span><span class="o"><</span><span class="n">Stream</span><span class="o"><</span><span class="n">Item</span><span class="o">=</span><span class="n">Item</span><span class="p">,</span><span class="w"> </span><span class="n">Error</span><span class="o">=</span><span class="n">Box</span><span class="o"><</span><span class="n">Error</span><span class="o">>>></span><span class="w"></span>
<span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">client</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Client</span><span class="o">::</span><span class="n">new</span><span class="p">(</span><span class="n">handle</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">Box</span><span class="o">::</span><span class="n">new</span><span class="p">(</span><span class="n">stream</span><span class="o">::</span><span class="n">unfold</span><span class="p">(</span><span class="n">after</span><span class="p">,</span><span class="w"> </span><span class="n">move</span><span class="w"> </span><span class="o">|</span><span class="n">cont_token</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">url</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">match</span><span class="w"> </span><span class="n">cont_token</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">ct</span><span class="p">)</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="n">format</span><span class="o">!</span><span class="p">(</span><span class="s">"{}?after={}"</span><span class="p">,</span><span class="w"> </span><span class="n">URL</span><span class="p">,</span><span class="w"> </span><span class="n">ct</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="nb">None</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">None</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">};</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">req</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Request</span><span class="o">::</span><span class="n">new</span><span class="p">(</span><span class="n">Method</span><span class="o">::</span><span class="n">Get</span><span class="p">,</span><span class="w"> </span><span class="n">url</span><span class="p">.</span><span class="n">parse</span><span class="p">().</span><span class="n">unwrap</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">client</span><span class="p">.</span><span class="n">request</span><span class="p">(</span><span class="n">req</span><span class="p">).</span><span class="n">from_err</span><span class="p">().</span><span class="n">and_then</span><span class="p">(</span><span class="n">move</span><span class="w"> </span><span class="o">|</span><span class="n">resp</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">status</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">resp</span><span class="p">.</span><span class="n">status</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="n">resp</span><span class="p">.</span><span class="n">body</span><span class="p">().</span><span class="n">concat2</span><span class="p">().</span><span class="n">from_err</span><span class="p">().</span><span class="n">and_then</span><span class="p">(</span><span class="n">move</span><span class="w"> </span><span class="o">|</span><span class="n">body</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">status</span><span class="p">.</span><span class="n">is_success</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">serde_json</span><span class="o">::</span><span class="n">from_slice</span><span class="o">::<</span><span class="n">ItemsResponse</span><span class="o">></span><span class="p">(</span><span class="o">&</span><span class="n">body</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">map_err</span><span class="p">(</span><span class="n">Box</span><span class="o">::<</span><span class="n">Error</span><span class="o">>::</span><span class="n">from</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">Err</span><span class="p">(</span><span class="n">format</span><span class="o">!</span><span class="p">(</span><span class="s">"HTTP status: {}"</span><span class="p">,</span><span class="w"> </span><span class="n">status</span><span class="p">).</span><span class="n">into</span><span class="p">())</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">map</span><span class="p">(</span><span class="n">move</span><span class="w"> </span><span class="o">|</span><span class="n">items_resp</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="p">(</span><span class="n">stream</span><span class="o">::</span><span class="n">iter_ok</span><span class="p">(</span><span class="n">items_resp</span><span class="p">.</span><span class="n">items</span><span class="p">),</span><span class="w"> </span><span class="n">items_resp</span><span class="p">.</span><span class="n">continuation_token</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="p">}))</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">flatten</span><span class="p">())</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="cp">#[derive(Deserialize)]</span><span class="w"></span>
<span class="k">struct</span><span class="w"> </span><span class="n">ItemsResponse</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">items</span><span class="o">:</span><span class="w"> </span><span class="n">Vec</span><span class="o"><</span><span class="n">Item</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="cp">#[serde(rename = </span><span class="s">"continuationToken"</span><span class="cp">)]</span><span class="w"></span>
<span class="w"> </span><span class="n">continuation_token</span><span class="o">:</span><span class="w"> </span><span class="nb">Option</span><span class="o"><</span><span class="n">String</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>While <a href="https://play.rust-lang.org/?gist=556ff0d112272b3c2da36c3d03aac52e&version=stable">this code</a>
may be a little challenging to decipher at first,
it’s not out of line compared to how working with <code>Future</code>s and <code>Stream</code>s looks like in general.
In either case, you can expect a lot of <code>.and_then</code> callbacks :)</p>
<p>There is one detail here that I haven’t mentioned previously, though.
It relates to the <code>stream::iter_ok</code> and <code>Stream::flatten</code> calls
which you may have already noticed.<br>
The issue with <code>stream::unfold</code> is that it only allows to yield an item <em>once</em> per closure invocation.
For us, this is too limiting: a single batch response from the <span class="caps">API</span> will contain many such items,
but we have no way of “splitting” them.</p>
<p>What we can do instead is to produce a <code>Stream</code> of entire <em>batches</em> of items,
at least at first, and then <code>flatten</code> it.
What <code>Stream::flatten</code> does here is to turn a nested <code>Stream<Stream<Item>></code>
into a flat <code>Stream<Item></code>. The latter is what we eventually want to return,
so all we need now is to create this nested <em>stream of streams</em>.</p>
<p>How? Well, that’s actually pretty easy.</p>
<p>We can already deserialize a <code>Vec<Item></code>
from the <span class="caps">JSON</span> response — that’s our item batch! —
which is essentially an <em>iterable</em> of <code>Item</code>s<sup id="fnref:5"><a class="footnote-ref" href="#fn:5" rel="footnote">5</a></sup>.
Another utility function from the <code>stream</code> module,
namely <a href="https://docs.rs/futures/*/futures/stream/fn.iter_ok.html"><code>stream::iter_ok</code></a>,
can readily turn such iterable into a “immediate” <code>Stream</code>.
Such <code>Stream</code> won’t be asynchronous at all
— its items will have been ready from the very beginning —
but it will still conform to the <code>Stream</code> interface,
enabling it to be <code>flatten</code>ed as we request.</p>
<h4>But wait! There is a bug!</h4>
<p>So in the end, is this the solution we’re looking for?…</p>
<p>Well, almost. First, here’s the expected usage of the function we just wrote:</p>
<div class="highlight"><pre><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">core</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">tokio_core</span><span class="o">::</span><span class="n">reactor</span><span class="o">::</span><span class="n">Core</span><span class="o">::</span><span class="n">new</span><span class="p">().</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
<span class="n">core</span><span class="p">.</span><span class="n">run</span><span class="p">({</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">continuation_token</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">None</span><span class="p">;</span><span class="w"> </span><span class="c1">// start from the beginning</span>
<span class="w"> </span><span class="n">items</span><span class="p">(</span><span class="o">&</span><span class="n">core</span><span class="p">.</span><span class="n">handle</span><span class="p">(),</span><span class="w"> </span><span class="n">continuation_token</span><span class="p">).</span><span class="n">for_each</span><span class="p">(</span><span class="o">|</span><span class="n">item</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">println</span><span class="o">!</span><span class="p">(</span><span class="s">"{:?}"</span><span class="p">,</span><span class="w"> </span><span class="n">item</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="nb">Ok</span><span class="p">(())</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="p">}).</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
</pre></div>
<p>While this is more complicated than the plain <code>for</code> loop in Python,
most of it is just Tokio boilerplate.
The notable part is the invocation of <code>items()</code>,
where we pass <code>None</code> as a continuation token to indicate that
we want the entire sequence, right from its beginning.</p>
<p>And since we’re talking about fetching long sequences,
we would indeed expect a <em>lot</em> of items.
So it is probably quite surprising to hear
that the stream we’ve created here will be completely <em>empty</em>.</p>
<p>…What? How?!</p>
<p>If you look again at the source code of <code>items()</code>,
the direct reason should be pretty easy to find.
The culprit lies in the <code>return None</code> branch of the first <code>match</code>.
If we don’t pass <code>Some(continuation_token)</code> as a parameter to <code>items()</code>,
this branch will be hit <em>immediately</em>,
terminating the stream before it had a chance to produce anything.</p>
<p>It may not be very clear how to fix this problem.
After all, the purpose of the <code>match</code> was to detect the <em>end</em> of the sequence,
but it apparently prevents us from <em>starting</em> it in the first place!</p>
<p>Looking at the problem from another angle,
we can see we’ve conflated two distinct states of our stream
— “before it has started” and “after it’s ended” — into a single one (“no continuation token”).
Since we obviously don’t want to make the <code>after</code> parameter mandatory
— users should be able to say “Give me everything!” —
we need another way of telling those two states apart.</p>
<p>In terms of Rust types, it seems that <code>Option<String></code> is no longer sufficient
for encoding all possible states of our <code>Stream</code>.
Although we could try to fix that in some ad-hoc way (e.g. by adding another <code>bool</code> flag),
it feels cleaner to just define a new, dedicated type.
For one, this allows us to designate a name for each of the states in question,
improving the readability and maintainability of our code:</p>
<div class="highlight"><pre><span class="k">enum</span><span class="w"> </span><span class="n">State</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">Start</span><span class="p">(</span><span class="nb">Option</span><span class="o"><</span><span class="n">String</span><span class="o">></span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="n">Next</span><span class="p">(</span><span class="n">String</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="n">End</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>Note that we can put this definition directly inside the <code>items()</code> function,
without cluttering the module namespace.
All the relevant details of our <code>Stream</code> are thus nicely contained
<a href="https://play.rust-lang.org/?gist=42af60531f18509e181c4e30a46eb077&version=stable">within a single function</a>:</p>
<div class="highlight"><pre><span class="k">fn</span><span class="w"> </span><span class="n">items</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">handle</span><span class="o">:</span><span class="w"> </span><span class="o">&</span><span class="n">Handle</span><span class="p">,</span><span class="w"> </span><span class="n">after</span><span class="o">:</span><span class="w"> </span><span class="nb">Option</span><span class="o"><</span><span class="n">String</span><span class="o">></span><span class="w"></span>
<span class="p">)</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">Box</span><span class="o"><</span><span class="n">Stream</span><span class="o"><</span><span class="n">Item</span><span class="o">=</span><span class="n">Item</span><span class="p">,</span><span class="w"> </span><span class="n">Error</span><span class="o">=</span><span class="n">Box</span><span class="o"><</span><span class="n">Error</span><span class="o">>>></span><span class="w"></span>
<span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// (definition of State enum can go here)</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">client</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Client</span><span class="o">::</span><span class="n">new</span><span class="p">(</span><span class="n">handle</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">Box</span><span class="o">::</span><span class="n">new</span><span class="p">(</span><span class="n">stream</span><span class="o">::</span><span class="n">unfold</span><span class="p">(</span><span class="n">State</span><span class="o">::</span><span class="n">Start</span><span class="p">(</span><span class="n">after</span><span class="p">),</span><span class="w"> </span><span class="n">move</span><span class="w"> </span><span class="o">|</span><span class="n">state</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">cont_token</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">match</span><span class="w"> </span><span class="n">state</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">State</span><span class="o">::</span><span class="n">Start</span><span class="p">(</span><span class="n">opt_ct</span><span class="p">)</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="n">opt_ct</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">State</span><span class="o">::</span><span class="n">Next</span><span class="p">(</span><span class="n">ct</span><span class="p">)</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">ct</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="n">State</span><span class="o">::</span><span class="n">End</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">None</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">};</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">url</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">match</span><span class="w"> </span><span class="n">cont_token</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">ct</span><span class="p">)</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="n">format</span><span class="o">!</span><span class="p">(</span><span class="s">"{}?after={}"</span><span class="p">,</span><span class="w"> </span><span class="n">URL</span><span class="p">,</span><span class="w"> </span><span class="n">ct</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="nb">None</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="n">URL</span><span class="p">.</span><span class="n">into</span><span class="p">(),</span><span class="w"></span>
<span class="w"> </span><span class="p">};</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">req</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Request</span><span class="o">::</span><span class="n">new</span><span class="p">(</span><span class="n">Method</span><span class="o">::</span><span class="n">Get</span><span class="p">,</span><span class="w"> </span><span class="n">url</span><span class="p">.</span><span class="n">parse</span><span class="p">().</span><span class="n">unwrap</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">client</span><span class="p">.</span><span class="n">request</span><span class="p">(</span><span class="n">req</span><span class="p">).</span><span class="n">from_err</span><span class="p">().</span><span class="n">and_then</span><span class="p">(</span><span class="n">move</span><span class="w"> </span><span class="o">|</span><span class="n">resp</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">status</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">resp</span><span class="p">.</span><span class="n">status</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="n">resp</span><span class="p">.</span><span class="n">body</span><span class="p">().</span><span class="n">concat2</span><span class="p">().</span><span class="n">from_err</span><span class="p">().</span><span class="n">and_then</span><span class="p">(</span><span class="n">move</span><span class="w"> </span><span class="o">|</span><span class="n">body</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">status</span><span class="p">.</span><span class="n">is_success</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">serde_json</span><span class="o">::</span><span class="n">from_slice</span><span class="o">::<</span><span class="n">ItemsResponse</span><span class="o">></span><span class="p">(</span><span class="o">&</span><span class="n">body</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">map_err</span><span class="p">(</span><span class="n">Box</span><span class="o">::<</span><span class="n">Error</span><span class="o">>::</span><span class="n">from</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">Err</span><span class="p">(</span><span class="n">format</span><span class="o">!</span><span class="p">(</span><span class="s">"HTTP status: {}"</span><span class="p">,</span><span class="w"> </span><span class="n">status</span><span class="p">).</span><span class="n">into</span><span class="p">())</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">map</span><span class="p">(</span><span class="n">move</span><span class="w"> </span><span class="o">|</span><span class="n">items_resp</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">next_state</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">match</span><span class="w"> </span><span class="n">items_resp</span><span class="p">.</span><span class="n">continuation_token</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">ct</span><span class="p">)</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="n">State</span><span class="o">::</span><span class="n">Next</span><span class="p">(</span><span class="n">ct</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="nb">None</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="n">State</span><span class="o">::</span><span class="n">End</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">};</span><span class="w"></span>
<span class="w"> </span><span class="p">(</span><span class="n">stream</span><span class="o">::</span><span class="n">iter_ok</span><span class="p">(</span><span class="n">items_resp</span><span class="p">.</span><span class="n">items</span><span class="p">),</span><span class="w"> </span><span class="n">next_state</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="p">}))</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">flatten</span><span class="p">())</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>Sure, there is a little more bookkeeping required now,
but at least all the items are being emitted by the <code>Stream</code> as intended.</p>
<hr />
<p>You can see the complete source in
<a href="https://play.rust-lang.org/?gist=42af60531f18509e181c4e30a46eb077&version=stable">the playground here</a>.</p>
<div class="footnote">
<hr />
<ol>
<li id="fn:1">
<p>Furthermore, the token doesn’t have to come as part of the <span class="caps">HTTP</span> response body.
Some <span class="caps">API</span> providers (such as GitHub) may use the <code>Link:</code> header
to point directly to the next <span class="caps">URL</span> to query. <a class="footnote-backref" href="#fnref:1" rev="footnote" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:2">
<p>This example uses “traditional”, synchronous Python code.
However, it should be easy to convert it to the asynchronous equivalent
that works in Python 3.5 and above,
provided you can replace <code>requests</code> with some async <span class="caps">HTTP</span> library. <a class="footnote-backref" href="#fnref:2" rev="footnote" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:3">
<p>If you are curious whether other languages could express it better,
you can check the
<a href="https://hackage.haskell.org/package/conduit-1.2.13/docs/Data-Conduit-List.html#v:unfold"><code>Data.Conduit.List.unfold</code> function</a>
from the Haskell’s <a href="https://hackage.haskell.org/package/conduit"><code>conduit</code> package</a>.
For most intents and purposes, it is their equivalent of <code>stream::unfold</code>. <a class="footnote-backref" href="#fnref:3" rev="footnote" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:4">
<p>Coincidentally, you can create iterators in the very same manner
through the <a href="https://docs.rs/itertools/0.7.6/itertools/fn.unfold.html"><code>itertools::unfold</code> function</a>
from the <a href="https://docs.rs/itertools/"><code>itertools</code> crate</a>. <a class="footnote-backref" href="#fnref:4" rev="footnote" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn:5">
<p>In more technical Rust terms, it means <code>Vec</code> implements the <code>IntoIterator</code> trait,
allowing anyone to get an <code>Iterator</code> from it. <a class="footnote-backref" href="#fnref:5" rev="footnote" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
</ol>
</div>Terminating a Stream in Rust2017-12-16T21:17:00+01:00Karol Kuczmarskitag:xion.io,2017-12-16:post/code/rust-stream-terminate.html<p>Here’s a little trick that may be useful in dealing with
<a href="https://docs.rs/futures/0.1.17/futures/stream/trait.Stream.html">asynchronous <code>Stream</code>s</a> in Rust.</p>
<p>When you consume a <code>Stream</code> using
<a href="https://docs.rs/futures/0.1.17/futures/stream/trait.Stream.html#method.for_each">the <code>for_each</code> method</a>,
its default behavior is to finish early should an error be produced by the stream:</p>
<div class="highlight"><pre><span class="kn">use</span><span class="w"> </span><span class="n">futures</span><span class="o">::</span><span class="n">prelude</span><span class="o">::*</span><span class="p">;</span><span class="w"></span>
<span class="kn">use</span><span class="w"> </span><span class="n">futures</span><span class="o">::</span><span class="n">stream</span><span class="p">;</span><span class="w"></span>
<span class="kn">use</span><span class="w"> </span><span class="n">tokio_core</span><span class="o">::</span><span class="n">reactor</span><span class="o">::</span><span class="n">Core</span><span class="p">;</span><span class="w"></span>
<span class="kd">let</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">stream</span><span class="o">::</span><span class="n">iter_result</span><span class="p">(</span><span class="n">vec</span><span class="o">!</span><span class="p">[</span><span class="nb">Ok</span><span class="p">(</span><span class="mi">1</span><span class="p">),</span><span class="w"> </span><span class="nb">Ok</span><span class="p">(</span><span class="mi">2</span><span class="p">),</span><span class="w"> </span><span class="nb">Err</span><span class="p">(</span><span class="kc">false</span><span class="p">),</span><span class="w"> </span><span class="nb">Ok</span><span class="p">(</span><span class="mi">3</span><span class="p">)]);</span><span class="w"></span>
<span class="kd">let</span><span class="w"> </span><span class="n">fut</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">s</span><span class="p">.</span><span class="n">for_each</span><span class="p">(</span><span class="o">|</span><span class="n">n</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">println</span><span class="o">!</span><span class="p">(</span><span class="s">"{}"</span><span class="p">,</span><span class="w"> </span><span class="n">n</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="nb">Ok</span><span class="p">(())</span><span class="w"></span>
<span class="p">});</span><span class="w"></span>
</pre></div>
<p>In more precise terms, it means that
<a href="https://docs.rs/futures/0.1.17/futures/future/trait.Future.html">the <code>Future</code></a>
returned by <code>for_each</code> will resolve with the first error
from the underlying stream:</p>
<div class="highlight"><pre><span class="c1">// Prints 1, 2, and then panics with "false".</span>
<span class="n">Core</span><span class="o">::</span><span class="n">new</span><span class="p">().</span><span class="n">unwrap</span><span class="p">().</span><span class="n">run</span><span class="p">(</span><span class="n">fut</span><span class="p">).</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
</pre></div>
<p>For most purposes, this is perfectly alright;
errors are generally meant to propagate, after all.</p>
<p>Certain <em>kinds</em> of errors, however, are better off silenced.
Perhaps they are expected to pop up during normal program operation,
or maybe their occurrence should merely <em>affect</em> program execution in a particular way,
and not halt it outright.
In a simple case like above, you can of course check what <code>for_each</code> itself has returned,
but that doesn’t scale to building larger <code>Stream</code> pipelines.</p>
<p>I encountered a situation like this myself when using
<a href="https://docs.rs/crate/hubcaps/0.4.2">the <em>hubcaps</em> library</a>.
The code I was writing was meant to
<a href="https://docs.rs/hubcaps/0.4.2/hubcaps/search/struct.SearchIssues.html#method.iter">search for GitHub issues</a>
within a specific repository.
In GitHub <span class="caps">API</span>, this is accomplished by sending a search query like <code>repo:$OWNER/$NAME</code>,
which may result in a rather obscure <span class="caps">HTTP</span> error (<a href="https://httpstatuses.com/422">422 Unprocessable Entity</a>)
if the given repository doesn’t actually exist.
But I didn’t care about this error; should it occur, I’d simply return an empty stream,
because doing so was more convenient for the larger bit of logic that was consuming it.</p>
<p>Unfortunately, the <code>Stream</code> trait offers no interface that’d target this use case.
There are only a few methods that even allow to look at errors mid-stream,
and even fewer that can end it prematurely.
On the flip side, at least we don’t have to consider <em>too</em> many combinations
when looking for the solution ;)</p>
<p>Indeed, it seems there are only two <code>Stream</code> methods that are worthy of our attention:</p>
<ul>
<li><a href="https://docs.rs/futures/0.1.17/futures/stream/trait.Stream.html#method.then"><code>Stream::then</code></a>,
because it allows for a closure to receive all stream values (items <em>and</em> errors)</li>
<li><a href="https://docs.rs/futures/0.1.17/futures/stream/trait.Stream.html#method.take_while"><code>Stream::take_while</code></a>,
because it accepts a closure that can end the stream early (but only based on items, not errors)</li>
</ul>
<p>Combining them both, we arrive at the following recipe:</p>
<ul>
<li>
<p>Inside a <code>.then</code> closure, look for <code>Err</code>ors that you consider non-fatal
and replace them with a special item value.
The natural choice for such a value is <code>None</code>.
As a side effect, this forces us to convert the regular (“successful”) <code>item</code>s into <code>Some(item)</code>,
effectively transforming a <code>Stream<Item=T></code> into <code>Stream<Item=Option<T>></code>.</p>
</li>
<li>
<p>Looks for the special value (i.e. <code>None</code>) in the <code>.take_while</code> closure
and terminate the stream when it’s been found.</p>
</li>
<li>
<p>Finally, convert the wrapped items back into their original form using <code>.map</code>,
thus giving us back a <code>Stream</code> of <code>T</code><span class="quo">‘</span>s.</p>
</li>
</ul>
<p>Applying this technique to our initial example,
we get something that looks like this:</p>
<div class="highlight"><pre><span class="kd">let</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">stream</span><span class="o">::</span><span class="n">iter_result</span><span class="p">(</span><span class="n">vec</span><span class="o">!</span><span class="p">[</span><span class="nb">Ok</span><span class="p">(</span><span class="mi">1</span><span class="p">),</span><span class="w"> </span><span class="nb">Ok</span><span class="p">(</span><span class="mi">2</span><span class="p">),</span><span class="w"> </span><span class="nb">Err</span><span class="p">(</span><span class="kc">false</span><span class="p">),</span><span class="w"> </span><span class="nb">Ok</span><span class="p">(</span><span class="mi">3</span><span class="p">)])</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">then</span><span class="p">(</span><span class="o">|</span><span class="n">r</span><span class="o">|</span><span class="w"> </span><span class="k">match</span><span class="w"> </span><span class="n">r</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">Ok</span><span class="p">(</span><span class="n">r</span><span class="p">)</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="nb">Ok</span><span class="p">(</span><span class="nb">Some</span><span class="p">(</span><span class="n">r</span><span class="p">)),</span><span class="w"> </span><span class="c1">// no-op passthrough of items</span>
<span class="w"> </span><span class="nb">Err</span><span class="p">(</span><span class="kc">false</span><span class="p">)</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="nb">Ok</span><span class="p">(</span><span class="nb">None</span><span class="p">)</span><span class="w"> </span><span class="c1">// non-fatal error, terminate the stream</span>
<span class="w"> </span><span class="nb">Err</span><span class="p">(</span><span class="n">e</span><span class="p">)</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="nb">Err</span><span class="p">(</span><span class="n">e</span><span class="p">),</span><span class="w"> </span><span class="c1">// no-op passthrough of other errors</span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">take_while</span><span class="p">(</span><span class="o">|</span><span class="n">x</span><span class="o">|</span><span class="w"> </span><span class="n">future</span><span class="o">::</span><span class="n">ok</span><span class="p">(</span><span class="n">x</span><span class="p">.</span><span class="n">is_some</span><span class="p">()))</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">map</span><span class="p">(</span><span class="nb">Option</span><span class="o">::</span><span class="n">unwrap</span><span class="p">);</span><span class="w"></span>
</pre></div>
<p>If we now try to consume this stream like before:</p>
<div class="highlight"><pre><span class="n">Core</span><span class="o">::</span><span class="n">new</span><span class="p">().</span><span class="n">run</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">s</span><span class="p">.</span><span class="n">for_each</span><span class="p">(</span><span class="o">|</span><span class="n">n</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nb">println</span><span class="o">!</span><span class="p">(</span><span class="s">"{}"</span><span class="p">,</span><span class="w"> </span><span class="n">n</span><span class="p">);</span><span class="w"> </span><span class="nb">Ok</span><span class="p">(())</span><span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="p">).</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
</pre></div>
<p>it will still end after the first two items,
but without producing any errors afterwards.</p>
<hr />
<p>For a more reusable version of the trick, you can check
<a href="https://play.rust-lang.org/?gist=1bf2199460258be6674c9c7a7a157f50&version=stable">this gist</a>;
it adds a <code>Stream::take_while_err</code> method through an <a href="http://xion.io/post/code/rust-extension-traits.html">extension trait</a>.</p>
<p>This isn’t a perfect solution, however, because it requires <code>Box</code>ing even on nightly Rust<sup id="fnref:1"><a class="footnote-ref" href="#fn:1" rel="footnote">1</a></sup>.
We can fix that by introducing a dedicated <code>TakeWhileErr</code> stream type,
similarly to what native <code>Stream</code> methods do.
I leave that as an exercise for the reader ;-)</p>
<div class="footnote">
<hr />
<ol>
<li id="fn:1">
<p>This is due to a limitation in <a href="https://github.com/rust-lang/rust/issues/34511">the <code>impl Trait</code> feature</a>
which prevents it from being used as a return type of trait methods. <a class="footnote-backref" href="#fnref:1" rev="footnote" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>Recap of the gisht project2017-11-24T17:52:00+00:00Karol Kuczmarskitag:xion.io,2017-11-24:post/programming/gisht-recap.html<p>In this post, I want to discuss some of the experiences I had with a project
that I recently finished, <a href="https://github.com/Xion/gisht"><em>gisht</em></a>.
By “finished” I mean that I don’t anticipate developing any new major features for it,
though smaller things, bug fixes, or non-code stuff, is of course still very possible.</p>
<p>I’m thinking this is as much “done” as most software projects can ever hope to be.
Thus, it is probably the best time for a recap / summary / postmortem / etc. —
something to recount the lessons learned, and assess the choices made.</p>
<h4>Some context</h4>
<p>The original purpose of <em>gisht</em> was to facilitate download <span class="amp">&</span> execution of GitHub gists
straight from the command line:</p>
<div class="highlight"><pre><span class="nv">$ </span>gisht Xion/git-outgoing <span class="c"># run the https://gist.github.com/Xion/git-outgoing gist</span>
</pre></div>
<p>I initially wrote <a href="https://github.com/Xion/gisht.py">its first version in Python</a>
because I’ve accumulated a sizable number of small <span class="amp">&</span> useful scripts
(for Git, Unix, Python, etc.) which were all posted as gists.
Sure, I could download them manually to <code>~/bin</code> every time I used a new machine
but that’s rather cumbersome, and I’m quite lazy.</p>
<p>Well, lazy <em>and</em> impatient :)
I noticed pretty fast that the speed tax of Python
is basically unacceptable for a program like <em>gisht</em>.</p>
<p>What I’m referring to here is not the speed of code execution, however,
but only the <em>startup time</em> of Python interpreter.
Irrespective of the machine, operating system, or language version,
it doesn’t seem to go lower than about one hundred milliseconds;
empirically, it’s often 2 or 3 times higher than that.
For the common case of finding a cached gist (no downloads)
and doing a simple <code>fork</code>+<code>exec</code>,
this startup time was very noticeable and extremely jarring.
It also precluded some more sophisticated uses for <em>gisht</em>,
like putting its invocation into the shell’s <code>$PROMPT</code><sup id="fnref:1"><a class="footnote-ref" href="#fn:1" rel="footnote">1</a></sup>.</p>
<h4>Speed: delivered</h4>
<p>And so the obvious solution emerged:
let’s <a href="https://transitiontech.ca/random/RIIR">rewrite it in Rust</a>!…</p>
<p>Because if I’m executing code straight from the internet,
I should at least do it in a <em>safe</em> language.</p>
<p>But jokes aside, it is obvious that a language compiling to native code
is likely a good pick if you want to optimize for startup speed.
So while the choice of Rust was in large part educational
(<em>gisht</em> was one of my first projects to be written in it),
it definitely hasn’t disappointed there.</p>
<p>Even without any intentional optimization efforts,
the app still runs <em>instantaneously</em>.
I tried to take some measurements using the <code>time</code> command,
but it never ticked into more than 0.001s.
Perceptively, it is at least on par with <code>git</code>,
so that’s acceptable for me :)</p>
<h4>Can’t segfault if your code doesn’t build</h4>
<p>Achieving the performance objective wouldn’t do us much good, however,
if the road to get there involved excessive penalties on productivity.
Such negative impact could manifest in many ways,
including troublesome debugging due to a tricky runtime<sup id="fnref:2"><a class="footnote-ref" href="#fn:2" rel="footnote">2</a></sup>,
or difficulty in getting the code to compile in the first place.</p>
<p>If you had even a passing contact with Rust,
you’d expect the latter to be much more likely than the former.</p>
<p>Indeed, Rust’s very design eschews runtime flexibility to a ridiculous degree
(in its “safe” mode, at least),
while also forcing you to absorb subtle <span class="amp">&</span> complex ideas
to even get your code past the compiler.
The reward is increased likelihood your program will behave as intended —
although it’s definitely not on the level of “if it compiles, it works”
that can be offered by Haskell or Idris.</p>
<p>But since <em>gisht</em> is hardly mission critical,
I didn’t actually care too much about this increased reliability.
I don’t think it’s likely that Rust would buy me much over something like modern C++.
And if I were to <em>really</em> do some kind of cost-benefit analysis of several languages
— rather than going with Rust simply to learn it better —
then it would be hard to justify it over something like Go.</p>
<h4>It scales</h4>
<p>So the real question is: has Rust <em>not hampered</em> my productivity too much?
Having the benefit of hindsight,
I’m happy to say that the trade-off was definitely acceptable :)</p>
<p>One thing I was particularly satisfied with was the language’s <em>scalability</em>.
What I mean here is the ability to adapt as the project grows,
but also to start quickly and remain nimble
while the codebase is still pretty small.</p>
<p>Many languages (most, perhaps) are naturally tailored towards the large end,
doing their best to make it more bearable to work with big codebases.
In turn, they often forget about helping projects take off in the first place.
Between complicated build systems and dependency managers (Java),
or a virtual lack of either (C++),
it can be really hard to get going in a “serious” language like this.</p>
<p>On the other hand, languages like Python make it very easy to start up
and achieve relatively impressive results.
Some people, however, report having encountered problems
once the code evolves past certain size.
While I’m actually
<a href="http://xion.io/post/programming/long-live-dynamic-languages.html">very unsympathetic</a> to those claims,
I realize perception plays a significant role here,
making those anecdotal experiences into a sort of self-fulfilling prophecy.</p>
<p>This perception problem should almost certainly spare Rust,
as it’s a natively compiled and statically typed language,
with a respectable type system to boot.
There is also <a href="https://servo.org/">some evidence</a>
that the language works well in large projects already.
So the only question that we might want to ask is:
how easy it is to actually <em>start</em> a project in Rust,
and carry it towards some kind of <abbr title="Minimum Viable Product"><span class="caps">MVP</span></abbr>?</p>
<p>Based on my experiences with <em>gisht</em>,
I can say that it is, in fact, quite easy.
Thanks mostly to the impressive Swiss army knife of <code>cargo</code>
— acting as both package manager and a rudimentary build system —
it was almost Python-trivial to cook a “Hello World” program
that does something tangible, like
<a href="https://github.com/Xion/gisht/blob/de1be876784d671dd84618c3a15d0836f9fd5697/src/main.rs">talk to a <span class="caps">JSON</span> <span class="caps">API</span></a>.
From there, it only took a few coding sessions to grow it
into a <a href="https://github.com/Xion/gisht/tree/5c156cb">functioning prototype</a>.</p>
<h4>Abstractions galore</h4>
<p>As part of rewriting <em>gisht</em> from Python to Rust,
I also wanted to fix some longstanding issues that limited its capabilities.</p>
<p>The most important one was the hopeless coupling to GitHub
and their particular flavor of gists.
Sure, this is where the project even got its name from,
but people use a dozen of different services to share code snippets
and it should very possible to support them all.</p>
<p>Here’s where it became necessary to utilize
the abstraction capabilities that Rust has to offer.
It was somewhat obvious to
<a href="https://github.com/Xion/gisht/blob/3fc443dc9986612fd46b4311ca2ecbc613a15cf9/src/gist.rs#L16">define a <code>Host</code> trait</a>
but of course its exact form had to be
<a href="https://github.com/Xion/gisht/commit/26746dfc2eac68b67753f71148eb9897a861914e#diff-9d0a9c0911fa012f0fcf8ca56b43f8c5">shaped</a>
over <a href="https://github.com/Xion/gisht/commit/1e54ad05480b42089977171f10d4727beca5f835#diff-9d0a9c0911fa012f0fcf8ca56b43f8c5">numerous iterations</a>.
Along the way, it even turned out that <code>Result<Option<T>></code> and <code>Option<Result<T>></code>
are sometimes <a href="https://github.com/Xion/gisht/blob/d9c30e69d58b2a4e5608e6c8a1aa6392133b490f/src/hosts/mod.rs#L44">both necessary</a>
as return types :)</p>
<p>Besides cleaner architecture,
another neat thing about an explicit abstraction is
the ability to slice a concept into smaller pieces —
and then put <em>some of them</em> back together.
While the <code>Host</code> trait could support a very diverse set of gist services and <em>pastebins</em>,
many of them turned out to be just a slight variation of one central theme.
Because of this similarity, it was possible to introduce
a single <a href="https://github.com/Xion/gisht/blob/d2e78b1f5ee4616b1d5eb7067c3c5dd0ce9e2fe4/src/hosts/simple.rs#L26"><code>Basic</code> implementation</a>
which handles multiple services through varying sets of <span class="caps">URL</span> patterns.</p>
<p>Devices like these aren’t of course specific to Rust:
interfaces (traits) and classes are a staple of <span class="caps">OO</span> languages in general.
But some other techniques were more idiomatic;
the concept of <a href="https://doc.rust-lang.org/std/iter/trait.Iterator.html">iterators</a>, for example,
is flexible enough to accommodate
<a href="https://github.com/Xion/gisht/blob/4fa347c6197190b0f6c68dd548efc28287a5859f/src/hosts/github.rs#L354">looping over GitHub user’s gists</a>,
even as they read directly from <span class="caps">HTTP</span> responses.</p>
<h4>Hacking time</h4>
<p>Not everything was sunshine and rainbows, though.</p>
<p>Take <em>clap</em>, for example.
It’s mostly a very good crate for parsing command line arguments,
but it couldn’t <em>quite</em> cope with the unusual requirements that <em>gisht</em> had.
To make <code>gisht Foo/bar</code> work alongside <code>gisht run Foo/bar</code>,
it was necessary to
<a href="https://github.com/Xion/gisht/commit/0eff00e31f94f3856558ebb1f6655a9e6fc50ca6#diff-7397f82f682a49eb62e2b056118124d0">analyze <code>argv</code></a>
before even handing it over to <code>clap</code>.
This turned out to be
<a href="https://github.com/Xion/gisht/commit/e7ab06a01d4675947965ec82fc6f3ec5a2517c89#diff-7397f82f682a49eb62e2b056118124d0">surprisingly tricky</a>
to get right.
Like,
<a href="https://github.com/Xion/gisht/commit/69e8aad4a1743beb57184dc38150ef02b306a0a1#diff-7397f82f682a49eb62e2b056118124d0">really</a>
tricky, with
<a href="https://github.com/Xion/gisht/commit/acadcfa0a97fe52584fbf8198541baa8733cb0a5#diff-7397f82f682a49eb62e2b056118124d0R58">edges cases</a>
and
<a href="https://github.com/Xion/gisht/commit/a9eb599168f5b6821aefa46dde0b0a89a41cd4e6#diff-7397f82f682a49eb62e2b056118124d0R562">stuff</a>.
But as it is often the case in software,
the answer turned out to be yet another layer of indirection plus
<a href="https://github.com/Xion/gisht/blob/30df960052f2a03270bbb5ca1f7be0920978007d/src/args.rs#L457-L578">a copious amount of tests</a>.</p>
<p>In another instance, however, a direct library support was crucial.</p>
<p>It so happened that <em>hyper</em>, the crate I’ve been using for <span class="caps">HTTP</span> requests,
didn’t handle the <code>Link:</code> response header out of the box<sup id="fnref:3"><a class="footnote-ref" href="#fn:3" rel="footnote">3</a></sup>.
This was a stumbling block that prevented the gist iterator (mentioned earlier)
from correctly handling pagination in the responses from GitHub <span class="caps">API</span>.
Thankfully, having <a href="https://docs.rs/hyper/0.11.7/hyper/header/trait.Header.html">the <code>Header</code> abstraction</a> in <em>hyper</em>
meant it was possible to add the missing support in
<a href="https://github.com/Xion/gisht/blob/30df960052f2a03270bbb5ca1f7be0920978007d/src/ext/hyper.rs">a relatively straighforward manner</a>.
Yes, it’s <a href="https://github.com/Xion/gisht/blob/30df960052f2a03270bbb5ca1f7be0920978007d/src/ext/hyper.rs#L23">not a universal implementation</a>
that’d be suitable for <em>every</em> <span class="caps">HTTP</span> client,
but it does the job for <em>gisht</em> just fine.</p>
<h4>Test-Reluctant Development</h4>
<p>And so the program kept growing steadily over the months,
most notably through
<a href="https://github.com/Xion/gisht/blob/30df960052f2a03270bbb5ca1f7be0920978007d/src/hosts/mod.rs#L101">more and more gist hosts</a>
it could now support.</p>
<p>Eventually, some of them would fall into a sort of twilight zone.
They weren’t as complicated as GitHub to warrant writing a completely new <code>Host</code> instance,
but they also couldn’t be handled via
<a href="https://github.com/Xion/gisht/blob/30df960052f2a03270bbb5ca1f7be0920978007d/src/hosts/common/basic.rs#L28">the <code>Basic</code> structure</a> alone.
A good example would be <a href="http://sprunge.us/">sprunge.us</a>:
mostly an ordinary pastebin,
except for its optional syntax highlighting
which may add some “junk” to the otherwise regular URLs.</p>
<p>In order to handle those odd cases,
I went for a classic wrapper/decorator pattern which, in its essence,
boils down to something like this:</p>
<div class="highlight"><pre><span class="k">pub</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="n">Sprunge</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">inner</span><span class="o">:</span><span class="w"> </span><span class="n">Basic</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">impl</span><span class="w"> </span><span class="n">Sprunge</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="n">new</span><span class="p">()</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">Sprunge</span><span class="p">{</span><span class="n">inner</span><span class="o">:</span><span class="w"> </span><span class="n">Basic</span><span class="o">::</span><span class="n">new</span><span class="p">(</span><span class="n">ID</span><span class="p">,</span><span class="w"> </span><span class="s">"sprunge.us"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="s">"http://sprunge.us/${id}"</span><span class="p">,</span><span class="w"> </span><span class="p">...)}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">impl</span><span class="w"> </span><span class="n">Host</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">Sprunge</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// override & wrap methods that require custom logic:</span>
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="n">resolve_url</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">url</span><span class="o">:</span><span class="w"> </span><span class="o">&</span><span class="kt">str</span><span class="p">)</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="nb">Option</span><span class="o"><</span><span class="n">io</span><span class="o">::</span><span class="nb">Result</span><span class="o"><</span><span class="n">Gist</span><span class="o">>></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">url_obj</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">try_opt</span><span class="o">!</span><span class="p">(</span><span class="n">Url</span><span class="o">::</span><span class="n">parse</span><span class="p">(</span><span class="n">url</span><span class="p">).</span><span class="n">ok</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="n">url_obj</span><span class="p">.</span><span class="n">set_query</span><span class="p">(</span><span class="nb">None</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">inner</span><span class="p">.</span><span class="n">resolve_url</span><span class="p">(</span><span class="n">url_obj</span><span class="p">.</span><span class="n">to_string</span><span class="p">().</span><span class="n">as_str</span><span class="p">())</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="c1">// passthrough to the `Basic` struct for others:</span>
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="n">fetch_gist</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">gist</span><span class="o">:</span><span class="w"> </span><span class="o">&</span><span class="n">Gist</span><span class="p">,</span><span class="w"> </span><span class="n">mode</span><span class="o">:</span><span class="w"> </span><span class="n">FetchMode</span><span class="p">)</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">io</span><span class="o">::</span><span class="nb">Result</span><span class="o"><</span><span class="p">()</span><span class="o">></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">inner</span><span class="p">.</span><span class="n">fetch_gist</span><span class="p">(</span><span class="n">gist</span><span class="p">,</span><span class="w"> </span><span class="n">mode</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="c1">// (etc.)</span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>Despite the noticeable boilerplate of a few pass-through methods,
I was pretty happy with this solution, at least initially.
After a few more unusual hosts, however,
it became cumbersome to fix all the edge cases
by looking only at the final output of the inner <code>Basic</code> implementation.
The code was evidently asking for some <em>tests</em>,
if only to check how the inner structure is being called.</p>
<p>Shouldn’t be too hard, right?… Yeah, that’s what I thought, too.</p>
<p>The reality, unfortunately, fell very short of those expectations.
Stubs, mocks, fakes —
<a href="https://testing.googleblog.com/2013/07/testing-on-toilet-know-your-test-doubles.html"><em>test doubles</em></a>
in general —
are a dark and forgotten corner of Rust
that almost no one seems to pay any attention to.
Absent a proper library support — much less a language one —
the only way forward was to roll up my sleeves
and implement
<a href="https://github.com/Xion/gisht/blob/30df960052f2a03270bbb5ca1f7be0920978007d/src/testing/inmemory_host.rs">a fake <code>Host</code></a>
from scratch.</p>
<p>But that was just the beginning.
How do you seamlessly inject this fake implementation into the wrapper
so that it replaces the <code>Basic</code> struct for testing?
If you are not careful and go for the “obvious” solution — a trait object:</p>
<div class="highlight"><pre><span class="k">pub</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="n">Sprunge</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">inner</span><span class="o">:</span><span class="w"> </span><span class="n">Box</span><span class="o"><</span><span class="n">Host</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>you’ll soon realize that you need not just a <code>Box</code>, but at least an <code>Rc</code> (or maybe even <code>Arc</code>).
Without this kind of shared ownership,
you’ll lose your chance to interrogate the test double once you hand it over to the wrapper.
This, in turn, will heavily limit your ability to write effective tests.</p>
<p>What’s the non-obvious approach, then?
The full rationale would probably warrant a separate post,
but the working recipe looks more or less like this:</p>
<ul>
<li>
<p>First, <em>parametrize</em> the wrapper with its inner type:
<code>pub struct Sprunge<T: Host> { inner: T }</code>.</p>
</li>
<li>
<p>Put that in an internal module with the correct visibility setup:</p>
<div class="highlight"><pre><span class="kn">mod</span><span class="w"> </span><span class="n">internal</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="n">Sprunge</span><span class="o"><</span><span class="n">T</span><span class="o">:</span><span class="w"> </span><span class="n">Host</span><span class="o">></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">pub</span><span class="p">(</span><span class="n">super</span><span class="p">)</span><span class="w"> </span><span class="n">inner</span><span class="o">:</span><span class="w"> </span><span class="n">T</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
</li>
<li>
<p>Make the regular (“production”) version of the wrapper into an <em>alias</em>,
giving it the type parameter that you’ve been using directly<sup id="fnref:4"><a class="footnote-ref" href="#fn:4" rel="footnote">4</a></sup>:</p>
<div class="highlight"><pre><span class="k">pub</span><span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="n">Sprunge</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">internal</span><span class="o">::</span><span class="n">Sprunge</span><span class="o"><</span><span class="n">Basic</span><span class="o">></span><span class="p">;</span><span class="w"></span>
</pre></div>
</li>
<li>
<p>Change the <code>new</code> constructor to instantiate the <code>internal</code> type.</p>
</li>
<li>
<p>In tests, create the wrapper with a fake <code>inner</code> object inside.</p>
</li>
</ul>
<p>As you can see in
<a href="https://github.com/Xion/gisht/blob/30df960052f2a03270bbb5ca1f7be0920978007d/src/hosts/sprunge.rs">the real example</a>,
this convoluted technique removes the need for any pointer indirection.
It also permits you to
<a href="https://github.com/Xion/gisht/blob/30df960052f2a03270bbb5ca1f7be0920978007d/src/hosts/sprunge.rs#L152">access the out-of-band interface</a>
that a fake object would normally expose.</p>
<p>It’s a shame, though, that so much work is required for something
that should be very simple.
As it appears, testing is still a neglected topic in Rust.</p>
<h4>Packing up</h4>
<p>It wasn’t just Rust that played a notable role in the development of <em>gisht</em>.</p>
<p>Pretty soon after getting the app to a presentable state,
it became clear that a mere <code>cargo build</code> won’t do everything
that’s necessary to carry out a complete build.
It <em>could</em> do more, admittedly,
if I had the foresight to explore <a href="http://doc.crates.io/build-script.html">Cargo build scripts</a>
a little more thoroughly.
But overall, I don’t regret dropping back to my trusty ol’ pick: Python.</p>
<p>Like in a few previous projects, I used the <a href="http://pyinvoke.org">Invoke task runner</a>
for both the crucial and the auxiliary automation tasks.
It is a relatively powerful tool
— and probably the best in its class in Python that I know of —
though it can be a bit capricious if you want to
<a href="https://github.com/Xion/gisht/blob/30df960052f2a03270bbb5ca1f7be0920978007d/tasks/release/__init__.py#L69">really fine-tune it</a>.
But it does make it much easier to organize your automation code,
to reuse it between tasks, and to (ahem) <em>invoke</em> those tasks in a convenient manner.</p>
<p>In any case, it certainly beats a collection of disconnected Bash scripts ;)</p>
<p>What have I automated in this way, you may ask?
Well, a couple of small things; those include:</p>
<ul>
<li>
<p>embedding of the current Git commit hash into the binary,
to help identify the exact revision in the logs of any potential bug reports<sup id="fnref:5"><a class="footnote-ref" href="#fn:5" rel="footnote">5</a></sup></p>
</li>
<li>
<p>after a successful build,
<a href="https://github.com/Xion/gisht/blob/30df960052f2a03270bbb5ca1f7be0920978007d/tasks/build.py#L40">replacing</a>
the <em>Usage</em> section in <em><span class="caps">README</span></em> with the program’s <code>--help</code> output</p>
</li>
<li>
<p>generating <a href="https://github.com/Xion/gisht/blob/30df960052f2a03270bbb5ca1f7be0920978007d/tasks/build.py#L96">completion scripts</a>
for popular shells by invoking the binary with a magic hidden flag (courtesy of <em>clap</em>)</p>
</li>
</ul>
<p>Undoubtedly the biggest task that I relegated to Python/Invoke,
was the preparation of
<a href="https://github.com/Xion/gisht/blob/30df960052f2a03270bbb5ca1f7be0920978007d/tasks/release/__init__.py"><em>release packages</em></a>.
When it comes to the various Linuxes (currently Debian and Red Hat flavors),
this wasn’t particularly complicated.
Major thanks are due to the amazing <a href="https://github.com/jordansissel/fpm"><em>fpm</em> tool</a> here,
which I recommend to anyone who needs to package their software in a distro-compatible manner.</p>
<p>Homebrew, however — or more precisely, <span class="caps">OS</span> X itself — was quite a different story.
Many, <a href="https://github.com/Xion/gisht/commits/a5423a63d10221c50faa1cb30a999a85286853a1">many</a>
failed attempts were needed to even get it to build on <a href="https://travis-ci.org">Travis</a>,
and the additional dependency on Python was
<a href="https://github.com/Xion/gisht/blob/30df960052f2a03270bbb5ca1f7be0920978007d/ci/travis/before_install-osx.sh#L15">partially to blame</a>.
To be fair, however, most of the pain was exclusively due to OpenSSL;
getting that thing to build is always <a href="https://github.com/sfackler/rust-openssl/issues/255">loads of “fun”</a>,
especially in such an opaque and poorly debuggable environment as Travis.</p>
<h4>The wrap</h4>
<p>There’s probably a lot of minor things and tidbits I could’ve mentioned along the way,
but the story so far has most likely covered all the important topics.
Let’s wrap it up then, and highlight some interesting points in the classic <em>Yay/Meh/Nay</em> manner.</p>
<h5>Yay</h5>
<ul>
<li>
<p>It was definitely a good choice to rewrite <em>gisht</em> specifically in Rust.
Besides all the advantages I’ve mentioned already,
it is also worth noting that the language went through about 10 minor version bumps
while I was working on this project.
Of all those new releases,
I don’t recall a single one that would introduce a breaking change.</p>
</li>
<li>
<p>Most of the Rust ecosystem (third-party libraries) was a joy to use,
and very easy to get started with.
Honorable mention goes to <em>serde_json</em> and how easy it was to
<a href="https://github.com/Xion/gisht/commit/a0655a6a5b86c05df5665e3bc7f1512f2476c9e4">transition the code</a>
from <em>rustc_serialize</em> that I had used at first.</p>
</li>
<li>
<p>With a possible exception of sucking in node.js as a huge dependency of your project
and using Grunt, there is probably no better way of writing automation <span class="amp">&</span> support code than Python.
There may eventually be some Rust-based task runners that could try to compete,
but I’m not very convinced about using a compiled language for this purpose
(and especially one that takes so long to build).</p>
</li>
</ul>
<h5>Meh</h5>
<ul>
<li>
<p>While <a href="https://docs.rs/clap">the <em>clap</em> crate</a> is quite configurable and pretty straightforward to use,
it does lack at least <a href="https://github.com/kbknapp/clap-rs/issues/568">one feature</a>
that’d be very nice for <em>gisht</em>.
Additionally, working with raw <em>clap</em> is often a little tedious,
as it doesn’t assist you in translating parsed flags into your own configuration types,
and thus requires
<a href="https://github.com/Xion/gisht/blob/30df960052f2a03270bbb5ca1f7be0920978007d/src/args.rs#L138-L182">shuffling those bits</a>
manually<sup id="fnref:6"><a class="footnote-ref" href="#fn:6" rel="footnote">6</a></sup>.</p>
</li>
<li>
<p>Being a <em>defacto</em> standard for continuous integration in open-source projects,
<a href="https://travis-ci.org">Travis <span class="caps">CI</span></a> could be a <em>little</em> less finicky.
In almost every project I decide to use it for,
I end up with about half a dozen commits
that frantically try to fix silly configuration issues,
all before even a simple <em>.travis.yml</em> works as intended.
Providing a way to test <span class="caps">CI</span> builds locally would be an obvious way to avoid this churn.</p>
</li>
</ul>
<h5>Nay</h5>
<ul>
<li>
<p>Testing in Rust is such a weird animal.
On one hand, there is a first-class, out-of-the-box support for unit tests
(and even integration tests) right in the toolchain.
On the other hand, the relevant parts of the ecosystem are immature or lacking,
as evidenced by the dreary story of mocking and stubbing.
It’s no surprise that there is a long way to catch up to languages with the strongest testing culture
(Java and C#/.<span class="caps">NET</span><sup id="fnref:7"><a class="footnote-ref" href="#fn:7" rel="footnote">7</a></sup>), but it’s disappointing to see Rust outclassed
<a href="https://github.com/google/googletest">even by C++</a>.</p>
</li>
<li>
<p>Getting anything to build reliably on <span class="caps">OSX</span> in a <span class="caps">CI</span> environment is already a tall order.
But if it involves things as OpenSSL, then it quickly goes from bad to terrible.
I’m really not amused anymore how this “Just Works” system often turns out to hardly work at all.</p>
</li>
</ul>
<p>Since I don’t want to end on such a negative note,
I feel compelled to state the obvious fact: every technology choice is a trade-off.
In case of this project, however, the drawbacks were <em>heavily</em> outweighed by the benefits.</p>
<p>For this reason, I can definitely recommend the software stack I’ve just described
to anyone developing non-trivial, cross-platform command line tools.</p>
<div class="footnote">
<hr />
<ol>
<li id="fn:1">
<p>This is not an isolated complaint, by the way,
as the interpreter startup time has recently emerged as <a href="https://lwn.net/Articles/730915/">an important issue</a>
to many developers of the Python language. <a class="footnote-backref" href="#fnref:1" rev="footnote" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:2">
<p>Which may also include a practical <em>lack</em> thereof. <a class="footnote-backref" href="#fnref:2" rev="footnote" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:3">
<p>It does handle it <a href="https://docs.rs/hyper/0.11.7/hyper/header/struct.Link.html">now</a>, fortunately. <a class="footnote-backref" href="#fnref:3" rev="footnote" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:4">
<p>Observant readers may notice that we’re exposing a technically private type (<code>internal::Sprunge</code>)
through a publicly visible type alias. If that type was <em>actually</em> private,
this would trigger a compiler warning
which is slated to become <a href="https://github.com/rust-lang/rust/issues/34537">a hard error</a>
at some point in the future. But, amusingly, we can fool the compiler by making it a
<em>public type</em> inside a <em>private module</em>, which is exactly what we’re doing here. <a class="footnote-backref" href="#fnref:4" rev="footnote" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn:5">
<p>This has since been rewritten and is now done in
<a href="https://github.com/Xion/gisht/blob/30df960052f2a03270bbb5ca1f7be0920978007d/build.rs"><em>build.rs</em></a>
— but that’s only because I implemented
<a href="https://github.com/rust-lang/cargo/pull/3929">the relevant Cargo feature</a> myself :) <a class="footnote-backref" href="#fnref:5" rev="footnote" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
<li id="fn:6">
<p>For an alternative approach that doesn’t seem to have this problem,
check <a href="https://docs.rs/structopt_derive">the <em>structopt</em> crate</a>. <a class="footnote-backref" href="#fnref:6" rev="footnote" title="Jump back to footnote 6 in the text">↩</a></p>
</li>
<li id="fn:7">
<p>Dynamically typed languages, due to their rich runtime,
are basically a class of their own when it comes to testing ease,
so it wouldn’t really be fair to hold them up for comparison. <a class="footnote-backref" href="#fnref:7" rev="footnote" title="Jump back to footnote 7 in the text">↩</a></p>
</li>
</ol>
</div>