Thiago's weblogZola2022-02-03T00:00:00+00:00https://www.thiagovarela.com.br/atom.xmlFast APIs with Rust and Axum - Part 1 (in progress)2022-02-03T00:00:00+00:002022-02-03T00:00:00+00:00https://www.thiagovarela.com.br/fast-apis-with-rust-and-axum-1/<blockquote>
<p>Can we move fast building rust web applications? Being <strong>pragmatic</strong> matters, this is my attempt to show how (or if?) we can take Rust's recent ecosystem growth to <strong>practical</strong> applications. </p>
</blockquote>
<h2 id="first-things-first">First things first</h2>
<p>If you are unfamiliar with Rust, it's a fun and long journey that starts here: <a href="https://rustup.rs/" target="_blank">https://rustup.rs/</a>. Follow along to install Rust and Cargo.</p>
<p>The remaining content assumes you are a beginner like me, master of copy/paste, but knows what <a href="https://doc.rust-lang.org/cargo/" target="_blank">Cargo</a> is.</p>
<p>We'll use a bunch of Rust libraries to write our API.</p>
<p><em><a href="https://docs.rs/axum/latest/axum/" target="_blank">axum</a> is a web application framework that focuses on ergonomics and modularity.</em></p>
<p>Standing on the shoulders of:</p>
<p><em><a href="https://docs.rs/tokio/latest/tokio/" target="_blank">tokio</a> is a runtime for writing reliable network applications without compromising speed.</em></p>
<p><em><a href="https://docs.rs/hyper/latest/hyper/" target="_blank">hyper</a> is a fast and correct HTTP implementation written in and for Rust.</em></p>
<p><em><a href="https://docs.rs/tower/latest/tower/" target="_blank">tower</a> Tower is a library of modular and reusable components for building robust networking clients and servers.</em></p>
<p>As you can read, lots of nice adjectives.</p>
<p><strong>What are we doing anyway?</strong></p>
<p>We'll build a <code>Hello World</code>, with a test. 🥱 so boring. Structuring the test is the most important thing here. </p>
<h2 id="getting-started">Getting started</h2>
<pre data-lang="sh" style="background-color:#2b303b;color:#c0c5ce;" class="language-sh "><code class="language-sh" data-lang="sh"><span>> cargo </span><span style="color:#bf616a;">new --bin</span><span> api_project
</span><span>> cd </span><span style="color:#bf616a;">api_project
</span><span>> mkdir </span><span style="color:#bf616a;">tests
</span><span>> touch </span><span style="color:#bf616a;">src/lib.rs
</span></code></pre>
<p>By the way, <code>cargo new</code> will add the <code>Cargo.toml</code> file where we manage dependencies and a <code>main.rs</code> file that is the binary entrypoint.</p>
<p>The first thing we want is the ability to create <a rel="noopener" target="_blank" href="https://doc.rust-lang.org/book/ch11-03-test-organization.html#integration-tests">integration tests</a> to ensure our application is doing what is supposed to, as opposed to a simple <code>it compiles</code>, the <code>lib.rs</code> is what makes the magic for our binary project.</p>
<p>Let's add some dependencies to <code>Cargo.toml</code>, we'll keep tokio with full features for the sake of this experiment, same with tower as we need it sooner then we think.</p>
<pre data-lang="toml" style="background-color:#2b303b;color:#c0c5ce;" class="language-toml "><code class="language-toml" data-lang="toml"><span>[dependencies]
</span><span style="color:#bf616a;">axum </span><span>= "</span><span style="color:#a3be8c;">0.4</span><span>"
</span><span style="color:#bf616a;">tokio </span><span>= { </span><span style="color:#bf616a;">version </span><span>= "</span><span style="color:#a3be8c;">1</span><span>", </span><span style="color:#bf616a;">features </span><span>= ["</span><span style="color:#a3be8c;">full</span><span>"] }
</span><span style="color:#bf616a;">tower </span><span>= { </span><span style="color:#bf616a;">version </span><span>= "</span><span style="color:#a3be8c;">0.4</span><span>", </span><span style="color:#bf616a;">features </span><span>= ["</span><span style="color:#a3be8c;">full</span><span>"] }
</span></code></pre>
<p>Our <code>lib.rs</code> initially will have a public function to retrieve an axum router.</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">use </span><span>axum::{routing::get, Router};
</span><span>
</span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">application</span><span>() -> Router {
</span><span> </span><span style="color:#65737e;">// Here we are creating a new Router with a route /health, that has a get handler
</span><span> Router::new().</span><span style="color:#96b5b4;">route</span><span>("</span><span style="color:#a3be8c;">/health</span><span>", </span><span style="color:#96b5b4;">get</span><span>(health))
</span><span>}
</span><span>
</span><span style="color:#65737e;">// This is the handler for the /health endpoint, the return type is a string slice that I purposely pronounce 'string'.
</span><span>async </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">health</span><span>() -> &</span><span style="color:#b48ead;">'static str </span><span>{
</span><span> "</span><span style="color:#a3be8c;">OK</span><span>"
</span><span>}
</span></code></pre>
<p>Cool, now let's write a test.</p>
<pre data-lang="sh" style="background-color:#2b303b;color:#c0c5ce;" class="language-sh "><code class="language-sh" data-lang="sh"><span>> cd </span><span style="color:#bf616a;">tests
</span><span>> mkdir </span><span style="color:#bf616a;">shared
</span><span>> touch </span><span style="color:#bf616a;">shared/mod.rs
</span><span>> touch </span><span style="color:#bf616a;">health.rs
</span></code></pre>
<p>The <code>shared/mod.rs</code> will look like this, and this is so we can share between many test cases.</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">use </span><span>axum::Router;
</span><span>
</span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">test_app</span><span>() -> Router {
</span><span> api_project::application()
</span><span>}
</span></code></pre>
<p>Now, back to <code>tests/health.rs</code>, <del>we'll make this compile and comment line by line.</del> </p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">use </span><span>axum::{http::{StatusCode, Request}, body::Body};
</span><span style="color:#b48ead;">use </span><span>tower::ServiceExt;
</span><span>
</span><span style="color:#b48ead;">mod </span><span>shared;
</span><span>
</span><span>#[</span><span style="color:#bf616a;">tokio</span><span>::</span><span style="color:#bf616a;">test</span><span>]
</span><span>async </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">test_health_endpoint</span><span>() {
</span><span> </span><span style="color:#b48ead;">let</span><span> app = shared::test_app();
</span><span> </span><span style="color:#b48ead;">let</span><span> request = Request::get("</span><span style="color:#a3be8c;">/health</span><span>");
</span><span>
</span><span> </span><span style="color:#b48ead;">let</span><span> request = request.</span><span style="color:#96b5b4;">body</span><span>(Body::empty()).</span><span style="color:#96b5b4;">unwrap</span><span>();
</span><span> </span><span style="color:#b48ead;">let</span><span> response = app.</span><span style="color:#96b5b4;">oneshot</span><span>(request).await.</span><span style="color:#96b5b4;">unwrap</span><span>();
</span><span>
</span><span> assert_eq!(response.</span><span style="color:#96b5b4;">status</span><span>(), StatusCode::</span><span style="color:#d08770;">OK</span><span>);
</span><span>}
</span></code></pre>
<p>We are adding axum re-exports of http <code>Request</code>, <code>Body</code> and <code>StatusCode</code> that are used throughout the code.
The <code>#[tokio::test]</code> is a macro to assign the async function to run using tokio runtime. </p>
<p>We're getting an instance of the Router through our shared module;</p>
<p>Then we build a GET request to <code>/health</code>;</p>
<p>We rebind (because we no longer need the builder) with <code>let request</code> making our request builder become a request to <code>/health</code> with an empty body.</p>
<p>Then things can get a bit interesting, we need to run the request through the router and get a response. Wat? </p>
<p>If you see again, we added a <code>use tower::ServiceExt;</code> at the top, which applies a bit of extra functionality to structs implementing <code>tower::Service</code>, and <code>axum::Router</code> does it. So we can use the <a rel="noopener" target="_blank" href="https://docs.rs/tower/latest/tower/trait.ServiceExt.html#method.oneshot">Oneshot</a> utility function.</p>
<p>With oneshot, we can now have a response like if it was ran throuh an actual server and we can play with its result.</p>
<p>Back to the terminal, let's run <code>cargo test</code> and see how it goes:</p>
<pre data-lang="sh" style="background-color:#2b303b;color:#c0c5ce;" class="language-sh "><code class="language-sh" data-lang="sh"><span>> cargo </span><span style="color:#bf616a;">test
</span><span>
</span><span> </span><span style="color:#bf616a;">Running</span><span> tests/health.rs (target/debug/deps/health-a801109dcc6aab35)
</span><span>
</span><span style="color:#bf616a;">running</span><span> 1 test
</span><span style="color:#bf616a;">test</span><span> test_health_endpoint ... ok
</span></code></pre>
<p>Now, we know it works, but for the sake of clarity, we should expand a little bit more on this specific line:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">let</span><span> request = request.</span><span style="color:#96b5b4;">body</span><span>(Body::empty()).</span><span style="color:#96b5b4;">unwrap</span><span>();
</span><span style="color:#b48ead;">let</span><span> response = app.</span><span style="color:#96b5b4;">oneshot</span><span>(request).await.</span><span style="color:#96b5b4;">unwrap</span><span>();
</span></code></pre>
<p>In Rust, handling errors is mainly(or maybe not, it depends) done by the Result type, that contains a successful value or an error.
.unwrap() consumes the success or terminates the current thread. Quite reasonable for a test suite, right?</p>
<p><code>let request = request.body(Body::empty()).unwrap();</code> is creating a <code>Request<Body></code> that can be consumed by oneshot.</p>
<p><code>let response = app.oneshot(request).await.unwrap();</code> is calling oneshot with the request, in return it will give a <code>future</code>, and to briefly explain what that is, it is something that can run asynchronously. A future by itself does nothing unless we <code>await</code> on them, that's the keyword kicking in to execute the future (inside tokio because of that nice macro at the top of the test), then unwrapping its Ok value. </p>
<p>Phew, finally we have the contents of the response.</p>
<p>This is really basic, but we can now replicate the same thing to other test cases and start building easier abstractions to make tests a more pleasant experience.</p>
<p>But where is the borrow checker, and and lifetimes? 😱! I left out several Rust concepts from this because what I would like to show is that we can do things that are somewhat easy to explain. </p>
<h2 id="running-the-server">Running the server</h2>
<p>So far, we worked in the <code>lib.rs</code> and tests folder. </p>
<p>Le's make use of our application in the <code>main.rs</code> and run the server:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">use </span><span>std::net::SocketAddr;
</span><span>
</span><span>#[</span><span style="color:#bf616a;">tokio</span><span>::</span><span style="color:#bf616a;">main</span><span>]
</span><span>async </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">main</span><span>() -> Result<(), tower::BoxError> {
</span><span> </span><span style="color:#b48ead;">let</span><span> app = api_project::application();
</span><span>
</span><span> </span><span style="color:#b48ead;">let</span><span> address: SocketAddr = "</span><span style="color:#a3be8c;">[::0]:4000</span><span>".</span><span style="color:#96b5b4;">parse</span><span>()?;
</span><span>
</span><span> println!("</span><span style="color:#a3be8c;">Listening on </span><span style="color:#d08770;">{}</span><span>", address);
</span><span>
</span><span> axum::Server::bind(&address)
</span><span> .</span><span style="color:#96b5b4;">serve</span><span>(app.</span><span style="color:#96b5b4;">into_make_service</span><span>())
</span><span> .await?;
</span><span>
</span><span> Ok(())
</span><span>}
</span></code></pre>
<blockquote>
<p>The question mark operator (?) unwraps valid values or returns errornous values. This is an elegant way of improving readability as long as the code is prepared to handle the Result type.</p>
</blockquote>
<p>Back to the terminal, on the root folder where Cargo.toml is, run <code>cargo run</code> and we should see something like:</p>
<pre data-lang="sh" style="background-color:#2b303b;color:#c0c5ce;" class="language-sh "><code class="language-sh" data-lang="sh"><span style="color:#bf616a;">Finished</span><span> dev </span><span style="color:#b48ead;">[</span><span>unoptimized + debuginfo</span><span style="color:#b48ead;">]</span><span> target(s) </span><span style="color:#bf616a;">in</span><span> 0.03s
</span><span> </span><span style="color:#bf616a;">Running </span><span>`</span><span style="color:#bf616a;">target/debug/api_project</span><span>`
</span><span style="color:#bf616a;">Listening</span><span> on </span><span style="color:#b48ead;">[</span><span>::</span><span style="color:#b48ead;">]</span><span>:4000
</span></code></pre>
<p>Cool, you can hit the endpoint in the browser or other client.</p>
<h2 id="error-handling">Error handling</h2>
<p>A very important piece of an API is how we give back information to API consumers of something that did not go well with the request.</p>
<p>Let's dig a bit how Axum helps in that front.</p>
Failure story2021-05-10T00:00:00+00:002021-05-10T00:00:00+00:00https://www.thiagovarela.com.br/failure-story/<p>Before things get lost in memory, I'd like to share what happened with a project I had back in 2017.</p>
<h2 id="the-project">The project</h2>
<p>The project, called <code>avaliando</code> (Portuguese for <code>assessing</code>), was an ed-tech side project that aimed at helping teachers optimizing their time while doing assessments, by doing automatic corrections.</p>
<p>It came to my hands through an informal chat with a coworker, and I guess after I was in, I messed it all up 😂</p>
<p>With my ideas, the project became more of a social endeavour. I wanted to help public schools, teachers and students by giving them a tool to build, correct and track exams, apply adaptive learning based on the individual historical data and guide them through a better learning experience. </p>
<p>Our initial goal was to apply this in a very poor and unequal state in Brazil's northeast, not by choice but because we lived there.</p>
<p>Briding a gap for kids in public school and leveling them up a bit seemed a very interesting idea.</p>
<h2 id="what-went-wrong">What went wrong</h2>
<p>So, this was basically my naivety at trying to do something good. I confess that I spent only about 10% of my time on this, all else had to go to the actual paying job, even so I tried to do my research and see what we could do with this, and we did something, sort of.</p>
<p>Our MVP was okish to onboard a school and teacher and allow them to create exams, print and feedback to the system the <a rel="noopener" target="_blank" href="https://en.wikipedia.org/wiki/Optical_mark_recognition">OMR</a>(<code>optical mark recognition</code>), using an Android app. As you can see, this was a total lack of reality check, trying to put a specific technology to solve a problem that requires a lot of groundwork.</p>
<p>I'll list some of the issues I remember:</p>
<ul>
<li>Since this was an education project, it was really hard to figure out funding, even with some positive feedback from local governments they wanted to see everything working before <code>paying</code>(or going through the contract process). -- not to mention I'm really bad at it.</li>
<li>The government was only interested in Portuguese and Mathematics aiming the PISA assessment. </li>
<li>Building up a decent question bank to allow teachers to easily create and adapt to their own class reality was hard, we had no money ;)</li>
<li>Where was my head when I thought that a school teacher would apply OMR exams to kids in remote areas? Even with scanners, I was creating a problem.</li>
<li>When I mentioned I messed up, I think I broadened the scope of the project too much for a small team of 4 with none of them full time. </li>
<li>I had a strong bias towards the value I was proposing, and anything other than that was simply not good enough. </li>
</ul>
<h2 id="lessons-learnt">Lessons learnt</h2>
<p>(maybe I'm stating the obvious)</p>
<ul>
<li>Your time is scarce, it is bad to try to tackle a bunch of things altogether. </li>
<li>Tech folks alone can't do much, we need help from people that understand the business.</li>
</ul>
Experimenting unikernels2021-04-06T00:00:00+00:002021-04-06T00:00:00+00:00https://www.thiagovarela.com.br/experienting-unikernels/<blockquote>
<p>Unikernels are specialised, single-address-space machine images constructed by using library operating systems. - <a rel="noopener" target="_blank" href="http://unikernel.org/">unikernel.org</a></p>
</blockquote>
<p>The first time I heard the term was last week, since then I did some experimentation with <a rel="noopener" target="_blank" href="https://nanovms.com/">Nanos</a> and I plan to take a look at <a rel="noopener" target="_blank" href="http://osv.io">OSv</a>. </p>
<p>With Nanos I was able to deploy a simple Rust http service to EC2, tweaking a bit their <a rel="noopener" target="_blank" href="https://ops.city">Ops</a> tool and setting up a CDK Stack to make things easier. Got stuck when trying to work with environment variables at runtime.</p>
<h2 id="but-why">But why?</h2>
<p>Well, I'm quite excited learning this stuff, though I know nothing about the inner bits of OSes and kernels and virtualisations. It seems intriguing, being able to optimise your software (that runs in a virtualised cloud) and reduce the cost and consumption (cpu, memory, energy).</p>
<p>It's a bit scary to dedicate time to unikernels, if you browse the web you'll find very little. The first impression is that it is an old concept not broadly accepted by the enterprise, seems folks are happy with K8s and containers. At the same time we see solutions like <a rel="noopener" target="_blank" href="https://firecracker-microvm.github.io/">Firecracker</a> being used by AWS Lambda. </p>
<p>I like the idea of a very limited toolset, that we have to overcome those limitations in our software, bringing loads of creativity. Not to mention the benefits, small, precise and safe.</p>
<p>Worst case scenario, I'll learn something new. </p>
How I plan to use this weblog.2021-01-21T00:00:00+00:002021-01-21T00:00:00+00:00https://www.thiagovarela.com.br/how-i-plan-to-use-this-weblog/<h2 id="content">Content</h2>
<p>Well, putting some of my thoughts in the open is one thing.
I'll probably talk about web, a couple of programming languages if my knowledge of them let me.
Apply some <a rel="noopener" target="_blank" href="https://indieweb.org/">IndieWeb</a> concepts as well.</p>
<h2 id="tech-behind">Tech behind</h2>
<p>After months trying to decide what tooling I would be using that can satisfy my needs of content publishing, I ended up deciding not to satisfy myself and just start, otherwise this would be again another year that I'm keen on doing things, but don't. </p>
<p>Nowadays there are plenty ways of hosting a website without paying for a server (or very little), like Netlify, Vercel, S3, but when it comes to analytics they charge you by a lot.</p>
<p><a rel="noopener" target="_blank" href="https://www.netlify.com/products/analytics/">Netlify Analytics</a> cost $9 per month, sorry but no.</p>
<p><a rel="noopener" target="_blank" href="https://vercel.com/docs/analytics#limits">Vercel</a> is like 1 day analytics for the free tier.</p>
<p><a rel="noopener" target="_blank" href="https://aws.amazon.com/cloudfront/">AWS CloudFront</a> gives a very nice analytics dashboard included in their pricing, but it can't serve subfolder root files from S3 if not using Lambda functions, on top of that Route53 isn't supporting CNAME to APEX domains. I'm sure moving out of Route53 and creating a lambda function would solve the problem, but let's avoid big tech for small things.</p>
<p>Ffs, I just want to check if a web page is being accessed. Some offline HTTP log analyzer (remember <a rel="noopener" target="_blank" href="https://awstats.sourceforge.io">awstats</a>?) would suffice. - <a rel="noopener" target="_blank" href="https://goaccess.io">GoAccess</a> seems a decent approach.</p>
<hr />
<h3 id="my-requirements">My requirements</h3>
<ul>
<li>No git deployments.</li>
<li>Simplest analytics through the http logs, no tracking.</li>
<li>Local content files - actually I wanted dynamic content but gave up the idea.</li>
<li>On top of local files, CLI command to publish (like awscli s3 sync)</li>
<li>Comments? Erm... </li>
</ul>