Like most, I like to learn a new programming language by doing. At the moment, I’m enjoying the catharsis of completing unimplemented Rosetta Code tasks in Crystal and vlang. During some exploration of the Crystal programming language, I took on the Synchronous Concurrency task to pour some of my knowledge of the CSP pattern from Go, into Crystal. The task asks you to communicate between multiple threads of execution within a process:
I’ve been getting rather excited about the Crystal programming language recently and I’ve finally gotten round to creating a very simple CockroachDB-backed Crystal API! In this example, I’m going to create a very contrived API to manage todos. When the API is up and running, you’ll be able to list the todos stored in the database, find a todo by its ID, and add new ones. These use cases lend themselves nicely to demonstrating 3 of the DB::QueryMethods.
In this post, I create a quick and dirty Go web server to demonstrate password reset functionality. My goal is to show you the basics of a password reset flow, so to keep things simple, I’ve taken a couple of shortcuts. Please read this for lots of best practice advice on this process. No HTTPS - The HTTP server accepts passwords in plaintext. No CAPTCHAs - If you want to prevent against bruteforce attempts to discover user accounts, you’ll want to implement this at the point the user submits their reset request.
Testing is vitally important to the success of software and software that connects to databases is no exception. There are a number of ways to test database code, ranging from mocks, to full integration tests that talk to an actual database. When writing mocks, I’ve had some great experiences with Data Dog’s sqlmock and I’d recommend that you check it out. In this post however, I’m going to devle into the steps involved in integration testing a CockroachDB database using docker-compose.
MySQL has a useful data type called ENUM which is an enumeration of values to validate against a given column. An ENUM value when used, occupies 1 byte, rather than the full length of the string value. ENUM values can be created as follows: CREATE TABLE `person` ( `id` BIGINT(20) NOT NULL AUTO_INCREMENT, `country` ENUM('DE', 'FR', 'UK', 'US') NOT NULL, PRIMARY KEY (`id`) ); …and updated as follows: ALTER TABLE `person` MODIFY COLUMN `country` ENUM('DE', 'FR', 'UK', 'US', 'ZW') NOT NULL AFTER `id`; In CockroachDB, there is no ENUM type (although this is being discussed here and here).
The cockroach demo command is my current favourite dev tool. It’s instant access to an N-node CockroachDB cluster, allowing you to get up and running with an enterprise cluster straight from your dev environment, reducing feedback loops massively by giving you all the features you need to create with confidence. If you start a cluster with cockroach demo, and don’t provide any arguments for --demo-locality, you’ll get the default values that are provided by the demo code.
CockroachDB’s Geo-partitioning allows you to partition data using location information like country codes. If you’re wanting to partition your data by-country, your data structure will likely already have the necessary components to do so. For example, the following table lends itself to being partitioned on its “country” column, meaning rows containing a country code of ‘DE’, ‘FR’, and ‘UK’ can be pinned to European data centres, while rows containing a country column of ‘US’ can be pinned to North American data centres etc:
On October 17th, I gave a talk at the inaugural east-cost multi-cloud conference, ESCAPE in Manhattan, organised and hosted by Cockroach Labs. I wanted to deliver this talk for a number of reasons: The tech industry isn’t talking about these issues as much as it should be. There are very few tech conferences where this kind of talk would be encouraged. I know next to nothing about the subject and there’s no greater kick up the arse to learn something than commit to speaking at a conference.
I recently gave a talk at a meetup in London and posed the following set of questions to the audience: “Who develops applications that talk to a database?” 70% of hands up. “Who develops applications that talk to a database and tests with production amounts of data?” 60% of hands up. “Who develops applications that talk to a database and tests with production amounts of data during development?
Anyone who ends up writing a load tester will soon arrive at the need for something to execute a function a number of times per second for a number of seconds… In the wonderful load testing CLI tool hey, JBD accomplishes this elegantly with channels, optionally configuring and waiting on a ticker channel: var throttle <-chan time.Time if b.QPS > 0 { throttle = time.Tick(time.Duration(1e6/(b.QPS)) * time.Microsecond) } ... if b.
In a previous post, I create a simple semaphore that took advantage of channel write blocking to acheive a maximum level of concurrency. Since then, I’ve turned this code into a package to document its usage and get some feedback. Background My goal in writing semaphore was to move the responsibility of concurrency limiting from the developer into a library that provides a declarative interface for it. The following code shows a traditional sync.
If you’ve ever needed to kick off multiple goroutines from func main, you’d have probably noticed that the main goroutine isn’t likely to hang around long enough for the other goroutines to finish: package main import ( "fmt" "time" ) func main() { go run(1, "A") go run(5, "B") } func run(iter int, name string) { for i := 0; i < iter; i++ { time.Sleep(time.Second) fmt.Println(name) } } It’ll come as no surprise that this program outputs nothing and exits with an exit code of 0.
When developing a service, you’ll want to ensure that you have decent test coverage of its endpoints. I’ve seen endpoint coverage achieved in a number of ways, some interesting, some interesting and I’d like to share my method. If you’ve arrived here, you’re probably no stranger to validating input to endpoints and it’s this use case that I’ll be demonstrating here. First and foremost, I’m using table-drive tests. In the context of service testing, they allow me to make multiple, requests to a service, using similar request bodies, without changing the test code I’m using to invoke the service, or polluting my test cases with huge, largely identical code.
One of the most useful go build flags I’ve used recently is tags. Its purpose is to look at the build tags you’ve specified and use them to drive what’s sent to the compiler for compilation. The tags flag is also available to the go test command, which means you can toggle swathes of tests on and off, without having to resort to code changes such as using flag in your TestMain function or - heaven forbid - writing logic into your tests themselves.
The benefits of thorough testing can’t be understated, especially when it comes to anything that’s in any way magical. For a language that avoids magic, I’d consider Go’s named return values and defer pretty magical, when used together. A big assumption and a very subtle syntax oversight and this magic would have lead to a serious production issue had it not been for a test that saved the day.
I ran into a curious little gotcha with Go’s iota construct today and I wanted to share it with you. Without cheating, what would you expect the following code to output? package main import "fmt" const ( one = 1 << iota two ) func main() { fmt.Println(one, two) } If you said 1 2, you’d be correct! There’s nothing fishy going on here. Now let’s suppose we need to add another constant.
Go’s httptest package provides a really simple, elegant way to test your HTTP services. It allows you to create requests to and capture the responses from anything that implements the http.Handler interface: type Handler interface { ServeHTTP(ResponseWriter, *Request) } There are many ways to acheive this and each is tested slightly different. A colleague of mine recently asked for some help testing his HTTP server and I’m hoping that this post might help others test theirs, regardless of how they’ve implemented it.
It’s important to be explicit about the dependencies in your application. If your Server struct requires access to a database, it makes sense to force consumers to provide it with the means to connect to that database during creation. Peter Bourgon’s brilliant Go best practices, six years in post makes a brilliant case for other mandatory explicit dependencies and I urge you to read it. What about optional dependencies? When starting a new application, I’m always faced with the decision of how best to manage optional dependencies.
Semaphores are like mutexes that allow N threads to access a shared resource instead of just 1. Go’s buffered channels make creating semaphore-like behaviour a doddle. To create a semaphore that allows 10 threads to concurrently access a shared resource, its simply make(chan struct{}, 10) etc. The only thing our threads need to lock/own is an item of the buffered channel, so it makes sense to use an empty struct, as it uses zero memory.
Channels are Go’s implementation of Tony Hoare’s CSP concurrency model. Rather than reiterate the basics, I’ll dive straight into some silly channel tricks that I’ve found to be useful. Note that these examples are designed to be easy-to-follow, rather than easy-to-copy-straight-into-production. Prevent sender from blocking Sending to a full channel blocks by design. This prevents fast senders from saturating a channel (and your available memory) but it’ll penalise them by forcing them to block, instead of penalising slow readers for not keeping up.
From time to time, you’ll want to serialise/deserialise a struct whose default serialisation seems counter-intuitive. Take for example, the JSON serlisation of time.Duration: json.NewEncoder(os.Stdout).Encode(time.Hour) // result: 3600000000000 I don’t event think science knows what this number is. Asking people to configure 1 hour as “3600000000000” is not only cruel, it’s asking for trouble; miss a zero and you’ve had it. A much friendlier and more natural alternative is to allow for the confguration of time.
Let’s Encrypt is rocking the SSL boat and the water’s warm. If you need that swanky EV banner, you’re happy to pay $€£¥ for the privilege and you want to manually renew/replace expired certificates, this post is probably not for you. tl;dr Running services in containers (Docker in particular) is becoming more and more popular, as is securing stuff for free with Let’s Encrypt. This post shows you how to combine the two.
I like a good index. I especially like the handy MongoDB TTL indexes you can apply to date columns to take care of old data automatically. My latest personal project manages secrets and is backed by BoltDB, a charming embedded key/value store, written in Go. Being as simple as it is, BoltDB doesn’t have the concept of item expiry. I spent a while Googling for elegant solutions but ended up (as we all often do), rolling my own.
Struct tags in Go provide metadata to fields and are used heavily by the encoders in the stdlib’s encoding package. Here’s a typical use case for a struct tag: import "encoding/json" type ServerConfig struct { Port int `json:"port"` APIKey string `json:"apiKey"` } func main() { bytes, _ := json.Marshal(ServerConfig{ Port: 1234, APIKey: "something secret", }) fmt.Println(string(bytes)) } The struct tags in this example give Go’s JSON encoder an explicit name to use when marshalling/unmarshalling ServerConfig structs.
Go’s tooling continues to delight and this one’s a real hidden gem… I’ve just discovered, with the help of exago.io, the testing/quick package in Go’s standard library. It’s a very simple blackbox testing package, which repeatedly executes a given block of your code with values you wouldn’t think to try yourself. Within minutes of discovering it, I’m already starting to think differently about how I write exported functions and here’s why.
Introduction Foreword This project is by no means complete! If you’d like to get involved and cause some destruction, I’d love to have some contributors and pull requests! github.com/codingconcepts/albert For my latest hackathon project, I decided to roll-my-own chaos monkey. Why not just use the Netflix Simian Army suite I hear you cry? The answer’s simple, we don’t use Spinnaker for continuous delivery and that’s an essential part of their chaos monkey.
I was one of many developers lured to Go by its promise of obscenely high concurrency. I estimate that in my first 37 minutes of playing with Go, I’d spun up in excess of 78 trillion goroutines. I’m also one of just as many developers who are now in love with Go because of its interfaces. Recap In Go, interfaces are implemented implicitly, meaning there’s none of this: public interface IAnimal { void Move(); } public class Dog : IAnimal { public void Move() { } } Only this:
I’m a big fan of optimising (?:as early as I possibly can|only when absolutely necessary). There’s a great case for not wasting time optimising things which may never need optimising. Especially when those efforts affect readability. Happily, I’ve enjoyed a bit of both (that’s to say “writing code and then optimising it”, not “wasting time, then making my code hard to read”) in a recent project and I’d like to share the experience.
semver, or “semantic versioning” is a simple but effective way to version an application. In the past, I’ve relied on my CI process (Jenkins, BuildMaster, ActiveBatch etc.) to keep track of my application’s version. This method became non-deterministic and hard to manage when there were multiple versions of my application in different branches of code. Enter semver semver aims to simplify the process of versioning your application by placing you in control of your application’s version by way of a .
ud is a little CLI I wrote which talks to the Urban Dictionary API and returns the top result by “thumbs up” responses. I wrote it as part of the Go training material I’m compiling for my colleagues, as it demonstrates the following: Writing an HTTP client Project layout Vendoring Installation $ go get -u github.com/codingconcepts/ud Usage As with the website, some of the definitions and examples don’t leave much to the imagination, so user discretion is advised!
The appleboy gin-jwt JWT middleware for Gin allows you to plug JWT into your Gin router with little to no fuss. Recently, I’ve used it to store basic user information to allow my router code to intelligently handle different user roles etc. Step 1 - Grab Gin and gin-jwt $ go get github.com/gin-gonic/gin $ go get gopkg.in/appleboy/gin-jwt.v2 Step 2 - Write some code Rather than stepping through line-by-line, I’ll describe what we’re doing using in-line comments (in a format you can just copy and paste into your own editor to play around with).
The rake build automation tool is a Ruby implementation of the make tool, which has been around since ever. I tried it out for the first time as part of a Go application I’m currently pushing into Production and found it a joy to work with. So much so, that I’ve taken to creating a Rakefile for projects that aren’t destined for deployment, as a wrapper for repetitive tasks like go build -o thing.