Writing tests for your static website: Jekyll, Hugo

Standard

Static site generators like Jekyll or Hugo are awesome to quickly publish a website online. Thanks to GitHub and Netlify, you can leverage powerful collaboration tools and free hosting to have a website up and running quickly. You’ll be able to deploy and update your website in minutes without worrying about hosting. This is super powerful.

One thing I’ve not seen a lot for static websites which is present in traditional software is tests: software you write to prove or make sure that your code does what you expect it to do. Sure, static websites have way less code than libraries or backends, but still: you can quickly have tens of posts and hundreds of lines of YAML in data files. It makes sense to write quick tests to ensure things like required keys are present for posts or foreign key consistency in data files. Tests ensure you have a high quality content on your website and can avoid broken layout which you would detect after browsing your static website.

Writing tests for Jekyll

How would you write tests for a Jekyll website? At the end of the day, static websites are composed of data files (usually in YAML) and content files in Markdown. Standard programming languages (Python, Ruby, PHP) can easily parse these files and you can write assertions about the content of them. These tests should be executed after every git push to perform continuous integration. You can use a platform like CircleCI or GitHub Actions to do this.

Jekyll tests code sample

Here is a sample code to run tests on CircleCI for Markdown posts: making sure required keys are there, tags are present and come from a predefined list, images and Twitter usernames have an expected format. These tests are written using Python 3.6 but you can use whatever programming language you like. You can also write tests for data files in YAML. It gets powerful when you write tests combining data files and content files in Markdown.

These tests run in less than 10 seconds on CircleCI after pushing your code and can quickly catch small mistakes. This is a straightforward way to improve the quality of your static website.

Go client for Updown

Standard

What is Updown?

Over the weekend, I’ve been working on creating a Go client for updown.io. Updown lets you monitor websites and online services for an affordable price. Checks can be performed for HTTP, HTTPS, ICMP and a custom TCP connection down to every 30s, from 4 locations around the globe. They also offer status pages, like the one I use for Teen Quotes. I find the design of the application and status pages really slick. For all these reasons, I use Updown for personal and freelance projects.

A Go REST client

I think that it’s the first time I wrote a REST API client in Go, and I feel pretty happy. My inspiration for the package came from the Godo package, the Go library for DigitalOcean. It helped me start and structure my files, structures and functions.

The source code is available on GitHub, under the MIT license. Here is a small glance at what you can do with it.

package main

import (
    "github.com/antoineaugusti/updown"
)

func main() {
    // Your API key can be retrieved at https://updown.io/settings/edit
    client := updown.NewClient("your-api-key", nil)
    // List all checks
    checks, HTTPResponse, err := client.Check.List()
    // Finding a token by an alias
    token, err := client.Check.TokenForAlias("Google")
    // Downtimes for a check
    page := 1 // 100 results per page
    downs, HTTPResponse, err := client.Downtime.List(token, page)
}

Enjoying working with Go again

I particularly enjoyed working with Go again, after a few months without touching it. I really like the integration with Sublime Text, the fast compilation, static typing, the golint (linter for Go code, that even takes into account variable names and comments) and go fmt (automatic code formatting) commands. I knew and I experienced once again that developing with Go is fast and enjoyable. You rapidly end up with a code that is nice to read, tested and documented.

Feedback

As always, feedback, pull-requests, or kudos are welcomed! I did not achieved 100% coverage as I was quite lazy and opted for integration tests, meaning tests actually hit the real Updown API when they are performed.

Developing and deploying a modulus checking API

Standard

Following my latest post about a Go package to validate UK bank account numbers, I wanted to offer a public API to let people check if a UK bank account number is valid or not. I know that offering a Go package is not ideal for everyone because for the moment Go is not everywhere in the tech ecosystem, and it’s always convenient to have an API you can send requests to, especially in a frontend context. My goal was to offer a JSON API, supporting authentication thanks to a HTTP header and with rate limits. With this, in the future you could adapt rate limits to some API keys, if you want to allow a larger amount of requests for some clients.

Packages I used

I wanted to give cloudflare/service a go because it lets you build quickly JSON APIs with some default endpoints for heartbeat, version information, statistics and monitoring. I used etcinit/speedbump to offer the rate limiting functionality and it was very easy to use. Note that the rate limiting functionality requires a Redis server to store request counts. Finally, I used the famous codegangsta/negroni to create middlewares to handle API authentication and rate limits and keeping my only controller relatively clean.

Deploying behind Nginx

My constraints were the following:

  • The API should only be accessible via HTTPS and HTTP should redirect to HTTPS.
  • The Golang server should run on a port > 1024 and the firewall will block access to everything but ports 22, 80 and 443
  • The only endpoints that should be exposed to the public are /verify, /version and /heartbeat. Statistics and monitoring should be accessible by administrators on localhost through HTTP

I ended up with this Nginx virtual host to suit my needs, I’m not sure if it can be simpler:

geo $is_localhost {
  default 0;
  127.0.0.1/32 1;
}

server {
    listen 80;
    listen 443 ssl;

    server_name modulus.antoine-augusti.fr localhost.antoine-augusti.fr;

    ssl_certificate /etc/nginx/ssl/modulus.antoine-augusti.fr.crt;
    ssl_certificate_key /etc/nginx/ssl/modulus.antoine-augusti.fr.key;

   if ($is_localhost) {
      set $test A;
   }

    if ($scheme = http) {
      set $test "${test}B";
    }
    
    # Redirect to HTTPS if not connecting from localhost
    if ($test = B) {
      return 301 https://$server_name$request_uri;
    }
    
    # Only the following endpoints are accessible to people not on localhost
    location ~ ^/(verify|heartbeat|version)  {
      include sites-available/includes/dispatch-golang-server;
    }

    # Default case
    location / {
      # Not on localhost? End of game
      if ($is_localhost = 0) {
        return 403;
      }
      # Forward request for people on localhost
      include sites-available/includes/dispatch-golang-server;
    }
}

And for sites-available/includes/dispatch-golang-server:

proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $host;
proxy_pass http://127.0.0.1:8080;

With this, I can still access the reserved endpoints by opening a SSH tunnel first with ssh -L4242:127.0.0.1:80 [email protected] and going to http://localhost.antoine-augusti.fr:4242/stats after.

Note that the Golang server is running on port 8080 and it should be monitored by Supervisor or whatever you want to use.

Grabbing the code and a working example

First of all, the API is available on GitHub under the MIT license so that you can deploy and adapt it yourself. If you want to test it first, you can use the API key foo against the base domain https://modulus.antoine-augusti.fr. Here is a cURL call for the sake of the example:

curl -H "Content-Type: application/json" -H "Api-Key: foo" -X POST -d '{"sort_code": "308037", "account_number": "12345678"}' https://modulus.antoine-augusti.fr/verify

Note that this API key is limited to 5 requests per minute. You’ve been warned 🙂 Looking for more requests per month or SLA, drop me a line.

Decorator pattern and repositories

Standard

My use case

Lately I’ve been using a lot the decorator pattern with repositories on Teen Quotes. My use case is somewhat simple: I use a relational database (MySQL) but sometimes I what to cache the results of some queries in a key-value store (Redis / Memcached for instance). With something like that, I don’t need to hit my database for queries that are always run or are slow to run, I’ll hit my key-value store instead. It’ll reduce the pressure on my database and will give some results faster for the application.

The decorator pattern

If you’re not familiar with the decorator pattern yet, it’s quite simple to use and I’m sure you’ll love it in no time. Basically, the decorator pattern allows behavior to be added to an individual object, either statically or dynamically, without affecting the behavior of other objects from the same class. As Richard Bagshaw said, the idea is that you take an object, and you wrap this object in another object which provides additional functionality, and you keep wrapping extra classes repeatedly for each additional requirement.

If you want to see some real world examples, continue to read this blog post or go directly to Laracasts.

Some code

I’ll show you something I’ve been working on last week: the ability to add tags to quotes. A “tag” is like a category for a “quote” (a post, an article, whatever you want to call it). I’m using Laravel with Eloquent for my relational database. I’ve created an interface called TagRepository.

namespace TeenQuotes\Tags\Repositories;

use TeenQuotes\Tags\Models\Tag;
use TeenQuotes\Quotes\Models\Quote;

class DbTagRepository implements TagRepository {

  /**
   * Create a new tag
   *
   * @param  string $name
   * @return \TeenQuotes\Tags\Models\Tag
   */
  public function create($name)
  {
    return Tag::create(compact('name'));
  }

  /**
   * Get a tag thanks to its name
   *
   * @param  string $name
   * @return \TeenQuotes\Tags\Models\Tag|null
   */
  public function getByName($name)
  {
    return Tag::whereName($name)->first();
  }

  /**
   * Add a tag to a quote
   *
   * @param  \TeenQuotes\Quotes\Models\Quote $q
   * @param  \TeenQuotes\Tags\Models\Tag $t
   */
  public function tagQuote(Quote $q, Tag $t)
  {
    $q->tags()->attach($t);
  }

  /**
   * Remove a tag from a quote
   *
   * @param  \TeenQuotes\Quotes\Models\Quote $q
   * @param  \TeenQuotes\Tags\Models\Tag $t
   */
  public function untagQuote(Quote $q, Tag $t)
  {
    $q->tags()->detach($t);
  }

  /**
   * Get a list of tags for a given quote
   *
   * @param  \TeenQuotes\Quotes\Models\Quote $q
   * @return array
   */
  public function tagsForQuote(Quote $q)
  {
    return $q->tags()->lists('name');
  }

  /**
   * Get the total number of quotes having a tag
   *
   * @param  \TeenQuotes\Tags\Models\Tag $t
   * @return int
   */
  public function totalQuotesForTag(Tag $t)
  {
    return $t->quotes()->count();
  }
}

Pretty simple stuff, I’m sure you’ve seen this multiple times. Let’s move on to the interesting part: the caching layer. We will create a new class CachingTagRepository implementing the same interface TagRepository. The key thing is that we’ll require a TagRepository class to be given in the constructor of this new class. Ultimately, we will pass the DB layer here.

 
namespace TeenQuotes\Tags\Repositories;

use Cache;
use TeenQuotes\Tags\Models\Tag;
use TeenQuotes\Quotes\Models\Quote;

class CachingTagRepository implements TagRepository {

  /**
   * @var \TeenQuotes\Tags\Repositories\TagRepository
   */
  private $tags;

  public function __construct(TagRepository $tags)
  {
    // The key thing is here: we assume we've already
    // a class that is implementing the interface.
    // We can rely on that!
    $this->tags = $tags;
  }

  /**
   * Create a new tag
   *
   * @param  string $name
   * @return \TeenQuotes\Tags\Models\Tag
   */
  public function create($name)
  {
    return $this->tags->create($name);
  }

  /**
   * Get a tag thanks to its name
   *
   * @param  string $name
   * @return \TeenQuotes\Tags\Models\Tag|null
   */
  public function getByName($name)
  {
    $callback = function() use ($name)
    {
      return $this->tags->getByName($name);
    };

    return Cache::rememberForever('tags.name-'.$name, $callback);
  }

  /**
   * Add a tag to a quote
   *
   * @param  \TeenQuotes\Quotes\Models\Quote $q
   * @param  \TeenQuotes\Tags\Models\Tag $t
   */
  public function tagQuote(Quote $q, Tag $t)
  {
    Cache::forget($this->cacheNameForListTags($q));

    $keyTotal = $this->cacheNameTotalQuotesForTag($t);

    if (Cache::has($keyTotal))
      Cache::increment($keyTotal);

    return $this->tags->tagQuote($q, $t);
  }

  /**
   * Remove a tag from a quote
   *
   * @param  \TeenQuotes\Quotes\Models\Quote $q
   * @param  \TeenQuotes\Tags\Models\Tag $t
   */
  public function untagQuote(Quote $q, Tag $t)
  {
    Cache::forget($this->cacheNameForListTags($q));

    $keyTotal = $this->cacheNameTotalQuotesForTag($t);

    if (Cache::has($keyTotal))
      Cache::decrement($keyTotal);

    return $this->tags->untagQuote($q, $t);
  }

  /**
   * Get a list of tags for a given quote
   *
   * @param  \TeenQuotes\Quotes\Models\Quote $q
   * @return array
   */
  public function tagsForQuote(Quote $q)
  {
    $key = $this->cacheNameForListTags($q);

    $callback = function() use($q)
    {
      return $this->tags->tagsForQuote($q);
    };

    return Cache::remember($key, 10, $callback);
  }

  /**
   * Get the total number of quotes having a tag
   *
   * @param  \TeenQuotes\Tags\Models\Tag $t
   * @return int
   */
  public function totalQuotesForTag(Tag $t)
  {
    $key = $this->cacheNameTotalQuotesForTag($t);

    $callback = function() use ($t)
    {
      return $this->tags->totalQuotesForTag($t);
    };

    return Cache::remember($key, 10, $callback);
  }

  /**
   * Get the key name when we list tags for a quote
   *
   * @param  \TeenQuotes\Quotes\Models\Quote $q
   * @return string
   */
  private function cacheNameForListTags(Quote $q)
  {
    return 'tags.quote-'.$q->id.'.list-name';
  }

  /**
   * Get the key name to have the number of quotes
   * having a tag
   *
   * @param  \TeenQuotes\Tags\Models\Tag $t
   * @return string
   */
  private function cacheNameTotalQuotesForTag(Tag $t)
  {
    return 'tags.tag-'.$t->name.'.total-quotes';
  }
}

You see, we do some things before (or after) calling the initial implementation, to add some functionalities (here a caching layer). Sometimes we directly defer to the initial implementation (see the create method).

Bonus: registering that in the IoC container

Let’s bind our TagRepository interface to the caching layer and the storage layer in a service provider!

namespace TeenQuotes\Tags;

use Illuminate\Support\ServiceProvider;
use TeenQuotes\Tags\Repositories\CachingTagRepository;
use TeenQuotes\Tags\Repositories\DbTagRepository;
use TeenQuotes\Tags\Repositories\TagRepository;

class TagsServiceProvider extends ServiceProvider {

  /**
   * Bootstrap the application events.
   *
   * @return void
   */
  public function boot()
  {
      //
  }

  /**
   * Register the service provider.
   *
   * @return void
   */
  public function register()
  {
      $this->registerBindings();
  }

  private function registerBindings()
  {
      $this->app->bind(TagRepository::class, function()
      {
          return new CachingTagRepository(new DbTagRepository);
      });
  }
}

Et voilà ! Happy coding!

Why Twitter was down for 5 hours. It’s all about calendars.

Standard

Twitter was down for many users this Monday, between midnight and 5 AM, CET. Android app users were logged out and the API was not working during a few hours. A bug in a line of code caused the service to think that it was 29 December, 2015.

Gregorian VS ISO calendars

The ISO week numbering system uses the YYYY format for the year instead of the Gregorian calendar’s yyyy. It then looks at which week of the year it is, and then uses a date digit with 1 starting on Monday. So, for example, Tuesday of the 50th week of 2014 would have been 2014-W50-2 in ISO week format. The problem comes in when January 1 of the new year ends up falling on a date that doesn’t get along well with the ISO week format. The first day of 2015 will start on a Thursday, whereas the ISO standard expects the first week of the year to start on “the Monday that contains the first Thursday in January.” In 2014-2015, that would be January 1st, 2015. That’s why Twitter believed that we were in 2015 on Monday, because “the first Thursday in January” is in fact this week.

Afterwards, some parts of the system believed we were almost a year later, while other were perfectly aware that we were still in 2014. Users were not able to log in (and were logged out) because tokens have a limited lifetime and therefore are not valid a year later.

Dates are hard to handle

Dates are really hard to handle when programming, and it’s very easy to make a mistake without breaking a thing until the day it will eventually explode everything because calendars or timezones are different. The same kind of mistake happened two years ago with the “Do Not Disturb” feature on iPhones, and it isn’t the last time it will happen for sure.

Using JavaScript / jQuery to detect if someone is using AdBlock

Standard

Let’s face it: ads are not cool. I know, but sometimes you have no other choice than to use them to pay your server / your coffee / whatever. And if you are using ads, you know that some people will not see your ads because they are using a plugin that blocks ads on your website. You want to reach these visitors and just say

“Hey, we know ads are not cool, but please make an exception for us or give us a small amount of money, just to support our work”

On Teen Quotes I am showing a friendly message in place of the only ad to say what I have to say. Here is what people with an ad blocker are seeing:
adblock

But let’s get back to the point: detect visitors that are using an ad blocker with JavaScript.

Some code

Let’s write some code. You’re going to write the most simple JS file you have ever written. Ready? Here it is:

var isNaughtyVisitor = false;

BOOM! That was fast, isn’t it? In fact, we don’t care about the variable name or its value, we just want to define a new variable. But this variable needs to be defined in a file called ads.js that you will include in your HTML. Why so? This is were it gets funny. Ad blockers are a little dumb, so when they see a file named something like ad.js or ads.js, they will block the request. And if the request was blocked… your variable will not be defined!

The second step is just to test in your regular JS file of your application if the variable was defined. This file needs to be include after the previous file, ads.js. You can add something like this:

$(document).ready(function() {
    // The div that will add to the DOM if the visitor is 
    // using an ad blocker
    var div = 'Your amazing HTML block here';
    
    // The script was never called, probably using an ad blocker
    if (typeof(isNaughtyVisitor) == "undefined") {
        // Insert the div wherever you want
        $("#footer").before(div);
        // Send the event to Google Analytics if you want to track
        ga('send', 'event', 'ads', 'hidden');
    // A friendly user!
    } else {
        // Send the event to Google Analytics if you want to track
        ga('send', 'event', 'ads', 'displayed');
    }
});

Pretty darn simple, and really effective. The downside is that friendly users will have to make an extra HTTP request. Yep, I know, it’s not fair for them.

Using ImageMagick to crop images

Standard

ImageMagick is a PHP library for image processing and image generation. It is a requirement for a popular library, stojg/crop that offers various cropping techniques. The library can be installed easily via Composer, but it requires ImageMagick to work.

Installing ImageMagick on Ubuntu / Debian

You are lucky, there is a package for that. It is just a one line command:

sudo apt-get install php5-imagick

It will grab ImageMagick, build the PHP extension and enable it in your php.ini file. You may have to restart your web server or PHP-FPM to reflect the changes.

Using ImageMagick on Travis CI

If you are running a continuous integration test suite, you may be interested to test some methods that are using the ImageMagick library under the hood. In order to do that, the ImageMagick library needs to be installed on the Travis’ VM. At the time, the ImageMagick library is already pulled in for every VM, but the PHP extension is not enabled by default. We will still try to grab the library before enabling it, just to be cautious. You’ll have to edit your travis.yml file and add the following lines to your before_script:

before_script:
  - sudo apt-get update -q
  - sudo apt-get install -y imagemagick
  - printf "\n" | pecl install imagick

The last line is a very dirty hack: when running the command pecl install imagick, you have to hit “Enter” to autodetect some settings. That’s why you need to pipe a new line to this command. Dirty, but it works, you’ll be able to test methods using ImageMagick afterwards.

Fighting against suicide on Teen Quotes

Standard

Teen Quotes lets teenagers share their daily thoughts and feelings. Since people speak their mind, we receive happy quotes about friendship and love, regular life inspiring quotes and sad quotes. And sometimes, unfortunately, really really sad quotes.

At Teen Quotes, we have a very strict publishing policy. Every quote gets reviewed by a human before being published. If we don’t like a quote, we can edit it a little bit to make it publishable or we can refuse it.

How to manage refused quotes

In the past, when we received a quote that we didn’t want to publish, here is what happened:

  • The submitted quote goes through moderation and gets refused. As an immediate result, it will never be published on Teen Quotes.
  • The author of the quote receives an email telling that its quote was refused with a few hints to improve: verify the English, be sure that a similar quote wasn’t published before and try to write an original quote.

This type of email is sent a lot because we refuse around 75 % of the submitted quotes. We would be super happy to explain in every email why this specific quote was refused but it would take too much time during moderation.

But I am convinced that sometimes we should contact the author of the quote, in specific situations. One of the possible situation is described in this post’s title: suicide promotion / suicide thoughts in a quote.

Submitting super sad quotes is a real alarm

As already said, users submit quotes on Teen Quotes to inspire others. Most of our visitors come just to get inspired, to relate to others, to put words on feelings. We deeply think that we shouldn’t publish quotes that promote suicide thoughts. Everyone has ups and downs, but suicide should not be an option. So we protect other members from Teen Quotes by refusing such quotes. And today, we are going one step further.

Trying to help authors that need assistance

In the upcoming major version of Teen Quotes, if a user submits a really sad quote, we will send him a personal email to make sure that everything is right and we will include some useful links if he needs to get help from non-profit organisation around the world. As a software engineer, it is usually difficult to have an impact in someone’s life. Moving a button 2 pixels to the left will not make a huge difference. We deeply hope that we will be able to help people with this feature. It really matters to us.

Laravel package for the recommendation system Easyrec

Standard

During a few days in August I have coded a Laravel wrapper for the recommendation system Easyrec. If you want to display most viewed items, best rated items, related items or something like that, a recommendation system is the perfect way to go. If you are familiar with Machine Learning techniques, you now that a recommendation system is something very difficult. If you remember, Netflix offered $ 1,000,000 to improve its collaborative filtering algorithm.

Easyrec provides a REST API that you can call for free. This is something very convenient if don’t have a lot of data and that you still want to use a recommender system for your web service.

Features overview

After registering the service provider and the alias in your app/config/app.php file, it will be super easy to use! Here is a little overview of the available functions:

Easyrec::view(42, 'Post 42', 'http://example.com/posts/42', 200, null, null, 'POST'); // User #200 has viewed post #42
Easyrec::rate(42, 8, 'Book 42', 'http://example.com/books/42', 200, null, null, 'BOOK'); // User #200 has rated 8/10 book #42
Easyrec::mostViewedItems(10, 'WEEK', 'BOOK'); // Retrieves the 10 most viewed books for the last week
Easyrec::bestRatedItems(10, 'MONTH', 'POST'); // Retrieves the 10 best rated posts for the last month

Documentation and download

Of course the package is available via Composer. The full documentation can be found on GitHub: github.com/AntoineAugusti/laravel-easyrec.

Do not hesitate to open up an issue if something is not working as expected. Pull-requests are very welcome also!

Error handlers order in Laravel

Standard

Error handlers are a neat feature of Laravel. They allow you to deal with custom exceptions so that you know what to do when something wrong happens. If you want to play with error handlers, you need to know two vital things:

  1. If an exception handler returns a response, that response will be sent to the browser and no other error handlers will be called ;
  2. Handlers are called based on the type-hint of the Exception they handle. If two handlers handle the same exception, the last one will be called first.

About error handlers order

An example:

App::error(function(MyCustomException $exception, $code)
{
	// This handler will not be called first
});

App::error(function(MyCustomException $exception, $code)
{
	// This handler will be called first
});

Error handlers and responses

If your last handler returns a response, your previous handler will not be called because of the first rule.

App::error(function(MyCustomException $exception, $code)
{
	// This handler will not be called AT ALL
});

App::error(function(MyCustomException $exception, $code)
{
	// This handler will be called first
	$data = ['title' => 'Error occured'];

	// We are returning a response: no other error handlers will be called
	return Response::view('errors.default', $data, $code);
});

Playing with error handlers order

Let’s say you want an error handler to be called at the bottom of the stack. This is very useful for example if you are using a package that registers an error handler and you want one of your error handlers to be called after. Laravel provides the App::pushError() function that registers an error handler at the bottom of the stack.

App::error(function(MyCustomException $exception, $code)
{
	// This handler will be called first
	// If we are returning a response here, no other error handlers will be called
});

App::pushError(function(MyCustomException $exception, $code)
{
	// This handler will be called at the end
	$data = ['title' => 'Error occured'];

	return Response::view('errors.default', $data, $code);
});