How Teal Keeps Their API, Tests and Documentation in Sync

Published on

October 8, 2020

There's a common problem in web development of creeping divergence between API documentation and API functionality. Its so easy to update functionality and forget to update the documentation! We can also just accidentally not align the two right from the start. This is solved by using a tool that tests the API and generates documentation at the same time. In the Rails ecosystem, we have the RSWAG gem which almost prevents divergence:

Rswag extends rspec-rails "request specs" with a Swagger-based DSL for describing and testing API operations. You describe your API operations with a succinct, intuitive syntax, and it automatically runs the tests. Once you have green tests, run a rake task to auto-generate corresponding Swagger files and expose them as YAML or JSON endpoints

I said Rswag almost prevents divergence because it does not enforce a few key items that ensure divergence is eradicated. This blog post shows what is really necessary to ensure the problem is solved, and also offers a few patterns to help with keeping the code DRY and clean.

Part 1: What RSwag gives you out of the box, and the problems that still exist

Part 2: A simple solve using Ruby JSON Schema Validator and the problems that still exist

Part 3: A robust solve that uses Json Schema Builder along with some patterns
‍
‍

Part 1: What RSwag gives you out of the box, and the problems that still exist

Let's assume RSWAG installed and you have a basic Article model, associated route and controller as follows:

# app/models/article.rb
class Article < ApplicationRecord
end

# config/routes.rb
Rails.application.routes.draw do
  mount Rswag::Api::Engine => '/api-docs' # added by Rswag
  mount Rswag::Ui::Engine => '/api-docs'  # added by Rswag
  resources :articles, only: :create
end

# app/controllers/articles_controller.rb
class ArticlesController < ApplicationController  
  def create
    create_params = params.require(:article).permit(:title, :body)                    
    article = Article.create!(create_params)
    render json: article, status: :created
  end
end

‍

Let's also assume you have a basic RSwag test set up for the above endpoint:

# specs/integration/articles_spec.rb
RSpec.describe "Articles", type: :request, swagger_doc: "v1/swagger.json" do
  path "/articles"  do
    post "creates an article" do
      tags "Article"
      consumes "application/json"
      produces "application/json"

      parameter name: :params, in: :body, schema: {
        type: :object,
        properties: {
            article: { 
              type: :object, 
              properties: { title: { type: :string }, body: { type: :string } }, 
              required: [ 'title', 'body' ] 
            }
        }
      }
      
      response '201', 'article created' do
        let(:params) { 
          { article: { title: 'foo', content: 'bar' } }
        }

        schema type: :object,
          properties: {
            article: { 
              type: :object, 
              properties: { id: {type: :integer }, title: { type: :string }, body: { type: :string } }, 
              required: [ 'id', 'title', 'body' ] 
            }  
          }

        run_test!
      end
    end
  end
end

‍

When we run the test, it passes, and per the RSwag promise we can generate some Swagger documentation with rails rswag, start up our server and navigate to localhost:3000/api-docs to see the docs.
‍

The Problem

But what is actually getting tested here? The response status code, given the params. That's all.

Given the goal of preventing creeping divergence, this is a massive issue: There is no coupling that ensures that the request schema is what is actually being used in the test! (and similarly, there is nothing testing that the response object matches the response schema)

If you look at the code above, you can see that the request object defined by let(:params) could be different than the schema defined in the request specification. The tests will still pass. This completely fails the promise of coupling between tests and documentation.

We want some sort of validation that the test's object passed to the controller and the request schema match.

We want something like:

# pseudo code:
it "matches the request object to the documented request schema" do      
    expect(request_object).to match_schema(request_schema)
end

‍
‍

Part 2: A simple solve using Ruby JSON Schema Validator and the problems that still exist

We can achieve this with the Ruby JSON Schema Validator gem, which offers a simple way to check if a JSON blob adheres to a JSON schema.

With the gem installed we can re-write the actual tests:

# specs/integration/articles_spec.rb
require "swagger_helper"

RSpec.describe "Articles", type: :request, swagger_doc: "v1/swagger.json" do
  path "/articles"  do
    post "creates an article" do
      tags "Article"
      consumes "application/json"
      produces "application/json"

      # notice we pull out the request and response schemas.
      expected_request_schema = {
        type: :object,
        properties: {
            article: { 
              type: :object, 
              properties: { title: { type: :string }, body: { type: :string } }, 
              required: [ 'title', 'body' ] 
            }
        }
      }

      expected_response_schema = {
        type: :object,
        properties: {
          id: { type: :integer }, 
          title: { type: :string }, 
          body: { type: :string }
        },
        required: [ 'id', 'title', 'body' ]
      }

      parameter name: :params, in: :body, schema: expected_request_schema
      
      response '201', 'article created' do
        let(:params) { 
          { article: { title: 'foo', body: 'bar' } }
        }

        before do |example|
          submit_request(example.metadata)
        end

        schema expected_response_schema

        it "matches the documented request schema" do |example|
          JSON::Validator.validate!(expected_request_schema, params, strict: true)
        end

        it "matches the documented response schema" do  |example|
          json_response = JSON.parse(response.body)
          JSON::Validator.validate!(expected_response_schema, json_response, strict: true)
        end

        it 'returns a valid 201 response' do |example|
          expect(response.status).to eq(201)
        end
      end
    end
  end
end

‍

Now if we decided to add a parameter to the test's request object, but forget to change the request schema, we get a failed test! This means there can never be a divergence between the API functionality, the test suite and the docs! For example if we change the params being sent to the controller to include the param extra, then the test will throw:
‍

     JSON::Schema::ValidationError:
       The property '#/article' contained undefined properties: 'extra'

Are we done? No! There are still a few problems remaining which really reveal themselves as you try to apply the above to an entire application. The first problem is that with the method above, we will potentially start to have duplicates of schemas. This might happen as we build out various update or index endpoints. Furthermore, if we create an Author class, and an author has multiple articles, does that mean when we test our author GET show endpoint (which contains an array of articles), we need to copy/paste the article schema into the author schema?? Yikes. That doesn't feel very DRY. We are getting into the world of building and composing schemas. That means it's time for some more help.
‍
‍

Part 3: A robust solve that uses Json Schema Builder along with some patterns

Enter the Json Schema Builder. With this gem, we are going to do a few things. The first is that we are going to consolidate our schemas into a specs/schemas directory. While we lose the visual convenience of the schemas and tests being in the same file, placing schemas into a common directory enables easier schema composition. The second change is that since this gem comes with its own validation feature, we can stop using (and remove) the aformentioned Ruby JSON Schema Validator gem.

We are going to add some lines to the top of the swagger_helper.rb file so we know where to find our awesome schema definitions:

# specs/swagger_helper.rb
require 'rails_helper'
require "json/schema_builder"

# this block ensures if any schema element is extra or missing, the test fails.
JSON::SchemaBuilder.configure do |opts|
  opts.validate_schema = true
  opts.strict = true
end

# let spec know where your shiny new schemas are!
Dir["./spec/schemas/*.rb"].each { |file| require file }

RSpec.configure do |config|
    # ...

With this in place we can create our schemas:

# spec/schemas/article_schemas.rb
module SpecSchemas 

  class ArticleCreateRequest
    include JSON::SchemaBuilder
    def schema
      object do
        object :article do
          string :title, required: true
          string :body, required: true
        end
      end
    end
  end

  class ArticleResponse
    include JSON::SchemaBuilder

    def schema
      object do
        number :id, required: true
        string :title, required: true
        string :body, required: true
      end
    end
  end
end

A shoutout here to the schema builder gem. It's DSL is fantastic and allows for very readable schemas.

We can now update our integration test to use these brand new schemas:

require "swagger_helper"

RSpec.describe "Articles", type: :request, swagger_doc: "v1/swagger.json" do
  path "/articles"  do
    post "creates an article" do
      tags "Article"
      consumes "application/json"
      produces "application/json"
      expected_request_schema = SpecSchemas::ArticleCreateRequest.new

      parameter name: :params, in: :body, schema: expected_request_schema.schema.as_json
      
      response '201', 'article created' do
        expected_response_schema = SpecSchemas::ArticleResponse.new
        let(:params) { 
          { article: { title: 'foo', body: 'bar' } }
        }

        before do |example|
          submit_request(example.metadata)
        end

        schema(expected_response_schema.schema.as_json)

        it "matches the documented request schema" do |example|
          errors = expected_request_schema.schema.fully_validate(params)
          expect(errors).to be_empty          
        end

        it "matches the documented response schema" do  |example|
          json_response = JSON.parse(response.body)
          errors = expected_response_schema.schema.fully_validate(json_response)
          expect(errors).to be_empty
        end

        it 'returns a valid 201 response' do |example|
          expect(response.status).to eq(201)
        end
      end
    end
  end
end

This is starting to look better but we still have a bit more to do. Right now there is a lot of duplication that we will see once we add more integration tests. What we really want to do is wrap everything up in a reusable, shared test that we can add to various endpoints.. The three tests above (checking for request structure, response structure and status code) should be included in every endpoint test, so let's move them to a shared test.

First let's make sure that we required our shared examples. In your swagger_helper.rb file add the following line:

# /specs/swagger_helper.rb
Dir["./spec/support/**/*.rb"].sort.each { |file| require file }

Then you can define a shared example:

# spec/support/integration/shared_examples.rb
RSpec.shared_examples "a JSON endpoint" do |expected_response_status|
    before do |example|
      submit_request(example.metadata)
    end
  
    describe "response status" do
      it "returns expected response status" do
        expect(response.status).to eq(expected_response_status)
      end
    end

    describe "request body" do
      it "matches the documented request schema" do |example|
          errors = expected_request_schema.schema.fully_validate(params)
          puts params if errors.count > 0 # for debugging
          expect(errors).to be_empty          
      end
    end
  
    describe "response body" do
      let(:json_response) { JSON.parse(response.body) }
  
      it "matches the documented response schema" do  |example|
        errors = expected_response_schema.schema.fully_validate(json_response)
        puts json_response if errors.count > 0 # for debugging
        expect(errors).to be_empty
      end
    end
end

And finally in your spec, you can utilize the shared example::

# spec/integration/articles_spec.rb
require "swagger_helper"

RSpec.describe "Articles", type: :request, swagger_doc: "v1/swagger.json" do
  path "/articles"  do
    post "creates an article" do
      tags "Article"
      consumes "application/json"
      produces "application/json"
      expected_request_schema = SpecSchemas::ArticleCreateRequest.new

      parameter name: :params, in: :body, schema: expected_request_schema.schema.as_json
      
      response '201', 'article created' do
        expected_response_schema = SpecSchemas::ArticleResponse.new
        schema(expected_response_schema.schema.as_json)
        
        let(:params) { 
          { article: { title: 'foo', body: 'bar' } }
        }

        before do |example|
          submit_request(example.metadata)
        end

        it_behaves_like "a JSON endpoint", 201 do
          let(:expected_response_schema) { expected_response_schema }
          let(:expected_request_schema) { expected_request_schema }
        end
      end
    end
  end
end

‍

Now when you make another endpoint, your API functionality and documentation will remain in sync with just a few lines of code! What will you do with all of your reclaimed spare time? I hear a lot of people are learning to bake sourdough bread these days...

‍

About the Author

Keith Gould

Connect with

Keith

on LinkedIn

How Teal Keeps Their API, Tests and Documentation in Sync

Part 1: What RSwag gives you out of the box, and the problems that still exist

The Problem

Part 2: A simple solve using Ruby JSON Schema Validator and the problems that still exist

Part 3: A robust solve that uses Json Schema Builder along with some patterns

About the Author

More on the Blog

Job Search Organizer Software

How to Stay Motivated During a Job Search

Your Guide to Remote HR Jobs

Take a step towards career confidence.