How Does TDD Really Work (Pt 4)

Endpoint stubbing for external APIs

How many web tutorials have you watched where someone builds a simple todo list?

How many simple todo lists have you been asked to build for money? Not many? No, me neither.

Previous episode

To work!

I’ve come across another novel problem today. As I’ve said I’m working with server rendered templates so I’m e2e testing routes that are pre-rendered with all their data before they reach the browser.

Today I’m making a table that shows some data fetched from an external API. External APIs present some problems particularly in e2e testing because I don’t have complete control over the data.

Bit by bit

Working in the smallest increments possible is usually the best way to work when doing TDD. So first I’ll start with some super simple tests, get them to pass as quickly as possible then refactor them to make them more rigorous then make those pass and so on and so on.

The first thing I do is a couple of tests to show the table when it’s empty and then show it again with some stubbed data in it.

cy.test('empty table', () => {
  cy.visit('/employees');
  cy.contains('Nothing to show');
});

cy.test('table with employees', () => {
  cy.visit('/employees');
  cy.contains('Amazing admin').should('exist');
});

Then I stub the data in an “/employees” endpoint that renders the table:

async function getEmployees() {
  return []
}

route.get(
  '/employees',
  async (req, res) => {
    const employees = await getEmployees();
    res.render(
      'templates/table',
      {
        employees,
      },
    )
  },
);

This causes the empty table test to pass then I’ll add some stubbed data into getEmployees to get the second test to pass.

But here’s the problem: how do I get the second test to pass without failing the original one?

Usually in e2e testing I have control over the database, so if I want the view to show an empty table I just wipe the records from the test database and then load the view. I can’t do that with an external API though. I can’t even stub the data for the test because the browser doesn’t have access to the part of the code that calls the external API.

I’m used to working with headless apps which make this problem easy to solve. You simply fetch the API data from the browser, after the view has rendered, then in the test you can intercept the endpoint that fetches the API data.

But if the API fetch happens on the server I’d have to stub the whole endpoint to test it. If I do that, I’m not really testing anything much at all, because most of the logic is run on the server.

Keep calm and carry on

I haven’t written the code that actually does the API fetch yet, so first I stub the data, which gets the second test to pass and causes the first to fail:

async function getEmployees() {
  return [{
    name: 'Amazing admin',
    id: 1
  }];
}

route.get(
  '/employees',
  async (req, res) => {
    const employees = await getEmployees();
    res.render(
      'templates/table',
      {
        employees,
      },
    )
  },
);

As I said earlier a great approach to writing TDD code is to get your test passing as soon as possible, then update the test again to make it fail before moving on to the next step. This usually means stubbing fake data that makes your test artificially pass, then you can update the test again or write a new test to make it more thorough, making sure the test fails, to then write more of the feature to make it pass again.

Flags

So my new test is passing because it’s looking for a full table, which my stubbed data is filling, but the old test that looks for an empty table is failing, again because of the stubbed data.

How do I pass stubbed data for the first test but empty data for the second? Then later, I’m going to have a problem where, I won’t want to use the real API for any of my tests. How do I hook up the real API for production but keep controlled, stubbed or empty, data for tests?

It occurred to me that Continuous Integration developers use “flags” to separate parts of their code allowing features for some users but not others. I’d been working under the assumption that I shouldn’t adapt my business code to allow for testing but in this case I can’t see any way of adapting the code purely from the test.

So here’s my solution, I use two fixture endpoints to give me the test results I want. Then, I hide them from production with an env variable.

First I make the route more modular:

// This isolates the part that will eventually be used to call the API
async function getEmployees() {
  return [{
    name: 'Amazing admin',
    id: 1
  }];
}

// Using next() allows me to split one controller 
// into two more reusable functions...

// The first to add data to the body
async function addEmployeesToBody(req, res, next) {
  req.body.employees = await getEmployees();
  next();
}

// The second to use that body and render the right template
function renderEmployeesTable(req, res) {
  return res.render('templates/table', req.body);
}

// Now I can just stack them in route.get
route.get(
  '/employees',
  addEmployeesToBody,
  renderEmployeesTable,
);

Then I create two test routes, hidden by my env variable:

// ...further down the file

if (process.env.NODE_ENV !== 'production') {

  // __test__/empty because it's returning an empty array
  route.get(
    '/employees/__test__/empty',
    (req, res, next) => {
      req.body.employees = [];
      next();
    },
    renderEmployeesTable,
  );

  // __test__/stubbed because it's returning the stubbed data
  route.get(
    '/employees/__test__/stubbed',
    (req, res, next) => {
      req.body.employees = [{ name: 'Amazing admin', id: 1 }];
      next();
    },
    renderEmployeesTable,
  );
}

Then I use the test routes in my tests instead:

cy.test('empty table', () => {
  cy.visit('/employees/__test__/empty');
  cy.contains('Nothing to show');
});

cy.test('table with employees', () => {
  cy.visit('/employees/__test__/stubbed');
  cy.contains('Amazing admin').should('exist');
});

This way I’m testing as much of my application code as possible, without testing the external API that I use to get the employee data.

This isn’t the same as using flags in CI but the principle is that if you can add code for different app environments (production, development, test) you can also make endpoints for different environments.

This works as long as I don’t have to update too many places when I want to change the real “/employees” endpoint. If I can use as much of the production code in my stubbed endpoints as possible, they’ll closely match the real functionality of the app and I’ll only have to update the real code when I change things.

To draw the circle even tighter around the API code I add a callback function to the second controller:

// Call my actual API...
async function getEmployees() {
  return axios.get('/external-api');
}

// Here I'm passing in the part of the code I'm going to stub,
// but still doing as much of the logic that actually gets used in
// my production app as possible...
async function addEmployeesToBody(getEmployeesCallback) {
  return (req, res, next) => {
    req.body.employees = await getEmployeesCallback();
    next();
  }
}

function renderEmployeesTable(req, res) {
  return res.render('templates/table', req.body);
}

// The real endpoint uses the real external api call...
route.get(
  '/employees',
  addEmployeesToBody(getEmployees),
  renderEmployeesTable,
);

if (process.env.NODE_ENV !== 'production') {

  // Now I can also use addEmployeesToBody as well as 
  // renderEmployeesTable in my test endpoints
  route.get(
    '/employees/__test__/empty',
    addEmployeesToBody(() => Promise.resolve([])),
    renderEmployeesTable,
  );

  route.get(
    '/employees/__test__/stubbed',
    addEmployeesToBody(() =>
      Promise.resolve([{ name: 'Amazing admin', id: 1 }])
    ),
    renderEmployeesTable,
  );

  // And I can easily add a new one to test API errors...
  route.get(
    '/employees/__test__/error',
    addEmployeesToBody(() => Promise.reject('API Error')),
    renderEmployeesTable,
  );
}

This way I can add error catching to addEmployeesToBody.

Yet another reason to keep things modular

I’m beginning to understand more and more why modular coding practices are advocated by the dev community at large. When I’m testing my code I can more easily swap out parts of the code for other parts, while re-using as much of my actual production code as possible.

If I hadn’t been writing tests while working on this feature there’s no way I would have written addEmployeesToBody in this way, passing the api call into a callback.

async function addEmployeesToBody(getEmployeesCallback) {
  return (req, res, next) => {
    req.body.employees = await getEmployeesCallback();
    next();
  }
}

I would just call the endpoint to get the employees within the function, which would look clearer and easier to read. But notice how, by doing this, I’m making my code extendable. Now, if I wanted to swap out a different API service, I only have to update the callback being passed into addEmployeesToBody and the rest of my code can stay the same.

That’s a benefit that has nothing to do with testing, even-though I only wrote it this way to make the code testable.