How I built a mini cache engine (Part 1)

Hi there, this is part one of a two part series.

I want to give you a mini guide on how I built a mini cache engine layer for a web application. This part covers content that is the same for everyone, and part two covers caching for logged in users.

What is the cache layer?

In the context of this blog, the cache layer refers to the layer that sits between your frontend and a backend which caches responses to improve response times, and reduce the load on the backend service.

Why would you need this?

Imagine you have an application which displays content to users, like Stories, News, Feeds, Blogs, etc., and it needs to be a fully rendered page so it can be indexed by Google.

You might say, oh right, let's instead prebuild the pages so that they're cached (that works, yes) and static.

But when the content changes often, prebuilding can't keep up, so you'd server render (SSR) the pages instead. And with SSR, every page visit renders on the server, which means a call to the backend, which sometimes means a database read, even when nothing has changed since the last visit.

So anyone spamming a page is basically spamming your backend too (not fun).

That's where the cache layer comes in. Serve the same response again, and only bother the backend when you actually have to.

Here's what I did

It didn't start as an engine initially. The first version was a catch-all route that matched the request, built a cache key from it, and cached the response. No user caching, just content that looks the same for everyone.

The code below uses ufo for URL handling.

1. Figuring out what to cache

Not every backend call is worth caching, so for each one, I wondered...

How often is it read?
How long can it be old before anyone notices?

An about page can be a day old and nobody will complain, but the latest posts? Not more than a minute behind.

So instead of throwing a number at every call, I gave each kind of data a name, and kept all the cache times in one file.

// utils/cache-times.ts
const minutesCache = (n: number) => n;
const hoursCache = (n: number) => n * 60;

export type CacheTypes = "static" | "content" | "dynamic";

export const cacheTimes: Record<CacheTypes, number> = {
  static: hoursCache(24),
  content: hoursCache(1),
  dynamic: minutesCache(1),
};

2. How to now cache it?

With the cache times sorted, the next question was where the caching actually happens.

Instead of a cached route for every backend endpoint, there's one catch-all route, /api/cache/..., that every cached request goes through. The cache time rides along in the URL, so the route knows how long to keep each response.

On the client, a small helper builds that URL.

// utils/cached-get.ts
export type CacheConfig = { type: CacheTypes };

export function cachedGet<T>(path: string, cacheConfig: CacheConfig) {
  const minutes = cacheTimes[cacheConfig.type];
  return $fetch<T>(`/api/cache/${minutes}/${path}`);
}

Each call then declares its own cache config, right where the request is made.

getPosts(page: number) {
  const cacheConfig: CacheConfig = { type: "content" };
  return cachedGet(`blog/posts?page=${page}`, cacheConfig);
}

So getPosts(2) goes to /api/cache/60/blog/posts?page=2.

Initially, during the first few times of using the cache engine, I had placed the duration at the end of the requests, and realized after lots of inconvenient network checks that browsers usually show just the last part of the URL :(

And then, for the server, a catch-all route which will extract the duration, along with all other properties of the request, so it can be forwarded

// server/api/cache/[...path].ts
import { withQuery } from "ufo";

export default defineEventHandler(async (event) => {
  const [minutes, ...rest] = getRouterParam(event, "path")!.split("/");
  const path = rest.join("/");
  const query = getQuery(event);

  const fetchFromBackend = defineCachedFunction(
    () => $fetch(path, { baseURL: useRuntimeConfig().apiBase, query }),
    {
      name: "api",
      maxAge: Number(minutes) * 60,
      swr: false,
      getKey: () => withQuery(path, query),
    },
  );

  return fetchFromBackend();
});

The key is the path plus the query string, so blog/posts?page=1 and blog/posts?page=2 are cached separately.

3. How do I now invalidate this cache?

They say cache invalidation is one of the hardest things in programming, and yes, it absolutely is. However, I still went the easy and simple route (since I'm lazy, of course).

Just wait for it to expire. Every entry is saved with the cache time from its URL. Once that time runs out, the next request fetches fresh data from the backend, and most of the time that's all you will really need.

Or create a way to bust it manually. If you can't wait for it to expire, like right after publishing a post, a request can ask for a fresh copy by adding (in this case) reset to the query, e.g. blog/posts?page=1&reset=1.

For the implementation

Pull out the reset from the query (so it's not added to the cache key),

// server/api/cache/[...path].ts
const { reset, ...query } = getQuery(event);

// ...then in the defineCachedFunction options
shouldInvalidateCache: () => reset === "1",

4. What could go wrong?

Param order. blog/posts?page=1&limit=10 and blog/posts?limit=10&page=1 ask for the same thing, but they're different strings. Since the key is built from the query string, they end up as two separate cache entries, and two backend requests.

So before building the key, the query gets its keys sorted.

const objectKeySorter = (obj: Record<string, any>) =>
  Object.fromEntries(
    Object.entries(obj).sort(([a], [b]) => a.localeCompare(b)),
  );

// ...then in the defineCachedFunction options
getKey: () => withQuery(path, objectKeySorter(query)),

Now both URLs share one key, whichever order the params come in.

Everyone fetching at once. When a popular entry expires, every request that comes in before the fresh one is saved misses the cache and goes straight to the backend, which is exactly what we were trying to avoid in the first place.

So, keep a map of requests still in progress, and let late arrivals wait for the one already running,

const inFlight = new Map<string, Promise<unknown>>();

export function once<T>(key: string, fetcher: () => Promise<T>) {
  if (inFlight.has(key)) return inFlight.get(key) as Promise<T>;

  const promise = fetcher().finally(() => inFlight.delete(key));
  inFlight.set(key, promise);
  return promise;
}

The finally is important here. It removes the entry once the request is done (whether it worked or failed), so the map never keeps growing.

Conclusion

That's the cache for content everyone sees the same way. In part 2, I'll go through caching for logged in users, which was the hardest part of the whole thing.

Thank you for taking the time to read! :)

How I built a mini cache engine (Part 1)

Share this post