Security audit

calm audit tests the authorization of your API from the outside. It calls every route as every kind of user and checks each answer against your policies and field rules. It runs on a throwaway database, so your data is never touched.

npx calm audit

Policies and field rules say what should happen. The audit shows what actually happens, including bugs in services, hooks and custom code that a static check cannot see.

What it does#

1. Boots your app (your own calm/ and app/ code) with createTestApp on a throwaway database on MONGODB_TEST_URI (default mongodb://127.0.0.1:27017). Rate limits and login lockout are off for the run, and the database is dropped at the end.
2. Signs up one user per persona: the owner, another user, an admin, and one user for each custom role your policies use. Anonymous calls carry no token.
3. Seeds a valid record for every resource from its field definitions, referenced resources first. Records the owner cannot create through the API are created through the service. A resource whose required fields it cannot fill (a .pattern(), a .check() or a Standard Schema without a default) gets fewer checks.
4. Calls list, read, create, update and delete on every resource as every persona, and compares each answer with the exposure matrix.
5. Runs probes against each resource and against the auth routes.

Without a user resource (the minimal template), only anonymous calls are made.

What it checks#

Some cells are skipped, not failed: an allowed action that answers 404 on a resource whose service has its own scope() (the scope may hide the record on purpose), and actions decided by a policy function.

Reading the output#

On a new project from the api template:

  Auditing on a throwaway database...

  CalmAPI audit

  post /posts  (custom scope)
    action  Anonymous  Signed-in user  Owner    Admin
    list    ✔ allow    ✔ allow         ✔ allow  ✔ allow
    read    ~ allow    ~ allow         ✔ allow  ✔ allow
    create  ✔ deny     ✔ allow         ✔ allow  ✔ allow
    update  ✔ deny     ✔ deny          ✔ allow  ✔ allow
    delete  ✔ deny     ✔ deny          ✔ allow  ✔ allow

  user /users
    action  Anonymous  Signed-in user  Owner    Admin
    list    ✔ deny     ✔ deny          ✔ deny   ✔ allow
    read    ✔ deny     ✔ deny          ✔ allow  ✔ allow
    create  ✔ deny     ✔ deny          ✔ deny   ✔ allow
    update  ✔ deny     ✔ deny          ✔ allow  ✔ allow
    delete  ✔ deny     ✔ deny          ✔ deny   ✔ allow

  Probes
  ✔ mass assignment: post.slug rejected with 422
  ✔ mass assignment: post.author rejected with 422
  ✔ mass assignment: post.publishedAt rejected with 422
  ✔ operator injection: post x[$ne]=1 -> 400
  ✔ limit cap: post limit=100000 -> 400
  ✔ unknown filter: post definitelyNotAField=1 -> 400
  ✔ error hygiene: post bad id -> 404
  ✔ operator injection: user x[$ne]=1 -> 400
  ✔ limit cap: user limit=100000 -> 400
  ✔ unknown filter: user definitelyNotAField=1 -> 400
  ✔ error hygiene: user bad id -> 404
  ✔ forged session token -> 401
  ✔ token in query string ignored -> 401
  ✔ login does not reveal accounts 401 / 401

  All checks passed · 52 passed · 2 skipped (custom scope or policy) · 54 checks

Each table shows the expected answer from your policies, and a mark for what the API did:

The post reads are skipped because post.service.js hides drafts from everyone but their author and admins.

When a check fails#

Here notes are private to their author:

// app/resources/note/note.resource.js
import { resource, f } from '#calm';

export default resource('note', {
  fields: { text: f.string().required(), author: f.ref('user').owner() },
  policy: { list: 'owner', read: 'owner', create: 'authenticated', update: 'owner', delete: 'owner' },
});

But the service overrides list() and forgets the owner scope:

// app/resources/note/note.service.js (a bug)
import { ResourceService } from '#calm';

export default class NoteService extends ResourceService {
  async list() {
    const items = await this.model.find({}).lean();
    return { items, total: items.length };
  }
}

The audit reports it under the resource, with one line per failed cell:

  note /notes
    action  Anonymous  Signed-in user  Owner        Admin
    list    ✔ deny     ✖ own (200)     ✖ own (200)  ✖ own (200)
    read    ✔ deny     ✔ deny          ✔ allow      ✔ deny
    create  ✔ deny     ✔ allow         ✔ allow      ✔ allow
    update  ✔ deny     ✔ deny          ✔ allow      ✔ deny
    delete  ✔ deny     ✔ deny          ✔ allow      ✔ deny
    ✖ list as Signed-in user: returned 2 record(s) owned by someone else
    ✖ list as Owner: returned 1 record(s) owned by someone else
    ✖ list as Admin: returned 5 record(s) owned by someone else

For the whole project (the api template plus note), the summary line reads 3 failed · 74 passed · 2 skipped (custom scope or policy) · 79 checks. Field leaks are listed under Field leaks, one line each (for example ✖ customer.phone returned to anonymous (read /customers)), and failed probes get a ✖ under Probes.

The fix is to keep the request context, so owner scoping still applies: call super.list(query, ctx), or build the filter with this.filterFor(ctx, filter) (see Services and hooks).

Options#

calm audit [--json] [--ci]

Each entry in results looks like this:

{ "resource": "note", "action": "list", "persona": "owner", "expected": "own", "status": 200, "ok": false, "note": "returned 1 record(s) owned by someone else" }

ok is true, false, or null for a skipped cell.

In CI and devtools#

The audit needs a MongoDB server, like npm test. Add it to the pipeline next to your tests (see Testing):

MONGODB_TEST_URI=mongodb://127.0.0.1:27017 npx calm audit --ci

The same audit runs from the devtools Security panel (Run audit, on a throwaway database on your dev server's MongoDB), and coding agents can run it with the calm_audit MCP tool. Run it after changing policies, fields or services.