Every item below was found the same way: by checking the thing that was supposed to enforce a rule, instead of the document that said it was enforced.
That is not a methodology we chose because it is rigorous. It is one we arrived at after a written-down rule turned out not to be enforced, three times in the same week.
The pattern
A gate can be written down, agreed to, and absent. Three examples of that pattern, all from this site, all found in one pass:
- A publishing standard said
assets must be non-empty. The schema that was supposed to enforce it had no such rule — a resource with assets: [] built fine.
- A set of cross-resource checks (duplicate ids, broken internal references) existed and had tests. Nothing on the build path called it. It guarded the test fixtures and nothing else.
- An SEO check confirmed markdown tables were rendering. Rendering was never the question.
None of these three was visible from reading the standard that claimed each was enforced. All three were visible in about a minute of checking whatever was supposed to do the enforcing — a schema, a call graph, a compiled stylesheet.
What we changed
Each one is now a gate that fails the build, and each gate was checked by breaking the thing it guards and confirming it went red. A gate that has never been seen to fail is not evidence of anything.
The redirect finding also became a tool, because a result that lives only in a report has a short life.