Rectangle One: Scoring & ATS Methodology

Version: 0.3.3, last updated May 2026.

This document describes how Rectangle One scores resumes, customises them for specific applications, generates cover letters, audits ATS (Applicant Tracking System) compatibility, and how we measure whether each of those flows is reliable, valid, and fair. We publish this so users and reviewers can replicate our claims rather than taking them on trust.

The features covered are:

• Resume Scoring: produces 0-100 numeric scores against a fixed rubric (§2). Reliability and fairness are measured statistically across a 45-resume corpus (§4.1, §4.2).
• Application Customisation: rewrites a resume for a specific role and job description. Quality is audited via prompt-promise checks on a 6-pair golden set: keyword uplift, title integrity (no cross-domain rewrites on mismatch pairs), and numeric faithfulness (§4.7).
• Cover Letter: generates a tailored cover letter. Audited via prompt-promise checks: word count, opener discipline, sentence-subject variety, role/organisation mention, and numeric faithfulness against the resume (§4.7).
• ATS Audit: a deterministic module that runs the same checks every time, no AI involved (§3).

Generative features additionally pass a fairness evaluation that perturbs demographic markers (name, school, address) on the same resume and asserts the rewrite is stable across the perturbations (§4.7, §4.10).

1. What we score

Rectangle One produces three distinct signals:

General quality and application fit are produced by an AI model against a fixed rubric (see §2). ATS readiness is a pure-code module with no AI calls; it runs the same checks every time, deterministically (see §3).

2. The scoring rubrics

2.1 General resume scoring

Four parameters, each scored 0-100 by the model, equally weighted at 25%:

The headline 0-100 score is the equal-weighted mean of the four parameters.

2.2 Application-fit scoring

Three parameters, with role alignment given the largest single share so the headline fit number primarily tracks how directly the resume matches the specific role and job description:

Both rubrics are versioned at v0.3.1. Any prompt or weight change that could move scores by more than the stability band (§4.1) requires a major version bump.

Rubric changelog:

• v0.1.0: initial prompt + previous-score anchoring (removed: anchoring made scores depend on history rather than content).
• v0.2.0 (April 2026): added NEUTRALITY REQUIREMENTS block; fairness evaluation reduced confirmed outliers from 7/45 to 5/45.
• v0.3.0 (April 2026): rewrote PROFESSIONAL PRESENTATION with five observable sub-criteria (mechanical correctness, English-variant consistency, tense/voice consistency, idea-level conciseness, professional register); removed the unobservable "two lines" length rule; added three concrete neutrality examples (employer parity, school parity, name/address parity). Per-parameter max stdDev dropped from 8.66 → 5.77.

3. Deterministic ATS audit

The ATS audit runs the following checks on every resume without calling an AI model. Every finding is reproducible from the same input.

Findings carry one of three severities. The default score penalties are:

Some checks carry a documented scoreImpact override where the evidence and user effect are not binary. Examples: multi-column layouts are −12; sidebars are −8; common-but-not-safelisted fonts such as Roboto are −2; decorative or handwriting fonts such as Indie Flower are −10; readable palette cautions are −2; dark-background or low-contrast palettes range from −8 to −12 depending on background luminance and measured body/heading contrast. The score object records both the total penalty and penalty by category.

Penalties subtract from a base of 100 (floored at 0) to produce the readiness score, banded as:

The audit is pure code. Every rule, threshold, and severity is in source; we chose this path so that every score is reproducible and candidates are not subject to a black-box third-party verdict on their employability.

3.1 Fonts and design gradation

The app distinguishes between an ATS-friendly badge and a graded score impact. The badge is intentionally conservative: it marks only designs we are confident about. The score underneath is more nuanced.

Published guidance supports the ordering of these risks, not our exact numeric penalties. The specific scoreImpact values are Rectangle One's deterministic calibration so the audit preserves that ordering without flattening every non-perfect design choice into the same penalty.

For fonts, common resume-safe/system fonts receive no finding. Common, legible fonts outside the conservative safelist, such as Roboto, Inter, Lato, Ubuntu, or Quicksand, produce a low-impact info finding. Decorative, handwriting, or novelty fonts produce a higher-impact warning. This reflects the evidence: guidance consistently recommends common legible fonts and warns against decorative formatting, but does not justify treating every modern readable sans font as equally risky as a script font.

For layout, multi-column and sidebar templates carry larger score impacts than font cautions because ATS guidance is much stronger and more consistent on reading order, text boxes, tables, headers/footers, and graphic regions.

Source basis for this ordering:

• Massachusetts Institute of Technology Career Advising & Professional Development, Make your resume ATS-friendly: use ATS-oriented formatting, a common and legible font, supported file types, and test the resume before submission.
• Suzanne Taylor, How To Write an ATS Resume (With Template and Tips), Indeed Career Guide (updated December 15, 2025): use a simple ATS-friendly template, clear section headings, avoid headers/tables/graphics, and choose standard fonts such as Arial, Calibri, or Times New Roman.

3.2 Colour and ATS parsing

We do not treat black-and-white as inherently higher scoring than every colour palette. The documented risk is narrower: ATS/OCR and print workflows are more reliable when core resume text stays high-contrast on a light or white background. Conservative accent colour is acceptable when it does not carry core text or reduce contrast.

Source basis for this rule:

• Massachusetts Institute of Technology Career Advising & Professional Development, Make your resume ATS-friendly, recommends ATS-oriented formatting, common legible fonts, supported files, and testing before submission.
• Suzanne Taylor, How To Write an ATS Resume (With Template and Tips), Indeed Career Guide (updated December 15, 2025), recommends simple ATS-ready templates, clear headings, avoiding headers/tables/graphics, and standard fonts.
• Lotus Buckner, Using Color on a Resume: Pros, Cons and How To Do It Well, Indeed Career Guide (updated December 11, 2025), and Indeed Editorial Team, How to Use Colours for a Resume (With Design Tips), Indeed Career Guide Canada (updated November 21, 2025), treat black-and-white as the safe default, allow colour when used conservatively, and warn against distracting or low-readability colour choices.
• W3C WAI, Understanding SC 1.4.3 Contrast (Minimum) (Level AA), provides the concrete 3:1, 4.5:1, and 7:1 contrast thresholds that we use as a deterministic readability proxy.

Implementation: palette checks use WCAG relative-luminance contrast maths as a deterministic proxy for scanner/readability risk. This is a readability heuristic, not a claim that ATS vendors themselves publish WCAG-based scoring rules. A palette is badged ATS-friendly when:

Dark/reversed backgrounds, body/secondary text contrast below 4.5:1, or very low primary-heading contrast produce a warning. Readable palettes that miss only the higher-confidence band produce an info finding. This means a high-contrast light palette can score the same as black and white; low-contrast or dark/strongly-coloured-background palettes cannot. A saturated palette with good body contrast can score better than one with unreadable body text, but it still loses ATS confidence when the page background itself is strongly coloured or primary section-heading colours fall below the highest-confidence threshold.

Current palettes badged as ATS-friendly:
Classic Black & White; Rectangle One Theme; Professional Blue; Modern Green; Serene Blue & Gray; Almost Monochrome; Mist & Blue; Muted Charcoal; Creative Orange & Emerald; Elegant Purple & Gold; Royal Navy & Gold; Rose Gold Glamour; Powder Blue Serenity; Slate & Sky; Cool Gray & Blue; Cosmic Latte; Earthy Tones; Rustic Red & Brown; Minty Fresh; Forest Canopy; Ocean Breeze; Enchanted Forest Deep; Emerald City Vista; Lavender & Pink.

4. How we measure ourselves

We treat scoring as a measurement problem and apply standard psychometric and ML-evaluation techniques. The eval harness is gated on a separate run mode so the suites are excluded from the default test run; they make live AI API calls.

4.1 Reliability: does the same resume get the same score?

Plain English: if you re-score a resume without changing anything, the number should not move much.

For each golden resume we run the scoring flow N=3 times at temperature 0 with a fixed prompt version, and report standard deviation and max-min spread on both the headline score and each individual parameter.

Why a band rather than zero? LLMs are not bit-deterministic even at temperature 0: floating-point non-associativity in GPU kernels, batching, and tied-logit sampling produce small per-call variation. The honest claim is a measured band of consistency, not determinism.

Measured (rubric v0.3.0, 45-resume bulk corpus, N=3 runs each, April 2026):

The headline score sits well inside its ±5 band on every resume in the corpus. The per-parameter max stdDev dropped from 8.66 (rubric v0.2.0) to 5.77 after the professional_presentation rewrite in v0.3.0. The 3.3% of per-parameter cells (out of 45 × 4 = 180) that exceed stdDev 5 are within the model's own intrinsic run-to-run variation and are disclosed here rather than masked.

For context. Published research on human resume screening shows wide disagreement between independent reviewers; inter-rater reliability is typically reported in the moderate range (κ ≈ 0.4–0.7), and the same resume rated by two trained reviewers can move by double-digit points (Highhouse 2008; Hunter & Hunter 1984). Rectangle One's run-to-run variation on identical input is materially smaller than human reviewer disagreement on the same artefact, but the framing of this comparison matters: humans disagree across reviewers, the AI disagrees with itself across runs. They are not the same axis. We publish ours as measured self-consistency and do not claim it as inter-rater equivalence.

In the product we content-hash-cache scores keyed by the rubric version, sanitised resume data, and language. An unchanged resume returns a bit-identical score instantly. Rubric version bumps force a fresh score.

4.2 Validity: do our scores agree with reality?

Plain English: when we rank resumes from worst to best, our order should broadly match the order an independent ground-truth source produces.

Status: deferred. Validity testing requires curator-scored ground truth: resumes that two reviewers have independently scored against the published rubric. That set is intentionally empty pre-launch: it gets populated only as scoring sessions happen with at least two human reviewers.

The bulk-tier corpus (§5) is not valid for Validity testing. We sourced it from public resume datasets with no rubric-aligned scores attached; consistency with our rubric is precisely what we are trying to measure, so using bulk labels would be circular.

When the curator set reaches a meaningful size (target: 50 resumes distributed across all bands, two-reviewer median scores) we will add a validity suite asserting Spearman ρ ≥ 0.6 between model output and curator score, and publish the result here with the dataset description.

We do not claim validity at launch. Landing-page wording must reflect this until the curator set ships.

4.3 Fairness: do equally-qualified candidates get equal scores?

For each golden resume we generate perturbed copies that change only demographically-correlated attributes:

We measure the absolute change in headline score across all perturbations and publish both the distribution and any individual cases that exceed our stability band. The intent is that demographic perturbations should be statistically indistinguishable from re-run noise.

The fairness suite runs against the full bulk corpus, which spans 25 tech and 20 non-tech resumes across 20+ occupational categories so the perturbations are exercised across a broad slice of real candidates rather than a single archetype.

Progression across rubric versions (45 resumes × 5 perturbations = 225 perturbation calls per run, April 2026):

Two-run agreement protocol. For rubric v0.3.0 we applied a replicate run before the v4 run to distinguish genuine bias signals from model noise. A resume is classified as a confirmed fairness outlier only if it fails in both independent runs. Under this protocol applied to v0.3.0 pre-fix, 44/45 resumes confirmed pass. The one confirmed outlier (jr-0076) was traced to a flawed perturbation design: replacing the institution name with "Northern Community College" while leaving the degree level as "Master's" created an internally incoherent credential. The school.community-college perturbation has been corrected in v0.3.0 to also normalise the degree to "Associate of Applied Science", making the test coherent.

Focused-replicate study (v0.3.0, 6 suspect resumes × 5 perturbations × N=4 paired replicates per cell, 144 calls). When two independent runs disagreed on individual cells, we could not tell whether a single failure represented systematic bias or per-call LLM jitter. A focused replicate harness repeats baseline and perturbed scoring N=4 times for each suspect cell and reports the mean shift against the per-cell noise band. A cell is reclassified as a confirmed bias outlier only when the mean shift exceeds the threshold across all four replicates. Findings: jr-0048 and jr-0076 were confirmed clean (max meanΔ ≤ 3 across all perturbations); jr-0079 and ra-0012 showed score shifts on multiple non-name perturbations that trace to internal coherence dependencies in those resumes rather than name-pattern bias; jr-0082 showed a deterministic, name-isolated drop of 6 points specifically on the West-African-name perturbation (std=0 across 4 replicates) while the same resume was stable for South Asian, Anglo, and Arabic names. This is the kind of signal (small, isolated, reproducible) that a single 45-resume bulk run would not reliably detect, and it is exactly the failure mode the methodology is designed to catch. Rubric v0.3.1 addresses it directly by strengthening the neutrality block with explicit West, East, and Southern African name examples and a self-check instruction that requires the model to invalidate any reasoning citing a name, school, employer, or location.

For reference, the headline-score reliability stdDev cap is 4.04; a fairness delta of 5–8 points is within the same magnitude as the model's own re-run variation on identical input. Deltas of 10+ are above re-run noise and we treat them as genuine signals worth investigating.

Scope note. The current fairness suite targets Resume Scoring only. Application-fit scoring runs at a higher temperature and is not yet covered by the automated bias sweep; it is a tracked pre-launch follow-up with higher urgency because a hiring-manager framing is inherently more identity-sensitive than a rubric framing. Until measured, we do not claim fairness on application-fit scores.

4.4 Acceptance thresholds (v0.3.1)

Thresholds will be reviewed once we have telemetry from real-world resumes; they will be tightened (or, where the data shows we are over-claiming, loosened with explicit disclosure) before v1.0.

4.5 Continuous improvement

We re-run the reliability and fairness suites against the bulk corpus on every rubric change and after each major model update. Any case that exceeds a published threshold is logged, attributed to a specific cause (prompt ambiguity, protected-attribute leakage, parsing edge case), and used as a prompt-engineering target for the next rubric revision.

We publish measured numbers, including individual cases where they exceed our band, because we would rather be honest about AI-scoring behaviour than claim deterministic precision the technology does not offer. The numbers improve between rubric versions; the methodology stays the same.

4.6 Eval coverage by feature

The — cells under Fairness and Validity for generative features reflect that those features produce free-text rather than numeric scores, so the perturbation and correlation methodology does not apply directly. Quality checks for generative features are deterministic (rule-based) wherever possible; AI-as-judge is reserved for specificity/coherence dimensions that rules cannot capture.

4.7 Generative-feature evals: measured results

Generative features do not produce a numeric score, so the methodology shifts from statistical reliability/fairness to prompt-promise auditing: we encode every claim the prompt makes ("under 300 words", "no filler phrases", "never invent a metric") as a deterministic check, and assert all promises hold across a fixed golden set of 6 resume × JD pairs spanning good-fit, partial-fit, and mismatch scenarios across backend, HR, PM, marketing, and tech-lead roles.

Application Customisation (temperature 0.5, April 2026, run on google/gemini-3.1-flash-lite-preview):

Mean keyword-coverage uplift +0.18; minimum per-pair uplift +0.10; zero high-severity novel numeric tokens across all suggestions; titles on both mismatch pairs verifiably preserved (no cross-domain forcing). Acceptance threshold: mean uplift ≥ 0.10, per-pair ≥ −0.02, zero forced cross-domain title rewrites on mismatch pairs, ≤ 2 high-severity novel numerics across the run.

Cover Letter (temperature 0.7, April 2026, run on google/gemini-3.1-flash-lite-preview):

Quantified-evidence rate (soft signal): 83.3% of letters cite at least one resume-grounded metric. Acceptance thresholds: word count ∈ [180, 330], zero forbidden openers, zero filler phrases, ≤ 2 consecutive "I" sentence starts, organisation and role both mentioned, zero novel numeric tokens.

Generative fairness (April 2026, run on google/gemini-3.1-flash-lite-preview): for each of three baseline pairs (jd-marketing-mismatch, jd-pm-good, jd-tech-lead-partial), the candidate's name (3 demographic variants), school (community-college variant), and address (non-metro variant) are perturbed and the customisation + cover-letter features are re-run. Acceptance bands enforced per perturbation: keyword-uplift Δ ≤ 0.10, novel-numeric count rise ≤ 1, title-align verdict does not regress away from the fit-correct expectation, organisation/role mention is not lost. All 36 perturbed cells across both features pass.

Faithfulness coverage. Both generative features share a faithfulness guard (§6.2). The customisation eval also enforces it on every suggestion individually so a pair cannot pass on coverage alone if any suggestion smuggles a fabricated metric.

4.8 Model coverage: what the numbers cover and what they don't

All measured numbers above (reliability, fairness, customisation, cover-letter) were produced against a single primary model: the slug pinned internally per flow and surfaced in the admin UI as the per-flow "Evaluated" badge. Each flow has a fixed temperature (Resume Score 0.0, Application Customisation 0.5, Cover Letter 0.7) that the eval harness pins to the production setting.

We do not assume these results transfer to other models. Different LLMs vary materially in:

• instruction adherence on hard-constraint blocks (title integrity, numeric integrity); a model that ignores the constraint will fail the customisation eval even if the currently-evaluated model passes it;
• temperature-0 determinism (per-call jitter differs across providers and changes the reliability stdDev band);
• protected-attribute behaviour: fairness deltas for the same prompt vary by model and must be re-measured.

The in-app admin model selector shows an "Evaluated" badge next to model and feature combinations covered by this methodology, and a warning when an admin selects a non-evaluated model for a feature. The warning is informational, not blocking; admins may experiment with new models, but until those combinations have been run through the eval harness we do not stand behind the published numbers for them.

4.9 What we are not: a note on third-party ATS scorers

We sometimes get asked whether Rectangle One uses Affinda, Sovren, Daxtra, or similar commercial parsing/scoring APIs. It does not. Our ATS readiness signal (§3) is a deterministic pure-code module; every rule, threshold, and severity is in source. We chose this path so that (a) every score is reproducible offline, (b) we can publish the rules verbatim rather than treating them as proprietary, and (c) candidates are not subject to a black-box third-party verdict on their employability. The trade-off is that we do not claim rendering-equivalence with any specific commercial ATS; we model the common ATS ingestion failure modes that published research and recruiter-facing literature consistently identify (non-standard fonts, multi-column layouts, header/footer regions, embedded images, missing standard sections, weak quantification, JD-keyword gaps).

4.10 Known limitations

We publish these openly so reviewers can weight the claims accordingly:

• Bulk corpus is N=45. Effects smaller than ~3 percentage points will not reliably surface. We expand the corpus when we add new occupational categories or perturbations.
• Replicate counts are conservative. Reliability uses N=3, focused fairness replicates use N=4 per cell. With observed σ ≈ 0.4 on aggregate scores and σ_max ≈ 4 on the most volatile sub-scores, N=4 has SE ≈ 2 on the headline, borderline for the |Δ| ≤ 5 fairness bar. Bumping to N=10 across the board would bring SE to ~1.3; we trade replicate count against iteration time accordingly.
• Single model evaluated (§4.8). Other models available in the product have not been measured.
• Validity is deferred (§4.2). Until the curator set is populated we make no claim that our scores correlate with independent expert judgement; the reliability and fairness numbers are about consistency, not correctness.
• Application-fit score deprioritised. It runs at temperature 0.3 and is not yet in the fairness sweep. The product framing is that the candidate already knows their fit; the score is advisory.
• Residual fairness drift on non-protected attributes. The focused-replicate study found that one resume (ra-0012) drops on multiple non-African name perturbations; this is name-anchoring on the underlying resume content rather than protected-attribute bias, but it does mean the model is more name-sensitive than we would like. Tracked for the next rubric revision.
• Generative features are not fairness-tested for parity. Customisation and cover-letter rewrites are tested for prompt-promise adherence and faithfulness, not for demographic parity in the rewrites themselves. Pre-launch follow-up.

5. Datasets we use

The eval harness loads resumes from two tiers:

The bulk corpus is sourced from two complementary public datasets:

A third dataset, cnamuangtoun/resume-job-description-fit (8,000 resume-JD pairs labelled "No Fit" / "Potential Fit" / "Good Fit"), is the intended source for application-fit validity testing (Spearman ρ of scores vs ground-truth fit labels). The validity suite has not been written yet; tracked as a pre-launch item.

All datasets are anonymised before use: name, email, phone, and address are replaced with synthetic values. This does not affect fairness testing, since the perturbation suite overwrites those same fields with its own values before scoring. Employers, institutions, and prose are preserved because the rubric reads them.

We never bundle the original datasets with the application.

6. Guardrails on the AI features

Beyond the eval harness, the AI features ship with two static guardrails:

6.1 Prompt-injection resistance

User-controlled text (resume content, job description, custom instructions) is wrapped in a tagged container with an explicit system clause instructing the model to treat the contents as data, not instructions. This is applied to all features that accept user-controlled text. Unit and integration tests exercise the wrapping and regression behaviour.

6.2 Faithfulness guard for AI rewrites

When any feature proposes a suggestion, a code-only guard scans the suggested text for tokens that do not appear in the original passage or the wider resume:

• High severity: novel numeric tokens (years, percentages, dollar figures). Surfaced in the UI as an amber “Verify this number” pill.
• Low severity: novel acronyms or multi-word proper nouns. Surfaced as a muted “New term added” pill.

Pure code, no AI. Acts as a safety net against hallucinated metrics or fabricated employer names. The pills are informational; they never block apply, since some additions are legitimate (e.g. a new sub-bullet the candidate asked for).

7. References

• ATS formatting/design: Massachusetts Institute of Technology Career Advising & Professional Development. Make your resume ATS-friendly. https://capd.mit.edu/resources/make-your-resume-ats-friendly/
• ATS formatting/design: Suzanne Taylor. How To Write an ATS Resume (With Template and Tips). Indeed Career Guide, updated December 15, 2025. https://www.indeed.com/career-advice/resumes-cover-letters/ats-resume-template
• ATS colour/readability: Lotus Buckner. Using Color on a Resume: Pros, Cons and How To Do It Well. Indeed Career Guide, updated December 11, 2025. https://www.indeed.com/career-advice/resumes-cover-letters/color-on-resume
• ATS colour/readability: Indeed Editorial Team. How to Use Colours for a Resume (With Design Tips). Indeed Career Guide Canada, updated November 21, 2025. https://ca.indeed.com/career-advice/resumes-cover-letters/colours-for-resume
• ATS colour/readability proxy: W3C Web Accessibility Initiative. Understanding SC 1.4.3 Contrast (Minimum) (Level AA). https://www.w3.org/WAI/WCAG22/Understanding/contrast-minimum.html
• Eval methodology: Zheng et al. (2023), Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena; Eugene Yan, Evals for AI Products; Hamel Husain, Your AI Product Needs Evals; Chip Huyen, Designing Machine Learning Systems.
• Human inter-rater reliability context: Highhouse, S. (2008). "Stubborn reliance on intuition and subjectivity in employee selection." Industrial and Organizational Psychology, 1(3), 333-342; Hunter, J. E. & Hunter, R. F. (1984). "Validity and utility of alternative predictors of job performance." Psychological Bulletin, 96(1), 72-98.
• Fairness perturbation methodology: Buolamwini, J. & Gebru, T. (2018). Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification; Bertrand, M. & Mullainathan, S. (2004). "Are Emily and Greg More Employable than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination."

8. Versioning

This document is versioned alongside the scoring rubric and ATS audit module. Any change to either that would alter scores by more than the stability band requires a major version bump and a fresh run of the eval harness, with a changelog entry recording before/after deltas.

Current version: 0.3.3 (May 2026).