Dishaya / Careers / Research Scientist, AI Evaluation

Research Scientist, AI Evaluation

Measure what good means when the answer must be right: own the science behind the quality bar, the honest accuracy numbers we publish, and the discipline that keeps what we claim calibrated to what our measurements support.

Research · Remote-First · Founding Equity

About Dishaya

Dishaya makes research people can actually send. One question in; a client-ready report, deck, and source ledger out, with every claim checked against the exact passage it cites and labeled verified, partial, unverified, or contradicted. What fails the check is disclosed, never hidden. We are early, independent, honest about both, and building toward one ten-year outcome: "Verified by Dishaya" becoming a mark a reader trusts before they read. More in About and Principles.

Why This Role Matters

The product's promise is that every claim is checked and every failure is disclosed. That promise is only as strong as the measurements behind it. This role owns the science of those measurements: defining what a good verified answer is, proving how often we deliver one, and making sure the numbers we publish about ourselves are numbers we would trust if a competitor published them. We would rather report a modest true number than an impressive soft one, and this role is how that stays true as we grow.

What You'll Work On

• The honest numbers: designing the measurements behind every accuracy claim we publish, so the number means what a skeptical reader assumes it means.
• Failure taxonomies: naming, counting, and ranking the ways verified research goes wrong, so improvement effort lands on the failures that matter most.
• Calibration: keeping every public claim aligned to what the evidence supports, including saying less when the measurements are thin.
• Raising the quality bar itself: evaluations that catch problems before users do, and that get harder as the product gets better.

We describe work by the outcomes you will own rather than by our internal systems; you will see everything on the inside from day one.

Responsibilities

• Design, run, and defend the evaluations that define what a good verified answer is.
• Own the accuracy numbers we publish; a number that cannot be reproduced does not ship.
• Build and maintain the failure taxonomy, and turn it into priorities the team acts on.
• Keep public claims calibrated to evidence, and say so plainly when a claim outruns the measurement behind it.
• Write things down: methods, results, uncertainty, and honest negative results.

Required Qualifications

• Strong empirical background in ML evaluation, NLP, or an adjacent field, with statistical rigor you can demonstrate rather than cite.
• Published or shipped evaluation work: papers, production evaluation suites, or public accuracy reporting you designed and stood behind.
• You understand sampling, measurement error, and uncertainty well enough to know when a number is not real, and you say so.
• Clear, calm written communication.

Preferred Qualifications

• Experience evaluating systems built on large language models in a product people paid for.
• You have published quality or accuracy numbers that faced outside scrutiny and held.
• You have turned evaluation findings into product decisions, not just reports.

Nice To Have

• Reviewing, organizing, or replication work in the empirical ML community.
• Public writing on measurement, calibration, or why most published numbers are softer than they look.

What Success Looks Like

• 30 days: you have reproduced our current quality measurements yourself, written down exactly what they do and do not support, and corrected at least one claim we were making about our own accuracy.
• 90 days: a failure taxonomy exists with real counts behind it, the team's improvement work is prioritized by it, and the accuracy reporting we publish is grounded in measurements you can defend line by line.
• 365 days: our public accuracy reporting stands up to hostile scrutiny from a motivated skeptic, the quality bar is measurably higher than when you arrived, and the proof is reproducible by someone who is not you.

Team Principles

• Honesty over fluency, in the product and in code review.
• Delete before you add; every abstraction earns its keep.
• Evidence over enthusiasm; direction comes from users.
• Small, senior, trusted; you own outcomes, not tickets.

Benefits

• Founding-level equity; early means it matters.
• Remote-first, judgment over time zones.
• The hardware and tools you need, without a procurement dance.
• Direct access to how the company runs: numbers, decisions, reasons.

Interview Process

1. Intro conversation (30 minutes): the honest state of the company, and what you want to build.
2. Craft deep-dive: real decisions inside work you shipped.
3. Paid working session: scoped, close to the real job, never spec work we ship.
4. References and a clear written offer, fast.

Equal Opportunity

Dishaya is an equal opportunity employer. We evaluate candidates on craft, judgment, and alignment with how we work, never on race, color, religion, gender, gender identity or expression, sexual orientation, national origin, disability, age, or veteran status.