Bayesconsensus

BayesConsensus

[](./CHANGELOG.md) [](./LICENSE) [](./CHANGELOG.md) []() []() []() []()

[](https://www.star-history.com/#davccavalcante/bayesconsensus&type=timeline&legend=top-left)

> Universal, zero-runtime-dependency NPM library and CLI that resolves disagreement across a fleet of agents, judges, or models with the Dawid-Skene model. It estimates each voter's reliability and the true answer at the same time, from the votes alone, and returns a consensus label with a calibrated confidence and a per-agent reliability audit, for Massive Intelligence (IM) systems.

BayesConsensus is the consensus layer for multi-agent Massive Intelligence (IM). When you ask several agents, judges, or models the same question and they disagree, you need a single answer you can defend. Plain majority vote is the reflexive choice, and it is blind: it counts heads, so a bloc of unreliable voters outvotes a smaller set of reliable ones, and it returns a bare tally with no honest measure of how sure to be. BayesConsensus uses the right primitive. The Dawid and Skene (1979) expectation-maximization algorithm estimates, simultaneously and with no ground truth, the latent true label of every item AND a confusion matrix for every agent, the diagonal of which is that agent's reliability. It then returns the maximum-a-posteriori consensus label with a calibrated confidence, weighting each vote by how reliable its caster has proven to be. It is the jury foreman for a fleet of non-human entities.

Core promise: zero required runtime dependencies, two-line setup, reliability-weighted consensus instead of a blind majority, a calibrated confidence and full posterior on every verdict, a per-agent confusion matrix as a reliability audit, missing votes treated as first-class, online updates as votes arrive, calibration measured by simulation rather than asserted, a tamper-evident SHA-256 audit trail, a node-free core that runs in Node, edge runtimes, and the browser, ESM plus CJS dual distribution, and a framework-agnostic tool adapter that binds to any MCP server or tool-calling backend with no hard dependency.

---

Why BayesConsensus

Aggregating votes from a fleet of agents is an estimation problem, not a counting problem. You want the answer the reliable voters point to, and you want to know which voters are reliable, without ever being told the right answers in advance. The two reflexive approaches both fail. Majority vote (count every voter equally) lets an unreliable bloc win and reports a tally that is not a probability, three of four agents saying "safe" is not a 75 percent chance of safe. Trusting one designated expert throws away the wisdom in the rest of the fleet and has no recourse when the expert is wrong. BayesConsensus uses the correct primitive: it infers each agent's reliability from the pattern of agreement and disagreement across many items, and weights every vote by that reliability while it infers the true labels, in one coupled estimation.

What sets it apart from counting votes:

• Weights by reliability, not by headcount. Each agent gets a confusion matrix learned from the data, and its votes count in proportion to its estimated reliability. A rubber-stamp agent that always says the same thing is down-weighted automatically; a consistently accurate agent pulls hard. The reliable minority can overrule the unreliable majority.
• Learns who is reliable with no answer key. Reliability is recovered by expectation-maximization from the votes alone, never from labelled ground truth. You do not tell BayesConsensus which agents are good; it finds out from the structure of their agreement across items.
• Returns a calibrated confidence, not a tally. Every verdict carries the full posterior over labels, and the winning probability is a calibrated confidence you can threshold, route on, or escalate from, not a raw fraction of votes.
• Audits every agent. The per-agent confusion matrix and a single reliability score are first-class outputs. This is how you find the rubber stamp, the contrarian, and the silently-degrading model in your fleet.
• Treats missing votes as first-class. Agents need not vote on every item, and an item need not be seen by every agent. The model sums only over the votes that were actually cast, never imputes or pads, so a sparse, real-world voting matrix is the normal case.
• Updates online. Observe votes as your agents report them and query the current verdict at any time; the fit runs lazily and the calibrated confidence sharpens as evidence accumulates. The full state is the label space plus the votes, so a snapshot round-trips through JSON with no hidden state.
• Measures its own calibration. A confidence is only trustworthy if it matches the empirical accuracy at that confidence, so calibration is measured by simulation, drawing a fleet from known reliabilities, fitting the model, and scoring accuracy and Brier against the truth, not asserted.
• Proves what was decided. A tamper-evident SHA-256 hash-chained audit trail of consensus verdicts and decisions, the evidence a review asks for when a non-human entity resolved a disagreement on your behalf.
• Checks its own assumption. Dawid-Skene assumes agents err independently given the true label, the assumption a fleet of near-identical models breaks. The built-in diversity diagnostic measures the error correlation across the fleet, reports the effective number of independent voters, and warns when the agreement is lockstep, so an inflated confidence is caught rather than trusted.
• Updates incrementally. A streaming OnlineConsensus keeps running sufficient statistics and folds each batch in with a warm-started EM step, so reliability evolves as data arrives; optional exponential forgetting lets it adapt when an agent's accuracy drifts, rather than averaging the drift away.
• Tells you when not to use it. When every agent is equally weak, there is no reliability signal to learn and majority vote is the better baseline; the benchmark measures exactly this regime and the model documents it rather than overclaiming.

---

Install

pnpm add @takk/bayesconsensus
# or: npm install @takk/bayesconsensus
# or: yarn add @takk/bayesconsensus
# or: bun add @takk/bayesconsensus

The core has zero required runtime dependencies. Every @takk sibling is an optional peer; install only what you compose with.

---

Quickstart

import { ConsensusModel } from "@takk/bayesconsensus";

// Four agents label whether a code snippet is safe. Three are competent; one is a rubber stamp.
const jury = new ConsensusModel({ labels: ["safe", "unsafe"] });
jury.observe("snippet-1", "reviewer-a", "unsafe");
jury.observe("snippet-1", "reviewer-b", "unsafe");
jury.observe("snippet-1", "reviewer-c", "unsafe");
jury.observe("snippet-1", "rubber-stamp", "safe"); // always says "safe", whatever the truth

const verdict = jury.consensus("snippet-1");
console.log(verdict.label, verdict.confidence, verdict.distribution);
// "unsafe", ~0.94, { safe: 0.06, unsafe: 0.94 } -- the rubber stamp's dissent is down-weighted

observe folds one vote in (an agent assigning a label to an item). consensus returns that item's maximum-a-posteriori label, its calibrated confidence, and the full posterior, fitting the Dawid-Skene model lazily on first query. The verdict weights each vote by the caster's learned reliability, so a single unreliable dissenter does not flip a confident answer.

---

Aggregate the whole fleet, and audit every agent

import { ConsensusModel } from "@takk/bayesconsensus";

const jury = new ConsensusModel({ labels: ["yes", "no"] }).observeVotes(votes);

const report = jury.aggregate();   // fit once, return every item's verdict and every agent's reliability
report.converged;                  // did EM converge
report.classPrior;                 // the inferred prevalence of each label across the fleet
report.items;                      // per item: consensus label, confidence, full posterior
report.agents;                     // per agent: confusion matrix, accuracy, per-class accuracy

const stamp = jury.reliability("rubber-stamp");
stamp.accuracy;                    // a single reliability score in [0, 1]
stamp.confusionMatrix;             // the learned K x K matrix; the diagonal is per-class accuracy
stamp.perClassAccuracy;            // accuracy broken down by true label

aggregate fits the model once and returns a posterior for every item plus a reliability report for every agent. reliability exposes one agent's learned confusion matrix and accuracy. This is the audit a multi-agent system needs: it surfaces the rubber stamp, the contrarian, and the model whose accuracy is quietly drifting.

---

Majority vote versus Dawid-Skene

import { fitDawidSkene, majorityVote } from "@takk/bayesconsensus";

const mv = majorityVote(votes);          // the blind baseline: count heads, ties broken deterministically
const fit = fitDawidSkene(votes, {       // the reliability-weighted estimate
  labels: ["a", "b", "c"],
  mode: "soft",                          // "soft" (default, calibrated) or "hard" (Fast Dawid-Skene, faster)
});

fit.consensus.get("item-1");             // { item, label, confidence, distribution }
fit.confusion;                           // per-agent confusion matrices
fit.classPrior;                          // inferred label prevalence
fit.converged;                           // convergence of the marginal log-likelihood

Use majorityVote for the free baseline and fitDawidSkene for the functional core when you want the raw fit rather than the ConsensusModel facade.

---

Online updates, and serialization for free

import { ConsensusModel } from "@takk/bayesconsensus";

const jury = new ConsensusModel({ labels: ["approve", "reject"] });
jury.observe("pr-42", "agent-1", "approve");
jury.observe("pr-42", "agent-2", "reject");
jury.consensus("pr-42").confidence;   // ~0.50, a genuine split

jury.observe("pr-42", "agent-3", "approve");
jury.observe("pr-42", "agent-4", "approve");
jury.consensus("pr-42").confidence;   // sharpens as agreement accrues

// The full state is the label space plus the votes; a snapshot is plain JSON.
const restored = ConsensusModel.fromSnapshot(JSON.parse(JSON.stringify(jury.snapshot())));

Observe votes as they arrive and query at any time. The fit is lazy and the confidence tightens with evidence. A snapshot is the label space and the votes, so it serializes and restores with no hidden state.

---

Check the fleet for independence

Dawid-Skene assumes agents err independently given the true label. A fleet of near-identical models breaks that assumption, and their agreement then overstates the evidence, inflating the confidence. The diversity diagnostic measures it.

import { ConsensusModel, diversityReport, temperDistribution } from "@takk/bayesconsensus";

const model = new ConsensusModel({ labels: ["yes", "no"] }).observeVotes(votes);
const report = model.diversity();

report.meanAgreement;        // raw lockstep, reference-free; near 1 means the fleet votes together
report.meanErrorCorrelation; // how often agents are wrong on the same items, the independence-breaking pattern
report.effectiveVoters;      // N / (1 + (N - 1) * correlation): the headcount discounted for correlation
report.confidenceDiscount;   // effectiveVoters / N, the factor to temper an over-confident verdict by
report.independenceHolds;    // false when the agents fail together
report.warning;              // a plain-language caution, present only when independence is doubtful

// Discount an over-confident verdict back toward honesty.
const verdict = model.consensus("item-1");
const honest = temperDistribution(verdict.distribution, report.confidenceDiscount);

The strongest form, diversityReport(votes, reference), takes a trusted reference labelling (a small labeled set) and measures error correlation directly. Without ground truth, meanAgreement is the blunt, always-identifiable signal: a unanimous-but-wrong fleet of similar models cannot be told apart from a unanimous-and-right one from the votes alone, so high agreement among models that share training data is itself the warning.

---

Stream votes with OnlineConsensus

ConsensusModel re-fits from scratch on each query. OnlineConsensus is the true incremental fit: it keeps running sufficient statistics and folds each batch in with a warm-started EM step, so reliability evolves as data arrives. Optional exponential forgetting lets it adapt when an agent's accuracy drifts.

import { OnlineConsensus } from "@takk/bayesconsensus";

const online = new OnlineConsensus({ labels: ["yes", "no"], forgetting: 0.7 });

// Fold a batch in; get the verdicts for that batch, scored with the just-updated model.
const verdicts = online.update([
  { item: "claim-1", agent: "model-a", label: "yes" },
  { item: "claim-1", agent: "model-b", label: "no" },
]);

online.reliability("model-a"); // evolves with every batch
online.classPrior();           // the current label prevalence

With forgetting below 1 the accumulated counts decay before each batch, so recent behavior outweighs the stale past, the model tracks an agent that degrades rather than averaging it with its better history.

---

Entry points

Fourteen subpath exports, each importable on its own. The core is node-free; only node touches a Node built-in.

| Import | What it gives you | |---|---| | @takk/bayesconsensus | The full toolkit barrel: the facade, the Dawid-Skene core, confusion and majority primitives, calibration, the diversity diagnostic, streaming, the adapter, and the audit chain. | | @takk/bayesconsensus/consensus | ConsensusModel, the facade most callers use: observe, fit, consensus, reliability, aggregate, diversity, snapshot. | | @takk/bayesconsensus/dawidskene | The Dawid-Skene EM core: fitDawidSkene, the log-space E-step and M-step, soft and hard modes. | | @takk/bayesconsensus/confusion | Confusion-matrix construction, diagonal Dirichlet smoothing, row-normalization, and the log form. | | @takk/bayesconsensus/vote | The branded ids and the VoteSet collector that builds the indexed vote matrix EM consumes. | | @takk/bayesconsensus/majority | majorityVote and weighted-vote baselines, also the deterministic EM initializer. | | @takk/bayesconsensus/calibration | Simulation-measured accuracy versus majority, Brier score, and reliability bins. | | @takk/bayesconsensus/diversity | diversityReport and temperDistribution, the error-correlation and effective-voter independence check. | | @takk/bayesconsensus/online | OnlineConsensus, the streaming incremental fit with optional exponential forgetting. | | @takk/bayesconsensus/adapter | The framework-agnostic tool descriptor and JSON-safe handler, name bayes_consensus. | | @takk/bayesconsensus/audit | Append-only SHA-256 hash-chained audit log of consensus verdicts, via Web Crypto. | | @takk/bayesconsensus/node | JSON and CSV loaders for vote files. | | @takk/bayesconsensus/edge | The node-free core, re-exported for edge runtimes and the browser. |

---

A tool for a non-human entity

The adapter exposes BayesConsensus as a framework-agnostic tool a non-human entity can call to resolve disagreement across the ensemble it operates. The descriptor shape, a name, a description, a JSON Schema, and a handler, matches what MCP servers and tool-calling APIs expect, and input arriving from a model is parsed defensively. Only votes (item, agent, label) are accepted, never any other agent state.

import { bayesConsensusTool } from "@takk/bayesconsensus/adapter";

// Register bayesConsensusTool with your MCP server or tool-calling API; its name is "bayes_consensus".
const result = bayesConsensusTool.handler({
  votes: [
    { item: "claim-1", agent: "model-a", label: "true" },
    { item: "claim-1",

…

## Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [davccavalcante](https://github.com/davccavalcante)
- **Source:** [davccavalcante/bayesconsensus](https://github.com/davccavalcante/bayesconsensus)
- **License:** Apache-2.0
- **Homepage:** https://davccavalcante.github.io/bayesconsensus/

Install and usage instructions live in the source repository linked above.