gramajo/nouns-proposal-predictor

Nouns DAO Proposal Predictor (DistilBERT)

Fine-tuned DistilBERT that reads a Nouns DAO proposal's title + description and predicts whether it passed. Trained on gramajo/nouns-proposals (982 proposals, pass/fail labels).

⚠️ Read this before using the model

Do not use this model to predict real proposal outcomes. It is published as a negative result and as a reproducible artifact, not as a working predictor. See the numbers below and decide for yourself.

Results

Accuracy is meaningless without a baseline. The bar is majority-class accuracy — what you'd score by ignoring the proposal entirely and always guessing the more common outcome. A model only tells you something if it clears that bar.

run	n train	n test	test pass rate	majority baseline	accuracy	lift	AUC	beats baseline?
A_stratified_random	785	197	50.2%	50.2%	58.9%	+8.6%	0.648	yes
B_temporal_breakeven	785	197	28.4%	71.6%	73.1%	+1.5%	0.651	yes
C_post_breakeven_only	147	50	28.0%	72.0%	72.0%	+0.0%	0.365	NO

What each run asks:

• A — stratified random split over all 982 proposals. Regime held constant. Does the text carry signal at all?
• B — temporal holdout at the BreakEven break. Train on props 1–785, test on 786–982. Does a model trained before the "BreakEven" voting bloc transfer to after it?
• C — post-break only, stratified inside the new regime. Within the new regime, is text predictive? (n is small — read with caution.)

The BreakEven regime change

Around proposal ~786, a bloc of large Nouns holders ("BreakEven") began voting down spend proposals to bring outflows in line with revenue. The pass rate shifts sharply across that boundary:

• Proposals 1–785: 55.9% pass
• Proposals 786–982: 28.4% pass

This is a structural break in the decision rule, and the new rule is largely not a function of the proposal text — it's a function of who is voting and what the treasury looks like. Run B is therefore a test of transfer across concept drift, not a test of whether text is informative. Run A is the cleaner test of the latter.

Training details

• Base model: distilbert-base-uncased, max_length 512, batch size 16, lr 3e-5, up to 6–8 epochs
• Early stopping on eval_loss (not F1 — F1 rewards a degenerate always-PASS predictor)
• No class weighting. An earlier version used balanced class weights derived from the pre-break training distribution; on the post-break test set this pushed the model further toward predicting PASS, making the drift look worse than it was.
• Seed: 42. Exact split indices are published in splits.json in this repo.

Reproduce it

Everything needed to check these numbers is public:

• Dataset: https://huggingface.co/datasets/gramajo/nouns-proposals
• Exact split IDs: splits.json in this repo
• Raw metrics: results.json in this repo
• Training notebook: nouns_predictor_experiments.ipynb in this repo

Limitations

• Text-only. No proposer identity, no proposer track record, no requested amount, no treasury balance at submission, no voting history — i.e. none of the variables most likely to actually determine a Nouns vote.
• 982 examples is small for a transformer. Run C in particular is trained on ~150 examples.
• Labels are binary pass/fail; abstentions, cancellations, and vetoes are not modeled here.
• Single seed. Results have not been averaged over multiple seeds and should be treated as indicative, not precise.