teaching ml to hate marvel movies

It turns out that I’ve rated 1,325 films and series on filmaffinity. That’s years of free labels just sitting there, so why not use them?

I scraped my own ratings, glued every public score I could find onto them, trained a model, and hooked it up to a bot that tells me what to watch every monday. 🍿

scraping like a gentleman

Filmaffinity has no api, so it’s html or nothing. I’m only taking my own data, but that still doesn’t give me the right to hammer a free site, so the scraper behaves itself:

• it waits 2 to 4 random seconds between requests, with exponential backoff on top.
• it honors Retry-After whenever the server sends one.
• a cloudflare 403 means stop. I didn’t write a retry loop or bolt on a proxy pool¹.
• every page gets cached to disk once, one file per url, so re-running the analysis never touches the network.

Soulmates (the users whose ratings correlate most with mine, a filmaffinity feature) need my session cookie. The scraper reads it from the environment, only ever sends it to filmaffinity, and never logs it.

Then I went shopping for more opinions. wikidata maps filmaffinity ids to imdb, metacritic and rotten tomatoes ids, and I queried it in batches of 200. For imdb ratings I used the official datasets.

what 1,325 ratings say about me

Well, turns out I’m harsh. My mean is 5.71 with a standard deviation of 1.83, while the filmaffinity crowd averages 6.63 on the same titles with a spread of 0.93. So I use the whole scale and the crowd basically lives between 6 and 7.

OK, so who actually agrees with me? Here’s the Spearman correlation of each source with my rating:

source	ρ
my soulmates	0.66
filmaffinity average	0.59
imdb	0.46
metacritic users	0.41
rt audience	0.37
metascore	0.35
tomatometer	0.34

A few dozen strangers who rate like me beat every critic aggregator out there, and the professional critics finish dead last. If you ever needed a one-table pitch for collaborative filtering, there you go.

And it gets worse for critics. Critic darlings with an imdb score under 7 get a 5.0 average from me (n=16), while crowd-pleasers that critics panned get 6.3 (n=24). When I disagree with the crowd, I still end up closer to the crowd than to the critics.

The rest is roughly what you’d expect from someone who watches way too many films. Documentaries average 6.76 and action 4.79. Lanthimos is my guy: 8 films, mean 8.5, 1.8 points above filmaffinity. Von Trier gets 7.9 across 12. Tim Burton is the clearest negative. And aug is cursed, sitting 1.19 points under filmaffinity across 241 titles².

the model: stack all the things

The target is my rating, and the model gets 177 features: filmaffinity average and votes, soulmates, imdb, metacritic, genres, topics, country, year, runtime, and the people. Directors, writers, cast and crew go through out-of-fold target encoding, so a director’s score only ever comes from films the model isn’t being tested on³.

On top of that sits a mlxtend StackingCVRegressor. Four base models (ridge, histogram gradient boosting, an rbf svr and ExtraTrees) feed a non-negative ridge meta-regressor. Once you normalize the weights, svr and ExtraTrees get about a third each, hgb 22% and ridge 13%.

Here’s the 5-fold cv:

model	mae	r²
stacking	0.959	0.568
blend (previous model)	0.973	0.558
gradient boosting	0.986	0.539
ridge	0.997	0.541
soulmates average, calibrated	1.103	0.443
filmaffinity average, calibrated	1.162	0.378
my mean	1.520

That means 61% of predictions land within one point of what I actually gave. I also get a P(≥7), Platt-scaled, with a Brier score of 0.148 versus 0.239 if you just always guess the base rate. The range it shows for each film comes from the 10th and 90th percentiles of the residuals, which go from -1.63 to +1.41.

is the stack actually better?

Plenty of ml blog posts skip this part, so I ran mlxtend’s 5x2cv paired t-test.

Against the soulmates baseline the stack wins with t=6.66, p=0.001, and against the filmaffinity line with t=10.21, p=0.0002. Both are real.

Against the old blend, though, it’s t=2.21, p=0.08. Not significant. The stack is a bit better and I can’t prove it, so why keep it? The bias-variance decomposition shows where the gain comes from: bias² is about the same for everything (around 1.55), but variance drops from 0.23 for hgb to 0.10 for the stack. It’s less jumpy from fold to fold, and that’s the entire improvement. I’m happy shipping a model for that reason as long as I’m upfront about it.

what it actually looks at

This is grouped permutation importance, i.e. how much mae gets worse when you shuffle a whole block of features:

Soulmates (+0.186), filmaffinity average (+0.154) and imdb (+0.080) do the heavy lifting. Topics and genres add a bit, and everything else is under 0.01.

Forward feature selection over blocks tells the same story. You start with imdb alone (mae 1.126), then add soulmates (1.069), topics (1.038), cast, genres, filmaffinity average, writer and year. The best subset uses 13 blocks and hits 0.994. Runtime, format, cinematographer and composer make it worse. Nobody picks a film for the composer lol.

Were the external scores worth the scraping? Barely: mae went from 0.985 to 0.972 averaged over three seeds. I still store rotten tomatoes but the model doesn’t use it, because it added nothing the others didn’t already have.

superheroes are a solved problem

I also ran association rules (apriori, mlxtend again) to find combos of attributes that predict “liked” (≥7, base rate 40%) or “disliked” (≤5, base rate 38%).

The disliked side is brutal. Topic “Superheroes”: 33 films, 33 disliked 💀 Confidence 1.00, lift 2.61. Action + USA + comic sits at 0.95, and action with imdb under 6.5 and filmaffinity under 6 is 39 out of 39.

The liked side is softer and more fun. Nordic origin: 42 films, 74% liked, lift 1.86. Tv series + crime: 30 films, 73%. 2010s miniseries: 71%. Apparently I’m a sad Scandinavian detective trapped in a Spaniard’s body, kek⁴.

the modays bot

How often would I remember to run a model by hand? Roughly never, so it went into grok bot.

There’s a teammate agent called filmaffiniter that knows the project and how to run every step of it. On top of that, a weekly routine fires every mon and:

1. refreshes my ratings and soulmates (incrementally, from cache),
2. retrains if there are enough new ratings,
3. scores the films in my filmaffinity lists that I haven’t rated yet,
4. checks what’s playing at my local cinema this week,
5. checks for new episodes of the open series I like, via tvmaze,
6. and sends me a short report in the chat.

Here’s an example of the weekly pipeline:

And yes, you can be as specific as you want:

fun facts & bottom line

• about 57% of my taste is explainable from public data. The other 43% is mood, who I watched it with, and The Room⁵.
• people who rate like you are a better signal than any critic aggregator, so if I were building a recommender from scratch, that’s where I’d start.
• stacking got me lower variance and a p-value of 0.08. A modest win, and I’m fine calling it modest.
• if you scrape, scrape your own stuff, go slow, and stop when the site tells you to.
• I’m terrible at remembering to run things, which is why this one runs on a weekly schedule.

Ask me on modays whether the bot got it right. 🤞

1. there’s also a 45-second wall-clock deadline per request, enforced with SIGALRM, because a socket that hangs forever means a scraper that hangs forever. ↩

2. my theory is that aug is when distributors dump everything they don’t believe in. My other theory is that it’s 40 degrees outside and I’m grumpy. ↩

3. without that, the model “learns” that Lanthimos is great by peeking at my rating of the very Lanthimos film it’s being tested on. The cv scores look amazing right up until you predict something new. ↩

4. turns out “grey sky, dead body, divorced cop” is a genre and I’m its target audience. ↩

5. the model predicted 2.3 for The Room. I gave it a 7. It’s a masterpiece of a different kind, and no feature will ever capture that, kek. ↩