Cinemath: Good Movies Do Better at the Box Office; Also Helps to Be Expensive, PG-13, and a Sequel

by     Posted 1 year, 306 days ago

transformers-dark-moon-rotten-tomatoes

It is common to argue that there is a divide between critics and audiences. Critics prefer arthouse dramas, preferably in black-and-white; audiences like things that go boom. Thanks to countless explosions and few positive reviews, the Transformers sequels became a lightning rod for this argument. Compare the 35s of Transformers: Dark of the Moon: 35% on Rotten Tomatoes, $350 million at the domestic box office. Later in the year, a critical darling like Warrior (83% on RT) managed just over $13 million. Is this the general pattern, or are these exceptions to the rule?

For the latest entry of Cinemath, I crunched the numbers to answer this very question. Thankfully the answer is that good movies statistically do better at the box office. After the jump, I try to quantify this effect with an equation that predicts box office using the Rotten Tomatoes score, budget, theater count, plus whether the movie is a sequel and/or rated PG-13.

Background

I gathered data on the 2011 box office for this year-end report, so it was easy enough to do a preliminary check of whether reviews were relevant in 2011. This scatterplot graphs the domestic box office gross against the Rotten Tomatoes score for all the wide releases in 2011.

box office 2011 rotten tomatoes

It looks like a blob of dots until you graph the trend line, which has a noticeably positive slope of about 0.68. This is too simplistic, not considering other factors that contribute to box office, but gives you a good idea of where we are headed.

To incorporate other factors, I developed a linear regression model. The result is an equation with the variable we want to predict (box office gross) on one side, and the input variables (e.g. RT score) on the other side. With regression analysis, you should test out many different factors to identify which are statistically significant to identify which yield the best model. I ultimately landed on these five:

rotten tomatoes box office statistics

  • RT – The Rotten Tomatoes critics score on a scale from 0-100. While not a perfect measure, it is a nice quantitative representation of the critical reception to a movie. I also tested using the IMDB rating or just the “Top Critics” RT score as the critical reception variable. The “All Critics” RT score is a better predictor than both the IMDB rating the “Top Critics” score, so I used the former.

  • Budget – The listed production budget in millions of dollars. I am always skeptical about publicly available budgets. Studios have no reason to be forthcoming about what they actually spend on a movie, and so aren’t. These numbers also don’t include the marketing budget for a film, which should have a direct impact on the box office. But the public numbers do give you a sense of scale. For instance, Transformers 3 ($195) cost a lot more to make and market than Warrior ($25 million). Until studios are willing to let me look at their books (please?), this will have to do.

  • Theaters – The theater count at widest release. This is simple economics. The wider the release, the greater the accessibility to the audience, both in terms of geography and available showtimes.

  • harry potter transformers box office statisticsSequel – A binary variable that indicates whether or not the movie is a sequel. Sequels are typically the continuation of a previously successful brand, and so have an intuitive advantage at the box office.

  • PG13 – A binary variable that indicates whether or not the movie is rated PG-13. This is the optimal rating for demographic variety. There is no age restriction on who can see the movie, but the rating also signals the intention to attract older audiences.

I evaluated other factors, like the month of release, the studio in charge, and whether it was animated. Ultimately, the above five factors lead to the best model that is also relatively explainable.

Results

The regression analysis was conducted in Minitab using the 2011 wide releases. There were 143 movies that screened in at least 600 theaters; this analysis is limited to the movies for which I could find a budget. The goal was to find an equation that would predict total domestic box office gross. I rounded the calculations for cleaner numbers, which resulted in this equation:

Gross = –80 + 0.6×RT + 0.5×Budget + 0.025×Theaters + 50×Sequel + 20×PG13

social network movie box office equationThe R2 value is used to measure how good a regression model is. (In statistics, this is bluntly referred to as goodness of fit.) The scale of R2ranges from 0 to 1, where 0 means a bad fit that doesn’t explain any of the variation in the data and 1 means a good fit that perfectly explains all of the variation in the data. For the 2011 data, the R2 for this model is 0.65. There is no strict rule about what makes a good R2. But given the limitation on the data I was able to collect (all publicly available sources) and the complicated process of making and releasing and marketing a movie, I did not expect to achieve an R2 greater than 0.5 with just five variables. So I consider this a good fit.

I tested this model on the 2010 wide releases for validation. Even though the 2010 data was not incorporated into the model formulation, the R2is 0.61 when applied to the 2010 data. This suggests that the model is not unique to 2011, and may be valid for predicting box office grosses going forward.

(Disclaimer: for this next part of the analysis to be valid, you need to satisfy certain assumptions that this model does not satisfy. I explain in more detail in the appendix, which is meant more for stats nerds than movie fans. These numbers are intended more as a guideline than an exact numerical calculation, and should be taken with a grain of salt.)

mission impossible 4 box office statisticsThe coefficient in front of each variable represents how much you would expect the gross to increase with a one-unit increase in the predictor variable. For example, the 0.6 in front of RT suggests that every percentage point of the RT score corresponds to about $0.6 million in domestic box office gross. Here is the full breakdown, illustrated by how it works for Mission Impossible – Ghost Protocol on the right:

  • An extra 10% on RT is worth about $6 million at the box office.

  • An increase of $1 million in budget is worth about $0.5 million at the domestic box office. At first, that looks like a loss. But keep in mind that this does not include international box office, DVD sales, or ancillary revenue.

  • An increase of 1,000 theaters is worth about $25 million at the box office.

  • Sequels are predicted to earn about $50 million more at the box office.

  • A PG-13 movie is predicted to earn about $25 million more at the box office.

These calculations aren’t exact. For instance, the equation predicts Dylan Dog would make –$24.5 million, when it actually grossed $1.2 million (basically the numerically possible equivalent). And the model isn’t really equipped to handle the megahits, totally underestimating the likes of Harry Potter and the Deathly Hallows, Transformers 3, Twilight: Breaking Dawn, and The Hangover 2. That may mean I am missing an important factor or need to tweak the model. On the other hand, those could just be outliers that can’t be predicted using basic data analysis. As a counterexample, the model would have been ready for the surprise success of Rise of the Planet of the Apes, predicting $178.6 million for the PG-13, critic-approved sequel that ultimately grossed $176.8 million.

Conclusion

rise-of-the-planet-of-the-apes-movie-image-03I do need to stress again that the exact numbers here are not gospel, but give you a general idea of what matters at the box office. I went into this with the research question, “Do moviegoers care if a movie is good?” The answer appears to be yes, with a high correlation between positive reviews and financial success. There is some causation here, where a critic or a friend recommends the movie, and as a direct result someone goes to see that movie. There is also just correlation—critics tend to enjoy the same movies that the general public likes, despite Transform-ed anecdotal evidence to the contrary. Regardless, the bottom line is encouraging: studios have financial incentive to put an effort into making good movies. Ideally, the product will be both good and marketable.  And sure, budget, rating, brand awareness, release timing are all important, too.  But “good” has value on its own.  For movie lovers, that is a beautiful thing.

All data comes from the The Numbers, Box Office Mojo, and Rotten Tomatoes.

If you are a stats nerd, head to the appendix on page 2 for the detailed output. Otherwise, meet me in the comments for anything you want to discuss.

Page 2: Appendix




Please Like Collider on Facebook

Pages: 1 2

Comments:

FB Comments

  • Lizard King

    Brendan, I always look forward to your Cinemath articles. Possibly one of my favorite things to read on the entire internet. Informative, well explained, interesting, and unique. Each article has been fantastic. Keep up the awesome work!

    • http://twitter.com/#!/colliderbrendan Brendan Bettinger

      Thanks! Wish I could do them more frequently.

  • Anarvin

    Great read as usually Brendan! : )

  • Tim

    In my opinion, Rotten Tomatoes is the worst thing to happen to cinema. People now see a movie based on what others say instead of forming their own opinion. Sad. For example: I hated Iron Man, but it got a fresh rating.

    • D. McHugh

      I don’t know. I used to take RT much more seriously when it was just the Top Critics of major media outlets. Now, it seems any Joe Blow with a computer in his bedroom can sign up and review movies. These opinions from so many random people from non-exiistant or self-created websites can easily drag down the overall rating of a great film while elevating a mediocre one. While I never see a film solely based on any one person’s opinion, I still consider what my go-to Big 3 say: Roger Ebert (Chicago Times), Peter Travers (Rolling Stone) and Richard Roeper.

      • Clay

        My top 3 are Ebert, Michael Phillips and James Berardinelli.

      • Alex

        While it does seem that the availability is there for just “anyone” to be able to submit to RT, there are certain guidelines that demand a certain level of quality of writing, consistency and time being a reviewer/number of reviews written. It is not as easy as it seems.

    • drod

      I know what you mean, while I loved Iron Man I tend to disagree with critics for example Public Enemies is my favorite movie and it only received a 69% on RT.

      • Donovan McLean

        Which is fresh.

    • Max

      Not relevant at all to what you’re saying, but most people love Iron Man

  • David

    Awesome work. As someone who worked in mathematics at the college level, I’m impressed. And I just had to test the formula myself. The Social Network grossed around $96 million, and the formula gives around $91 mil. I always find myself laughing at box office forecasts – one “widely trusted” website had That’s My Boy opening for twenty million this weekend. I knew that was a joke when I first saw it. It would be an interesting follow-up to discover a more accurate opening weekend predictor.

  • paul tracy

    ironically, collider pushes the hype muscle machine trash. but, this is at least a positive step in the right direction.

    :: next on collider, perfumes inspired by the dark knight rises ::

    • Konrad-ko-man-der

      It’s called Sex Panther by Odeon. It’s illegal in nine countries… Yep, it’s made with bits of real panther, so you know it’s good.

      • paul tracy

        holy shit that made me laugh.

        nicely done.

        ~ pp

  • Ricky Johnston

    Is there anyway you can share the data set. I would love to give this as a homework assignment to my class

    • http://twitter.com/#!/colliderbrendan Brendan Bettinger

      Sure. Will you get in touch with me: colliderbrendan at gmail dot com.

  • t

    I didn’t understand a word of this article but it spoke to me.

    • Ray

      OMG. Me Too. I’m Like Huh but I do know the Man speaks the Truth.

      All Honestly. I wanna see a movie. I’ll will see no matter what critics says. I respect their opinions though.

  • enzo

    If all critics gave a movie 51% it would have a 100% value on RT. Metacritic is a better service using the actual average score with prominent critics carrying more weight.

    I would have liked to have seen Brenden’s numbers using metacritic, but still, the premise that critics evaluations actually count for something is reassuring.

    Other variables: Genre inclination, Star/Actor and Director prominence. Critics always cut slack for a Scorsese or de Niro flick. And documentaries always seem to score higher while comedies score lower.

    • http://twitter.com/#!/colliderbrendan Brendan Bettinger

      Metacritic introduces their own bias in trying to assign a number to every review, especially since each critic uses a different scale. Then they weight the average according to perceived importance. Plus, they have smaller sample sizes—the Piranha 3DD is based on just 13 critics.

      RT only needs to determine whether a review is positive or negative (admittedly with some error). And they have larger sample sizes (compare the 43 reviews for Piranha 3DD). I don’t trust Metacritic, and prefer RT for statistical analysis.

      • Bonobo

        Rotten Tomatoes also has an “Avarage Rating” for each movie, where they do basically the same thing as metacritic, but from their larger sample size. I think it would be interesting to see how that rating fits into your model, perhaps alongside the rotten/fresh rating.

        The RT Average Rating is a number I personally find more telling than the tomatometer, and both of them together give a much clearer indication of the critical response. Sometimes a 90%+ on RT can just mean that everyone agrees that a particular movie is mildly entertaining.

        Take Ben Affleck’s The Town for instance. A solid but unspectacular thriller which has a puzzling 94% on the Tomatometer, but a much more reasonable 7,8 average rating. Compare that to Eternal Sunshine of the Spotless Mind, an original and fascinating film, which has a 93% on the Tomatometer, but a much stronger 8.4 average rating.

        I think people often have undue expectations of films with high RT scores, as they equate such scores with high praise, not universal (but perhaps unenthusiastic) praise. This causes them to loose faith in the validity of Rotten Tomatoes (I know a few such people myself), so I really hope they will play up the Avarage Rating a bit.

  • Armand

    If anyone has been paying attention to the highest grossing movies of each year, they’d notice that it’s been a great one for the last 5 years! Starting with 08′ The Dark Knight, 09′ Avatar, 10′ Toy Story, 11′ Harry Potter 7-2, 12′ – As great as the Dark Knight Rises could be, the Avengers is destined to be the top grossing movie of the year, still a great movie.

    • Arnold

      U forgot about THE HOBBIT.

      • Armand

        The Hobbit is going to be a great movie, but it wont beat the Avengers at the Box Office, seeing as its part one, those who feel cheated by the story being split two (who are complete morons) may just skip it in the theaters, rent it, and just go see part 2, just my opinion though, anything could happen

  • Lance

    A thoughtful bit of analysis, and the results are definitely interesting. Thanks for posting this, Brendan!

  • John M.

    Love the math. Trendlines are truth!

  • Anarvin

    Am I just bad at math, or the formula doesnt apply for almost any of this years releases?

  • J

    This is a really cool idea and I do think there is a correlation to RT% and actual gross but I dont see any way it can work with a formula. For example, Dark Shadows comes out to 132 million and its actual domestic gross was 74 million. That could have been because of Burtons recent films disappointing and the amount of competition (two factors that cant be broken down to a number). Hunger Games is not a sequel but it should be treated like one because of its large fanbase. Regardless, the formula gives 133 million and the domestic gross just crossed 400 million. There are movies like Paranormal Activity, where the budget (or lack thereof) has no bearing on the gross. Unless 3D charges are added in, it cant calculate any movie that charges an extra $3. Marketing is generally not released to the public but it can make a movie a megahit (Avengers) or even more of a flop (Green Lantern). The number of theaters is a worldwide number yet the formula is only calculating domestic. I love RT and trust it to determine if I should spend $10 or $1, but there are far too many other factors to definitively say there is a connection. This is an excellent and ambitious article so i am impressed, but i think it can be improved (but not perfected).

    • http://twitter.com/#!/colliderbrendan Brendan Bettinger

      I tried to address most of your concerns in the text, and there is a better model in the appendix. But yes, you can always find examples that succeed or fail on factors I can’t account for.

      Number of theaters is domestic. I would have preferred to look at the worldwide gross, but did not because I only had information on domestic theater counts.

  • Pingback: La formula matematica per calcolare in anticipo il successo di un film « News «

  • Jed D

    The studios have internally been using linear regression for years to “predict” box office (as well as ancillary markets). There are several problems in predicting box office. One is causality (the number of theaters doesn’t cause higher box office, but it’s correlated because, in order to get higher box office, you have to have a large number of theaters). Another is that the RT score isn’t available when you are trying to decide how much to spend to market the film, let alone when you greenlight the picture.

    What jumps out at me, eyeballing the scatter plots, is that if you limit the analysis to the “hits” (the movies that pay the bills) — say films with box office of $150 million or greater, the trend line looks like it would be either flat or is possibly negatively correlated with the RT score. And the R2 measure looks like it would be will well under .5.

  • Andy

    Assume the studios fib about the budgets by around 25%. For many years now, that’s what they figure they can get away with without being called out-and-out liars. When the budget is under $15 million, they will, however tell the truth because that’s when they get concessions from the unions.

  • sofasobad

    Brendan, why did you only look at movies that played in more than 600 theatres? Don’t some of the best-reviewed prestige arthouse films play in far fewer theatres than that? Doesn’t setting a high limit on the theatre count skew the analysis?

  • Pingback: 21 Jump Street | I Hate Your Favourite Movie

  • Pingback: 2012: Year Of The Female Action Hero? | USA Press

  • Pingback: 2012: Year Of The Female Action Hero? - US Mag - The Breaking News | US Mag - The Breaking News

  • Pingback: 2012: Year Of The Female Action Hero? | LatestMusicHeadlines.com

  • Pingback: 2012: Year Of The Female Action Hero? | Simply Boundless Entertainment

  • Pingback: Good Movies Actually Do Better at the Box Office | Gear Diary

  • Pingback: Good Movies Actually Do Better At The Box Office

  • Pingback: Cinemath: 2013 Box Office Stats. A Year-End Report on the 2013 Box Office | Collider

Copyright ©2005 - 2014. All Rights Reserved. Built by topLingo

Click Here