Blog

Today is even more random than usual, as any possibility of applying my brain to a new creative topic is toast. It looks like I'm finally circling down to being able to close the investigation that I've been working on the last two months. And I set up my new laptop last night and have yet to go through and do a complete functionality verification regarding programs and data-transfer, so I don't dare do anything on either the old or new machine that involves changing files yet. But fortunately I can do some more nattering on the question of what sorts of useful meaning can be squeezed from Amazon and Goodreads book ranking data.

I've had a delightful amount of engagement on this topic, and somewhat unusually it's come from a lot of different intersections of my online life. People are popping over from the LesFic mailing lists to comment, and Tuesday's blog got a mention on File 770, which resulted in pointers to some other similar data-crunching that ties in to the topic.

[Note: these two posts were originally posted on LiveJournal then migrated to Dreamwidth. Comments on the May 3, 2016 post can be found in DW here, and those on the DW version of this post can be found here.]

Correlation of Rating Volume and Sales

A commenter on File 770 pointed me to a fascinating analysis by SFF author Mark Lawrence that looks at the correlation between the number (but not magnitude) of Goodreads ratings and book sales. He used his own sales data plus data provided from colleagues writing in similar genres who were willing to provide numbers. (Note that book contracts sometimes explicitly prohibit authors from discussing income or sales numbers in public.) Lawrence's conclusions are stated as:

[T]here is a pretty close relationship between the number of Goodreads ratings a book has and its total sales. PROVIDING that the books are of a similar age and the same genre.

(A rough squint at his graph suggests that the correlation for his specific dataset works out to around $7-8 per Goodreads rating.)

Correlation of Goodreads Rating Volume and Average Rating

(This originally appeared in slightly different form as a comment I made at File 770.)

Having poked around a bit on Goodreads, I’ve pulled a data set from a list entitled “Best Historical Fantasy” (the genre chosen to be thematically comparable to one of the sets I pulled from Amazon) and I took the top 100 titles from that list. Goodreads lists (in contrast to the sales-based Amazon Top 100 lists) are created by reader/reviewers and books are added and up-voted by other reader/reviewers. Internal ranking in the list is dependent on some combination of the average rating and the number of different people recommending the book for the list.

Overall, the results confirmed my analysis from Amazon with a couple of interesting details.

The first interesting detail is that the Goodreads correlation (at least for this dataset) continues roughly linear throughout the ranking scale, rather than losing linearity toward the bottom of the scale. My knee-jerk hypothesis is that this relates to the ability to leave a ranking on Goodreads without leaving a review. This may suppress the number of lower rankings on Amazon because someone who doesn't care for a book may be less motivated to say so if they have to do it in a text review rather than a one-click star ranking.

The second interesting detail is something that most people who pay attention already know anecdotally, which is that Goodreads rankings have a wider spread. This may be simply a consequence of the "suppression of lower rankings" effect on Amazon noted above. (To be clear: I'm not saying that Amazon actively suppresses lower rankings, only that the structure of the interface may have that as an effect.)

The lowest ranking in the Amazon Historical Fantasy Top 100 was 4.0, while Goodreads rankings (though obviously not for the same exact books) went down to 3.1 In addition to the abovementioned suppression effect, the suggested interpretations for Goodreads ratings are also shifted down from Amazon ratings. If you go by the suggested interpretations, an Amazon 4* is a Goodreads 3*. Amazon rating distributions are more likely to descend from a maximum at 5-stars, while Goodreads distributions--even for popular books--are more likely to have a maximum at 4-starts, indicating an overall shift of the curve. More on this below.

The third interesting observation is more particular. As I was entering the rating/# of reviews data, I started noticing a particular name standing out as an outlier. Not an outlier in terms of matching the average-rating/no.-of-ratings trend, but a set of books–all by the same author–who fell high in the list (based on the number of people voting them for the list) while having surprisingly few ratings. So although I wasn’t tracking author or title in general, I flagged every time that author came up in the list: a total of 11 times (out of 100 books). When the books were sorted in descending order of average rating, that author occupied 7 of the top 10 slots (average ratings of 4.7 or more) and completely followed the overall correlation of having relatively few reviews. (In a dataset where a handful of titles had rating entries in the 6 figures, all but one of those 7 had been rated fewer than 100 times.) Having taken a look at the author’s bibliography on wikipedia (and I’m not going to name names because that’s not the point), the best hypothesis seems to be that this is someone with a small but extremely dedicated readership. That dedication extended to ensuring the authors position on the list by voting for the books there, but reflects the general conclusion that the average rating must be understood in the context of the overall number of ratings.

(A commenter on File 770 suggested an alternate interpretation than "small group of enthusiastic fans", noting that some Goodreads authors engage in list-vote trading to move each others' books higher in lists. I have no speculations on this particular author's situation.)

Correlation Between Average Amazon Rating and Average Goodreads Rating for Popular Books

While it's an easy anecdotal observation that average Amazon ratings tend to be higher than average Goodreads ratings, I wanted to confirm this impression with data. So I started with Amazon's Top 100 Historical Fantasy books, looking only at Kindle sales this time (for consistency), and stripping out omnibus editions (to avoid redundancy). I used Amazon data as the starting point because I figured sales figures were harder to game than Goodreads lists.

Because I was going to have to search on each book individually in Goodreads, I only took the top 40 from the Amazon list, which trimmed down to 33 when omnibus editions and not-yet-released titles were excluded. I then calculated the Amazon:Goodreads ratio, as well as the absolute difference. I played around with these results in several ways, plotting them as a curve distribution and running a mean and standard deviation.

At a very rough approximation, the absolute difference in reviews seems to follow a standard curve: average = 0.4 stars, standard deviation = 0.2 stars. In fact, roughly 90% of the data fell within one standard deviation of the mean.

The ratio of reviews isn't quite as pretty a distribution, but has an average ratio of 1.09 with a standard deviation of 0.06. Here only 80% of the data fell within one StDev.

I leave it to a better statistician than I to say something meaningful. I expect that differences in average rating may be much more variable for books with smaller distributions, due to the larger effects of individual choice (both in what rating to give and in which rating site to participate in).

Only one book in the set had a higher average ranking in Goodreads than in Amazon. It's a pre-release listing and has relatively few reviews, which may account for its position as an outlier. Its position on the Amazon Top 100 list seems to be due entirely to the fact that it's a 47 North publication (i.e., an Amazon imprint) and "sales" are probably artificially inflated by internal promotion activities, such as making it available to "book club" arrangements (based on comments in the reviews). Some of the text reviews on Amazon are...um...harsh.

Some Random Overall Conclusions

Mark Lawrence’s sales correlation is very interesting. And it doesn’t necessarily mean that the average rating number by itself isn’t meaningful. I suspect that comparing the combined average rating + number of ratings data to the overall trend line for the relevant genre will tell you more than the absolute average rating alone, in terms of whether the reading community likes or doesn’t like the book. But all of that may be irrelevant to sales.

Just for fun, what does all this say about my books?

Daughter of Mystery has an average Amazon rating 0.5 stars higher than for Goodreads. Plotting the cumulative average Amazon Rating against total ratings over time, I'd say the average rating more or less settled in to a relatively stable number around the 15-20 review mark, but is still highly subject to single inputs. The first 5 reviews were all 5-star.

The Mystic Marriage has an average Amazon rating 0.7 stars higher than for Goodreads. The average Amazon ranking doesn't seem to have stabilized yet and the first 6 reviews were all 5-star. This book is still at the stage where the ratings are artificially high due to low numbers of readers. (Excuse me while I go sob in a corner for a while.)

Both books fall pretty solidly on the normal trendline for Lesbian Romance, but don't have enough data points to compare meaningfully to the Top 100 Historical Fantasy data. As for sales data...let's just say, "Inadequate data for meaningful analysis." (The next related phenomenon I examine may be changes in rating patterns for series books. My knee-jerk hypothesis is that later books in a series will tend to have increasingly higher average ratings because readers will typically continue with the series only if they liked it.)

It's very important to keep in mind that these comparisons and correlations are only meaningful when looking at sets of books with similar potential distribution. A lesbian romance with 200 Amazon ratings isn't comparable to a blockbuster best-seller with 200 Amazon ratings. The latter is only starting to scratch the surface of its most dedicated readership while the former may well have already saturated the market.

In conclusion, much as it pains me to admit it, I do myself no favors in begging my readers and fans to leave reviews if they weren't already inclined to do so. Strongarming existing readers into leaving ratings/reviews does not necessarily generate new readers. It certainly doesn't directly generate additional income. And to the extent that looking at the average rating + # of ratings provides useful data, artificially inflating one's average rating by solicitation to existing fans isn't meaningful.

Instead, the only useful thing to do is beg people to encourage other people to read my books. And the rating stats will fall out of that on their own.

Usually I like to focus this blog on the creative part of the writing process, but I'm in an unusual pause at the moment so I thought I'd talk about the analytic end. I know the common wisdom in mainstream publishing is that an author should pay no mind to reviews and ratings. At most, we should do comic readings of our one-star reviews to show how little we care. (Only cry in private behind locked doors.) So this essay isn't really for anyone whose book came out from a major publisher. But I have this weird bi-cultural existence, suspended between what I consider my "home" writing community (mainstream SFF) and the community in which I was published (small-press/self-published lesbian fiction), so I get a lot of opportunities to compare and contrast. This essay is for people who don't have mainstream publication and for people who may be bewildered by some of those cultural differences.

There is major anxiety within the LesFic community (quite possibly within all marginal publishing communities) around the crowd-sourced rating-and-review sites like Amazon and Goodreads. A big reason for this anxiety is that it's what they (we) have: they don't get a big publicity blitz, they don't get bookstore placement, they don't get advance reviews in all the highly-respected sites. What they do get is an aggregate of individual reader opinions when those readers are motivated enough to post them.

In my experience, this anxiety is expressed in two major ways. There is a strong community pressure that readers--that is, readers "within the community"--should only ever review and rate books that they absolutely love, and therefore that they will rate highly. This philosophical position is expressed explicitly by many LGBT review sites, and in social media forums for LGBT book communities. This anxiety walks hand in hand with the second: the tendency to react to less-than-perfect reviews as a personal attack. Given a supposed "community standard" of only reviewing/rating books you love, there is an interpretation that to rate a book badly (where "badly" is anything less than a four-star review…or sometimes less than a five-star one) must have been done out of personal malice against the author. That either the reviewer is deliberately giving a false opinion (because, of course, the book must be objectively excellent!), or that, even if they genuinely didn't care for it, the act of publicly expressing that opinion could only come from personal malice.

Viewed from within the community (and it is very much an expression of the assumption that the reader/writer/publisher nexus is a community whose purpose is to support each other against the world) this can look a lot more reasonable than it does from outside the community--where it tends to look fairly toxic.

But beyond the damage to the usefulness of ratings/reviews when only glowing opinions are authorized, there is a damage to authors' perceptions of their own work. Express skepticism of the usefulness of all-five-star ratings and some authors will loudly proclaim that their book is so great that of course it earned all those five-star ratings.

No. I believe that almost every book can earn some genuine and sincere five-star ratings. But no book is universally beloved. Let me repeat that with emphasis: NO BOOK IS UNIVERSALLY BELOVED.

Because I wanted to throw some data at this essay, I took a look at Amazon for the top 100 sellers in Historical Fantasy and the top 100 sellers in Lesbian Romance. You know who has spent a very long time in the top 10 books sold in Historical Fantasy? Diana Gabaldon's Outlander. Do you know how many one-star reviews Outlander has on Amazon? 749. Seven hundred and forty-fucking-nine one-star reviews (4% of the total). No book is universally beloved.

Do you know what unfavorable reviews and ratings mean? They mean that your book is engaging readers who are outside your narrow inner-core target audience. Not just that it's reaching them, but it's engaging them sufficiently to express their opinion in public. And up to a certain point (I'll talk about that point later) the more reviews you get, the lower your average rating is. Because the more people you engage, the more likely you are to engage people who may have liked your book but didn't absolutely love it. Sure, that hurts. It would be nice to be universally beloved. NO BOOK IS UNIVERSALLY BELOVED.

Ah, but I'm a data person, so I ask myself, is it possible to quantify to what extent a less-than-perfect average rating reflects getting your book in the hands of people outside your core target audience? Let's see.

I went through the top 100 Amazon sellers in the Historical Fantasy category and recorded the number of reviews and the average star rating. Then I calculated the average number of reviews at each rating. I mention Diana Gabaldon above because I ended up pulling her out as her own little category for the analysis. And then I plotted those pairs of data.

I had to put Diana Gabaldon on a separate y-axis that differed by nearly an order of magnitude from the rest of the data. But here's the take-away: from around an average rating of around 4.3 on up, a lower rating correlates with a larger engagement (expressed as overall number of reviews). This holds true for the overall average of that top 100 and it holds true in the specific case of Diana Gabaldon. The books that had only five-star reviews? (And keep in mind, these are ones that are currently in the top 100 sellers of the category.) None of them had more than four (4) reviews. Your first, most engaged reviewers are quite naturally going to be people from your core target audience. But an extremely high average is a sign that you haven't expanded far beyond that (yet).

Now let's take a look at the same sort of data for the Amazon category of lesbian romance. (In this case, I looked specifically at Kindle sales because for small press books the dynamics of e-book versus paper are peculiar.) I also cut the data off at a rating of 4.0, not only to compare better with the historical fantasy data (for which that was the lowest average rating) but because the values below that represented only one or two books each and so are less reliable for trending purposes.

And what do we find? Pretty much the same thing. In this case, a rating of 4.4 and up correlates very closely with the average number of reviews at that rating. No book that had an average five-star rating had more than five (5) reviews. Interestingly, if you look at the plot for the maximum number of reviews at each average-rating point, you get the same effect: a very strong correlation between number of reviews and a lower average rating.

Now, of course, at some point this effect breaks down. At some point a lower average rating does start to reflect people's opinion of the specific book, even in the aggregate. And that seems to start somewhere in the lower 4's depending on the data set. And these trends are looking at aggregate behavior. It doesn't mean that there's no difference at all between a 4.4 rating and a 5.0 rating. What it means is that the meaningful difference between a book that has a 5.0 rating with X number of reviews, and a book in the same marketing category with a 4.4 rating and 50X number of reviews is not necessarily one of quality. What it means is that the second book is reaching outside its core audience. And is engaging them.

As someone who has been working very hard to reach outside what my publisher believes to be my core target audience, this is what I keep reminding myself when I get a "meh" rating or review. It means that I've succeeded.

Yeah, ok, lots of spoilers in this review because I WANT TO WARN EVERYONE NOT TO SEE THIS FUCKING MOVIE!!!!! This is the platonic ideal of the Tragic Lesbian Boarding School Story.

* * *

This is a series of reviews of lesbian-themed movies originally inspired by a request for recommendations of "good movies involving lesbian romances that don't end up with the protagonists deeply unhappy, dead, or both." To this set of criteria I’ve added the question, “Is the story primarily about coming out?” This set of index questions can involve some spoilers, but I will usually only hide them for new releases.

Many of these movies are not currently in print. I'll link each to their imdb.com entry for reference. But for those currently available, Wolfe Video [https://www.wolfevideo.com] is the go-to distributor for lgbt movies.

* * *

Lost and Delerious (2001) is basically Dead Poet’s Society with girls. Except with the Bury Your Gays trope more explicitly gay. There's also a strong Psycho Lesbian trope, in that a thwarted lesbian relationship drives one character to increasingly bizarre and violent behavior and suicide. Hey, I told you there would be massive spoilers. Don't blame me if you're still reading.

This movie belongs to the genre of hot-house boarding school stories, in which same-sex relationships bloom and are cut off well before their prime. Mary, the new girl at an upper-crust all-girls boarding school ends up rooming with two girls, Tori and Paulie, who are involved in a hot-and-heavy relationship. All three have problematic relationships with mothers: Paulie’s birth mother gave her up for adoption and she is currently trying to track her down and contact her. (When she eventually succeeds in locating her, the woman refuses to allow contact.) Mary’s mother died three years ago and she feels she’s being sent to school to make room for her stepmother. Tori’s mother is trying to make her over into her own image as a socialite. A running subplot involves two of the school's teachers who are widely rumored to be lovers.

While Mary figures out she’s ok with pretending not to notice the sex going on in the next bed over, the balance is upset when Tori’s sister barges in one morning when the lovers are still naked in bed together. Tori freaks out about the potential for being outed and throws Paulie under the bus, claiming she was the sexual agressor and that she (Tori) is perfectly straight. To support this, Tori takes on a program of public heterosexuality, sneaking out to date and have sex with a random boy, selected due to a chance meeting. When Mary chooses to be supportive of Paulie, she takes the risk of being labelled a lesbian herself.

There’s a subplot where Paulie finds an injured Harris Hawk and secretly rehabilitates it in the woods. (I will now forego discussing the logistics of bird of prey rehabilitation as the event is clearly meant to be Deeply Symbolic and practicality need not intervene.) Paulie is the hawk, a fierce wounded creature. She makes bold symbolic gestures, including a chivalric declaration of love in the library while wearing her fencing gear and carrying an epee.

But both the girls are terrified to name their sexuality. Relevant quote, “I’m not a lesbian! I’m just Paulie in love with Tori and Tori’s in love with me.” In a late night encounter, Tori confesses she’ll never love anyone but Paulie but that they can never be together.

Paulie has always played the role of Bad Girl, which initially masks her acting out of her emotional crisis. As in Dead Poet’s Society, poetry and Shakespeare and drama are the medium through which strong emotions are expressed within this shrine of classical learning. This framing drives Paulie to challenge Tori’s boyfriend to a literal duel on the night of the big school formal (at which all the parents are present) and to cut in when Tori is dancing with her father, threatening a confrontation where she declares her love. Tori, terrified, rejects her. Mary is having her own issues, as her father fails to show up for the dance and Paulie taunts her into confessing that she hates her father, using Lady Macbeth’s “Unsex me” speech, and then recruits Mary as her second for the duel.

They meet the boys in the woods with swords, and the duel ensues, but Paulie’s using an unblunted sword and actually stabs her rival in the leg. The scene cuts to Mary running across the field where all the students and teachers are gathered in a picnic to find Tori, and we see the hawk flying up, called to Paulie where she stands on the rooftop of the school. We see Paulie begin to fall, then see the hawk flying away, and we see all the girls staring up at the roof in horror. But Mary, our viewpoint protagonist, is ok, because now we get a voiceover about the lesson she learned from the hawk and how now she’ll always remember her dead mother’s face.

This version of Lesbian Tragedy (the plot that Emma Donoghue classifies as “Rivals”) always marks out the butch character for death while allowing the femme character to recant and be redeemed. At the beginning of the movie, I don’t recall there being an obvious butch/femme distinction between the Paulie and Tori. But as the emotional crisis progresses, Paulie’s presentation becomes more and more masculinized, culminating in her wearing a suit at the school dance, envisioning herself as Tori’s knight, and more explicitly with the “Unsex me” speech. Tori drags herself by force into a normative female role by her pursuit of a heterosexual sexual experience. So rather than their gender perfomances locking them into the fates of their respective roles, once those fates were set in motion, the gendered roles claimed and assimilated them.

Given the context of the inspiration for this review series, this is definitely a Do Not Recommend. We hit all the bullet-points: Tori recanted, Paulie came out and died, everyone is unhappy. This is the sort of movie that could convince an entire generation of young lesbians that they are doomed. The fact that movies like this are still being made in the 21st century is a crime.

I've been on something of a book-buying and library spree lately for the Lesbian Historic Motif Project--not that I didn't already have plenty of material on my own shelves to last for another couple of years already! But it's always nice to have choice, and after covering a whole string of entire books, I wanted to fit in some shorter articles. By the time I finished with Castle's book last week, I'd actually expected to have the new front-end for the LHMP (and my blog as a whole) up and running, but the combination of an intensive project at work (for the last five weeks) and the push to get Mother of Souls out to the beta-readers meant that I've fallen down on my end of the necessary work. (My web masters are being very patient.) So I'll keep teasing you with the knowledge that Something New is coming Real Soon Now.