Measuring Homestuck (Data! Charts! Graphs!)

[NotEgbert]

---

Measuring Homestuck
Word Count, Update Frequency, and more

Redditor? See the r/Homestuck page for this post here.

This report is completely spoiler-free as long as you don't follow any links to HS pages.

---

Plenty of older threads attempt to speculate and/or estimate the "size" of Homestuck, or provide comparisons to popular benchmarks (Bible, LoTR, etc), or present metrics based on hand-collected data (update frequency, etc). This thread is an attempt to address the "how big is Homestuck" question (and others) with automated data ripped from the adventure, using established and experimental methods for equivalence and comparison to works in other media. Additionally, the exploration of Homestuck data for interesting trends and statistics is encouraged, because everyone loves graphs. EVERYONE.

I plan to update the underlying data source used in this post (and linked below) at the end of each act or sub-act. Currently, the collected data spans from the beginning of Homestuck to the end of A6I2, on this page. At the start of A6I3 or A7(?), I plan to officially update the data source in this post, and all dependent metrics/figures. Other users are free (and encouraged) to download, edit, expand, alter, redistribute, and use the linked Excel file(s) for their own exploration of Homestuck's data, and interesting finds or alterations posted in this thread will be added to the title post.

Without further ado:

---

Word, Page, and Media Counts

Number of words in Homestuck's text: 418,290

This metric includes:

• Page Title Text (e.g. "John: Quickly retrieve arms from drawer." = 6)
• Inner Page Text (e.g. "Your ARMS are in your MAGIC CHEST, pooplord!" = 8)
• Pester/Sprite/Dialog Names & Text (e.g. "TG: ok i can accept that" = 6)

It DOES NOT include:

• Text in Images & Flash (e.g. "ZOOSMELL POOPLORD" on this page)
• Next-Page Commands (which are repeated in following Title Text)
• Act, Intermission, and Button Labels (like "Show Pesterlog" and "[I]")

The "==>" and similar commands count as one word, in titles and page text.

Number of Adventure pages in Homestuck: 4,816

This metric includes:

• Regular Adventure Pages (e.g. page 1901)
• Flash/Image Only Pages (e.g. page 6009)
• Selection/Branching Pages (e.g. page 7037)

It DOES NOT include:

• Sweet Bro and Hella Jeff Comics
• Linked Pages from Page Text (e.g. this image from this page)

Number of images in Homestuck: 5,512
Number of image frames in Homestuck: 21,163

This metric includes:

• Single Adventure Page Images (e.g. page 1901 has 1 image)
• Extra Images Per-Page (e.g. page 1907 has 2 images)

It DOES NOT include:

• Sweet Bro and Hella Jeff Comics
• Linked Images from Page Text (e.g. this image from this page)
• "Scraps" (e.g. this image on this page)
• Header Images in the Scratch ("[o]") section

Frames are extracted automatically from each GIF file and counted regardless of duration.

Number of Flash Animations in Homestuck: 135

This metric includes all Flash files on Adventure pages, regardless of the [S] tag.

The following figure is a look at word count on a per-page basis, binned logarithmically.

This histogram shows us the largest bin of word count is the 33-64 range, with 854 pages. A very-close second is the 0-1 range, with 853. Pages with this few words are almost-always "minipages" like this one, with only a single-word title (usually "==>") and no body text. You can also see there are two pages in the "Extreme Outlier" bin of 4097+ words, which (as you might have expected) are the "recaps," this page and this one.

The following figure shows how word count grows over the course of the Adventure.

Small vertical jumps can be observed at 3574, 3888, and 5138, where the recaps are located in the story. This progression suggests a new recap is due soon! The trend overall is very-slightly concave up, implying a mild increase in average per-page word count over the course of the story so far.

Comparisons using these numbers to common literature standards follow in the next section.

---

Size by Comparison

This section uses data from a number of internet sources, including this page and this one, to compare Homestuck's length to that of famous novels and other common literary standard lengths. I cannot be responsible for the accuracy of data from these outside sources, but expect they are correct to within 10%. Anyone with better sources or data should reply to the thread, and I'll update this section.

This first figure is a comparison on raw word count alone.

Now, while impressive in its own right, this comparison fails to account for the variety of media and storytelling elements in Homestuck as compared to a traditional novel. We can see that, counting only the story words (using the metric explained in the last section), Homestuck is about "Two Crime and Punishments" or "Half a Bible." It is furthermore roughly-comprable in size to The Lord of The Rings trilogy, and David Foster Wallace's novel Infinite Jest. The scale of the story, by word count alone, is pretty amazing! However, a more applicable comparison should take into account the various other storytelling methods in Homestuck, including images and Flash animations; for this sort of comparison, we have to delve into the largely-subjective field of adjusted word count or word equivalence. The ultimate goal of this is to attempt to estimate, as closely as possible, the number of words it would take to convey Homestuck using text only.

While there is by no means a common standard for word equivalence of images, and absolutely no precedent for word-equivalence of Flash animations, this is a best effort based on available information and common-sense considerations, with conservative estimates employed whenever possible for what should be a a "safe-minimum" result. Word equivalence is fairly common in scientific and scholarly journals, which may individually have rules regarding image size and density as applicable to article length restrictions. This presentation by the American Journal of Neuroradiology goes over their particular rules on word-equivalence, which are excerpted in the following list:

• add 100 words to your total word count for every two brain scans or like-sized images.
• Each graph should be counted as 100 words.
• consider image submissions not outlined in this presentation as 100 words.
• arrays of images add (4 images = 400 words)

Of course, a radiographic brain scan presents a great deal more "equivalent information" than, say, this story image. The great variation in Homestuck's image complexity suggests that a flat counting metric may not be the best option, especially when animated GIFs are involved. After a great deal of experimentation, comparison, and screwing around, I've come up with the following metric for word-count equivalence of Homestuck's images:

N_words=9+0.6*S_img
...where S_img represents the size of each image file, in kiloBytes.

This puts the flashy-cracked-window image above at 16 words (~one 'good' sentence), while more complicated, animated images like this one are more lengthy (this one is ~118). As a good measure for evaluation, think about the number of words it would take to describe every detail in an image which is remotely pertinent to the story: for the first image, the fact that the window has four panes, in cracked, and is flashing with an otherwise blank "screen." Again, we're trying for a conservative estimate. While not every image in Homestuck is well-represented with this metric, the parameters have been selected so as to produce a good average (or final-sum) result. Equivalence on each individual image is not the intent.

For Flash animations, without any information or precedent, I've adopted a similar system:

N_words=20+0.1*S_Flash
...where S_Flash represents the size of each image file, in kiloBytes.

This makes the first Strife Page, a relatively-simple animation/game, worth about 70 words. By comparison, Cascade is worth a whopping 5,020. By my totally-qualitative estimate, this metric undercounts the small Flash files (which are many) and overcounts the huge ones (which are very few), which balances to somewhere-near-reasonable-on-average. A better metric for this might include the duration of each animation rather than the byte-size, but I have yet to find a machine-readable way to count the duration of each animation, and wouldn't even know where to begin on the game-like ones such as Myststuck.

In addition to the raw word count listed above, these two equivalence metrics yield an "adjusted word count" or "complexity count" for Homestuck, which should, as intended, represent "the number of words to convey the Homestuck story using only text."

Adjusted Word Count for Homestuck: 728,063

We can retroactively re-apply this number and adjusted counting scheme to our results from before, including the previous section.

This figure is a comparison using the Adjusted Word Count for Homestuck.

Using this comparative scheme, we can say Homestuck is "a little under one Bible" or "over one and a half LoTRs." Again, the parameters and methods used for word-count equivalence are very subjective and estimated, but are all intended to be conservative estimates and measures, based at least in-part on common standards, where available.

---

Adjusted Size Metrics

Continuing to update our prior graphs, the following adds the complexity measure, along with raw data size, to the accumulating word count line-graph from the first section.

This figure shows how story complexity grows page by page in Homestuck.

We're counting both Words and kiloBytes on the same Y-axis here, which might invite some poor inferences; the green line shouldn't be used in comparison to the others. They just fit so well on the same axe, I couldn't resist. Also, see that huge spike in story size around ~6000? Yeah, that's Cascade. The interesting spikes here-and-there made me want to look at how complexity (read: adjusted word count) is distributed across pages, so I made the following scatterplot:

This figure shows complexity for each page in Homestuck.

The big spikes (labeled with page number) are Recap 1, Recap 2, Recap 3, and.... Cascade. The blue line across the bottom is a simple Linear fit to the data, which shows a net-upward trend in page complexity. Most pages are "down in the grass" with a complexity around 10-100.

Finally, I was interested in the breakdown of Homestuck's size between media types. First off:

File Size of Homestuck: 622.56 MB

Includes text words, story images, and Flash Animations. Does not include page decorations or other HTML outside the comic frame.

This figure shows the breakdown in file size between Homestuck's storytelling mediums.

This chart speaks for itself; a tiny majority of Homestuck's file size is due to text, as compared to images and Flash. This is not unexpected by any means. Still, 2.5MB of text alone is nothing to be ashamed of!

This concludes the sections on content and comparison. Any ideas regarding other interesting methods of analysis or statistics should be posted as replies in this thread (or PMs to me!), and good ones will be added to this post and the shared Excel file.

---

Update Frequency in Homestuck

The next two plots look at how often, and when, Homestuck is updated.

This figure shows how often per-day Homestuck has been updated, since starting.

Moving averages as calculated center-weighted. The Yearly line (violet) makes a good metric for update frequency after averaging-out spikiness; Homestuck's period of most-frequent-updates is thus around Fall 2010. Notice also the large drop across all series around Fall 2011, when the comic went "on break."

We can also bin the updates-per-day data:

This figure shows the frequency of daily updates in Homestuck.

Seen here, most days (since Homestuck started on 4/13/2009) there are zero updates to the comic. On days that the comic is updated, there are most-commonly 4 updates per day. The largest number of updates in one day occurred on 7/8/2011, with 41 updates.

---

This concludes my "Measuring Homestuck" report. My Excel file with page data, charts, and calculations is available here, or attached; Homestuck fans are encouraged to edit, alter, append, expand, and use this data and these figures in their own work, and asked to repost any interesting finds to this thread in order to make them public and keep centralized. I only ask that, if possible, my forum username continue to be attached to all derivative works in the "Summary" tab, as it is currently. I would also like to add a very special thanks to Andrew Hussie for writing what might be "the greatest story ever told," which has kept me entertained for 3+ years running.

In Summary:
Number of words in Homestuck's text: 418,290
Number of Adventure pages in Homestuck: 4,816
Number of images in Homestuck: 5,512
Number of image frames in Homestuck: 21,163
Number of Flash Animations in Homestuck: 135
File Size of Homestuck: 622.56 MB
Adjusted Word Count for Homestuck (see above): 728,063

(last Updated Friday, 29 June 2012)
Please post or PM with questions/suggestions!

[Wesley Foxx]

Approved as fuck. Silly spam filter.

This is very cool and comprehensive (I'm actually still reading through it)

[BobbyBritish]

Wow this must have taken a long time but it's pretty interesting and I read it all. Here's hoping Homestuck can beat Harry Potter.

[SkaianRedeemer]

Hmm. If you got the word count for exploration-style flashes, how would that impact their adjusted word count? Data size of the text removed from calculation but word count added directly?

Excellent work on this all around.

[temporalPeregrine]

Wait, the largest number of updates was 66, a few days ago. Doesn't matter, this is really fucking amazing.

[Wesley Foxx]

As he said, this is only up through the end of A2I2

[temporalPeregrine]

I should read the introduction before reading the actual study.

[nsinned]

This is very comprehensive and well assembled (and interesting). I skimmed the text quickly, had a couple of questions, went back, and discovered that if I'd just read it carefully in the first place, the questions would already have been answered; the answers were right where it made sense for them to be.

Nice job!

[NotEgbert]

Thanks to everyone for the great reviews so far!

I plan on updating the title post with a "FAQ" once I get a few more questions or requests via this thread or PM. That'll probably happen sometime tonight or tomorrow, depending on activity.

I might also, if I can leverage the existing data-collection program, make a similar report for Problem Sleuth, which should be easier to classify due to the absence of Flash anyway. I'd like to know if there's any interest first.

[NotEgbert]

Hmm. If you got the word count for exploration-style flashes, how would that impact their adjusted word count? Data size of the text removed from calculation but word count added directly?

Excellent work on this all around.

I'm planning on eventually making a new metric for equivalence of flash and images, with one part of it being the potential to use OCR to grab text from the visuals and count the words directly. Alternately, this could be human-tasked, but I like the automation option. A compromise might be to "poll" users on what they think each flash file should be worth (using a select few examples), then use a machine learning or correlation model to predict the word counts of other animations based on metrics like size, duration, song duration, closeness to other Flashes, etc.

In this case, I'd probably add the "actual text words" of each Flash to the word count of the Page it's on, then generally loosen or reduce the equivalent words for the Flash to compensate.

Of course, anyone is free to download the excel file and start adding human-read stats (like words in Flash files, or duration) as new columns, then changing the calculation columns to use these new measures!

[The Ghost of Starscream]

This is absolutely fantastic. Excellent work.

[DvorakElt]

What are the other, smaller spikes around 4400- 4800?

[adimal]

Wasn't 6/26/2012 the largest update day with 66 pages?

[LTY]

This... is... FUCKING... AMAZING. I love statistics and stuff. ^^
Do you have assembled the data on your own? (If yes, do you put it online? Nevermind, just tell me where I can get them...) Found it! ^^
There are so many sweet things one could do... *shiver*...

EDIT: Maybe you could do the Page Complexity with an exponential scale to avoid this "down in the grass"-effect.

[tinaun]

You should do a "date modified" scan to find out what hours are the updates usually made.

And a listing of the most common words would be great.

[LTY]

You should do a "date modified" scan to find out what hours are the updates usually made.

And a listing of the most common words would be great.

Holy Moly, don't tell anyone, but those would be the things I would have requested as well. But I can say one thing, Homestuck ussually updates when I am asleep, and that drives me crazy... ^^

Other suggestions:
Frequency of character appearances.
Length of Pesterlogs. (maybe apportioned to the characters)
A cross between the Update frequency and the Update size (e.g. offsetting the Cascade hiatus against the size of Cascade, like average work per day for an update, where there are days prior without an update *bluh* difficult to describe)
Frequency of fish/hors/catpuns (could be a bit complicated)
Frequency of motherfuck
Frequency of glubbing ^^
Comparison of Jesus/Jegus and God/Gog frequency
Number of Deaths
Number of Ressurections
Number of explosions
Amount of Grist used - Number of things alchemized
Tigers catches (should be easy)
Number of smooches

Sweet Jegus... I'm home, I adore this thread by now, you don't even know...

[NotEgbert]

EDIT: Maybe you could do the Page Complexity with an exponential scale to avoid this "down in the grass"-effect.

Yes probably, though it would more likely be on a logarithmic or logistic function of size/other factors.

You should do a "date modified" scan to find out what hours are the updates usually made.

Unfortunately I don't think the former is possible, since the comic pages are php (really all served as the same file, with server-read parameters s= and p=) and therefore don't have any filesystem-attached "date modified" attribute. It's possible Hussie has this info in a column of his comic database, but it's not something I can grab from the result page sent by the server.

And a listing of the most common words would be great.

With the data I have right now (see the excel file), you could do this with page titles only, because I didn't include the full page text in the saved data for each page (too big/not useful in what I was attempting to measure initially). I will probably have the "most common words in page titles" done soon, and the more intensive one perhaps after a later update.

Frequency of character appearances.

Can probably do this to a limited extent, by looking for names followed by ":" in titles and chat handles followed by : in pester/sprite/dialoglogs. Not sure how accurate it would be due to writing style, etc, and of course it wouldn't count the "appearance" of each character in images and flash, only their names in page text.

Length of Pesterlogs.

Totally doable, and probably not-too-difficult to tag them with the characters involved. Might be difficult to organize/show though, I can't think of a good method offhand right now.

A cross between the Update frequency and the Update size (e.g. offsetting the Cascade hiatus against the size of Cascade, like average work per day for an update, where there are days prior without an update *bluh* difficult to describe)

Oh man that's a great idea. Will attempt this very soon.

Frequency of fish/hors/catpuns (could be a bit complicated)

Doable if I had a list of each pun-term used, which would probably require finding them all manually anyway...

Frequency of motherfuck
Frequency of glubbing ^^
Comparison of Jesus/Jegus and God/Gog frequency

Doable with page text, which again I don't currently include but probably will in a future dataset iteration.

Number of Deaths
Number of Ressurections
Number of explosions
Amount of Grist used - Number of things alchemized

Probably not doable with this data/method unless every one has a very particular word pattern associated with it, like "X has died," "X exploded." Also needs page text. This is a good candidate for human counting.

Tigers catches (should be easy)

You mean Got Tigers?
Not sure, probably falls int he same category as the last one.

Number of smooches

Same as the last two but might be slightly easier due to common language.

[NotEgbert]

Wasn't 6/26/2012 the largest update day with 66 pages?

As mentioned in the intro (and repeated in Wesley Foxx's replay above yours), this data only goes through the end of A6I2.

[NotEgbert]

What are the other, smaller spikes around 4400- 4800?

Here's a version of that plot with more labels on the other peaks:


In case this was missed, each number is a page suffix for Homestuck. For a given page number you can find the associated adventure page by adding it to the URL "http://www.mspaintadventures.com/?s=6&p=00"

So, the first new peak is at http://www.mspaintadventures.com/?s=6&p=004467 , etc.

[LTY]

Yes probably, though it would more likely be on a logarithmic or logistic function of size/other factors.

Dammit... I always mix them up. I meant a logarithmic scale.

You mean Got Tigers?
Not sure, probably falls int he same category as the last one.

That's what I was talking about... Sorry, my English sure has some blackouts sometimes. ^^

What about the ratio between Characters introduced/Characters dead. Wouldn't it be funny if it looked like this?

PS: I downloaded the Excel and I was like O.O That's amazing, did you count the words on your own? What is meant with "Estimated Page Use" and "Short Page" on the PageDate Sheet

[NotEgbert]

Dammit... I always mix them up. I meant a logarithmic scale.



That's what I was talking about... Sorry, my English sure has some blackouts sometimes. ^^

What about the ratio between Characters introduced/Characters dead. Wouldn't it be funny if it looked like this?

PS: I downloaded the Excel and I was like O.O That's amazing, did you count the words on your own? What is meant with "Estimated Page Use" and "Short Page" on the PageDate Sheet

Hah, no, I had a program count them for me with RegEx. It actually collected all of the data on that table autonomously, taking about an hour and a half.

"Estimated page use" was a metric I was attempting to derive which basically measured the number of pages it would take in a comic-style book (Like the current PS and HS books) to cover all of HS. So, based on the current books, you could fit ~4 adventure pages with one (static) image each onto one "book page," whereas images with animations could take up more like 1/2 or 3/4 of a page... etc. Short page was just a way to count the number of pages with 1-2 word titles and no text, like "==>", but the wordcount histogram proved a better way of doing that.

I never really finished it, so it didn't get included in the title post. Probably will eventually though.

[Kíeros]

This is... just... indescribable. Good job on compiling all of those statistics; I cannot wait to use some of them for some... purposes that I have.

[LTY]

With the data I have right now (see the excel file), you could do this with page titles only, because I didn't include the full page text in the saved data for each page (too big/not useful in what I was attempting to measure initially). I will probably have the "most common words in page titles" done soon, and the more intensive one perhaps after a later update.

You rang?...

[orngjce223]

What about the ratio between Characters introduced/Characters dead.

Isor has collected some statistics like that.

[NotEgbert]

You rang?...

O.O

This is great stuff. I'm going to add a link in the title post ASAP.

Did you manually enter the lists of verbs and characters, or scrub them from the pattern "CHAR: VERB..." ?
I'd love to see whatever calculation spreadsheets you're using. Maybe we should start a shared dropbox or Google doc for collaboration.