10

140 characters. How much memory would it take up ?

I'm trying to calculate how many tweets my EC2 Large instance Mongo DB can hold.

6 Answers 6

13

Twitter uses UTF-8 encoded messages.

UTF-8 code points can be up to six four octets long, making the maximum message size 140 x 4 = 560 8-bit bytes.

This is, of course, just for the raw messages, excluding storage overhead, indexing and other storage-related padding.

e: Twitter successfully let me post the message:

™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™

Yes, that's 140 trademark symbols, which are three octets each in UTF-8

4
  • 3
    UTF-8 encodes code points, not glyphs. Each code point can be up to 4 octets in UTF-8, UTF-16, and UTF-32. CESU-8 encoding can use 6 octets for a single code point, but that only comes from naive encoding of UTF-16 into UTF-8, and is not actually valid UTF-8.
    – Gabe
    May 14, 2011 at 5:20
  • Terminology and math updated, thank you for the clarification.
    – Charles
    May 14, 2011 at 5:23
  • 1
    Worth pointing out that every tweet isn't 140 characters, and that as Twitter's user base becomes more international (or doesn't), bigger code points will be used more (or less) frequently.
    – mlissner
    Oct 28, 2015 at 16:31
  • While that's true, the question asks about the size in the context of determining storage requirements, so thinking about the maximum size instead of the average size is more correct.
    – Charles
    Oct 28, 2015 at 17:00
3

Back in September, an engineer at Twitter gave a presentation that suggested it's about 200 bytes per tweet.

Of course you still have to account for overhead for your own metadata and the database itself, but 200 bytes/record is probably a good place to start.

3
0

Typically it's two bytes per character if you're storing Unicode as UTF-8, so that would mean 280 bytes max per tweet.

1
  • UTF-8 characters can be more than two octets in size. For example the trademark glyph is three octets long. Wikipedia suggests the maximum possible size is six (!!!) octets.
    – Charles
    May 14, 2011 at 4:37
0

Probably 284 bytes in memory ( 4 byte length prefix + length*2). Inside the DB I cannot say but probably 280 if the DB is UTF-8, you could add some bytes of overhead, for metadata etc.

0

Potentially of interest:
http://mehack.com/map-of-a-twitter-status-object
Anatomy of a Twitter Status Object

Also more about twitter character encoding:
http://dev.twitter.com/pages/counting_characters

1
0

It's technically stored as UTF-8, and in reality, the slide deck from a tweeter guy here http://www.slideshare.net/raffikrikorian/twitter-by-the-numbers gives the real stat about it:

140 characters, ~200 bytes

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.