I've heard people talking about "base 64 encoding" here and there. What is it used for?
-
9From the manual for base64_encode(): "This encoding is designed to make binary data survive transport through transport layers that are not 8-bit clean, such as mail bodies."– still_dreaming_1Commented Feb 28, 2019 at 16:36
-
1@still_dreaming_1 What does 8-bit clean look like?– IgorGanapolskyCommented Feb 12 at 15:32
-
1@IgorGanapolsky I'm not an expert at understanding the different possible "layers" traffic might go through, but as far as I understand it in this context, 8 bit clean would mean the data will not be reinterpreted/corrupted. If some "layer" along the way does interpret certain characters as control codes, this can prevent the data from being received as intended. But I wouldn't resort to base 64 encoding unless you can confirm this is required or you can confirm the data sometimes gets corrupted otherwise and using base 64 encoding prevents this. Sometimes it does help though.– still_dreaming_1Commented Feb 13 at 13:47
19 Answers
When you have some binary data that you want to ship across a network, you generally don't do it by just streaming the bits and bytes over the wire in a raw format. Why? because some media are made for streaming text. You never know -- some protocols may interpret your binary data as control characters (like a modem), or your binary data could be screwed up because the underlying protocol might think that you've entered a special character combination (like how FTP translates line endings).
So to get around this, people encode the binary data into characters. Base64 is one of these types of encodings.
Why 64?
Because you can generally rely on the same 64 characters being present in many character sets, and you can be reasonably confident that your data's going to end up on the other side of the wire uncorrupted.
-
131(In theory you could do base-80 encoding or something similar, but it would be significantly harder. Powers of two are natural bases for binary.) Commented Oct 14, 2008 at 15:08
-
16@yokees: There is no guarantee, they're just characters that are almost always safe. This is why there are multiple forms of Base-64 (en.wikipedia.org/wiki/Base-64).– user565869Commented Jan 11, 2013 at 21:28
-
11Does that mean that all network type data passing should use some kind of encoding? Commented Aug 9, 2016 at 3:27
-
8But why is base64 method used to encode string data? eg in javascript atob function Is there meaning the server to encode a json file to base64 format? Special characters could be a use case but why not utf8 in that case, are they equibalent? Any further resource regarding that would be greatly appreciated thank you. Commented Sep 30, 2016 at 15:33
-
8@TomRussell - I'm not sure where you are getting the "subset" idea from. The term "base-64" does refer to the number system. In ordinary decimal (base-10), we have 10 distinct symbols. In hexadecimal (base-16), we have 16 distinct symbols. In binary, we have 2 distinct symbols. Well, in base-64, we have 64 distinct symbols. So, it really is just the number system, in exactly the same way as those other number systems. Now, WHICH 64 symbols to use is a completely different matter, and in some cases, you need to use a different set of 64 symbols than in other cases.– John YCommented Aug 3, 2017 at 20:57
It's basically a way of encoding arbitrary binary data in ASCII text. It takes 4 characters per 3 bytes of data, plus potentially a bit of padding at the end.
Essentially each 6 bits of the input is encoded in a 64-character alphabet. The "standard" alphabet uses A-Z, a-z, 0-9 and + and /, with = as a padding character. There are URL-safe variants.
Wikipedia is a reasonably good source of more information.
-
5@CholthiPaulTtiopic: The results of encryption or compression, or sound/images/video. Commented Sep 5, 2016 at 14:07
-
1@CholthiPaulTtiopic: I'm afraid I have no idea what you mean by "what about storage" but at this point I think we're somewhat off-topic. Commented Sep 6, 2016 at 5:39
-
4@CholthiPaulTtiopic: I'd strongly avoid thinking in terms of "string binary". Binary data should be treated as binary data, and not treated as text. I've seen literally hundreds - possibly thousands - of questions on SO which basically boil down to people not taking enough care over this distinction. Commented Sep 6, 2016 at 7:29
-
3@still_dreaming_1 PHP calls them
binary strings
. (source)php.net/manual/en/function.pack.php Commented Feb 28, 2019 at 6:07 -
1
Years ago, when mailing functionality was introduced, so that was utterly text based, as the time passed, need for attachments like image and media (audio,video etc) came into existence. When these attachments are sent over internet (which is basically in the form of binary data), the probability of binary data getting corrupt is high in its raw form. So, to tackle this problem BASE64 came along.
The problem with binary data is that it contains null characters which in some languages like C,C++ represent end of character string so sending binary data in raw form containing NULL bytes will stop a file from being fully read and lead in a corrupt data.
For Example :
In C and C++, this "null" character shows the end of a string. So "HELLO" is stored like this:
H E L L O
72 69 76 76 79 00
The 00 says "stop here".
Now let’s dive into how BASE64 encoding works.
Point to be noted : Length of the string should be in multiple of 3.
Example 1 :
String to be encoded : “ace”, Length=3
- Convert each character to decimal.
a= 97, c= 99, e= 101
- Change each decimal to 8-bit binary representation.
97= 01100001, 99= 01100011, 101= 01100101
Combined : 01100001 01100011 01100101
- Separate in a group of 6-bit.
011000 010110 001101 100101
- Calculate binary to decimal
011000= 24, 010110= 22, 001101= 13, 100101= 37
- Covert decimal characters to base64 using base64 chart.
24= Y, 22= W, 13= N, 37= l
“ace” => “YWNl”
Example 2 :
String to be encoded : “abcd” Length=4, it's not multiple of 3. So to make string length multiple of 3 , we must add 2 bit padding to make length= 6. Padding bit is represented by “=” sign.
Point to be noted : One padding bit equals two zeroes 00 so two padding bit equals four zeroes 0000.
So lets start the process :–
- Convert each character to decimal.
a= 97, b= 98, c= 99, d= 100
- Change each decimal to 8-bit binary representation.
97= 01100001, 98= 01100010, 99= 01100011, 100= 01100100
- Separate in a group of 6-bit.
011000, 010110, 001001, 100011, 011001, 00
so the last 6-bit is not complete so we insert two padding bit which equals four zeroes “0000”.
011000, 010110, 001001, 100011, 011001, 000000 ==
Now, it is equal. Two equals sign at the end show that 4 zeroes were added (helps in decoding).
- Calculate binary to decimal.
011000= 24, 010110= 22, 001001= 9, 100011= 35, 011001= 25, 000000=0 ==
- Covert decimal characters to base64 using base64 chart.
24= Y, 22= W, 9= j, 35= j, 25= Z, 0= A ==
“abcd” => “YWJjZA==”
Base-64 encoding is a way of taking binary data and turning it into text so that it's more easily transmitted in things like e-mail and HTML form data.
It's a textual encoding of binary data where the resultant text has nothing but letters, numbers and the symbols "+", "/" and "=". It's a convenient way to store/transmit binary data over media that is specifically used for textual data.
But why Base-64? The two alternatives for converting binary data into text that immediately spring to mind are:
- Decimal: store the decimal value of each byte as three numbers: 045 112 101 037 etc. where each byte is represented by 3 bytes. The data bloats three-fold.
- Hexadecimal: store the bytes as hex pairs: AC 47 0D 1A etc. where each byte is represented by 2 bytes. The data bloats two-fold.
Base-64 maps 3 bytes (8 x 3 = 24 bits) in 4 characters that span 6-bits (6 x 4 = 24 bits). The result looks something like "TWFuIGlzIGRpc3Rpb...". Therefore the bloating is only a mere 4/3 = 1.3333333 times the original.
-
14Do I understand correctly, that 64 is the best choice as it is the highest power of two that is convertible to a printable ASCII character (there is 95 of them)?– vohoCommented Jan 18, 2017 at 9:21
-
If in both cases they're 24 bits, then isn't the bloating 1:1? Or When you say 4 characters that span 6 bits, do you mean that there's actually 8 bits per char but the first two are padded 0s? Commented Feb 27, 2019 at 5:43
-
1@Backwards_Dave Each 6 bits are expressed in 8 bits. So the bloating is 8:6, or 4:3. Commented Feb 27, 2019 at 19:24
-
@AtesGoral am I right in my assumption, that when you use Base256, you could map it 1:1 ? because 1 byte= 8bit = 256 possible characters?– MarcLCommented Jan 8, 2021 at 11:32
-
@user2774480 As a thought experiment, yes. But there's probably no practicality in using Base256. Commented Jan 13, 2021 at 20:06
Aside from what's already been said, two very common uses that have not been listed are
Hashes:
Hashes are one-way functions that transform a block of bytes into another block of bytes of a fixed size such as 128bit or 256bit (SHA/MD5). Converting the resulting bytes into Base64 makes it much easier to display the hash especially when you are comparing a checksum for integrity. Hashes are so often seen in Base64 that many people mistake Base64 itself as a hash.
Cryptography:
Since an encryption key does not have to be text but raw bytes it is sometimes necessary to store it in a file or database, which Base64 comes in handy for. Same with the resulting encrypted bytes.
Note that although Base64 is often used in cryptography is not a security mechanism. Anyone can convert the Base64 string back to its original bytes, so it should not be used as a means for protecting data, only as a format to display or store raw bytes more easily.
Certificates
x509 certificates in PEM format are base 64 encoded. http://how2ssl.com/articles/working_with_pem_files/
-
9It's actually easier, processingwise, to store bytes as bytes in a lot of cases. Even in a database, and especially in a file (if fixed-length records are used, or the bytes are the only content). Base64 is typically used when those bytes are intended to be transmitted somewhere, particularly over a channel that might lop off bits or interpret some of the bytes as control codes.– cHaoCommented Aug 25, 2012 at 5:51
-
I've never seen a hash written as unsigned 8 bit integers, 0,1,255,36...and displaying it with UTF-8 or any other encoding wouldn't make sense, how else would you display it other than with base64? Encryption keys and encrypted data are often stored in configuration and XML files where you cannot store the raw bytes. I agree if you can store it as raw bytes then by all means, but base64 is for those situations when you cannot. There are many uses of base64 beyond transmitting. These are simply two common scenarios where you will see it. Commented Aug 25, 2012 at 6:23
-
2You'd display the hash as hex, not decimal. For hashes, that is in fact far more common than base64.– cHaoCommented Feb 23, 2014 at 9:51
-
@cHao Yes, this is also common. Hex digits can represent any binary data, but base 64 has the advantage of taking up a lot less space since it uses more characters. Commented Feb 23, 2014 at 19:43
-
1You've got the size of SHA and MD5 reversed; SHA is usually (but not always) 256, and MD5 is 128. Commented Jun 11, 2020 at 19:43
In the early days of computers, when telephone line inter-system communication was not particularly reliable, a quick & dirty method of verifying data integrity was used: "bit parity". In this method, every byte transmitted would have 7-bits of data, and the 8th would be 1 or 0, to force the total number of 1 bits in the byte to be even.
Hence 0x01 would be transmited as 0x81; 0x02 would be 0x82; 0x03 would remain 0x03 etc.
To further this system, when the ASCII character set was defined, only 00-7F were assigned characters. (Still today, all characters set in the range 80-FF are non-standard)
Many routers of the day put the parity check and byte translation into hardware, forcing the computers attached to them to deal strictly with 7-bit data. This force email attachments (and all other data, which is why HTTP & SMTP protocols are text-based), to be convert into a text-only format.
Few of the routers survived into the 90s. I severely doubt any of them are in use today.
-
3This is an excellent point of discussion and an interesting history lesson, thanks. Commented Jun 5, 2015 at 13:43
-
1But I think the adoption of 7-bit ASCII was primarily driven by punched paper tape formats, and its origins lie in telegraphy rather than inter-computer communication, Commented Jul 1, 2020 at 20:41
From http://en.wikipedia.org/wiki/Base64
The term Base64 refers to a specific MIME content transfer encoding. It is also used as a generic term for any similar encoding scheme that encodes binary data by treating it numerically and translating it into a base 64 representation. The particular choice of base is due to the history of character set encoding: one can choose a set of 64 characters that is both part of the subset common to most encodings, and also printable. This combination leaves the data unlikely to be modified in transit through systems, such as email, which were traditionally not 8-bit clean.
Base64 can be used in a variety of contexts:
- Evolution and Thunderbird use Base64 to obfuscate e-mail passwords[1]
- Base64 can be used to transmit and store text that might otherwise cause delimiter collision
Base64 is often used as a quick but insecure shortcut to obscure secrets without incurring the overhead of cryptographic key management
Spammers use Base64 to evade basic anti-spamming tools, which often do not decode Base64 and therefore cannot detect keywords in encoded messages.
- Base64 is used to encode character strings in LDIF files
- Base64 is sometimes used to embed binary data in an XML file, using a syntax similar to ...... e.g. Firefox's bookmarks.html.
- Base64 is also used when communicating with government Fiscal Signature printing devices (usually, over serial or parallel ports) to minimize the delay when transferring receipt characters for signing.
- Base64 is used to encode binary files such as images within scripts, to avoid depending on external files.
- Can be used to embed raw image data into a CSS property such as background-image.
Some transportation protocols only allow alphanumerical characters to be transmitted. Just imagine a situation where control characters are used to trigger special actions and/or that only supports a limited bit width per character. Base64 transforms any input into an encoding that only uses alphanumeric characters, +
, /
and the =
as a padding character.
The base64 is a binary to a text encoding scheme that represents binary data in an ASCII string format. base64 is designed to carry data stored in binary format across the channels. It takes any form of data and transforms it into a long string of plain text. Earlier we can not transfer a large amount of data like files because it is made up of 2⁸ bit bytes but our actual network uses 2⁷ bit bytes. This is where base64 encoding came into the picture. But, what actually does base64 mean?
let’s understand the meaning of base64.
base64 = base+64
we can call base64 as a radix-64 representation.base64 uses only 6-bits(2⁶ = 64 characters) to ensure the printable data is human readable. but, how? we can also write base65 or base78, but why only 64? let’s prove it. base64 encoding contains 64 characters to encode any string. base64 contains:
10 numeric value i.e., 0,1,2,3,…..9.
26 Uppercase alphabets i.e., A,B,C,D,…….Z.
26 Lowercase alphabets i.e., a,b,c,d,……..z.
two special characters i.e., +,/. Depends upon your OS.
The steps followed by the base64 algorithm are as follow:
- count the number of characters in a String.
- If it is not multiple of 3 pad with special character i.e., = to make it multiple of 3.
- Encode the string in ASCII format.
- Now, it will convert the ASCII to binary format 8-bit each.
- After converting to binary format, it will divide binary data into chunks of 6-bits each.
- The chunks of 6-bit binary data will now be converted to decimal number format.
- Using the base64 Index Table, the decimals will be again converted to a string according to the table format.
- Finally, we will get the encoded version of our input string.
Base64 is a binary to a text encoding scheme that represents binary data in an ASCII string format. It is designed to carry data stored in binary format across the network channels.
Base64 mechanism uses 64 characters to encode. These characters consist of:
- 10 numeric value: i.e., 0,1,2,3,...,9
- 26 Uppercase alphabets: i.e., A,B,C,D,...,Z
- 26 Lowercase alphabets: i.e., a,b,c,d,...,z
- 2 special characters (these characters depends on operating system): i.e. +,/
How base64 works
The steps to encode a string with base64 algorithm are as follow:
- Count the number of characters in a String. If it is not multiple of 3, then pad it with special characters (i.e. =) to make it multiple of 3.
- Convert string to ASCII binary format 8-bit using the ASCII table.
- After converting to binary format, divide binary data into chunks of 6-bits.
- Convert chunks of 6-bit binary data to decimal numbers.
- Convert decimals to string according to the base64 Index Table. This table can be an example, but as I said, 2 special characters may vary.
Now, we got the encoded version of the input string.
Let's make an example: convert string THS to base64 encoding string.
Count the number of characters: it is already a multiple of 3.
Convert to ASCII binary format 8-bit. We got (T)01010100 (H)01001000 (S)01010011
Divide binary data into chunks of 6-bits. We got 010101 000100 100001 010011
Convert chunks of 6-bit binary data to decimal numbers.We got 21 4 33 19
Convert decimals to string according to the base64 Index Table. We got VEhT
It's used for converting arbitrary binary data to ASCII text.
For example, e-mail attachments are sent this way.
“Base64 encoding schemes are commonly used when there is a need to encode binary data that needs be stored and transferred over media that are designed to deal with textual data. This is to ensure that the data remains intact without modification during transport”(Wiki, 2017)
Example could be the following: you have a web service that accept only ASCII chars. You want to save and then transfer user’s data to some other location (API) but recipient want receive untouched data. Base64 is for that. . . The only downside is that base64 encoding will require around 33% more space than regular strings.
Another Example:: uenc = url encoded = aHR0cDovL2xvYy5tYWdlbnRvLmNvbS9hc2ljcy1tZW4tcy1nZWwta2F5YW5vLXhpaS5odG1s = http://loc.querytip.com/asics-men-s-gel-kayano-xii.html.
As you can see we can’t put char “/” in URL if we want to send last visited URL as parameter because we would break attribute/value rule for “MOD rewrite” – GET parameter.
A full example would be: “http://loc.querytip.com/checkout/cart/add/uenc/http://loc.magento.com/asics-men-s-gel-kayano-xii.html/product/93/”
I use it in a practical sense when we transfer large binary objects (images) via web services. So when I am testing a C# web service using a python script, the binary object can be recreated with a little magic.
[In python]
import base64
imageAsBytes = base64.b64decode( dataFromWS )
-
1
-
1@FelipeM slower, not faster. Base64 has 33% overhead (for the price of safety.)– JurajCommented Jun 5, 2020 at 14:25
The usage of Base64 I'm going to describe here is somewhat a hack. So if you don't like hacks, please do not go on.
I went into trouble when I discovered that MySQL's utf8 does not support 4-byte unicode characters since it uses a 3-byte version of utf8. So what I did to support full 4-byte unicode over MySQL's utf8? Well, base64 encode strings when storing into the database and base64 decode when retrieving.
Since base64 encoding and decoding is very fast, the above worked perfectly.
You have the following points to take note of:
Base64 encoding uses 33% more storage
Strings stored in the database wont be human readable (You could sell that as a feature that database strings use a basic form of encryption).
You could use the above method for any storage engine that does not support unicode.
-
8"You could sell that as a feature that database strings use a basic form of encryption" I like your style :D– ErcanCommented Sep 17, 2015 at 17:51
-
12"You could sell that as a feature that database strings use a basic form of encryption" what a horrible thing to say :D– AlexCommented Dec 28, 2016 at 14:17
-
4basic form of encryption against anyone who doesn't have the base64 decode algorithm rofl :D– EladianCommented Oct 9, 2017 at 13:18
-
2@Alex Not at all a "horrible thing to say". Second degree sensitive data is okay to be base64 encoded to make it unreadable by db administrators. It's not always necessary to have the highest level of encryption for every piece of data. For example, if you want to hide "comments" from a db administrator, then base64 is suitable for the job. Gratcias! Commented Jan 27, 2018 at 14:49
-
1It's worth mentioning that MySQL does now have support for all of Unicode, though for purposes of backwards compatibility, their
utf8
type is still three-bytes only; if you want the real thing, useutf8mb4
. Nice hack, but no longer necessary.– TRiGCommented Apr 3, 2018 at 10:49
Mostly, I've seen it used to encode binary data in contexts that can only handle ascii - or a simple - character sets.
To expand a bit on what Brad is saying: many transport mechanisms for email and Usenet and other ways of moving data are not "8 bit clean", which means that characters outside the standard ascii character set might be mangled in transit - for instance, 0x0D might be seen as a carriage return, and turned into a carriage return and line feed. Base 64 maps all the binary characters into several standard ascii letters and numbers and punctuation so they won't be mangled this way.
One hexadecimal digit is of one nibble (4 bits). Two nibbles make 8 bits which are also called 1 byte.
MD5 generates a 128-bit output which is represented using a sequence of 32 hexadecimal digits, which in turn are 32*4=128 bits. 128 bits make 16 bytes (since 1 byte is 8 bits).
Each Base64 character encodes 6 bits (except the last non-pad character which can encode 2, 4 or 6 bits; and final pad characters, if any). Therefore, per Base64 encoding, a 128-bit hash requires at least ⌈128/6⌉ = 22 characters, plus pad if any.
Using base64, we can produce the encoded output of our desired length (6, 8, or 10). If we choose to decide 8 char long output, it occupies only 8 bytes whereas it was occupying 16 bytes for 128-bit hash output.
So, in addition to security, base64 encoding is also used to reduce the space consumed.
Base64 can be used for many purposes.
The primary reason is to convert binary data to something passable.
I sometimes use it to pass JSON data around from one site to another, store information in cookies about a user.
Note: You "can" use it for encryption - I don't see why people say you can't, and that it's not encryption, although it would be easily breakable and is frowned upon. Encryption means nothing more than converting one string of data to another string of data that can be either later decrypted or not, and that's what base64 does.
-
7
-
2You're interpreting the definition of "encryption" far too literally. The word has evolved into something a fair bit more specific than its origins. Commented Jun 5, 2015 at 13:45