38

I'm doing a lot of string manipulation in C#, and really need the strings to be stored one byte per character. This is because I need gigabytes of text simultaneously in memory and it's causing low memory issues. I know for certain that this text will never contain non-ASCII characters, so for my purposes, the fact that System.String and System.Char store everything as two bytes per character is both unnecessary and a real problem.

I'm about to start coding my own CharAscii and StringAscii classes - the string one will basically hold its data as byte[], and expose string manipulation methods similar to the ones that System.String does. However this seems a lot of work to do something that seems like a very standard problem, so I'm really posting here to check that there isn't already an easier solution. Is there for example some way I can make System.String internally store data as UTF8 that I haven't noticed, or some other way round the problem?

11
  • Is .NET the best way to go here? It seems like C/C++ would be better options for manipulation of huge strings in memory.
    – Paul Sasik
    Aug 27, 2012 at 15:07
  • 5
    Jon Skeet investigated this last year msmvps.com/blogs/jon_skeet/archive/2011/04/05/… Aug 27, 2012 at 15:08
  • 2
    @Jon's article is a good one, but it really only focuses on a detailed analysis of memory usage, not replacing String per se. Also, FWIW, Jon put "wacky-ideas" and "evil-code" tags on the article. Aug 27, 2012 at 15:15
  • 1
    If you have Reflector or something similar, you could pull out a copy of the String class from the framework, and change its internal structure to use a byte array. Aug 27, 2012 at 15:31
  • 1
    @RobertHarvey - I would do that as well, unfortunately, it does not contains the most interesting bits, such as the famous InternalMarvin32HashString() method (I swear this method exists :-) more seriously, many stuff is unsafe and/or internal CLR calls. Aug 27, 2012 at 15:34

4 Answers 4

12

Well, you could create a wrapper that retrieves the data as UTF-8 bytes and converts pieces as needed to System.String, then vice-versa to push the string back out to memory. The Encoding class will help you out here:

var utf8 = Encoding.UTF8;
byte[] utfBytes = utf8.GetBytes(myString);

var myReturnedString = utf8.GetString(utfBytes);
13
  • +1, I investigated this problem myself when harvesting mass amounts of data for a real estate company and this solution, while a bit magical and seemingly janky, is pretty much the best thing I was able to come up with in C#.
    – tmesser
    Aug 27, 2012 at 15:10
  • this ends up into the UTF-16 encoded string object, by the way.
    – Tigran
    Aug 27, 2012 at 15:13
  • @Tigran, there is no way to get around that if you are going to use System.String at any point. You can, however, pull out subsections of the encoded byte array and write them out in a controlled way, leaving an upper limit on how many resources you're sucking up.
    – tmesser
    Aug 27, 2012 at 15:15
  • @Tigran please elaborate. utf8 is not really utf8?
    – paparazzo
    Aug 27, 2012 at 15:17
  • 3
    @Tigran - Yes it does. Unless the OP wants to completely forego everything about Strings that you get for free with the .NET Framework (which I strongly recommend against), at least some of the data he's working with will have to be converted to and from a UTF-16 System.String to work with it. but, the untold gigs of data he's working with overall can remain in UTF-8 (or even ASCII if he really is certain the data will not contain any non-ASCII characters).
    – KeithS
    Aug 27, 2012 at 15:18
6

As you've found, the CLR uses UTF-16 for character encoding. Your best bet may be to use the Encoding classes & a BitConverter to handle the text. This question has some good examples for converting between the two encodings:

Convert String (UTF-16) to UTF-8 in C#

1
  • Thanks. Marked this as the answer since the link contains lots of info about doing the conversion. I think the approach that you and KeithS suggest is probably the best compromise in my situation between maximum performance and getting some kind of solution that saves memory without taking too long to implement. Aug 28, 2012 at 8:02
3

Not really. System.String is designed for storing strings. Your requirement is for a very particular subset of strings with particular memory benefits.

Now, "very particular subset of strings with particular memory benefits" comes up a lot, but not always the same very particular subset. Code that is ASCII-only isn't for reading by human beings, so it tends to be either short codes, or something that can be handled in a stream-processing manner, or else chunks of text merged in with bytes doing other jobs (e.g. quite a few binary formats will have small bits that translate directly to ASCII).

As such, you've a pretty strange requirement.

All the more so when you come to the gigabytes part. If I'm dealing with gigs, I'm immediately thinking about how I can stop having to deal with gigs, and/or get much more serious savings than just 50%. I'd be thinking about mapping chunks I'm not currently interested in to a file, or about ropes, or about a bunch of other things. Of course, those are going to work for some cases and not for all, so yet again, we're not talking about something where .NET should stick in something as a one-size-fits-all, because one size will not fit all.

Beyond that, just the utf-8 bit isn't that hard. It's all the other methods that becomes work. Again, what you need there won't be the same as someone else.

7
  • 5
    It's not so strange. The OP wants strings that work the same way as System.String, but take up half the space. A drop-in replacement, in other words. Aug 27, 2012 at 15:12
  • @RobertHarvey Yes, but they e.g. won't want a O(n) length because they know they don't need it from the knowledge of their data. Someone with similar but not identical needs for a utf-8 based string will need a O(n) count because they aren't sticking to ASCII-only. The general problem comes up a lot, but the tiny details vary and that makes one guy's perfect drop-in replacement another guy's poison.
    – Jon Hanna
    Aug 27, 2012 at 15:14
  • Robert Harvey has it exactly. Jon - What I'm doing, very roughly, involves extensive cross-referencing between bits of text. As such, it would be very hard to avoid having the entire text in memory during the processing. Writing chunks I'm not immediately interested in to a file only to have to read them back a milisecond later would I imagine be dreadful for performance! (As well as making the code more complicated) Aug 27, 2012 at 15:24
  • 1
    Yep. Now, if I were you, I'd still do a hunt for someone having written a nice open-source utf8string class that matched what I needed, because sometimes we do get lucky. Even then though, I'd expect there to be some point where their clever memory-saver had to be cut out as a disaster to me, or where I could do a memory-saving trick that would have ruined them.
    – Jon Hanna
    Aug 27, 2012 at 16:16
  • 1
    Speaking of memory-saving tricks, do you know about ropes as in en.wikipedia.org/wiki/Rope_%28computer_science%29 as yet another example of something that is great for some people who have to deal with very large strings, and absolutely useless to others. Thought I'd mention it in case you were in the former camp :)
    – Jon Hanna
    Aug 27, 2012 at 16:20
0

As I can see your problem is that char in C# is occupying 2 bytes, instead of one.

One way to read a text file is to open it with :

    System.IO.FileStream fs = new System.IO.FileStream(file, System.IO.FileMode.Open);
    System.IO.BinaryReader br = new System.IO.BinaryReader(fs);

    byte[] buffer = new byte[1024];
    int read = br.Read(buffer, 0, (int)fs.Length);

    br.Close();
    fs.Close(); 

And this way you are reading the bytes from the file. I tried it with *.txt files encoded in UTF-8 that is 2 bytes per char, and ANSI that is 1 byte per char.

2
  • 2
    Do you mean UTF-16? UTF-8 will, like ANSI, be 1 byte per char for the particular data I'm asking about. But thanks, that is in fact exactly the way I'll be reading the data. Aug 27, 2012 at 15:44
  • @PhantomDrummer I actually tried UTF-8, notepads usual encoding, and it took 2 bytes per char :) glad to help
    – Thanatos
    Aug 27, 2012 at 15:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.