2

I've been developing a Rust library recently to try to provide fast access to a large database (the Unicode character database, which as a flat XML file is 160MB). I also want it to have a small footprint so I've used various approaches to reduce the size. The end result is that I have a series of static slices that look like:

#[derive(Clone,Copy,Eq,PartialEq,Debug)]
pub enum UnicodeCategory {
    UppercaseLetter,
    LowercaseLetter,
    TitlecaseLetter,
    ModifierLetter,
    OtherLetter,
    NonspacingMark,
    SpacingMark,
    EnclosingMark,
    DecimalNumber,
    // ...
}

pub static UCD_CAT: &'static [((u8, u8, u8), (u8, u8, u8), UnicodeCategory)] =
    &[((0, 0, 0), (0, 0, 31), UnicodeCategory::Control),
      ((0, 0, 32), (0, 0, 32), UnicodeCategory::SpaceSeparator),
      ((0, 0, 33), (0, 0, 35), UnicodeCategory::OtherPunctuation),
      /* ... */];

// ...

pub static UCD_DECOMP_MAP: &'static [((u8, u8, u8), &'static [(u8, u8, u8)])] =
    &[((0, 0, 160), &[(0, 0, 32)]),
      ((0, 0, 168), &[(0, 0, 32), (0, 3, 8)]),
      ((0, 0, 170), &[(0, 0, 97)]),
      ((0, 0, 175), &[(0, 0, 32), (0, 3, 4)]),
      ((0, 0, 178), &[(0, 0, 50)]),
      /* ... */];

In total, all the data should only take up around 600kB max (assuming extra space for alignment etc), but the library produced is 3.3MB in release mode. The source code itself (almost all data) is 2.6MB, so I don't understand why the result would be more. I don't think the extra size is intrinsic as the size was <50kB at the beginning of the project (when I only had ~2kB of data). If it makes a difference, I'm also using the #![no_std] feature.

Is there any reason for the extra binary bloat, and is there a way to reduce the size? In theory I don't see why I shouldn't be able to reduce the library to a megabyte or less.

As per Matthieu's suggestion, I tried analysing the binary with nm.

Because all my tables were represented as borrowed slices, this wasn't very useful for calculating table sizes as they were all in anonymous _refs. What I could determine was the maximum address, 0x1208f8, which would be consistent with a filesize of ~1MB rather than 3.3MB. I also looked through the hex dump to see if there were any null blocks that might explain it, but there weren't.

To see if it was the borrowed slices that were the problem, I turned them into non-borrowed slices ([T; N] form). The filesize didn't change much, but now I could interpret the nm data quite easily. Weirdly, the tables took up exactly how much I expected them to (even more weirdly, they matched my lower bounds when not accounting for alignment, and there was no space between the tables).

I also looked at the tables with nested borrowed slices, e.g. UCD_DECOMP_MAP above. When I removed all of these (about 2/3 of the data), the filesize was ~1MB when it should have only been ~250kB (by my calculations and the highest nm address, 0x3d1d0), so it doesn't look like these tables were the problem either.

I tried extracting the individual files from the .rlib file (which is a simple ar-format archive). It turns out that 40% of the library is just metadata files, and that the actual object file is 1.9MB. Further, when I do this to the library without the borrowed references the object file is 261kB! I then went back to the original library and looked at the sizes of the individual _refs and found that for a table like UCD_DECOMP_MAP: &'static [((u8,u8,u8),&'static [(u8,u8,u8)])], each value of type ((u8,u8,u8),&'static [(u8,u8,u8)]) takes up 24 bytes (3 bytes for the u8 triplet, 5 bytes of padding and 16 bytes for the pointer), and that as a result these tables take up a lot more room than I would have thought. I think I can now fully account for all the filesize.

Of course, 3MB is still quite small, I just wanted to keep the file as small as possible!

5
  • Interesting issue, however diagnosing this would be easier with a breakdown of what is taking the space here, and this is not apparent in the question (then again, I suppose if you had the breakdown the question would be quite different). As a result, it's unclear to me what kind of answer you are hoping for. Jul 23, 2016 at 18:05
  • @MatthieuM. I guess I was wondering if there are any unexpected things that might happen behind the scenes to increase the space used. I noticed in a similar library that he used #[inline(never)] on his data, but this is not legal syntax anymore. Would it be possible that rust might inline this data into each function I use it in? What tools/approaches can I use to diagnose space breakdown in a library? Sorry if the question is too unclear to be answered!
    – sourtin
    Jul 23, 2016 at 18:11
  • 1
    On linux I would use nm, however it would need some scripting on the output to get the size of each symbol and grouping the symbols per source. Jul 23, 2016 at 18:15
  • 1
    Is this a standard Rust rlib? Some other things to try (on Linux): ar tv foo.rlib show the archive members including sizes, and objdump -wh foo.rlib will show the summary information for the contained object file(s). Jul 23, 2016 at 22:19
  • Thank you both for all your help, nm and ar have helped me to explain everything about where the extra space comes from!
    – sourtin
    Jul 23, 2016 at 22:35

1 Answer 1

0

Thanks to Matthieu M. and Chris Emerson for pointing me towards the solution. This is a summary of the updates in the question, sorry for the duplication!

It seems that there are two reasons for the supposed bloat:

  1. The .rlib file outputted is not a pure object file, but is an ar archive file. Usually such a file would consist entirely of one or more object files, but rust also includes metadata. Part of the reason for this seems to be to obviate the need for separate header files. This accounted for around 40% of the final filesize.

  2. My calculations turned out to not be accurate for some of the tables, which also happened to be the largest ones. Using nm I was able to find that for normal tables such as UCD_CAT: &'static [((u8,u8,u8), (u8,u8,u8), UnicodeCategory)], the size was 7 bytes for each item (which is actually less than I originally anticipated, assuming 8 bytes for alignment). The total of all these tables was about 230kB, and the object file including just these came in at 260kB (after extraction), so this was all consistent.

    However, examining the nm output more closely for the other tables (such as UCD_DECOMP_MAP: &'static [((u8,u8,u8),&'static [(u8,u8,u8)])]) was more difficult because they appear as anonymous borrowed objects. Nevertheless, it turned out that each ((u8,u8,u8),&'static [(u8,u8,u8)]) actually takes up 24 bytes: 3 bytes for the first tuple, 5 bytes of padding, and an unexpected 16 bytes for the pointer. I believe this is because the pointer also includes the size of the referenced array. This added around a megabyte of bloat to the library, but does seem to account for the entire filesize.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.