1

I have following struct

struct Employee {
    id: u64,
    name: String,
}

I am serializing it with following code and then writing the serialized byte array to a file:

let emp = Employee {
    id: 1546,
    name: "abcd".to_string(),
};

let mut file = OpenOptions::new()
    .read(true)
    .write(true)
    .create(true)
    .open("hello.txt")
    .unwrap();

let initial_buf = &bincode::serialize(&emp).unwrap();

println!("Initial Buf: {:?}", initial_buf);

file.write(&initial_buf);
file.write(&[b'\n']);
file.flush();

file.seek(SeekFrom::Start(0)).unwrap();

let mut final_buf: Vec<u8> = Vec::new();

let mut reader = BufReader::new(file);

reader.read_until(b'\n', &mut final_buf).unwrap();

println!("Final Buf: {:?}", final_buf);

I get the following output:

Initial Buf: [10, 6, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 97, 98, 99, 100]
Final Buf: [10]
  • 2
    The newline (ascii 10) is in initial_buf, so it seems clear that it's originating from serialize, which you haven't provided. – trentcl May 7 '18 at 14:36
  • serialize is bincode::serialize – Devashish Dixit May 7 '18 at 14:36
2

Bincode's contract is that you give it a value to serialize and it gives you back bytes. The contract does not guarantee that the bytes you get back cannot contain a newline.

In your data the integer 1546 is 0x60A which is represented as the bytes [10, 6, 0, 0].

You should be able to work with Bincode data without any separators at all. The bincode::deserialize_from function will know where to stop reading.

| improve this answer | |
  • The problem with this approach is that the virtual cursor in file can be at any place. For example, if I want to read a random record from the middle of file, how can I seek to the starting location of that record? To accomplish this, I'll have to store the starting offsets of all the records separately. – Devashish Dixit May 8 '18 at 5:44
  • If you require the ability to seek, you could length-prefix each record. Generally I would expect this approach to be significantly more efficient than using a newline delimiter -- because in order to seek to a particular "line" of the file you need to look at every single byte before that line to determine whether it is a newline byte, whereas with length prefixes you can skip an entire record with one read. – dtolnay May 8 '18 at 10:14
  • Also, it would be great if somehow we can add this in bincode itself. BSON adds int32 at the beginning to specify the length of encoded byte array. – Devashish Dixit May 8 '18 at 11:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.