5

Some background:

I working on some java packages that need to take data, divide it and distribute to many Servers online. According to the user code (person using my packages), I will know which kind of buffer to use. I have seen The Netty Project that has some interesting concepts in mind and other solutions suggested here in the site, but not exactly what I'm looking for. I guess I can implement my requirements, but I like reusing better.

The actual Issue: Byte Buffer Requirements

Separate implementation from interface. Change between the two implementation.

Fixed Size:

  1. Direct memory usage.
  2. Zero Copy IO on distribution.

Growing Size

  1. Can grow to up to a certain Size.
  2. Keeps balance between reallocation and amount of memory in use.

Update 1: Decided to go with the Netty project library due to a lot of other benefits.

I wanted to give some details about my system. I need to distribute large amounts of data throughout a network of servers. I have very low amount of knowledge regarding the size of my buffers. Sometimes it can be fixed sizes, sometimes it can be in unknown size mode. I want to be able to develop a mechanism that enjoys the benefit of both worlds. I do have a defined event when I receive the final buffer for inspection.

I understand that buffers can perform better when they are directly mapped to memory. Netty provides this for me, but I can't make this buffer grow. I have dynamic buffer which can grow -Netty provides that too.

Couple of questions:

  1. should I copy the buffer to from dynamic to direct in case it is changed? Is there a way to switch modes?

  2. What do you think of implementing a class that holds an array of ChannleBuffers and exposes the same interface like ChannleBuffer while containing "inside" an array of buffers allocating new direct memory as needed.

  3. Do you have other solutions in mind ?

2
  • A pool of directly-allocated byte buffers (that is, pass/consume List<ByteBuffer>). Although, I am not sure what "Zero Copy IO on distribution" means :( Unfortunately ByteBuffer is itself a sealed class that doesn't conform to a common interface... so why not just whatever Netty has?
    – user166390
    Oct 15, 2011 at 19:18
  • from what i understand Netty supports "endless" buffers. I want to limit my user. Zero Copy means to avoid as much copying the buffers in order to send them, Netty is good for that.
    – qballer
    Oct 15, 2011 at 19:48

2 Answers 2

2

I would have a good look at scatter/gather NIO on arrays of direct byte buffers. The direct-ness gives you no-copy I/O, and the array gives you the mechanism to grow.

1

I found out that Netty supplied me with all the things I need and more.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.