161

I am making an HTTP get request to a website for an android application I am making.

I am using a DefaultHttpClient and using HttpGet to issue the request. I get the entity response and from this obtain an InputStream object for getting the html of the page.

I then cycle through the reply doing as follows:

BufferedReader r = new BufferedReader(new InputStreamReader(inputStream));
String x = "";
x = r.readLine();
String total = "";

while(x!= null){
total += x;
x = r.readLine();
}

However this is horrendously slow.

Is this inefficient? I'm not loading a big web page - www.cokezone.co.uk so the file size is not big. Is there a better way to do this?

Thanks

Andy

1

12 Answers 12

363

The problem in your code is that it's creating lots of heavy String objects, copying their contents and performing operations on them. Instead, you should use StringBuilder to avoid creating new String objects on each append and to avoid copying the char arrays. The implementation for your case would be something like this:

BufferedReader r = new BufferedReader(new InputStreamReader(inputStream));
StringBuilder total = new StringBuilder();
for (String line; (line = r.readLine()) != null; ) {
    total.append(line).append('\n');
}

You can now use total without converting it to String, but if you need the result as a String, simply add:

String result = total.toString();

I'll try to explain it better...

  • a += b (or a = a + b), where a and b are Strings, copies the contents of both a and b to a new object (note that you are also copying a, which contains the accumulated String), and you are doing those copies on each iteration.
  • a.append(b), where a is a StringBuilder, directly appends b contents to a, so you don't copy the accumulated string at each iteration.
16
  • 26
    For bonus points, provide an initial capacity to avoid reallocations as the StringBuilder fills up: StringBuilder total = new StringBuilder(inputStream.available());
    – dokkaebi
    Commented Jan 22, 2012 at 7:04
  • 11
    Doesn't this cut out new line characters? Commented May 10, 2012 at 4:35
  • 5
    don't forget to wrap the while up in try / catch like this: try { while ((line = r.readLine()) != null) { total.append(line); } } catch (IOException e) { Log.i(tag, "problem with readline in inputStreamToString function"); }
    – botbot
    Commented Jun 4, 2012 at 8:13
  • 4
    @botbot: Logging and ignoring an exception is not much better than just ignoring the exception... Commented Sep 9, 2012 at 16:56
  • 52
    It's amazing that Android doesn't have a built-in stream-to-string conversion. Having every code snippet on the web and app on the planet re-implement a readline loop is ridiculous. That pattern should have died with pea green in the 70s. Commented Sep 20, 2013 at 16:08
35

Have you tried the built in method to convert a stream to a string? It's part of the Apache Commons library (org.apache.commons.io.IOUtils).

Then your code would be this one line:

String total = IOUtils.toString(inputStream);

The documentation for it can be found here: http://commons.apache.org/io/api-1.4/org/apache/commons/io/IOUtils.html#toString%28java.io.InputStream%29

The Apache Commons IO library can be downloaded from here: http://commons.apache.org/io/download_io.cgi

7
  • I realize this is a late response, but just now happened to stumble across this via a Google search.
    – Makotosan
    Commented Aug 24, 2010 at 0:28
  • 66
    The android API doesn't include IOUtils
    – Charles Ma
    Commented Nov 10, 2010 at 12:09
  • 2
    Right, which is why I mentioned the external library that has it. I added the library to my Android project and it made it easy to read from streams.
    – Makotosan
    Commented Nov 15, 2010 at 18:49
  • where can I download this, and how did you import that into your android project?
    – RAW
    Commented Nov 24, 2011 at 6:57
  • 3
    If you have to download it, I wouldn't call it "built in"; nevertheless, I just downloaded it, and will give it a go. Commented Apr 2, 2014 at 18:22
17

Another possibility with Guava:

dependency: compile 'com.google.guava:guava:11.0.2'

import com.google.common.io.ByteStreams;
...

String total = new String(ByteStreams.toByteArray(inputStream ));
12

I believe this is efficient enough... To get a String from an InputStream, I'd call the following method:

public static String getStringFromInputStream(InputStream stream) throws IOException
{
    int n = 0;
    char[] buffer = new char[1024 * 4];
    InputStreamReader reader = new InputStreamReader(stream, "UTF8");
    StringWriter writer = new StringWriter();
    while (-1 != (n = reader.read(buffer))) writer.write(buffer, 0, n);
    return writer.toString();
}

I always use UTF-8. You could, of course, set charset as an argument, besides InputStream.

0
7

What about this. Seems to give better performance.

byte[] bytes = new byte[1000];

StringBuilder x = new StringBuilder();

int numRead = 0;
while ((numRead = is.read(bytes)) >= 0) {
    x.append(new String(bytes, 0, numRead));
}

Edit: Actually this sort of encompasses both steelbytes and Maurice Perry's

3
  • The problem is - I dont know the size of the thing im reading before i start - so might need some form of array growing as well. Inless you can query an InputStream or URL through http to find out how big the thing im retrieving is to optimise the size of the byte array. I have to be efficient as its on a mobile device which is the main problem! However thanks for that idea - Will give it a shot tonight and let you know how it handles in terms of performance gain! Commented Mar 24, 2010 at 17:40
  • I don't think the size of the incoming stream is that important. The above code reads 1000 bytes at a time but you could increase/decrease that size. With my testing it didn't make that much difference weather I used 1000/10000 bytes. That was just a simple Java app though. It may be more important on a mobile device.
    – Adrian
    Commented Mar 24, 2010 at 18:18
  • 5
    You could end up with an Unicode entity that is chopped into two subsequent reads. Better to read until some kind of boundary character, like \n, which is exactly what BufferedReader does. Commented Oct 16, 2011 at 19:41
4

Possibly somewhat faster than Jaime Soriano's answer, and without the multi-byte encoding problems of Adrian's answer, I suggest:

File file = new File("/tmp/myfile");
try {
    FileInputStream stream = new FileInputStream(file);

    int count;
    byte[] buffer = new byte[1024];
    ByteArrayOutputStream byteStream =
        new ByteArrayOutputStream(stream.available());

    while (true) {
        count = stream.read(buffer);
        if (count <= 0)
            break;
        byteStream.write(buffer, 0, count);
    }

    String string = byteStream.toString();
    System.out.format("%d bytes: \"%s\"%n", string.length(), string);
} catch (IOException e) {
    e.printStackTrace();
}
3
  • Can you explain why it would be faster?
    – Akhil Dad
    Commented Feb 2, 2016 at 5:32
  • It doesn't scan the input for newline characters, but just reads chunks of 1024 bytes. I'm not arguing this will make any practical difference.
    – heiner
    Commented Feb 2, 2016 at 10:41
  • any comments over @Ronald answer? He is doing the same but for a larger chunk equal to inputStream size. Also how different it is if I scan char array rather than byte array as Nikola answer? Actually I just wanted to know which approach is best in which case? Also readLine removes \n and \r but I seen even google io app code they are using readline
    – Akhil Dad
    Commented Feb 2, 2016 at 12:54
3

Maybe rather then read 'one line at a time' and join the strings, try 'read all available' so as to avoid the scanning for end of line, and to also avoid string joins.

ie, InputStream.available() and InputStream.read(byte[] b), int offset, int length)

2
  • Hmm. so it would be like this: int offset = 5000; Byte[] bArr = new Byte[100]; Byte[] total = Byte[5000]; while(InputStream.available){ offset = InputStream.read(bArr,offset,100); for(int i=0;i<offset;i++){ total[i] = bArr[i]; } bArr = new Byte[100]; } Is that really more efficient - or have i written it badly! Please give an example! Commented Mar 24, 2010 at 12:13
  • 2
    no no no no, I mean simply { byte total[] = new [instrm.available()]; instrm.read(total,0,total.length); } and if you then needed it as a String, use { String asString = String(total,0,total.length,"utf-8"); // assume utf8 :-) }
    – SteelBytes
    Commented Mar 25, 2010 at 0:44
2

Reading one line of text at a time, and appending said line to a string individually is time-consuming both in extracting each line and the overhead of so many method invocations.

I was able to get better performance by allocating a decent-sized byte array to hold the stream data, and which is iteratively replaced with a larger array when needed, and trying to read as much as the array could hold.

For some reason, Android repeatedly failed to download the entire file when the code used the InputStream returned by HTTPUrlConnection, so I had to resort to using both a BufferedReader and a hand-rolled timeout mechanism to ensure I would either get the whole file or cancel the transfer.

private static  final   int         kBufferExpansionSize        = 32 * 1024;
private static  final   int         kBufferInitialSize          = kBufferExpansionSize;
private static  final   int         kMillisecondsFactor         = 1000;
private static  final   int         kNetworkActionPeriod        = 12 * kMillisecondsFactor;

private String loadContentsOfReader(Reader aReader)
{
    BufferedReader  br = null;
    char[]          array = new char[kBufferInitialSize];
    int             bytesRead;
    int             totalLength = 0;
    String          resourceContent = "";
    long            stopTime;
    long            nowTime;

    try
    {
        br = new BufferedReader(aReader);

        nowTime = System.nanoTime();
        stopTime = nowTime + ((long)kNetworkActionPeriod * kMillisecondsFactor * kMillisecondsFactor);
        while(((bytesRead = br.read(array, totalLength, array.length - totalLength)) != -1)
        && (nowTime < stopTime))
        {
            totalLength += bytesRead;
            if(totalLength == array.length)
                array = Arrays.copyOf(array, array.length + kBufferExpansionSize);
            nowTime = System.nanoTime();
        }

        if(bytesRead == -1)
            resourceContent = new String(array, 0, totalLength);
    }
    catch(Exception e)
    {
        e.printStackTrace();
    }

    try
    {
        if(br != null)
            br.close();
    }
    catch(IOException e)
    {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
}

EDIT: It turns out that if you don't need to have the content re-encoded (ie, you want the content AS IS) you shouldn't use any of the Reader subclasses. Just use the appropriate Stream subclass.

Replace the beginning of the preceding method with the corresponding lines of the following to speed it up an extra 2 to 3 times.

String  loadContentsFromStream(Stream aStream)
{
    BufferedInputStream br = null;
    byte[]              array;
    int                 bytesRead;
    int                 totalLength = 0;
    String              resourceContent;
    long                stopTime;
    long                nowTime;

    resourceContent = "";
    try
    {
        br = new BufferedInputStream(aStream);
        array = new byte[kBufferInitialSize];
1
  • This is much faster than the above and accepted answers. How do you use "Reader" and "Stream" on android?
    – SteveGSD
    Commented May 22, 2014 at 0:30
1

If the file is long, you can optimize your code by appending to a StringBuilder instead of using a String concatenation for each line.

2
  • Its not that long to be honest - its the page source of the website www.cokezone.co.uk - so really not that big. Definitely less than 100kb. Commented Mar 22, 2010 at 12:57
  • Does anybody have any other ideas on how this could be made more efficient - or if this is even inefficient!? If the latter is true - why does it take SO long? I dont believe the connection is to blame. Commented Mar 22, 2010 at 18:06
1
    byte[] buffer = new byte[1024];  // buffer store for the stream
    int bytes; // bytes returned from read()

    // Keep listening to the InputStream until an exception occurs
    while (true) {
        try {
            // Read from the InputStream
            bytes = mmInStream.read(buffer);

            String TOKEN_ = new String(buffer, "UTF-8");

            String xx = TOKEN_.substring(0, bytes);
1

To convert the InputStream to String we use the BufferedReader.readLine() method. We iterate until the BufferedReader return null which means there's no more data to read. Each line will appended to a StringBuilder and returned as String.

 public static String convertStreamToString(InputStream is) {

        BufferedReader reader = new BufferedReader(new InputStreamReader(is));
        StringBuilder sb = new StringBuilder();

        String line = null;
        try {
            while ((line = reader.readLine()) != null) {
                sb.append(line + "\n");
            }
        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            try {
                is.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
        return sb.toString();
    }
}`

And finally from any class where you want to convert call the function

String dataString = Utils.convertStreamToString(in);

complete

-2

I am use to read full data:

// inputStream is one instance InputStream
byte[] data = new byte[inputStream.available()];
inputStream.read(data);
String dataString = new String(data);

Note that this applies to files stored on disk and not to streams with no default size.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.