0

I am using Eclipse to design an application for use on Android 2.1 utilizing the jsoup 1.6.1 core library to connect to a web page and scrape html. So far, the application, pasted below, connects to a webpage via jsoup the connect method, builds a document via the jsoup get method, it then selects all Table Row Elements, then gets all Elements of this table row via jsoup getAllElements, (in this case, Elements), tests these for text, and if containing a specific substring, converts the text to a String via jsoup's text method, and adds these to a ListArray collection of Strings, displaying them in an android ListView. Below are my 2 questions, and then the code.

The code seems to work on my emulator, returning row data, however, there is a bug where Android is displaying each row with the String information in duplicate that I cannot seem to track down.

Additionally, I wanted to know if there was a more optimal method for performing these tasks in a faster manner using jsoup or another java/android library, and wanted to inquire any suggestions. For instance, it seems like when I run this code on a page containing a lot of data that it takes a little longer than I may like. For instance, I am downloading the entire page and creating a jsoup doc prior to performing any operation on the data, or returning any of it to the user. If I were to connect to the page, search for the first Table Row html Element, test the child Elements for text containing a substring, I could potentially return information to the Android end-user more quickly and with less overhead of downloading the entire document. I was not sure which jsoup methods would deliver something to that end.

I am very new to jsoup and java, and somewhat less new (but still learning) in other languages like C and C#. Formatting the pasted text properly left something to be desired, and I was not able to use the prettify tag successfully when viewing in Chrome, and apologies in advance if I missed inputting a character or end block while copying and pasting the test. Thank you again.

Jsoupx1.Activity.java

    package jsoup.example1;
    import java.io.IOException;
    import java.util.ArrayList;
    import android.app.ListActivity;
    import android.os.Bundle;
    import org.jsoup.Jsoup;
    import org.jsoup.nodes.Document;
    import org.jsoup.nodes.Element;
    import org.jsoup.select.Elements;
    import android.widget.ArrayAdapter;
    import android.widget.ListView;
    import android.widget.Toast;
    import android.view.View;

    public class Jsoupx1Activity extends ListActivity {

    public void onCreate(Bundle icicle) {
            super.onCreate(icicle);


   //ArrayList tableRowStrings will be the recipient of table row data that matches a search criterion, 
   //stored in the form of strings.
   ArrayList<String> tableRowStrings = new ArrayList<String>();

    //Open a document from a valid URL using jsoup
    Document doc = null;
            try {
                doc = Jsoup.connect("http://www.w3schools.com/html/html_tables.asp").get();

            } catch (IOException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }


            //Once this document has loaded, select all <tr> tableRow elements into tableRows, with each
            //tableRow, collect all children, which will be <td>, into tableDatas Elements, test that Elements class for text,
            //if text exists, transfer the text of these children to a string, test that string for a substring, 
            //In this case, a date, if it contains this substring, add it to the created tableRowStrings ArrayList.
            //This function falsely returns a string with duplicated table row information.

            //Example Test Criteria, in reality this may be interfaced to a datetime or textbox widget
            //to facilitate an end-user defined search
            String testString  = "Apples";

            //select all <tr> or Table Row Elements
            Elements tableRows = doc.select("tr");

            //Load ArrayList with table row strings
            for (Element tableRow : tableRows){

                 Elements tableDatas = tableRow.getAllElements();

                 //Test if the the TableData contains Text, if so convert that text to a string rowData
                 if (tableDatas.hasText()){
                     String rowData = tableDatas.text();

                     if (rowData.contains(testString)){
                         tableRowStrings.add(rowData);
                     }

                 }

        } 

            ArrayAdapter<String> adapter = new ArrayAdapter<String>(this,
                    android.R.layout.simple_list_item_1, tableRowStrings);
            setListAdapter(adapter);
        }

    @Override
        protected void onListItemClick(ListView l, View v, int position, long id) {
            String item = (String) getListAdapter().getItem(position);
            Toast.makeText(this, item + " selected", Toast.LENGTH_LONG).show();
        }
    }

And the accompanying main.xml

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="fill_parent"
    android:layout_height="fill_parent"
    android:orientation="vertical" >

 <TextView   
    android:id="@+id/text"
    android:textStyle="bold"   
    android:layout_width="wrap_content" 
    android:layout_height="wrap_content" 
    /> 


<TextView xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="fill_parent"
    android:layout_height="fill_parent"
    android:padding="10dp"
    android:textSize="16sp" >
</TextView>

<ListView
  android:id="@+android:id/list"
  android:layout_width="fill_parent" 
   android:textColor="#444444"
    android:cacheColorHint="#00000000"
         android:layout_height="wrap_content" 
 >
</ListView>

</LinearLayout>

2 Answers 2

0

To fix the issue where the String had duplicate copies of the table data, I replaced the logic within the foreach block. Previous incorrectly performing version:

Elements tableDatas = tableRow.getAllElements();

//Test if the the TableData contains Text, if so convert that text to a string rowData
         if (tableDatas.hasText()){
             String rowData = tableDatas.text();

             if (rowData.contains(testString)){
                 tableRowStrings.add(rowData);
             }

         }

Updated version:

if (tableRow.hasText()){
    String rowData = tableRow.text();
    if(rowData.contains(testString)){
        tableRowStrings.add(rowData);
    }
}

This seems to fix the problem. It seems slightly faster as well now. I am not sure why it was collecting 2 copies of the table data in the previous version, and I guess that's why I may have referred to it as a bug, which may not be true, but just my misunderstanding of the code.

On questions of performance, perhaps a piece of it concerns my lack of understanding with the java garbage collector. For example, in my code, once I have created a document and extracted the required Elements, will the Garbage Collector calculate that I have no more operations for the document, and remove it?

The only other pressing question I have remains is there a way to open a connection to a website, begin scraping and return the first match to the end-user via android ListView, and the continuing along to search the html document, and then refreshing the ListView if/when more matching items are located?

0

After going back to the jsoup documentation, it appears that the Jsoup.connect(String url) method simply provides the interface to the web page or file html, method rather than downloading the entire document. The get() executes the request as a GET and parses the result, returning a parsed Document. Doing some testing, the Jsoup.connect().get() operation consumes the most time. It didn't look like any of the other Jsoup options would allow the granularity of parsing to deliver content faster. The other option would be to try a different library, or use a Web Dyno approach and the Jsoup.connect().post() method to connect to a proxy webpage which would itself provide the scraping functionality, along the lines of the android application in the following blog post, linked below. I am just waiting for the required 24 hour period to expire so I can accept the answer, unless I should just delete this post, I wasn't quite sure... http://ifdefined.com/blog/post/A-Hacker-News-app-for-Android.aspx

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.