0

I am trying to get table data from the following url:

Get Data from this URL

and I wrote this code with the help of jaunt API

package org.open.browser;

import com.jaunt.Element;
import com.jaunt.Elements;
import com.jaunt.JauntException;
import com.jaunt.UserAgent;

public class ICICIScraperDemo {

    public static void main(String ar[]) throws JauntException{
        
        UserAgent userAgent = new UserAgent();         //create new userAgent (headless browser)
        userAgent.visit("https://www.icicidirect.com/idirectcontent/Research/TechnicalAnalysis.aspx/companyprofile/inftec");     
       Elements links = userAgent.doc.findEvery("<div class=expander>").findEvery("<a>");  //find search result links
        String url = null;
        for(Element link : links) {
            if(link.innerHTML().equalsIgnoreCase("Company Details")){
                  url = link.getAt("href");
            }
        }
        /*userAgent = new UserAgent(); */        //create new userAgent (headless browser)
        userAgent.visit(url);   
        System.out.println(userAgent.getSource());
        Elements results = userAgent.doc.findEvery("<tr>").findEvery("<td>");
          System.out.println(results);
    }
}

But it didn't work.

Then I tried another API called htmlunit and wrote below code

public void htmlUnitEx(){
        String START_URL = "https://www.icicidirect.com/idirectcontent/Research/TechnicalAnalysis.aspx/companyprofile/inftec";
                try {
                    WebClient webClient = new WebClient(BrowserVersion.CHROME);
                    HtmlPage page = webClient.getPage(START_URL);
                    WebResponse webres = page.getWebResponse();
                    //List<HtmlAnchor> companyInfo = (List) page.getByXPath("//input[@id='txtStockCode']");
                     HtmlTable companyInfo = (HtmlTable) page.getFirstByXPath("//table");
                        for(HtmlTableRow  item : companyInfo.getBodies().get(0).getRows()){  
                            String label = item.getCell(1).asText();
                            System.out.println(label);
                             if(!label.contains("Registered Office")){
                                    continue ;
                                }
}
}

But this also not giving the result .

Can someone please help how to get the data from the above url and other Anchor url in a single session?

2
  • What do you mean "didn't work"? The <a> tags are all JavaScript. What is it you're trying to get? The real data loads in an iframe - use your browser tools to see the call to, for example, https://www.icicidirect.com/idirectcontent/basemasterpage/ContentDataHandler.ashx?icicicode=INFTEC
    – stdunbar
    Sep 6, 2018 at 2:14
  • @stdunbar Thanks for your comment ,do you mean the url for table data creation is different than url i posted ? Also these API wont work to get data from the Page? And if you saw this url when loading right hand side one table with data ,that data i want to store. Sep 6, 2018 at 2:17

1 Answer 1

0

Using HtmlUnit you can do this

    String url = "https://www.icicidirect.com/idirectcontent/Research/TechnicalAnalysis.aspx/companyprofile/inftec";

    try (final WebClient webClient = new WebClient(BrowserVersion.FIREFOX_60)) {
        HtmlPage page = webClient.getPage(url);
        webClient.waitForBackgroundJavaScript(1000);

        final DomNodeList<DomNode> divs = page.querySelectorAll("div.bigcoll");
        System.out.println(divs.get(1).asText());
    }

Two things to mention:

  • you have to wait after the getPage call a bit because some parts are created by javascript/AJAX
  • there are many way to find elements on a page (see Finding a specific element). I have done only a quick hack to show the code is working.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.