0

I'm trying to scrape the Steam Marketplace with JSoup, and after using the JSoup Cookbook and advice from other questions on this site, I'm stuck. I can successfully parse a single page, but I am having GREAT difficulty getting to the second page of the market list.

I know what the URLs look like on the subsequent pages. The first page looks like this:

http://steamcommunity.com/market/search?q=m4a1

The second page of items looks like this:

http://steamcommunity.com/market/search?q=m4a1#p2_default_desc

And the "page number" increases reasonably as expected in a web browser (I use Chrome).

So I built the web parser that way, and it always returns what looks like a new query on the original results (the first 10 items are shown again, and again, and again regardless of the "page" number in the URL...

So, I added in cookies to pass the session ID as well (and in one iteration, just passing back all cookies), but that didn't seem to work either. Any ideas?

Here's the Source Code I'm using. For now, it doesn't do anything fancy, just prints out the item description and stuff to the screen.

public class MarketWebParser {

    public static void main(String[] args) {

        MarketWebParser myParser = new MarketWebParser();
        myParser.parseMainUrl("http://steamcommunity.com/market/search?q=m4a1");
    }

    void parseMainUrl(String inputUrl) {
        try {
            int page = 1;
            int searchResultsEnd = 10;
            int searchResultsTotal = 1000;
            String sessionId;
            Map<String, String> steamCookies;
            Response res = Jsoup.connect(inputUrl).method(Method.POST).execute();
            steamCookies = res.cookies();
            sessionId = res.cookie("sessionid");
            System.out.println("Session: " + sessionId);

            while (searchResultsEnd != searchResultsTotal) {

                String pageAddition = "#p" + Integer.toString(page);
                String currUrl = inputUrl + pageAddition;
                System.out.println(currUrl);
                Document doc = Jsoup.connect(currUrl).cookie("sessionid",sessionId).get();
                Element content = doc.getElementById("searchResults");
                Elements marketLinks = content
                        .getElementsByClass("market_listing_row_link");
                for (Element link : marketLinks) {
                    System.out.println(link.text());
                }
                Element searchResultsTotalElem = doc
                        .getElementById("searchResults_total");
                searchResultsTotal = Integer.parseInt(searchResultsTotalElem
                        .text().replaceAll("[^\\d]", ""));
                page++;


                System.out.println("Progress: " + searchResultsEnd + " of "
                        + searchResultsTotal);
                searchResultsEnd = 10 * page;
                Thread.sleep(1000);
            }

        } catch (IOException e) {
            e.printStackTrace();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

    }
}

Here's the source code for the Webpage: (You'll notice that there's no URL for the link to the next page - to see that part of the web page, search for "searchResults_total". The button setup is just after that, but there's nothing there for me to hook to...). Note, I've removed some heading parts that I don't think are useful to get under the character count...

<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8" />
<title>Steam Community Market :: Showing results for: &quot;m4a1&quot;</title>

<!-- main body -->
<div id="BG_bottom">
    <div class="market_search_results_header">
        <h2 class="market_search_results_title">Showing results for:</h2>
                    <div style="display: inline-block; margin: 10px 0;">
                                    <a href="http://steamcommunity.com/market/search?" class="market_searchedForTerm">
                        &quot;m4a1&quot;<span class="removeIcon"></span>
                    </a>

                            </div>
            </div>
    <div id="mainContents">
        <div id="sideBar">
            <div id="findItems">
                <div class="market_search_sidebar_contents">
                    <h2 class="market_section_title">
                        Find Items                  </h2>
                    <div class="marker_search_sidebar_section_tip">
                        Looking for a specific item?                    </div>
                    <div class="market_search_box_container">
                        <form id="market_search" accept-charset="UTF8" action="http://steamcommunity.com/market/search" method="GET" >
                                                        <span class="game_button">
                                <span class="game_button_edge game_button_left"></span>
                                <span class="game_button_contents">
                                    <input class="filter_search_box market_search_sidebar_search_box" type="text" id="findItemsSearchBox" value="m4a1" placeholder="Search" name="q" autocomplete="off" tabindex="1" />
                                    <input class="market_search_submit_button" id="findItemsSearchSubmit" type="submit" value="Submit" tabindex="3" />
                                </span>
                                <span class="game_button_edge game_button_right"></span>
                                <span class="game_button_preload"></span>
                            </span>
                            <br>
                            <div style="text-align: right; padding-right: 15px; margin-top: 0.25em;">
                                <div id="market_search_advanced_show" class="btn_black btn_details btn_small" onclick="ShowAdvancedSearchOptions();">
                                    <span>
                                        <span style="float: left;">Show advanced options...</span>
                                        <span class="btn_details_arrow down"></span>
                                    </span>
                                </div>
                            </div>
                        </form>
                    </div>
                </div>
            </div>
        </div>
        <div id="searchResults" class="market_page_left">
            <div id="searchResultsTable" class="market_content_block market_home_listing_table market_home_main_listing_table market_listing_table market_listing_table_active">
                <div id="searchResultsRows">
                        <div class="market_listing_table_header">
        <div class="market_listing_right_cell market_listing_their_price market_sortable_column" data-sorttype="price">PRICE<span class="market_sort_arrow" style="display:none;"></span></div>
        <div class="market_listing_right_cell market_listing_num_listings market_sortable_column" data-sorttype="quantity">QUANTITY<span class="market_sort_arrow" style="display:none;"></span></div>
                <div class="market_sortable_column" data-sorttype="name"><span class="market_listing_header_namespacer"></span>NAME<span class="market_sort_arrow" style="display:none;"></span></div>
    </div>

<a class="market_listing_row_link" href="http://steamcommunity.com/market/listings/730/M4A1-S%20%7C%20Nitro%20%28Battle-Scarred%29?filter=m4a1" id="resultlink_0">
    <div class="market_listing_row market_recent_listing_row market_listing_searchresult" id="result_0">
                <img id="result_0_image" src="http://steamcommunity-a.akamaihd.net/economy/image/fWFc82js0fmoRAP-qOIPu5THSWqfSmTELLqcUywGkijVjZYMUrsm1j-9xgEObwgfEh_nvjlWhNzZCveCDfIBj98xqodQ2CZknz52YOLkDyRufgHMAqVMY-Y1ywf-CCk17fhiV9Sz-a8VFlS65cSPLbZ9ONofScfWXKCOY1uu6xluiKhcLpff9C7m3yvsPjtcDRru_2IHyuGZ-uw8OqQ-aC8/62fx62f" style="border-color: #D2D2D2;" class="market_listing_item_img" alt="" />      
                <div class="market_listing_right_cell market_listing_their_price">
            <span class="market_table_value">
                Starting at:<br/>
                <span style="color:white">&#36;3.05</span>
            </span>
            <span class="market_arrow_down" style="display: none"></span>
            <span class="market_arrow_up" style="display: none"></span>
        </div>
        <div class="market_listing_right_cell market_listing_num_listings">
            <span class="market_table_value">
                <span class="market_listing_num_listings_qty">65</span>
            </span>
        </div>

                <div class="market_listing_item_name_block">
            <span id="result_0_name" class="market_listing_item_name" style="color: #D2D2D2;">M4A1-S | Nitro (Battle-Scarred)</span>
            <br/>
            <span class="market_listing_game_name">Counter-Strike: Global Offensive</span>
        </div>
    </div>
</a>

<a class="market_listing_row_link" href="http://steamcommunity.com/market/listings/730/M4A1-S%20%7C%20Nitro%20%28Field-Tested%29?filter=m4a1" id="resultlink_1">
    <div class="market_listing_row market_recent_listing_row market_listing_searchresult" id="result_1">
                <img id="result_1_image" src="http://steamcommunity-a.akamaihd.net/economy/image/fWFc82js0fmoRAP-qOIPu5THSWqfSmTELLqcUywGkijVjZYMUrsm1j-9xgEObwgfEh_nvjlWhNzZCveCDfIBj98xqodQ2CZknz52YOLkDyRufgHMAqVMY-Y1ywf-CCk17fhiV9Sz-a8VFlG64NuDbq4qNtpKGpHSX_TSbw36v01s0_deep3Y9Cntj3i9OG0DXhPo8mgBkeaOpPI11cbT2t9i/62fx62f" style="border-color: #D2D2D2;" class="market_listing_item_img" alt="" />     
                <div class="market_listing_right_cell market_listing_their_price">
            <span class="market_table_value">
                Starting at:<br/>
                <span style="color:white">&#36;3.58</span>
            </span>
            <span class="market_arrow_down" style="display: none"></span>
            <span class="market_arrow_up" style="display: none"></span>
        </div>
        <div class="market_listing_right_cell market_listing_num_listings">
            <span class="market_table_value">
                <span class="market_listing_num_listings_qty">183</span>
            </span>
        </div>

                <div class="market_listing_item_name_block">
            <span id="result_1_name" class="market_listing_item_name" style="color: #D2D2D2;">M4A1-S | Nitro (Field-Tested)</span>
            <br/>
            <span class="market_listing_game_name">Counter-Strike: Global Offensive</span>
        </div>
    </div>
</a>

<a class="market_listing_row_link" href="http://steamcommunity.com/market/listings/730/M4A1-S%20%7C%20Cyrex%20%28Field-Tested%29?filter=m4a1" id="resultlink_2">
    <div class="market_listing_row market_recent_listing_row market_listing_searchresult" id="result_2">
                <img id="result_2_image" src="http://steamcommunity-a.akamaihd.net/economy/image/fWFc82js0fmoRAP-qOIPu5THSWqfSmTELLqcUywGkijVjZYMUrsm1j-9xgEObwgfEh_nvjlWhNzZCveCDfIBj98xqodQ2CZknz52YOLkDyRufgHMAqVMY_YvywW4CHYh18R6RtKuyLYDLVWq6ZzFZrUqMItNHpbRUqCOYg-rvEMx1qdcLseIpy253Cy4Mz9fUxDi-ToHhqbZ7WBrCDQm/62fx62f" style="border-color: #D2D2D2;" class="market_listing_item_img" alt="" />     
                <div class="market_listing_right_cell market_listing_their_price">
            <span class="market_table_value">
                Starting at:<br/>
                <span style="color:white">&#36;58.34</span>
            </span>
            <span class="market_arrow_down" style="display: none"></span>
            <span class="market_arrow_up" style="display: none"></span>
        </div>
        <div class="market_listing_right_cell market_listing_num_listings">
            <span class="market_table_value">
                <span class="market_listing_num_listings_qty">19</span>
            </span>
        </div>

                <div class="market_listing_item_name_block">
            <span id="result_2_name" class="market_listing_item_name" style="color: #D2D2D2;">M4A1-S | Cyrex (Field-Tested)</span>
            <br/>
            <span class="market_listing_game_name">Counter-Strike: Global Offensive</span>
        </div>
    </div>
</a>

<a class="market_listing_row_link" href="http://steamcommunity.com/market/listings/730/M4A1-S%20%7C%20Nitro%20%28Well-Worn%29?filter=m4a1" id="resultlink_3">
    <div class="market_listing_row market_recent_listing_row market_listing_searchresult" id="result_3">
                <img id="result_3_image" src="http://steamcommunity-a.akamaihd.net/economy/image/fWFc82js0fmoRAP-qOIPu5THSWqfSmTELLqcUywGkijVjZYMUrsm1j-9xgEObwgfEh_nvjlWhNzZCveCDfIBj98xqodQ2CZknz52YOLkDyRufgHMAqVMY-Y1ywf-CCk17fhiV9Sz-a8VFlG64NuDbq4qNtpKGpHSX_TSbw36v01s0_deep3Y9Cntj3i9OG0DXhPo8mgBkeaOpPI11cbT2t9i/62fx62f" style="border-color: #D2D2D2;" class="market_listing_item_img" alt="" />     
                <div class="market_listing_right_cell market_listing_their_price">
            <span class="market_table_value">
                Starting at:<br/>
                <span style="color:white">&#36;3.17</span>
            </span>
            <span class="market_arrow_down" style="display: none"></span>
            <span class="market_arrow_up" style="display: none"></span>
        </div>
        <div class="market_listing_right_cell market_listing_num_listings">
            <span class="market_table_value">
                <span class="market_listing_num_listings_qty">64</span>
            </span>
        </div>

                <div class="market_listing_item_name_block">
            <span id="result_3_name" class="market_listing_item_name" style="color: #D2D2D2;">M4A1-S | Nitro (Well-Worn)</span>
            <br/>
            <span class="market_listing_game_name">Counter-Strike: Global Offensive</span>
        </div>
    </div>
</a>

<a class="market_listing_row_link" href="http://steamcommunity.com/market/listings/730/M4A1-S%20%7C%20Guardian%20%28Minimal%20Wear%29?filter=m4a1" id="resultlink_4">
    <div class="market_listing_row market_recent_listing_row market_listing_searchresult" id="result_4">
                <img id="result_4_image" src="http://steamcommunity-a.akamaihd.net/economy/image/fWFc82js0fmoRAP-qOIPu5THSWqfSmTELLqcUywGkijVjZYMUrsm1j-9xgEObwgfEh_nvjlWhNzZCveCDfIBj98xqodQ2CZknz52YOLkDyRufgHMAqVMY_YvywW4CHZ_-_hmWNKx9rUSFlC249qCLbF9Y4oeH8DWWvCCbgn06xprhaJfe8CPqHm5iSrrOjwNCkfi8msBnuKZ-uw8ccIhRqM/62fx62f" style="border-color: #D2D2D2;" class="market_listing_item_img" alt="" />      
                <div class="market_listing_right_cell market_listing_their_price">
            <span class="market_table_value">
                Starting at:<br/>
                <span style="color:white">&#36;4.61</span>
            </span>
            <span class="market_arrow_down" style="display: none"></span>
            <span class="market_arrow_up" style="display: none"></span>
        </div>
        <div class="market_listing_right_cell market_listing_num_listings">
            <span class="market_table_value">
                <span class="market_listing_num_listings_qty">151</span>
            </span>
        </div>

                <div class="market_listing_item_name_block">
            <span id="result_4_name" class="market_listing_item_name" style="color: #D2D2D2;">M4A1-S | Guardian (Minimal Wear)</span>
            <br/>
            <span class="market_listing_game_name">Counter-Strike: Global Offensive</span>
        </div>
    </div>
</a>

<a class="market_listing_row_link" href="http://steamcommunity.com/market/listings/730/M4A1-S%20%7C%20Guardian%20%28Field-Tested%29?filter=m4a1" id="resultlink_5">
    <div class="market_listing_row market_recent_listing_row market_listing_searchresult" id="result_5">
                <img id="result_5_image" src="http://steamcommunity-a.akamaihd.net/economy/image/fWFc82js0fmoRAP-qOIPu5THSWqfSmTELLqcUywGkijVjZYMUrsm1j-9xgEObwgfEh_nvjlWhNzZCveCDfIBj98xqodQ2CZknz52YOLkDyRufgHMAqVMY_YvywW4CHZ_-_hmWNKx9rUSFlG64NuDbq4qN4seF8jZWvXSYgqru0g-hKlfJpKNp3npiym6aDtZUxa_rmIMmeXWpPI11YiABWgk/62fx62f" style="border-color: #D2D2D2;" class="market_listing_item_img" alt="" />     
                <div class="market_listing_right_cell market_listing_their_price">
            <span class="market_table_value">
                Starting at:<br/>
                <span style="color:white">&#36;3.97</span>
            </span>
            <span class="market_arrow_down" style="display: none"></span>
            <span class="market_arrow_up" style="display: none"></span>
        </div>
        <div class="market_listing_right_cell market_listing_num_listings">
            <span class="market_table_value">
                <span class="market_listing_num_listings_qty">135</span>
            </span>
        </div>

                <div class="market_listing_item_name_block">
            <span id="result_5_name" class="market_listing_item_name" style="color: #D2D2D2;">M4A1-S | Guardian (Field-Tested)</span>
            <br/>
            <span class="market_listing_game_name">Counter-Strike: Global Offensive</span>
        </div>
    </div>
</a>

<a class="market_listing_row_link" href="http://steamcommunity.com/market/listings/730/M4A1-S%20%7C%20Guardian%20%28Factory%20New%29?filter=m4a1" id="resultlink_6">
    <div class="market_listing_row market_recent_listing_row market_listing_searchresult" id="result_6">
                <img id="result_6_image" src="http://steamcommunity-a.akamaihd.net/economy/image/fWFc82js0fmoRAP-qOIPu5THSWqfSmTELLqcUywGkijVjZYMUrsm1j-9xgEObwgfEh_nvjlWhNzZCveCDfIBj98xqodQ2CZknz52YOLkDyRufgHMAqVMY_YvywW4CHZ_-_hmWNKx9rUSFlC249qCLbF9Y4oeH8DWWvCCbgn06xprhaJfe8CPqHm5iSrrOjwNCkfi8msBnuKZ-uw8ccIhRqM/62fx62f" style="border-color: #D2D2D2;" class="market_listing_item_img" alt="" />      
                <div class="market_listing_right_cell market_listing_their_price">
            <span class="market_table_value">
                Starting at:<br/>
                <span style="color:white">&#36;6.30</span>
            </span>
            <span class="market_arrow_down" style="display: none"></span>
            <span class="market_arrow_up" style="display: none"></span>
        </div>
        <div class="market_listing_right_cell market_listing_num_listings">
            <span class="market_table_value">
                <span class="market_listing_num_listings_qty">118</span>
            </span>
        </div>

                <div class="market_listing_item_name_block">
            <span id="result_6_name" class="market_listing_item_name" style="color: #D2D2D2;">M4A1-S | Guardian (Factory New)</span>
            <br/>
            <span class="market_listing_game_name">Counter-Strike: Global Offensive</span>
        </div>
    </div>
</a>

<a class="market_listing_row_link" href="http://steamcommunity.com/market/listings/730/M4A1-S%20%7C%20Nitro%20%28Factory%20New%29?filter=m4a1" id="resultlink_7">
    <div class="market_listing_row market_recent_listing_row market_listing_searchresult" id="result_7">
                <img id="result_7_image" src="http://steamcommunity-a.akamaihd.net/economy/image/fWFc82js0fmoRAP-qOIPu5THSWqfSmTELLqcUywGkijVjZYMUrsm1j-9xgEObwgfEh_nvjlWhNzZCveCDfIBj98xqodQ2CZknz52YOLkDyRufgHMAqVMY-Y1ywf-CCk17fhiV9Sz-a8VFlC249qCLbYuNY5JF5OECfOAYgir7k840_ILLcPcqCzpji7oP2xYXhvr-T8FzuSZ-uw8GGkTN28/62fx62f" style="border-color: #D2D2D2;" class="market_listing_item_img" alt="" />      
                <div class="market_listing_right_cell market_listing_their_price">
            <span class="market_table_value">
                Starting at:<br/>
                <span style="color:white">&#36;13.26</span>
            </span>
            <span class="market_arrow_down" style="display: none"></span>
            <span class="market_arrow_up" style="display: none"></span>
        </div>
        <div class="market_listing_right_cell market_listing_num_listings">
            <span class="market_table_value">
                <span class="market_listing_num_listings_qty">31</span>
            </span>
        </div>

                <div class="market_listing_item_name_block">
            <span id="result_7_name" class="market_listing_item_name" style="color: #D2D2D2;">M4A1-S | Nitro (Factory New)</span>
            <br/>
            <span class="market_listing_game_name">Counter-Strike: Global Offensive</span>
        </div>
    </div>
</a>

<a class="market_listing_row_link" href="http://steamcommunity.com/market/listings/730/M4A1-S%20%7C%20Nitro%20%28Minimal%20Wear%29?filter=m4a1" id="resultlink_8">
    <div class="market_listing_row market_recent_listing_row market_listing_searchresult" id="result_8">
                <img id="result_8_image" src="http://steamcommunity-a.akamaihd.net/economy/image/fWFc82js0fmoRAP-qOIPu5THSWqfSmTELLqcUywGkijVjZYMUrsm1j-9xgEObwgfEh_nvjlWhNzZCveCDfIBj98xqodQ2CZknz52YOLkDyRufgHMAqVMY-Y1ywf-CCk17fhiV9Sz-a8VFlC249qCLbYuNY5JF5OECfOAYgir7k840_ILLcPcqCzpji7oP2xYXhvr-T8FzuSZ-uw8GGkTN28/62fx62f" style="border-color: #D2D2D2;" class="market_listing_item_img" alt="" />      
                <div class="market_listing_right_cell market_listing_their_price">
            <span class="market_table_value">
                Starting at:<br/>
                <span style="color:white">&#36;6.90</span>
            </span>
            <span class="market_arrow_down" style="display: none"></span>
            <span class="market_arrow_up" style="display: none"></span>
        </div>
        <div class="market_listing_right_cell market_listing_num_listings">
            <span class="market_table_value">
                <span class="market_listing_num_listings_qty">85</span>
            </span>
        </div>

                <div class="market_listing_item_name_block">
            <span id="result_8_name" class="market_listing_item_name" style="color: #D2D2D2;">M4A1-S | Nitro (Minimal Wear)</span>
            <br/>
            <span class="market_listing_game_name">Counter-Strike: Global Offensive</span>
        </div>
    </div>
</a>

<a class="market_listing_row_link" href="http://steamcommunity.com/market/listings/730/M4A1-S%20%7C%20Guardian%20%28Well-Worn%29?filter=m4a1" id="resultlink_9">
    <div class="market_listing_row market_recent_listing_row market_listing_searchresult" id="result_9">
                <img id="result_9_image" src="http://steamcommunity-a.akamaihd.net/economy/image/fWFc82js0fmoRAP-qOIPu5THSWqfSmTELLqcUywGkijVjZYMUrsm1j-9xgEObwgfEh_nvjlWhNzZCveCDfIBj98xqodQ2CZknz52YOLkDyRufgHMAqVMY_YvywW4CHZ_-_hmWNKx9rUSFlG64NuDbq4qN4seF8jZWvXSYgqru0g-hKlfJpKNp3npiym6aDtZUxa_rmIMmeXWpPI11YiABWgk/62fx62f" style="border-color: #D2D2D2;" class="market_listing_item_img" alt="" />     
                <div class="market_listing_right_cell market_listing_their_price">
            <span class="market_table_value">
                Starting at:<br/>
                <span style="color:white">&#36;4.52</span>
            </span>
            <span class="market_arrow_down" style="display: none"></span>
            <span class="market_arrow_up" style="display: none"></span>
        </div>
        <div class="market_listing_right_cell market_listing_num_listings">
            <span class="market_table_value">
                <span class="market_listing_num_listings_qty">33</span>
            </span>
        </div>

                <div class="market_listing_item_name_block">
            <span id="result_9_name" class="market_listing_item_name" style="color: #D2D2D2;">M4A1-S | Guardian (Well-Worn)</span>
            <br/>
            <span class="market_listing_game_name">Counter-Strike: Global Offensive</span>
        </div>
    </div>
</a>
                </div>

        <div id="searchResults_ctn" class="market_paging" style="">
            <div class="market_paging_summary">
                Showing <span id="searchResults_start">1</span>-<span id="searchResults_end">10</span> of <span id="searchResults_total">364</span> results         </div>
            <div class="market_paging_controls" id="searchResults_controls">
                <span id="searchResults_btn_prev" class="pagebtn">&lt;</span>
                <span id="searchResults_links"></span>
                <span id="searchResults_btn_next" class="pagebtn">&gt;</span>
            </div>
            <div style="clear: both;"></div>
        </div>
                </div>
        </div>
        <div style="clear:both"></div>
    </div>
    <br />
</div></div>
<div id="market_advancedsearch_dialog" class="market_modal_dialog" style="display: none;">
    <div>
        <form id="market_advanced_search" accept-charset="UTF8" action="http://steamcommunity.com/market/search" method="GET" >
            <div id="market_advancedsearch_left">
                    <input class="filter_search_box market_search_sidebar_search_box" type="text" id="advancedSearchBox" value="" placeholder="Search" name="q" autocomplete="off" tabindex="1" />
                    <input class="market_search_submit_button" id="advancedSearchSubmit" type="submit" value="Submit" tabindex="3" />
                    <div id="market_advancedsearch_options">
                        <input class="market_search_checkbox" id="market_advanced_searchdescriptions_checkbox" type="checkbox" name="descriptions" value="1" title="By default, the Community Market will only look for matches in the items' names and types. If you check this box, all of the text in the items' descriptions will be searched too. Searching for matches in descriptions may cause your search to take more time to complete." tabindex="2" />
                        <label for="market_advanced_searchdescriptions_checkbox" title="By default, the Community Market will only look for matches in the items' names and types. If you check this box, all of the text in the items' descriptions will be searched too. Searching for matches in descriptions may cause your search to take more time to complete.">Include descriptions in search</label>
                    </div>
            </div>
            <div style="overflow: hidden">
                <div class="market_app_selector">
                    <div class="appselect" id="market_advancedsearch_appselect" onclick="ShowMenu( 'market_advancedsearch_appselect', 'market_advancedsearch_appselect_options', 'left' );">
                        <div id="market_advancedsearch_appselect_activeapp" class="option">
                            <div id="app_option_0_selected" class="popup_item market_advancedsearch_appname">
                                <span>All Games</span>
                            </div>
                        </div>
                        <div id="appselect_arrow"></div>
                    </div>
                    <div id="market_advancedsearch_appselect_options" class="popup_block_new" style="display: none;">
                        <div id="market_advancedsearch_appselect_options_apps" class="popup_body popup_menu">
                            <div id="app_option_0" class="popup_item popup_menu_item market_advancedsearch_appname">
                                <span>All Games</span>
                            </div>
                                                            <div id="app_option_238460" class="popup_item popup_menu_item market_advancedsearch_appname" data-appid="238460">
                                    <img src="http://cdn.akamai.steamstatic.com/steamcommunity/public/images/apps/238460/2f258aaff583d797812cdcf24830d5992f48733b.jpg" alt="BattleBlock Theater" /><span>BattleBlock Theater</span>
                                </div>
                                                            <div id="app_option_730" class="popup_item popup_menu_item market_advancedsearch_appname" data-appid="730">
                                    <img src="http://cdn.akamai.steamstatic.com/steamcommunity/public/images/apps/730/69f7ebe2735c366c65c0b33dae00e12dc40edbe4.jpg" alt="Counter-Strike: Global Offensive" /><span>Counter-Strike: Global Offensive</span>
                                </div>
                                                            <div id="app_option_570" class="popup_item popup_menu_item market_advancedsearch_appname" data-appid="570">
                                    <img src="http://cdn.akamai.steamstatic.com/steamcommunity/public/images/apps/570/0bbb630d63262dd66d2fdd0f7d37e8661a410075.jpg" alt="Dota 2" /><span>Dota 2</span>
                                </div>
                                                            <div id="app_option_238960" class="popup_item popup_menu_item market_advancedsearch_appname" data-appid="238960">
                                    <img src="http://cdn.akamai.steamstatic.com/steamcommunity/public/images/apps/238960/1110764aac57ac28d7ffd8c43071c75d5482a9c9.jpg" alt="Path of Exile" /><span>Path of Exile</span>
                                </div>
                                                            <div id="app_option_251970" class="popup_item popup_menu_item market_advancedsearch_appname" data-appid="251970">
                                    <img src="http://cdn.akamai.steamstatic.com/steamcommunity/public/images/apps/251970/cee4aeadcf73b7b00018c535fa3b850bfcdb9982.jpg" alt="Sins of a Dark Age" /><span>Sins of a Dark Age</span>
                                </div>
                                                            <div id="app_option_753" class="popup_item popup_menu_item market_advancedsearch_appname" data-appid="753">
                                    <img src="http://cdn.akamai.steamstatic.com/steamcommunity/public/images/apps/753/135dc1ac1cd9763dfc8ad52f4e880d2ac058a36c.jpg" alt="Steam" /><span>Steam</span>
                                </div>
                                                            <div id="app_option_440" class="popup_item popup_menu_item market_advancedsearch_appname" data-appid="440">
                                    <img src="http://cdn.akamai.steamstatic.com/steamcommunity/public/images/apps/440/e3f595a92552da3d664ad00277fad2107345f743.jpg" alt="Team Fortress 2" /><span>Team Fortress 2</span>
                                </div>
                                                            <div id="app_option_230410" class="popup_item popup_menu_item market_advancedsearch_appname" data-appid="230410">
                                    <img src="http://cdn.akamai.steamstatic.com/steamcommunity/public/images/apps/230410/2e271fb410bb8be5536a24d4114b42f9e5c91176.jpg" alt="Warframe" /><span>Warframe</span>
                                </div>
                                                    </div>
                    </div>
                </div>
                <div id="market_advancedsearch_filters" class="market_advancedsearch_filters">
                    Pick a game from the drop-down list above to see the available filters.             </div>
            </div>
        </form>
    </div>

    <div class="market_advancedsearch_bottombuttons">
        <a href="javascript:ResetSearchFilters();">Reset to default</a>
        <a href="javascript:document.forms['market_advanced_search'].submit();" class="item_market_action_button item_market_action_button_green nodisable">
            <span class="item_market_action_button_edge item_market_action_button_left"></span>
            <span class="item_market_action_button_contents">
                Search          </span>
            <span class="item_market_action_button_edge item_market_action_button_right"></span>
            <span class="item_market_action_button_preload"></span>
        </a>
    </div>
</div>

Thank you!

1 Answer 1

1

The hash in the url tells you that the part after the url is actually not transferred to the server, it is merely a site-internal reference. So you are out of luck this way. However, if you look at the site and analyze a bit the site structure and the network traffic (open developver tools -> network panel and trigger the loading of the next page) you find, that the next results are fetched using this url:

steamcommunity.com/market/search/render/?query=m4a1&start=10&count=10&search_descriptions=0&sort_column=default&sort_dir=desc

This returns a json string, which you may parse using a json library. I did not look deeper into this, but this is the route that should work.

1
  • That absolutely did it! Had to bring in GSON, as the resulting was JSON that then contained the inner HTML, but the result worked. Thanks! Aug 24, 2014 at 22:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.