1

I'm trying to use curl to automate requesting occupation codes from the Australian Bureau of Statistics web page. I'm getting a "Search Index does not exist" error when I do a POST request with Curl. I think the problem is that there are two inputs that I'm not entering, and so it is failing. I would appreciate any assistance on this:

Curl request:

curl --data "searchcontent=&helpdocid=[docid]&searchtext=121315&submit=Go" http://www.ausstats.abs.gov.au/ausstats/searchclass.nsf/(Searchattmnt)?openagent

The relevant code from the ABS site is this:

<!-- Start Search Content -->
<div id="titlemain"> Search Results</div><div id="middle"></div>
<form name="attmnt" method="post" action="http://www.ausstats.abs.gov.au/ausstats/searchclass.NSF/(Searchattmnt)?openagent"> 
<input type="hidden" name="searchcontent" value="">
<input type="hidden" name="helpdocid" value="[docid]">
<div id="content">
<h3>New Search :</h3>
<input type="text" name="searchtext" size =60 MAXLENGTH="255" value="">   
<INPUT VALUE="Go" TYPE="submit">
</form>
</div>
<!-- End Search Content --> 

Let me know if anything else would help. I'm staring down the barrel of some serious data entry if I can't get this to work.

EDIT: Just fixed up the typo in the curl request to avoid confusion.

3
  • Have you tried capturing the POST from the web page using a tool like Wireshark? That may be more accurate than trying to decipher the HTML. Feb 2, 2011 at 6:06
  • Might be a typo here, but &sumbit should be &submit (if it's even needed). Also, those hidden elements value may actually be set with javascript based on user action.
    – Bill N
    Feb 2, 2011 at 6:13
  • @Justin Morgan - No I haven't I'll try that. @Bill N - Yeah that would've been a typo copying it across. If that is the case, can I still use curl to submit this post, or will it prevent it? Feb 2, 2011 at 21:19

3 Answers 3

1

The problem is that curl's --data parameter sends the text exactly as you provide it, but you are not url-encoding reserved characters (specifically, the [ and ] characters), so the server will not be able to match up the value of the helpdocid field correctly.

On a side note, don't include the "submit=Go" field in your posted data. Only fields with both names and values get submitted.

Try this:

curl --data "searchcontent=&helpdocid=%5Bdocid%5D&searchtext=121315" http://www.ausstats.abs.gov.au/ausstats/searchclass.nsf/(Searchattmnt)?openagent
1
  • I used @Justin Morgan's suggestion and ran wireshark while I performed the request. This gave me the values that I required. In the end this was the curl request I used that worked: curl --data "searchcontent=1220.0Data%2BCubes-SuperTable25.06.091&helpdocid=17441354B1295875CA2571E6000AE9A0&searchtext=121318" http://www.ausstats.abs.gov.au/ausstats/searchclass.NSF/(Searchattmnt)?openagent Feb 3, 2011 at 0:34
1

I used @Justin Morgan's suggestion and ran wireshark while I performed the request. This gave me the values that I required. In the end this was the curl request I used that worked:

curl \
-d "searchcontent=1220.0Data%2BCubes-SuperTable25.06.091" \
-d "helpdocid=17441354B1295875CA2571E6000AE9A0" \
-d "searchtext=121318" \
http://www.ausstats.abs.gov.au/ausstats/searchclass.NSF/(Searchattmnt)?openagent
0

The Mozilla plug-in livehttpheader shows the string of urlencoded text used to post to the server.

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.