6

I am working on a challenging problem : finding a solution to get data after a booking process. Basically, I have a page with a form (SLIM FORM), that I need to automatically fill with informations coming from provider form (e.g. easyjet.com or hotels.com, any booking site basically). For instance : https://secure.booking.com/hotel/es/royal.html?sid=1c2bab12a0c64a541728840f52cd6401;errorc_checkin_invalid=checkin;errorc_intro_error_message_invalid=intro_error_message;errorv_stage=1;errorv_checkin=2011-07-05;errorv_hotel_id=90228;errorv_installment_count=1;errorv_hostname=www.booking.com;errorv_nr_rooms_9022801_80638194_0=1;errorv_interval=1 the information in my Booking is what i need to get.

enter image description here

I made some tests and here are what I found out, for now :

It's not possible to have both on the same page, because with cURL, there is no communication with the external server, and with iframes, it leaves the page ASAP the src of the iframe changes.

So, I decided that the booking process should happen on a dedicated page, in the domain of the booking provider (easyjet.com...)

1) Am I right to consider performing the booking on the real site, or is there a way to include the external website on my page and perform the whole process of booking in it (basically filling forms on departure, arrival date etc...)?

If not possible, I made some tests with cURL and came to this conclusion :

_ I will have to define fitted regex for each provider, and I am under the impression that some have mechanisms to identify cURL and block it. (e.g. lufthansa.com) But it works quite well with others ( booking.com )

I have 2 additionnal questions :

2) Are there better solutions than cURL to parse some HTML in a page (especially since it doesn't work if the URL doesn't include sessionID)? I was thinking maybe of using something like Selenium...

3) How can I trigger my cURL parsing on an other tab or window? (I was thinking about a system similar to bookmarks that can trigger some JavaScript code)

Thanks for your answers and sorry for the length :-)

Update : Based on answers I received, here are fresh thoughts : for big providers (easyjet, hotels.com etc...), I will use an API if available. For small providers (e.g. http://www.hotel-gare-clermont.com/en,1,6217.html ), I think the proxy solution is worth another one, and I won't receive any complaints on legal issues from "Hotel de la Gare", while adding visibility to those small providers. What do you think?

1
  • 3
    Do keep in mind that you should try to talk to the providers - they may be able to offer an API interface, they might object and block the server IP (although this is business for them, so it's unlikely)
    – Phil Lello
    May 4, 2011 at 0:58

3 Answers 3

7
+75

1) This is possible, but it has the side effect of being borderline illegal. You cannot just scape providers forms and reserve their pages in an iframe. If providers caught you doing it you would likely be sued.

What you need is a partnering agreement with the various providers. with this agreement, they would likely open up an API (Application Programming Interface) for you to use. This would allow you to more directly query their site and make bookings in a clean and approved way.


2) cURL is a great library, which does the job of fetching web pages very well. There are many examples around the internet for fetching a page to a string. In terms of parsing that string, in an ideal world you could use an XML parser. Unfortunately HTML pages are very badly constructed, which makes them difficult to parse. Most coders, when they have to parse HTML chunks tend to use regular expressions.

In order to get the session ID, your first cURL request should be to a login form on example.com. Fake the submit of a login form by trying to get http://example.com?username=bob&pass=secret. You can check for a valid login by looking for the text "successful login" or similar in the server response. You can get the session ID (if it is a cookie) from the response headers. Subsequent cURL requests should send your cookie.


3) cURL operates on server side, so has absolutely no knowledge of your tabs that are open. You could use Javascript to query tabs, but I bet most browsers will not allow you to do this for security reasons.

3
  • 1
    +1 for borderline illegal. this has also technical implications. since the data isn't generated to be machine-readable, they're free to change the output format. so using interfaces you're intended to use is not only more legal but also less prone to breaking should some layout change.
    – Nicolas78
    May 2, 2011 at 10:32
  • This is not intended to 'steal' any data from providers. I am working for a big GDS, the website is a GUI for Travel Agencies to perform booking (but not only), that's why they need to perform booking in easyjet.com or hotels.com and those data are required to do some statistical treatment. But contacting every provider that allows to book flight or hotel is a virtually impossible mission. But I agree when an API is available it's definitely the best solution. Does the proxy solution involve such legal issues? May 3, 2011 at 13:06
  • 2
    Don't get too caught up on the legal considerations: in my opinion, the practical ones are far more alarming! What happens when a prominent provider like hotels.com changes some of their forms? What happens if easyjet.com adds a captcha to one of theirs? Or what if Ryanair starts blocking your IP? Is it really in the interest of a big GDS to have a core piece of infrastructure built around a connection that can never be more than a weak hack?
    – Schwartzie
    May 4, 2011 at 2:39
6

Sending the user direct to the provider is a much more reliable solution, because you give your user control of the process. But of course, you lose control of the process :)

Alternatively, you have to create a proxy on your server that queries the site on behalf of your user:

end-user         yourdomain     easyjet
   |                 |             |
   |-----search----->|             |
   |<--booking form--|             |
   |---user's data-->|             |
   |                 |---forward-->|
   |                 |<--result----|
   |<--pass to user--|             |
   |                 |             |
   v                 v             v

To the end-user, the booking happens with you; to easyjet/lufthansa/whoever, you appear to be a customer. The problem is, each website is different, and you will have a lot of work adapting your system to what every (or most) site requires, and as you have already noticed, airlines do not want you to take their custom. That's why many brokers' sites (kelkoo, gocompare...) started out doing what you're planning, but ended up as glorified advertising.

4
  • Thanks for this very clear answer (sorry i cannot vote up cause of the bounty ;-) ) PS : on gocompare.com, it seems you can perform all the booking without going on the provider site... Apr 28, 2011 at 12:23
  • didn't realise gocompare was that good. I tried them for insurance, and all I got was a lot of data collected about me and very little accurate information back. But air travel isn't quite as complicated.
    – boisvert
    Apr 28, 2011 at 13:22
  • And what about the legal issues? Setting a proxy requires some authorization or something? May 3, 2011 at 14:26
  • 1
    There's no legal issue over the technical solution as such, but the owners of a website have rights over their work - visual design, commercial brand, user experience, and by engineering a system so that users systematically bypass the sellers' website, you may be breaching those rights. As xconspirisist says below, providers would sue, and whether you'd succeed would depend on details of what you do with their work.
    – boisvert
    May 3, 2011 at 20:48
0

The best method to use the API from the providers. The next comes cURL or IFrame.

You may use JSON/AJAX which supports cross site requests and you can manipulate the output.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.