Web Scrape Booking.com with JavaScript

This blog is originally posted to crawlbase blog.

How to Scrape Booking.com, a site that started in 1996, has become a significant endeavor in the travel and accommodation world. Millions of people use it for planning their trips because it’s known for being easy to use. You can do a lot on the platform, like booking hotels, finding vacation rentals, and getting flights and cars. The website is easy to navigate and a popular choice for travelers looking for different places to stay around the world.

In this blog post, we’ll look at how to use the Crawlbase Crawling API and JavaScript to get data from Booking.com. We’ll cover everything from getting hotel reviews and prices to capturing detailed location data. The main idea is to smoothly work through the challenges of Booking.com and share tips on effective scraping. Let’s explore the world of web scraping and see how Crawlbase Crawling API and JavaScript can help us to scrape useful data from Booking.com.

Understanding Booking.com Structure

HTML Structure of Booking.com Pages
Identifying Target Data Elements

Prerequisites for Scraping Booking.com

JavaScript Basics
Crawlbase API token
Setting Up the Environment

Fetching Booking.com Search Results Page

Scrape Booking.com Property Data

Data Processing and Storage

Extracting information from Data
Cleaning and Transforming Data
Saving Data in Different Formats

Ensuring Robust Scraping

Error Handling and Logging
Respecting Robots.txt
Avoiding IP Blocks and Captchas

Final Words

Frequently Asked Questions

Understanding Booking.com Structure

Booking.com html structure

HTML Structure of Booking.com Pages

In this section, we’ll look at how Booking.com arranges its content using HTML. It’s important to understand the Document Object Model (DOM) to scrape information effectively.

Overview of Booking.com DOM
- Check out the structure from top to bottom.
- Find the main HTML tags and features that shape the page.
Class and ID Naming Conventions
- Look into the importance of class and ID names.
- Get information using these element identifiers.
Nested Elements and Relationships
- See how elements are placed inside one another.
- Move through parent and child elements for thorough data extraction.

Identifying Target Data Elements

This part is all about figuring out the specific data you want to pull from Booking.com pages.

Defining Scraping Objectives
- Outline what information you need.
- Make clear goals for effective scraping.
Inspecting Elements with Browser Developer Tools
- Use browser tools to look at elements closely.
- Find and confirm the data you want accurately.
XPath and CSS Selectors for Targeting
- Introduce XPath and CSS selectors.
- Learn to make effective selectors to get exactly what you need.

Prerequisites for Scraping Booking.com

JavaScript basics

Before we start web scraping on Booking.com, it’s important to know some basics about JavaScript, the programming language we’ll be using. Get familiar with concepts like DOM manipulation, which helps us interact with different parts of a webpage, making HTTP requests to get data, and handling asynchronous operations for smoother coding. Knowing these basics will be really helpful as we work on our project.

Crawlbase API token

Now, let’s talk about getting the token we need from Crawlbase to make our Booking.com scraping work.

First, login to your Crawlbase account on their website.

Once you’re logged in, find the page called “Account Documentation“ inside your Crawlbase dashboard.

Crawlbase dashboard

Look for a code called “JavaScript token” on that page. Copy this code – it’s like a secret key that makes sure our code can talk to Booking.com properly.

Now that you have this token, you’re all set to finish setting up the other parts of our Booking.com scraping project for it to work smoothly.

Setting Up the Environment

Now that we have everything ready, let’s set up the tools we need for our JavaScript code. Follow these steps in order:

Create Project Folder:

Open your terminal and type mkdir booking_scraper to make a new folder for your project. You can name this folder whatever you want.

mkdir booking_scraper

Navigate to Project Folder:

Type cd booking_scraper to go into the new folder. This helps you manage your project files better.

cd booking_scraper

Create JavaScript File:

Type touch scraper.js to create a new file called scraper.js. You can name this file differently if you prefer.

touch scraper.js

Install Crawlbase Package:

Type npm install crawlbase to install a package called Crawlbase. This package is important for our project as it helps us interact with the Crawlbase Crawling API, making it easier to get information from websites.

npm install crawlbase

By doing these steps, you’re setting up the basic structure for your Booking.com scraping project. You’ll have a dedicated folder, a JavaScript file to write your code, and the necessary Crawlbase tool to make the scraping process smooth and organized.

Fetching Booking.com search result page

Once you have your API credentials and installed the Node.js library for web scraping, it’s time to work on your “scraper.js” file. Now, let’s pick the Booking.com search results page you want to scrape. For this example, we’re focusing on the Booking.com : San Francisco, California – Search Results.

To make the Crawlbase Crawling API work, you need to set up specific parameters and endpoints. First, ensure you’ve made the “scraper.js” file as explained earlier. Then, copy and paste the script below into that file. Lastly, run the script in your terminal with the command “node scraper.js.”

const { CrawlingAPI } = require('crawlbase');

const crawlbaseToken = 'YOUR_CRAWLBASE_JS_TOKEN';
const api = new CrawlingAPI({ token: crawlbaseToken });

const bookingPageURL =
'https://www.booking.com/searchresults.html?ss=San+Francisco%2C+California%2C+United+States&checkin=2023-12-25&checkout=2023-12-31&group_adults=2&no_rooms=1&group_children=0&selected_currency=USD';

api.get(bookingPageURL).then(handleCrawlResponse).catch(handleCrawlError);

function handleCrawlResponse(response) {
if (response.statusCode === 200) {
console.log(response.body);
}
}

function handleCrawlError(error) {
console.error(error);
}

HTML Response:

HTML response from Booking.com

Scrape Booking.com property data

In this example, we’ll guide you on how to scrape property information from a Booking.com search results page. This includes details like property thumbnail image, name, location, rating, reviews, and prices. To achieve this, we’ll use two JavaScript libraries: cheerio, commonly used for web scraping, and fs, which is often used for file system operations.

The provided JavaScript code utilizes the Cheerio library to scrape property details from a Booking.com search results page. It reads HTML content which you extracted from “scraper.js,” in previous step, and then processes it with Cheerio, and collects information such as property name, price, rating, reviews, and image URL. The script goes through each property listing and saves the data in a JSON array.

const fs = require('fs');
const cheerio = require('cheerio');

const htmlContent = fs.readFileSync('response-booking.html', 'utf8');
const $ = cheerio.load(htmlContent);

const allLocationData = [];

const cardSelector = '[data-testid="property-card"]';

$(cardSelector).each((index, card) => {
const currentCard = $(card);

const extractText = (selector) => currentCard.find(selector).text().trim();

const locationData = {
thumbnail: currentCard.find('[data-testid="image"]').attr('src'),
name: extractText('[data-testid="title"]'),
address: extractText('[data-testid="address"]'),
roomType: extractText('ul.ba51609c35 li.a6a38de85e div.abf093bdfe'),
rating: extractText('[data-testid="review-score"] div.a3b8729ab1.d86cee9b25'),
review: extractText('[data-testid="review-score"] div.abf093bdfe.f45d8e4c32'),
price: extractText('span[data-testid="price-and-discounted-price"]'),
additionalFees: extractText('div[data-testid="taxes-and-charges"]'),
availabilityLink: currentCard.find('[data-testid="availability-cta-btn"]').attr('href'),
};

allLocationData.push(locationData);
});

const jsonData = JSON.stringify(allLocationData, null, 2);

console.log(jsonData);

fs.writeFileSync('allLocationData.json', jsonData, 'utf8');

JSON Response:

Data Processing and Storage

Extracting Information from Data

After getting the data, the next step is to pull out the important information effectively. Learn techniques like regular expressions, DOM parsing libraries, and advanced HTML parsing methods. Adapt your approach to Booking.com’s specific structure to ensure your scraper can handle different data types and structures. This guarantees accuracy and completeness for insightful analysis.
Cleaning and Transforming Data

Raw scraped data often has issues like inconsistencies, missing values, or unnecessary information. This section covers ways to clean and process data, including advanced techniques like spotting outliers and managing duplicate entries. By dealing with these problems, you ensure your data is not only clean but also ready for various analytical tasks, like statistical analysis, visualization, or machine learning.
Saving Data in Different Formats

Once the data is extracted and cleaned, it’s crucial to choose the right storage format. Explore options like JSON for flexibility, CSV for simplicity, and databases like MySQL or MongoDB for power. This section goes beyond the basics, offering insights into optimizing data storage for quick retrieval, efficient querying, and maintaining data integrity. Picking the right storage format is essential for the long-term use and accessibility of your scraped data.

Ensuring Robust Scraping

Error Handling and Logging

Web scraping has its difficulties, and it’s crucial to handle errors effectively for a robust scraping process. Learn to recognize and predict potential errors in your scripts to address issues proactively. Incorporate thorough error handling and detailed logging to minimize data loss and gain insights into your scraper’s performance for ongoing optimization.
Respecting Robots.txt

Respecting a website’s robots.txt file is important for ethical scraping and a sustainable strategy. Understand the guidelines, including specific directives like those set by Booking.com. Configure your scraper to follow these rules not only for ethical reasons but also to establish a positive relationship with the website, preventing IP bans and ensuring long-term access to valuable data.
Avoiding IP Blocks and Captchas

Facing IP blocks and captchas is common in web scraping. Learn strategies to avoid detection and handle IP blocks gracefully, such as rotating IP addresses or using proxy servers. Understand how to navigate captchas effectively, either through automated solving or by incorporating human interaction into your scraping workflow. A thoughtful approach to these challenges is essential for maintaining discretion and ensuring uninterrupted scraping activities.

Final Words

This guide provides you information and tools to scrape data from Booking.com using JavaScript and the Crawlbase Crawling API. You get different information from Booking.com, like details about properties such as their name, where they are, how they’re rated, what people are saying in reviews, how much they cost, and small pictures called thumbnail images. Whether you’re new to web scraping or know a bit already, these ideas help you get going. If you want to try scraping on other online platforms like AirBnb, Expedia, Hotels.com, we have more guides for you to look into.

If you have any questions or need assistance with your scraping projects, the Crawlbase support team is available 24/7. Feel free to reach out to us for guidance, clarification, or any help you might need in your web scraping projects.

Frequently Asked Questions

What are the drawbacks of using the Booking API?

Using the Booking API has some drawbacks. First, it might not offer the same access or features as the Booking.com website. Changes to how the API works can make it tricky to integrate, and you may need to update it frequently. Dealing with authentication and API keys can be complicated. Users might have restrictions on how much data they can use, and there could be costs linked to the number of times they make requests to the API.

What tools/libraries can I use to scrape Booking.com with JavaScript?

To scrape Booking.com with JavaScript, you can use various tools and libraries. Some popular options are Puppeteer, Cheerio, and Nightmare.js. Puppeteer is a headless browser that allows automated interactions, while Cheerio is a lightweight library for parsing and manipulating HTML. Nightmare.js combines the functionality of both Puppeteer and Cheerio. These tools enable you to automate the process of navigating Booking.com, extracting data, and handling dynamic content, making it easier to scrape information from the website using JavaScript.

Can I share or sell the scraped data obtained from Booking.com?

No, sharing or selling scraped data obtained from Booking.com is likely to be a violation of Booking.com’s terms of service. It may also be illegal and unethical. Websites typically have policies against unauthorized data scraping and commercial use of their data. It’s important to respect the terms and conditions of Booking.com and obtain explicit permission if you intend to use or share the scraped data for any purpose other than personal use. Engaging in such activities without permission can lead to legal consequences.

How Can I Scrape Booking.com Without Getting Blocked?

If you want to scrape Booking.com without facing blocks, you can use the Crawlbase Crawling API. This API includes rotating proxies that help to control the frequency of your requests. With over 30+ Geo-locations, the Crawlbase Crawling API allows you to distribute your requests strategically. Additionally, you can use AI solutions to navigate around blocks and captchas. It’s essential to keep your scraping code up-to-date to handle any changes on the website. Crawlbase provides a reliable infrastructure and consistent support to make the scraping process better and reducing the chances of being blocked.