Wget Command in Linux with Examples

Introduction

GNU wget is a command-line program that allows you to download files from the internet. You can use wget to download files using HTTP, HTTPS, and FTP protocols. wget has a lot of features, including the ability to download multiple files, resume downloads, limit bandwidth, recursive downloads, download in the background, mirror a website, and more.

This tutorial uses practical examples and extensive descriptions of the most frequent options to demonstrate how to use the wget command. We will also address a few FAQs on wget Command in Linux.

Installing wget

Most Linux distributions now come with the wget package pre-installed.

Open your terminal, type wget, and press enter to see if the wget package is installed on your machine. The system will print wget: missing URL if you have wget installed. Otherwise, wget command not found will be shown.

If wget isn't already installed, you can do so using your distro's package manager.

Installing Wget on Ubuntu and Debian

sudo apt install wget

Installing Wget on CentOS and Fedora

sudo yum install wget

Wget Command Syntax

Let's go through the basics of the wget command before we get into how to use it.

The utility expressions for wget are as follows:

wget [options] [url]

options - The Wget options
url - The URL of the file or location you wish to download or synchronize.

How to Download a File with `wget`

When called without any options, wget will download the resource supplied in the [url] to the current directory in its most basic form.

We'll download the Linux kernel tar bundle in the following example:

wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.17.2.tar.xz

Wget begins by determining the domain's IP address, then connects to the remote server and begins the transfer.

Wget displays a progress bar beside the file name, file size, download speed, and expected download time during the download. You can find the downloaded file in your current working directory once the download is complete.

Use the -q option to disable the output.

wget will overwrite the file if it already exists and add .N at the end of the file name.

Saving the Downloaded File Under Different Name

Pass the -O option followed by the desired name to save the downloaded file under a different name:

wget -O latest-hugo.zip https://github.com/gohugoio/hugo/archive/master.zip

Instead of the original name, the command above will store the latest hugo zip file from GitHub as latest-hugo.zip.

Downloading a File to a Specific Directory

Wget saves the downloaded file in the current working directory by default. Use the -P option to save the file to a specific location:

wget -P /mnt/iso http://mirrors.mit.edu/centos/7/isos/x86_64/CentOS-7-x86_64-Minimal-2009.iso

Wget is told to save the CentOS 7 iso file to the /mnt/iso directory by the command above.

Limiting the Download Speed

Use the --limit-rate option to set a download speed limit. By default, speed is expressed in bytes per second. Add k for kilobytes, m for megabytes, and g for gigabytes to the end of the number.

The following command will download the Go binary at a maximum of 1MB per second:

wget --limit-rate=1m https://dl.google.com/go/go1.10.3.linux-amd64.tar.gz

When you don't want wget to use all of your available bandwidth, use this option.

Resuming a Download

The -c option allows you to resume a download. This is important if your connection stops during a huge file download, and you want to continue the previous one instead of starting over.

We'll use the following example to resume the Ubuntu 18.04 iso download:

wget -c http://releases.ubuntu.com/18.04/ubuntu-18.04-live-server-amd64.iso

wget will start the download from the beginning and overwrite the old file if the remote server does not support resuming downloads.

Downloading in Background

Use the -b option to download in the background. We're downloading the OpenSuse iso file in the background in the following example:

wget -b https://download.opensuse.org/tumbleweed/iso/openSUSE-Tumbleweed-DVD-x86_64-Current.iso

The output is forwarded to the wget-log file in the current directory by default. Use the tail command to keep track of the download's progress:

tail -f wget-log

Changing the Wget User-Agent

The remote server may be set to block the Wget User-Agent when downloading a file. To imitate a different browser in instances like this, use the -U option.

wget --user-agent="Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" http://wget-forbidden.com/

Using the command above, Firefox 60 will request the page from wget-forbidden.com.

Downloading Multiple Files

Use the -i option, followed by the path of a local or external file containing a list of the URLs to be downloaded, to download many files at once. Each URL should be on its own line.

Using the URLs indicated in the linux-distros.txt file, the following example explains how to obtain the Arch Linux, Debian, and Fedora iso files:

wget -i linux-distros.txt

http://mirrors.edge.kernel.org/archlinux/iso/2018.06.01/archlinux-2018.06.01-x86_64.iso
https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-9.4.0-amd64-netinst.iso
https://download.fedoraproject.org/pub/fedora/linux/releases/28/Server/x86_64/iso/Fedora-Server-dvd-x86_64-28-1.1.iso

URLs will be read from the standard input if you specify - as a filename.

Downloading via FTP

Specify the username and password as shown below to download a file from a password-protected FTP server:

wget --ftp-user=FTP_USERNAME --ftp-password=FTP_PASSWORD ftp://ftp.example.com/filename.tar.gz

Creating a Mirror of a Website

Use the -m option with wget to create a mirror of a website. By following and downloading all internal links as well as the website resources, this will build a complete local duplicate of the website (JavaScript, CSS, Images).

wget -m https://example.com

You'll need to provide a few extra parameters to the command above if you wish to browse the downloaded page locally.

wget -m -k -p https://example.com

The -k option instructs wget to transform links in downloaded documents so that they can be viewed locally. Wget will download all essential files for displaying the HTML page if you use the -p option.

Skipping Certificate Check

Use the --no-check-certificate option to download a file over HTTPS from a server with an incorrect SSL certificate:

wget --no-check-certificate https://domain-with-invalid-ss.com

Downloading to the Standard Output

wget will (flag -q) download and output the newest WordPress version to stdout (flag -O -) and pipe it to the tar utility, which will extract the archive to the /var/www directory in the following example.

wget -q -O - "http://wordpress.org/latest.tar.gz" | tar -xzf - -C /var/www

FAQs on wget Command in Linux

How do I use the `wget` command?

Using the wget command is simple. Open a terminal and type wget followed by the URL of the file you want to download. For example, wget https://example.com/file.txt will download the file "file.txt" from the specified URL.

Can the `wget` command resume interrupted downloads?

Yes, wget supports resuming interrupted downloads. If the remote server supports resume functionality, wget can continue the download from where it was paused or interrupted using the -c or --continue option.

Does the `wget` command follow links to download entire websites?

Yes, wget can recursively download entire websites by following links. Use the -r or --recursive option to enable recursive downloading. Additionally, you can use options like -np or --no-parent to restrict downloading to a specific directory.

Can `wget` limit the download speed to prevent excessive bandwidth usage?

Yes, wget allows you to limit the download speed using the --limit-rate option. Specify the desired speed value in bytes per second or with the suffixes 'k' or 'M' for kilobytes or megabytes per second, respectively.

Can the `wget` command convert links to work offline in downloaded HTML pages?

Yes, wget provides the --convert-links option, which modifies links in downloaded HTML pages to adapt them for offline browsing. It rewrites links to point to locally downloaded content, allowing you to browse the page without an internet connection.

Does `wget` support authentication for downloading password-protected files?

Yes, wget supports various authentication methods, including basic authentication and cookies. You can use the --user and --password options to provide the appropriate credentials for authentication.

Can `wget` download multiple files simultaneously?

Yes, wget can download multiple files simultaneously when provided multiple URLs as arguments. It spawns multiple connections to download the files concurrently, which can help expedite the downloading process.

Conclusion

You can use wget to download numerous files, resume incomplete downloads, mirror websites, and mix and match Wget features to meet your specific needs.

Visit the GNU wget Manual page to learn more about wget.

If you have any queries, please leave a comment below and we’ll be happy to respond to them.