Download in Geek Style: Use Wget (Part 1)

| Comments

I was getting questions about wget from my friends who have recently joined Linux. So, I am writing this article illustrating wget, for an average Linux user, in a little Geeky way.
In this article we will discuss several wget commands and split this article into two parts:
Basic Use 
Advanced Use

Starting with Wget – Basic Use:

Starting a Download with wget

It is easy, just write wget in terminal followed by the url of the file to download (download link)
wget "download link" (quotes not needed)

Downloading Multiple Files

Same as above command, just put the download links of all files one after other, followed by a space. Wget assumes everything written after “wget” command to be an URI (unless it starts with “-” sign which is for command switches\mods), and try to download all of them on “first come first serve” basis.
wget  "first link" "second link" "third link" and so on...(quotes not needed)

Logging Output(-o and -a)

When you start downloading with wget, it gives out the output (progress information) to console screen.  If you want you can get this output transferred to a specific file.
wget -o logs.txt "download link"
If you are continuing an download and don’t want the output logs to be rewritten, you can use -a switch. Unlike -o, -a switch appends the output logs into the desired text file. -o rewrites the file every time the command is run.
wget -a logs "download link"

Continuing a download (-c or –continue)

If a download couldn’t get completed when it was started, it can be continued from where it was left (only if server allows continuing downloads). Wget will scan the file in local storage and on the server and will start downloading from where the download ended last time and continue until both files are of equal size (in other words, download is complete).
wget -c "download link"
If the server don’t allow continuing downloads, wget will refuse the new download and this can ruin the contents of partially downloaded file.

Downloading a list of files (-i or –input-file)

Wget can get a text or even html file as input and will download all the links given in the file with -i switch. For example, if file is a plain text file containing one URI each line, following command is used
wget -i "list of files" (without quotes)
and if input file is an HTML file containing links to download, -F (–force-html) flag has to be used with above command. e.g.
wget -i "list of files" -F (without quotes)
When downloading multiple files to the same directory, what wget will do with the new files depends on many things. By default (if no extra switches like -N,-nc,-p or -r are used) wget will re-download the existing files and will  rename them as file, file.1, file.2 etc. You can change this behavior to what you want with following switches:

Timestamps (-N switch)

Using time stamps force wget to check the files which are downloaded for their creation time and then comparing it with with the files on the server. If  time stamp matches and both files are of same files, wget will skip that file and move on to next file available in the list.  If a new version of file is available on the server, then previously downloaded file is overwritten and new file is downloaded. This is very useful when downloading a long list of files.
wget -N "download link or links"
No Clobbering (-nc or –no-clobber)
When running with -nc or –no-clobber, wget will not download multiple newer copies of the file. Unlike -N switch, it will not download the file if newer versions of the file are available on the server, the previously downloaded version is kept. Remember, it is not clobbering which is prevented with -nc, it is multiple version saving which is prevented.
wget -nc “download link or links”

Saving Downloaded Files

By default Wget will save all the downloaded files in the same directory from which the terminal is running (usually your home folder). In order to direct the downloaded files to get saved in other location, use “-P” switch
wget -P “path where to save the download” “download link or list”

Stop Wget 

When you are running wget from Console (Terminal in other words), simply pressing “ctrl + C ” will stop wget. But in case you are running wget in background (with -b switch) or you have chosen to log the output with -o or -a switches, OR you closed the Terminal window by mistake, wget will go in background and can not be closed with ctrl + C. 

In that case following command will help:
killall wget
But this will stop all the instances of wget. In case you are running multiple wget instances, you will first need to find the PID of all wget commands and then close those you want.
Use this command to find the PIDs of all the running processes:
ps -A
This will show you a list of all the processes running. Find the wget processes and note their PIDs. Then use following command to kill specific process:
kill 9 "PID" where PID is the Process ID number, without quotes 
Well ! We can make the job easier. Pass following command:
(ps -A)|grep wget
and you will see something like this:

4860 ?        00:00:00 wget
4860 is PID of wget command (I have only one instance running at the moment, so only one). Use “kill 9 PID” command and it is all done.
Or even easier, use following:
pgrep wget (for getting PID of wget) OR pkill wget (to instantly stop wget)
 This list can go as long as we want. People get confused when they have to close wget running in background, mainly because they don’t know how to use command line.
But, this is Linux. There are a billion different ways to do one task…BELIEVE IT…: )

Some things you need to know

  • Wget saves downloaded files in the same directory from where Terminal is running. It is your Home Folder by default. You can change it using “cd” command or with above stated command (-P).
  • Wget will not stop even if you close the Terminal window, it simply goes to background and carry on its job until it is finished. It will not stop even if you log out and switch user without stopping it.
  • Wget will not automatically resume previously started downloads, you have to tell it to continue download with “-c” flag.
  • Wget has a huge set of commands which when run in right combinations can do variety of jobs, making your life easy. We will look at some of them in later article.

Circle Beat Of The Geek on Google Plus
OR Like us on Facebook  OR Follow on Twitter