Linux downloader customizable

Bash script to download set of files based on the list. You will find this a very useful script as you try to use this in many other programs. I will provide some examples to use this script to download images from a site based on keywords. This below script requires single argument which includes list of the HTML links to be downloaded.

#!/bin/bash

# Loop through each link
while [ `find "$1" -size +0` ]
do
url=`head -n1 "$1"`
head -n1 "$1" > temp
wget -Fi temp
ERRORCODE=$?

# if no error downloading the link
if [ $ERRORCODE -eq 0 ]; then
sed -si 1d "$1"

# if error downloading the link try later
# remove from the top and put it in the bottom
else
echo "ERROR: could not get url" 1>&2
echo "$url" >> "$1"
sed -si 1d "$1"
fi
done 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s