Application to download web site contents

Franc Zabkar wrote in
:

On Mon, 15 Aug 2011 09:09:59 -0500, Lostgallifreyan
put finger to keyboard and composed:

Sjouke Burry wrote in
:

Wget does need a bit of supervision, it surprised me once by trying to
download a complete .GOV website....
I stopped it after the first 15 GB
(usage: WGET -r url)

Ouch. Without knowing more than basic recursion switching, I'd aim to
find a small branch of the site I wanted intact, along with a few other
branches, and make a download list for them.

Thanks very much to all for your suggestions.

I found that the following command line worked well:

wget -r -nc -l 10 -np URL -o capture.log

... where -l = little L

I needed two attempts because the default number of levels (5) wasn't
enough.

The "-np" switch excludes parent directories, and "-nc" (no clobber)
prevents overwriting of files which have already been downloaded.

- Franc Zabkar

Useful. I thought there was a way to ignore parent directories, though I'd
thought that might be a default. (Although I once caught it making merry with
far more than I asked, once, too. But not 15 GB worth. I've used the -nc
switch which is very important for resuming a batch, it lets me shut down any
time I want, though it needs a resume switch (I forget what, but I think
there is one for use with -nc.

Franc Zabkar wrote in
:

wget -r -nc -l 10 -np URL -o capture.log


From the manual for v1.114, "Also beginning with Wget 1.7, if you use ‘-c’ on
a file which is of equal size as the one on the server, Wget will refuse to
download the file and print an explanatory message."

-c is the continue switch so this can replace -nc with same effect, but with
the added benefit of finishing large stuff at any time after an instance of
WGET was aborted, so long as the server allows resumption. If it doesn't, the
partial download is left untouched.

Franc Zabkar wrote:
On Mon, 15 Aug 2011 09:09:59 -0500, Lostgallifreyan
put finger to keyboard and composed:

Sjouke Burry wrote in
:

Wget does need a bit of supervision, it surprised me once by trying
to download a complete .GOV website....
I stopped it after the first 15 GB
(usage: WGET -r url)

Ouch. Without knowing more than basic recursion switching, I'd
aim to find a small branch of the site I wanted intact, along with a
few other branches, and make a download list for them.

Thanks very much to all for your suggestions.

I found that the following command line worked well:

wget -r -nc -l 10 -np URL -o capture.log

... where -l = little L

I needed two attempts because the default number of levels (5) wasn't
enough.

The "-np" switch excludes parent directories, and "-nc" (no clobber)
prevents overwriting of files which have already been downloaded.

- Franc Zabkar

You are one savvy person and I always appreciated you input in differeent
ngs.
Thanks again for all you ideas and input.
Buffalo