Application to download web site contents

In the past I used a small utility that could download the content of
a web site starting from a particular URL. You could specify whether
or not to follow off-site links, and you could limit the number of
levels. Unfortunately I can't remember it's name.

Could anyone recommend a similar application? I'm aware of Leech, but
the free version appears to be limited in some way.

- Franc Zabkar
--
Please remove one 'i' from my address when replying by email.

Franc Zabkar wrote:

In the past I used a small utility that could download the content
of a web site starting from a particular URL.

I don't know of any "small" utility that can grab web-content, other
than wget.exe.

You could specify whether or not to follow off-site links, and you
could limit the number of levels. Unfortunately I can't remember
it's name.

Could anyone recommend a similar application? I'm aware of Leech,
but the free version appears to be limited in some way.

HT Track:

http://www.httrack.com/

==========
HTTrack is a free (GPL, libre/free software) and easy-to-use offline
browser utility.

It allows you to download a World Wide Web site from the Internet to a
local directory, building recursively all directories, getting HTML,
images, and other files from the server to your computer. HTTrack
arranges the original site's relative link-structure. Simply open a page
of the "mirrored" website in your browser, and you can browse the site
from link to link, as if you were viewing it online. HTTrack can also
update an existing mirrored site, and resume interrupted downloads.
HTTrack is fully configurable, and has an integrated help system.
===========

It might work on win-98 as is, and probably almost certainly with
kernelEx.

In message , Franc Zabkar
writes:
In the past I used a small utility that could download the content of
a web site starting from a particular URL. You could specify whether
or not to follow off-site links, and you could limit the number of
levels. Unfortunately I can't remember it's name.

Webstripper? I used to have (I think) that (possibly in dial-up days!).

Could anyone recommend a similar application? I'm aware of Leech, but
the free version appears to be limited in some way.

- Franc Zabkar

(Sorry, no; there's still something called webstripper, but it costs $40
or 30 Euros.)
--
J. P. Gilliver. UMRA: 1960/1985 MB++G.5AL-IS-P--Ch++(p)Ar@T0H+Sh0!:`)DNAf

.... "Peter and out." ... "Kevin and out." (Link episode)

Franc Zabkar wrote in
:

In the past I used a small utility that could download the content of
a web site starting from a particular URL. You could specify whether
or not to follow off-site links, and you could limit the number of
levels. Unfortunately I can't remember it's name.

Could anyone recommend a similar application? I'm aware of Leech, but
the free version appears to be limited in some way.

- Franc Zabkar

Total recommendation for WGET. Commandline tool, indispensible stuff..
http://www.gnu.org/s/wget

It won't do multiesegment downloads,. but it's both polite and remorseless,
just what we need.

Whether it limits the extent of remote linkage I don't know, but I think it
does, it certainly has plenty of intelligent control on it.

Lostgallifreyan wrote:
Franc Zabkar wrote in
:

In the past I used a small utility that could download the content of
a web site starting from a particular URL. You could specify whether
or not to follow off-site links, and you could limit the number of
levels. Unfortunately I can't remember it's name.

Could anyone recommend a similar application? I'm aware of Leech, but
the free version appears to be limited in some way.

- Franc Zabkar

Total recommendation for WGET. Commandline tool, indispensible stuff..
http://www.gnu.org/s/wget

It won't do multiesegment downloads,. but it's both polite and remorseless,
just what we need.

Whether it limits the extent of remote linkage I don't know, but I think it
does, it certainly has plenty of intelligent control on it.

Wget does need a bit of supervision, it surprised me once by trying to
download a complete .GOV website....
I stopped it after the first 15 GB
(usage: WGET -r url)

"Franc Zabkar" wrote in message
...
In the past I used a small utility that could download the content of
a web site starting from a particular URL. You could specify whether
or not to follow off-site links, and you could limit the number of
levels. Unfortunately I can't remember it's name.

Could anyone recommend a similar application? I'm aware of Leech, but
the free version appears to be limited in some way.


Mr. Franc Zabkar

Try looking here First:

http://www.majorgeeks.com/
http://www.oldapps.com/

--
http://hot-text.ath.cx

"Franc Zabkar" wrote:

In the past I used a small utility that could download the content of
a web site starting from a particular URL. You could specify whether
or not to follow off-site links, and you could limit the number of
levels. Unfortunately I can't remember it's name.

Could anyone recommend a similar application? I'm aware of Leech, but
the free version appears to be limited in some way.

In good 'ol IE, there was Favorites, Add to Favorites, Make Available Offline,
with a Customize button to set how many levels and stuff. I think it only saved
as a singe huge "archive" file, instead of making a separate page-by-page
save that could be viewed outside IE.

Sjouke Burry wrote in
:

Lostgallifreyan wrote:
Franc Zabkar wrote in
:

In the past I used a small utility that could download the content of
a web site starting from a particular URL. You could specify whether
or not to follow off-site links, and you could limit the number of
levels. Unfortunately I can't remember it's name.

Could anyone recommend a similar application? I'm aware of Leech, but
the free version appears to be limited in some way.

- Franc Zabkar

Total recommendation for WGET. Commandline tool, indispensible
stuff.. http://www.gnu.org/s/wget

It won't do multiesegment downloads,. but it's both polite and
remorseless, just what we need.

Whether it limits the extent of remote linkage I don't know, but I
think it does, it certainly has plenty of intelligent control on it.

Wget does need a bit of supervision, it surprised me once by trying to
download a complete .GOV website....
I stopped it after the first 15 GB
(usage: WGET -r url)


Ouch. Without knowing more than basic recursion switching, I'd aim to find
a small branch of the site I wanted intact, along with a few other branches,
and make a download list for them. It could be awkward if I don't know how
much I missed on any branch, but mostly I use it on lists of single files so
I never explored it enough (and the branch scheme directly extends from this
simple method). I think maybe there's a limit to how smart a tool like that
can be, it's like a proxy scanner in that there's no substitute for trying to
limit it based on perceptions we can make, but it cannot..

"Sanity Clause" wrote in
:

In good 'ol IE, there was Favorites, Add to Favorites, Make Available
Offline, with a Customize button to set how many levels and stuff. I
think it only saved as a singe huge "archive" file, instead of making a
separate page-by-page save that could be viewed outside IE.


That could be a neat method, as you can probably resave from the bulk
archive, as single web pages with images. Not that I'd ever use a browser for
bulk tasks though, I'd always want to batch script something like that.

On Mon, 15 Aug 2011 09:09:59 -0500, Lostgallifreyan
put finger to keyboard and composed:

Sjouke Burry wrote in
:

Wget does need a bit of supervision, it surprised me once by trying to
download a complete .GOV website....
I stopped it after the first 15 GB
(usage: WGET -r url)

Ouch. Without knowing more than basic recursion switching, I'd aim to find
a small branch of the site I wanted intact, along with a few other branches,
and make a download list for them.

Thanks very much to all for your suggestions.

I found that the following command line worked well:

wget -r -nc -l 10 -np URL -o capture.log

... where -l = little L

I needed two attempts because the default number of levels (5) wasn't
enough.

The "-np" switch excludes parent directories, and "-nc" (no clobber)
prevents overwriting of files which have already been downloaded.

- Franc Zabkar
--
Please remove one 'i' from my address when replying by email.