Skip to content

website converter use more wget threads #1

@leonstafford

Description

@leonstafford

HTTrack or other multithreaded downloader would be useful, but less likely to be installed.

Possible workaround with wget: https://stackoverflow.com/a/7645245/1668057

wget -r -np -N [url] &
wget -r -np -N [url] &
wget -r -np -N [url] &
wget -r -np -N [url]

So, with -b and no-clobber, could be useful, but need to benchmark. seems they'd keep attempting the same URLs if starting point is the same or is there a list to crawl that they'd all share and grab from next "to crawl"?

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions