about summary refs log tree commit diff
path: root/crawler/top-1000-websites.txt (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Crawler: Remove prepending of https:// to each urlBaitinq2022-10-231-1001/+1001
| | | | | We now prepend it to the top-1000-urls list. This fixes crawled urls having two https://
* Crawler: Add skeleton crawler implementationBaitinq2022-10-201-0/+1001
Starts by filling a queue with the top 1000 most visited sites. "Crawls" each one (empty fn), and blocks for new elements on the queue.