Commit message (Collapse) | Author | Age | Files | Lines | |
---|---|---|---|---|---|
* | Crawler: Remove prepending of https:// to each url | Baitinq | 2022-10-23 | 1 | -1001/+1001 |
| | | | | | We now prepend it to the top-1000-urls list. This fixes crawled urls having two https:// | ||||
* | Crawler: Add skeleton crawler implementation | Baitinq | 2022-10-20 | 1 | -0/+1001 |
Starts by filling a queue with the top 1000 most visited sites. "Crawls" each one (empty fn), and blocks for new elements on the queue. |