about summary refs log tree commit diff
path: root/crawler (follow)
Commit message (Expand)AuthorAgeFilesLines
* Crawler: Replace String::from with .to_string()Baitinq2022-10-271-3/+6
* Crawler: Fix bad error handling with match handlingBaitinq2022-10-251-6/+9
* Crawler: Use async ClientBaitinq2022-10-252-7/+12
* Crawler: Shuffle crawled urlsBaitinq2022-10-252-4/+4
* Crawler: Add "correct" error handlingBaitinq2022-10-251-21/+23
* Crawler: Parse urls with the "url" crateBaitinq2022-10-252-25/+25
* Crawler: Add crawled url filterBaitinq2022-10-241-1/+8
* Crawler: Set queue size to 2222Baitinq2022-10-241-1/+1
* Crawler+Indexer: Rust cleanupBaitinq2022-10-231-3/+2
* Crawler: Replace println! with dbg!Baitinq2022-10-231-7/+7
* Crawler: Remove prepending of https:// to each urlBaitinq2022-10-232-1006/+1006
* Crawler: Only crawl 2 urls per urlBaitinq2022-10-231-0/+6
* Crawler: Change blockingqueue to channelsBaitinq2022-10-232-12/+12
* Crawler: Implement basic async functionalityBaitinq2022-10-222-39/+46
* Crawler: Add basic indexer communicationBaitinq2022-10-212-11/+48
* Crawler: Add Err string in the craw_url methodBaitinq2022-10-201-3/+3
* Crawler: Add indexer interaction skeletonBaitinq2022-10-201-1/+5
* Crawler: Wrap crawl response in Result typeBaitinq2022-10-201-18/+23
* Crawler: Normalise relative urlsBaitinq2022-10-201-2/+17
* Crawler: Remove duplicate parsed urlsBaitinq2022-10-202-0/+4
* Crawler: Add basic html parsing and link-followingBaitinq2022-10-202-9/+36
* Crawler: Add skeleton crawler implementationBaitinq2022-10-203-0/+1042
* Misc: Separate OSSE into componentsBaitinq2022-10-192-0/+15