OSSE - Overly Simple Search Engine :)

	Commit message (Collapse)	Author	Age	Files	Lines
*	Crawler: Fix bad error handling with match handling	Baitinq	2022-10-25	1	-6/+9
\|
*	Crawler: Use async Client	Baitinq	2022-10-25	1	-6/+11
\|
*	Crawler: Shuffle crawled urls	Baitinq	2022-10-25	1	-3/+2
\|
*	Crawler: Add "correct" error handling	Baitinq	2022-10-25	1	-21/+23
\|
*	Crawler: Parse urls with the "url" crate	Baitinq	2022-10-25	1	-25/+24
\| \| \| \| \|	This fixes relative urls, makes url filtering and validation better, and many other improvements.
*	Crawler: Add crawled url filter	Baitinq	2022-10-24	1	-1/+8
\| \| \| \|	This filters hrefs such as "/", "#" or "javascript:"
*	Crawler: Set queue size to 2222	Baitinq	2022-10-24	1	-1/+1
\|
*	Crawler+Indexer: Rust cleanup	Baitinq	2022-10-23	1	-3/+2
\| \| \| \| \| \|	Getting more familiar with the language so fixed some non optimal into_iter() usage, unnecessary .clone()s and unnecessary hack when we could just get a &mut for inserting into the indexer url database.
*	Crawler: Replace println! with dbg!	Baitinq	2022-10-23	1	-7/+7
\|
*	Crawler: Remove prepending of https:// to each url	Baitinq	2022-10-23	1	-5/+5
\| \| \| \| \|	We now prepend it to the top-1000-urls list. This fixes crawled urls having two https://
*	Crawler: Only crawl 2 urls per url	Baitinq	2022-10-23	1	-0/+6
\| \| \| \|	This makes it so that we dont get rate limited from websites.
*	Crawler: Change blockingqueue to channels	Baitinq	2022-10-23	1	-11/+11
\| \| \| \| \|	We now use the async-channel channels implementation. This allows us to have bounded async channels.
*	Crawler: Implement basic async functionality	Baitinq	2022-10-22	1	-39/+45
\|
*	Crawler: Add basic indexer communication	Baitinq	2022-10-21	1	-10/+46
\|
*	Crawler: Add Err string in the craw_url method	Baitinq	2022-10-20	1	-3/+3
\|
*	Crawler: Add indexer interaction skeleton	Baitinq	2022-10-20	1	-1/+5
\|
*	Crawler: Wrap crawl response in Result type	Baitinq	2022-10-20	1	-18/+23
\|
*	Crawler: Normalise relative urls	Baitinq	2022-10-20	1	-2/+17
\| \| \| \| \|	We now normalise urls starting with / (relative to root) and // (relative to protocol)
*	Crawler: Remove duplicate parsed urls	Baitinq	2022-10-20	1	-0/+3
\|
*	Crawler: Add basic html parsing and link-following	Baitinq	2022-10-20	1	-9/+34
\| \| \| \| \|	Extremely basic implementation. Needs max queue size, error handling, formatting of parsed links.
*	Crawler: Add skeleton crawler implementation	Baitinq	2022-10-20	1	-0/+40
\| \| \| \| \|	Starts by filling a queue with the top 1000 most visited sites. "Crawls" each one (empty fn), and blocks for new elements on the queue.
*	Misc: Separate OSSE into components	Baitinq	2022-10-19	1	-0/+3
	We now have a cargo workspace with the Crawler, Client and Indexer packages.