| Commit message (Collapse) | Author | Files | Lines | ||
|---|---|---|---|---|---|
| 2022-10-28 | Frontend: Logically structure html | Baitinq | 1 | -35/+39 | |
| 2022-10-28 | Frontend: Add local bootstrap files | Baitinq | 3 | -5/+18 | |
| 2022-10-27 | Fronted: Order search results by priority | Baitinq | 3 | -2/+20 | |
| We do this by implementing the PartialOrd and Ord traits into the CrawledResource struct and then using Itertools::sorted() on the display iterator. | |||||
| 2022-10-27 | Frontend: Use display_results() function for rendering CrawledResources | Baitinq | 1 | -5/+14 | |
| 2022-10-27 | Frontend: Fetch results from indexer | Baitinq | 3 | -33/+53 | |
| 2022-10-27 | Crawler: Abstract database word fetching with search_word_in_db() | Baitinq | 1 | -2/+10 | |
| 2022-10-27 | Indexer: Add /search with no query endpoint | Baitinq | 1 | -0/+6 | |
| Just returns []. | |||||
| 2022-10-27 | Crawler: Replace String::from with .to_string() | Baitinq | 1 | -3/+6 | |
| 2022-10-27 | Indexer: Setup permissive CORS | Baitinq | 3 | -1/+21 | |
| 2022-10-27 | Indexer: Return json from the /search endpoint | Baitinq | 3 | -7/+6 | |
| 2022-10-26 | Frontend: Add results field to the state and set dummy results | Baitinq | 1 | -2/+46 | |
| 2022-10-26 | Frontend: Add basic search_query state | Baitinq | 3 | -8/+158 | |
| 2022-10-26 | Frontend: Add basic layout | Baitinq | 2 | -1/+43 | |
| 2022-10-25 | Frontend: Update index.html to include boostrap | Baitinq | 1 | -3/+16 | |
| Also setup viewport and title. | |||||
| 2022-10-25 | Crawler: Fix bad error handling with match handling | Baitinq | 1 | -6/+9 | |
| 2022-10-25 | Crawler: Use async Client | Baitinq | 4 | -48/+152 | |
| 2022-10-25 | Indexer: Use CrawledResource structure as values in the reverse index db | Baitinq | 3 | -11/+45 | |
| This will allow us to integrate priorities and other improvements. | |||||
| 2022-10-25 | Indexer: Add "correct" error handling | Baitinq | 1 | -7/+7 | |
| 2022-10-25 | Crawler: Shuffle crawled urls | Baitinq | 3 | -4/+5 | |
| 2022-10-25 | Crawler: Add "correct" error handling | Baitinq | 1 | -21/+23 | |
| 2022-10-25 | Crawler: Parse urls with the "url" crate | Baitinq | 3 | -25/+26 | |
| This fixes relative urls, makes url filtering and validation better, and many other improvements. | |||||
| 2022-10-24 | Crawler: Add crawled url filter | Baitinq | 1 | -1/+8 | |
| This filters hrefs such as "/", "#" or "javascript:" | |||||
| 2022-10-24 | Flake: Add rust-analyzer package | Baitinq | 1 | -0/+1 | |
| 2022-10-24 | Crawler: Set queue size to 2222 | Baitinq | 1 | -1/+1 | |
| 2022-10-24 | Misc: Update build/run instructions | Baitinq | 1 | -2/+4 | |
| Now show how to run each module + yew frontend | |||||
| 2022-10-24 | Client->Frontend: Create yew frontend skeleton | Baitinq | 8 | -14/+238 | |
| We have replaced the client with a yew frontend. | |||||
| 2022-10-23 | Crawler+Indexer: Rust cleanup | Baitinq | 2 | -14/+6 | |
| Getting more familiar with the language so fixed some non optimal into_iter() usage, unnecessary .clone()s and unnecessary hack when we could just get a &mut for inserting into the indexer url database. | |||||
| 2022-10-23 | Crawler: Replace println! with dbg! | Baitinq | 1 | -7/+7 | |
| 2022-10-23 | Crawler: Remove prepending of https:// to each url | Baitinq | 2 | -1006/+1006 | |
| We now prepend it to the top-1000-urls list. This fixes crawled urls having two https:// | |||||
| 2022-10-23 | Crawler: Only crawl 2 urls per url | Baitinq | 1 | -0/+6 | |
| This makes it so that we dont get rate limited from websites. | |||||
| 2022-10-23 | Crawler: Change blockingqueue to channels | Baitinq | 3 | -19/+45 | |
| We now use the async-channel channels implementation. This allows us to have bounded async channels. | |||||
| 2022-10-23 | Indexer: Listen on 0.0.0.0 | Baitinq | 1 | -1/+1 | |
| 2022-10-22 | Indexer: Implement basic reverse index searching and adding | Baitinq | 3 | -15/+163 | |
| Very inefficient but kind of functional:::))))))) | |||||
| 2022-10-22 | Crawler: Implement basic async functionality | Baitinq | 3 | -93/+285 | |
| 2022-10-21 | Crawler: Add basic indexer communication | Baitinq | 2 | -11/+48 | |
| 2022-10-21 | Indexer: Add skeleton http rest endpoint functionality | Baitinq | 3 | -1/+539 | |
| /search and /resource endpoint. | |||||
| 2022-10-20 | Crawler: Add Err string in the craw_url method | Baitinq | 1 | -3/+3 | |
| 2022-10-20 | Crawler: Add indexer interaction skeleton | Baitinq | 1 | -1/+5 | |
| 2022-10-20 | Crawler: Wrap crawl response in Result type | Baitinq | 1 | -18/+23 | |
| 2022-10-20 | Crawler: Normalise relative urls | Baitinq | 1 | -2/+17 | |
| We now normalise urls starting with / (relative to root) and // (relative to protocol) | |||||
| 2022-10-20 | Crawler: Remove duplicate parsed urls | Baitinq | 3 | -0/+20 | |
| 2022-10-20 | Crawler: Add basic html parsing and link-following | Baitinq | 3 | -9/+1561 | |
| Extremely basic implementation. Needs max queue size, error handling, formatting of parsed links. | |||||
| 2022-10-20 | Crawler: Add skeleton crawler implementation | Baitinq | 4 | -0/+1051 | |
| Starts by filling a queue with the top 1000 most visited sites. "Crawls" each one (empty fn), and blocks for new elements on the queue. | |||||
| 2022-10-19 | Misc: Change to use "oxalica/rust-overlay" for the nix development shell | Baitinq | 3 | -26/+90 | |
| This fixes vscode not being able to find rust-analyzer and rust-src | |||||
| 2022-10-19 | Misc: Separate OSSE into components | Baitinq | 9 | -10/+56 | |
| We now have a cargo workspace with the Crawler, Client and Indexer packages. | |||||
 |
index : OSSE | |
| Overly Simple Search Engine :) |
| about summary refs log tree commit diff |