diff options
author | Baitinq <manuelpalenzuelamerino@gmail.com> | 2022-11-07 15:52:58 +0100 |
---|---|---|
committer | Baitinq <manuelpalenzuelamerino@gmail.com> | 2022-11-08 11:45:24 +0100 |
commit | 638c940159df496ac7fcd34d45830712c171e827 (patch) | |
tree | 391c5de5d203321354af16b9645992f0ce024b59 | |
parent | Frontend: Append "..." to truncated titles and descriptions (diff) | |
download | OSSE-638c940159df496ac7fcd34d45830712c171e827.tar.gz OSSE-638c940159df496ac7fcd34d45830712c171e827.tar.bz2 OSSE-638c940159df496ac7fcd34d45830712c171e827.zip |
Misc: Update Readme to be more user friendly
-rw-r--r-- | README.md | 96 | ||||
-rw-r--r-- | docs/frontend.png | bin | 0 -> 169239 bytes |
2 files changed, 83 insertions, 13 deletions
diff --git a/README.md b/README.md index b16d224..e8c6f66 100644 --- a/README.md +++ b/README.md @@ -1,29 +1,99 @@ -# OSSE (Overly Simple Searche Engine) (Oh-Si) +# OSSE Search Engine -An attempt to make a "functional" search engine while learning Rust :^) +> Overly Simple Search Engine - Making search engines simple?? \ +> Pronunciation: "oh-see" -## Building +![contributors](https://img.shields.io/github/contributors/baitinq/OSSE.svg) [![license](https://img.shields.io/github/license/baitinq/OSSE.svg)](https://github.com/Baitinq/OSSE/blob/master/LICENSE) [![PRs welcome](https://img.shields.io/badge/PRs-welcome-ff69b4.svg)](https://github.com/baitinq/OSSE/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22) ![Stars](https://img.shields.io/github/stars/baitinq/OSSE.svg) ![Forks](https://img.shields.io/github/forks/baitinq/OSSE.svg) [![code with hearth by Baitinq](https://img.shields.io/badge/%3C%2F%3E%20with%20%E2%99%A5%20by-Baitinq-ff1414.svg)](https://github.com/Baitinq) -### 1. Setup your environment +<img src="https://raw.githubusercontent.com/Baitinq/OSSE/master/docs/frontend.png" /> + + +## 🚩 Table of Contents + +- [Why?](#-why) +- [Usage](#-usage) +- [Features](#-features) +- [How it works](#-how-it-works) +- [Roadmap](#-roadmap) +- [Contributing](#-contributing) +- [License](#-license) + +## 🐂 Why? + +[Just for fun!](https://justforfunnoreally.dev) I really wanted to learn [Rust](https://rust-lang.org) and at the time I was really interested in how search engines worked, so there wasn't any better way of achieving both goals than with this very project! + +## 🤖 Usage + +This repository is a monorepo formed by the independent components that form the OSSE search engine. + +### Installing Dependencies + +* #### With [Nix](https://nixos.org/): -#### With Nix: ``` $ nix develop ``` -or -``` -$ nix-shell shell.nix -``` -#### Otherwise: +* #### Otherwise: -Install cargo through your preferred method. +Install [cargo](https://doc.rust-lang.org/cargo/) and [trunk](https://trunkrs.dev) with your preferred method (such as your favorite package manager). +### Running -### 2. Run! +* [Crawler](https://github.com/Baitinq/OSSE/tree/master/crawler) +``` +$ cargo run --bin crawler +``` + +* [Indexer](https://github.com/Baitinq/OSSE/tree/master/indexer) ``` $ cargo run --bin indexer -$ cargo run --bin crawler +``` + +* [Frontend](https://github.com/Baitinq/OSSE/tree/master/frontend) +``` $ trunk serve frontend/index.html --open ``` + +Once all the components are running, you can navigate to ```127.0.0.1:8080``` on your favorite web browser and start using OSSE! + +## 🎨 Features + +* [Completely Self-Hosted](https://en.wikipedia.org/wiki/Self-hosting_(web_services)) : OSSE does not use any external services, all you need is its three components (indexer, crawler & frontend) to have a "complete" search engine. +* [Custom Indexing and Ranking Algorithms](https://github.com/Baitinq/OSSE/tree/master/indexer) : OSSE uses its own open-source indexing and ranking algorithm, meaning that its code is reviewable and improvable by third parties, ensuring its technically and morally correct functionality. +* [Hackable]() : OSSE is built with extensibility & modularity in mind, so it is entirely feasible to replace or customize its various components. +* [Privacy Respecting]() : As a result of OSSE being completely independent, it does not send any metadata to any services. + + +## ⚙️ How it works + +The OSSE search engine is separated into three independent components: + +* ### [Indexer](https://github.com/Baitinq/OSSE/tree/master/indexer) +This component provides both the actual search engine indexer's implementation and the REST API used to search and add indexed resources. It uses [Actix Web](https://actix.rs) for the REST API (running on port 4444). For the implementation of the actual indexer data structure, we currently use a very simple reverse index implemented with a hashmap, so all the indexed resources are currently lost each time the indexer is restarted. + +* ### [Crawler](https://github.com/Baitinq/OSSE/tree/master/crawler) +This component is a simple recursive crawler that forwards the crawled raw HTML to the indexer. It uses [reqwest](https://docs.rs/reqwest/latest/reqwest) for fetching a predefined list of [root websites](https://github.com/Baitinq/OSSE/blob/master/crawler/top-1000-websites.txt) and parses them with [scraper](https://docs.rs/scraper/latest/scraper), sending the website contents to the indexer and extracting all its links, adding them to a queue of websites to be crawled. This process is "recursively" repeated indefinitely. + +* ### [Frontend](https://github.com/Baitinq/OSSE/tree/master/frontend) +This component is a simple web interface to the indexer. It allows users to search and visualize results in a user friendly way. It is currently built using [Yew](https://yew.rs), which allows us to write the frontend in rust and produce a "blazingly fast" Wasm based web-ui. + +## 🐾 Roadmap + +- [x] Add frontend +- [ ] Change indexer to use a ngram index instead of a reverse index +- [ ] Improve frontend +- [ ] Improve responsiveness of searching when the indexer is recieving info from crawlers +- [ ] Rust cleanup +- [ ] Improve page ranking algorithm + + +## 💬 Contributing + +> "If you have any ideas or patches, please do not hesitate to contribute to OSSE!" + +## 📜 License + +This software is licensed under the [BSD-2-Clause](https://github.com/baitinq/OSSE/blob/master/LICENSE) © [Baitinq](https://github.com/Baitinq). + diff --git a/docs/frontend.png b/docs/frontend.png new file mode 100644 index 0000000..c2742fa --- /dev/null +++ b/docs/frontend.png Binary files differ |