# Open-source Large Language Models via REST API for the MIT Community

SIPB, MIT students, and MIT researchers stand to benefit substantially from access to LLMs.  However, large language model (LLM) services are generally dominated by an oligopoly of profit-oriented companies with closed-source models and opaque privacy policies.  (OpenAI, Anthropic, Google, to name-drop.)

One of SIPB's fundamental goals is to provide the MIT community with access to computational facilities.  I believe the meaning evolves to keep up with the decades, and in this spirit, I'd like to propose a SIPB project that brings **open-source large language models** to the MIT community.  The aim is to offer this cutting-edge technology responsibly to augment **learning, teaching, & research** while ensuring user privacy.

## Teaching

As an educator myself, I want to highlight this project's potential for enhancing *teaching*.  Professors need to adapt their curricula in the face of the revolutionary rising access to AI models.  I believe it would be ideal to have faculty support and be actively involved in this project's development, and I'd like to help professors learn how to wield our service in their curricula for effective teaching.

## Core ideas

These are some core tenets & motivations I have in mind as I write this proposal.  There are trade-offs in focusing on these tenets versus others, so I'd appreciate some discussion.

1. **Responsibility**: As this is such a revolutionary/transformative technology we're considering, it's important to approach this project responsibly.  This project deserves motivated team members who recognize the importance of moving forward cautiously.

1. **Access**: Not everyone has access to the computational resources necessary for LLMs.  I believe SIPB can start & maintain a GPU cluster to bridge this access gap.

1. **Privacy**: As long as they adhere to some sensible agreements, I'd prefer users have the option to interact with the service with confidence that their inputs aren't being read.

## The plan

### Phase 1: Planning

Summer 2023?

- Determine **MVP**.
- Identify **hardware requirements** for running LLMs locally.
  - A sensible example goal would be for the hardware to handle **65b** LLaMA models, which is currently on the high end.  Such a hardware setup would be well-capable in handling the 30b models.
  - How much demand can we expect to get?  Even if our hardware can run a single 65b instance, that's not super desirable if everyone's requests are delayed.
  - What servers/clouds would SIPB need to set up/secure?
- Decide which open-source models to offer and initially to experiment with.
  - Review the [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard).
  - Browse [r/LocalLLaMA](https://reddit.com/r/LocalLLaMA) for anecdotes.
  - Choose a selection of models better suited for certain tasks, such as coding or language translation.
- How many **team members** are required for this project?
- Crystalize this project's plan (e.g. into a Gantt chart).
- **Purchase** the hardware.
- What are our terms of use?

#### Consider the implications

I recognize how the project has the potential to potentiate research, creative work, and even learning.  However, we need to move forward responsibly; users could develop an over-reliance on our service, hinder their critical thinking, or be academic dishonest ("SIPB-T, do my pset.").

We need to:

- **Collaborate with professors**, to understand how this service can benefit learning and to develop guidelines for acceptable uses for psets and research.
- **Consider [MIT's academic integrity policy](https://integrity.mit.edu/)**, which may evolve very soon, perhaps with SIPB's involvement.

### Phase 2: Implementation

Summer and Fall 2023?

- With the hardware secured, **experiment** with the models and pivot as needed.
- Set up **the compute infrastructure**.
- Lay the foundations of the API server.  What originally inspired me was [LLaMA.go](https://github.com/gotzmann/llama.go), a Golang implementation.  See also [OpenAI's API](https://platform.openai.com/docs/introduction).
  - API tokens?
  - How will we monitor/assess for intended versus malicious use?
  - Logging?
- **Test** the system.
  - I'd recommend a **Blue Team Red Team** setup.
  - This depends on our terms of use and any limitations we wish to encode in our models.
  - Unit testing, integration testing, load testing, user testing.
- Develop a **risk management plan**.

### Phase 3: Deployment & Maintenance

Fall or Spring 2023?

- Publicity? or keep hush?
- Rolling/phased? or full deployment?  Be informed by results and risk assessments from testing.
- Run **educational workshops**, demonstrating how students may use this service responsibly.  (I'd be particularly excited to run these!)
- Provide sufficiently comprehensive documentation on how to use the service.
- Keep the system up-to-date.
- Monitor use.
- Pivot as needed.

## Intended outcomes

A user-friendly REST API that enables the MIT community to interact with large language models, to foster a homely, efficient environment for research, teaching, & learning.

## Appendix A: Possible project names

No particular order.

* **MOSLRA** (MIT Open Source LLM RESTful API)
* **Oracle**
* **Orpheus**
* **SIPB-T**

Note: I fervently request that we do not anthropomorphize Tim the Beaver nor SIPB's fuzz ball into AI models.