MiniHF

MiniHF is an inference, human preference data collection, and fine-tuning tool for local language models. It is intended to help the user develop their prompts into full models. Normally when we prompt a language model we're forced to think in that models latent space. MiniHF lets you go the other direction: Imagine the ideal context in which your prompt could take place and then add it to the model. To make this possible MiniHF provides several powerful features:

• Lightweight web interface and inference server that lets you easily branch your session with the model into multiple completion chains and pick the best ones

• Make your own feedback dataset by writing with local language models such as StableLM and NeoX 20b.

• A monte carlo tree search (MCTS) based inference algorithm, Weave, which rejection samples from the model to improve output quality

• The ability to finetune both the underlying generator LoRa and the evaluator reward LoRa used for the tree search on your own custom dataset

• Easy bootstrapping of new document contexts and models using reinforcement learning from AI feedback (RLAIF)

• Easy install with minimal dependencies

If you want to discuss MiniHF with other users, we have a discord server.