Skip to main content

🖌 LLM Chat

Jarvislabs.ai Create LLM Chat

Use LLM Chat - An interesting open source software that lets you explore with different instruction-following fine-tuned LLM models in a chat kind of interface in 2 ways(Standalone Gradio and Discord bot applications).

The framework comes with more than 40+ preloaded models.

info

Due to certain technical challenges, we are temporarily not providing model weights. We are working actively on it, and will bring them back soon 😊

note

If you find the framework useful, please leave a star in authors github page.

Launch an instance (common step)

In both cases of Gradio and Discord bot, you need to start a instance with framework as LLM Chat. Also choose GPUs and storage as needed. By selecting LLM Chat as framework, your instance can access the NAS(Network Attached Storage) which already contains pre-downloaded weights of all models. This will save tons of your time.

note

If you are choosing the larger models you would need to pick more than 1 GPU with 40+ GPU VRam (A6000/A100). However, based on your choice of GPUs, the application will show the best possible options based on the amount of VRAMs and the number of model's parameters.

Jarvislabs.ai Create LLM Chat

Gradio application

Once the instance is up and running, click API button on your running instance. This will bring you to the web UI of the Gradio application.

Jarvislabs.ai Web LLM Chat

This may take a few seconds to load. You can start chatting with your favourite models.

Discord bot

Once the instance is up and running, click Jupyter icon on your running instance. This will bring you to the web UI of the JupyterLab.

Terminate running app

When the instance is started, we start the /home/launch_chat.sh from inside a tmux session. Access to the running script with the below command in JupyterLab terminal.

tmux attach -t llmchat

The output would look somthing like this.

Jarvislabs.ai Web LLM Chat

Hit CTRL + C, then the running application will stop. Whenever an instance is created as LLM Chat framework, it runs Gradio application. We want to run Discord bot while keeping the benefits of a full pack of pre-downloaded model weights. That is why we need to stop the running Gradio application, and run Discord bot manually instead.

note

Soon we will provide a way to choose Gradio or Discord bot when launching an instance through the dashboard UI.

Run Discord bot

Once the running default applicaion is stopped, run Discord bot with the below command. It assumes that you already created Discord Application at Discord Developer Portal. You need to replace YOUR_DISCORD_TOKEN with the token of your Discord Application. Also, find the model that you want from this list and replace MODEL_NAME with the model name.

export LLMCHAT_APP_MODE=DISCORD
export DISCORD_BOT_TOKEN="YOUR_DISCORD_TOKEN"
export DISCORD_BOT_MODEL_NAME="MODEL_NAME" # alpaca-lora-7b

python entry_point.py

(Optional) Setting up Discord application

In case, you don't have Discord application yet, follow the below instructions

  1. Go to Discord Developer Portal and login.
  2. Select New Application button on the top-right corner.
  3. Name your application and hit Create button.
  4. (Optional) choose app icon within the General Information on the right pane. This icon will determine the look of the bot in Discord.
  5. Select Bot on the right pane and enable all threes under Privileged Gateway Intents section(PRESENCE INTENT, SERVER MEMBERS INTENT, MESSAGE CONTENT INTENT).
    • Under the Build-A-Bot section, you can generate the token of your Discord application. Use it to replaceYOUR_DISCORD_TOKEN from the previous section.
  6. Select OAuth2 > URL Generator on the right pane and check Bot in the SCOPES section. It will make BOT PERMISSIONS section appear. Check Send Messages in BOT PERMISSIONS section.
  7. Copy the GENERATED URL and paste it in the browser, then you will be able to invite your Discord application(bot) to the Discord server of your choice(Basically, you should be Admin of the server to do this).