Nvidia adds new vision, speech, and language capabilities to ChatRTX - a free, local chatbot for PCs with Nvidia RTX graphics cards

Nvidia has expanded the capabilities of Chat for RTX (ChatRTX) by adding support for additional large language models (LLM) as well as vision and speech support in the latest version. The free, local chatbot does not need to send data online while working with user information, thus keeping chats private.

Different LLMs are tuned for different purposes – general chats, scientific papers, essay writing, and more. Some LLMs also have strict training to censor output, so this makes certain LLMs more useful than others. Previously available TensorRT-LLM models included the default Mistral model as well as Llama 2. Nvidia has opened ChatRTX to a few new ones, Gemma and ChatGLM3.

Gemma is a LLM developed by Google DeepMind released in 2024 as two and seven billion parameter models. The models are highly tuned to filter out sensitive speech, personal information, and risky or dangerous responses.

ChatGLM3 is a LLM developed by Zhipu AI and Tsinghua KEG released in 2023 as a six billion parameter model. The bilingual Chinese-English model was designed to be a strong, open-source competitor to OpenAI’s ChatGPT.

Nvidia has added OpenAI’s CLIP neural network to ChatRTX to automatically recognize images and associate related text. For example, if CLIP is fed an image of a television anchor, broadcast cameras, and lights in a room, it will label the image as a photo of a television studio. This enables ChatRTX to work with images without manual content labeling. OpenAI’s Whisper speech recognition system has also been added to ChatRTX, allowing users to speak prompts to ChatRTX.

Readers who want to try out a local chatbot can download Nvidia ChatRTX here as long as they meet the following system requirements. The local chatbot runs on PCs that have an Nvidia GeForce RTX 30 or 40 series graphics card. Ampere and Ada-based graphics cards with at least 8 GB of VRAM are also supported. Those that don’t have a compatible card can buy a fast Nvidia on Amazon. Other system requirements include Windows 11, 16 GB RAM, a minimum of 35 GB of storage space free, and driver version 535.11 or later.