ROASTY Videos

A Roadmap for Instruct-Tuned LoRA LLMs with Hybrid RAG and Multimodal Capabilities

RoastyAI charts a roadmap for developing and releasing an instruct-tuned Low-Rank Adaptation (LoRA) model that combines the strengths of RAG and instruction tuning while incorporating multimodal reasoning. A key innovation in our approach is leveraging real-world input-output pairs from the live Twitter bot @RoastyAI that specializes in roasting users to gain inputs for Reinforcement Learning from Human Feedback (RLHF). This dynamic dataset enables the alignment of the model toward generating genuinely funny, sarcastic, and contextually appropriate responses, marking a breakthrough in LLM humor and creativity. The bot’s interactions—consisting of user queries, contextual data (e.g., images, tone, or intent), and the bot’s responses—form a rich dataset of conversational examples that encapsulate authentic, human-like humor. This dataset is used to fine-tune the reward model, enabling the AI to evaluate the quality of its outputs based on human preferences for wit, sarcasm, and creativity. By integrating RLHF into the training loop, we refine the system's ability to navigate the nuances of humor, ensuring it generates responses that are not only contextually grounded but also genuinely funny and aligned with human expectations. This approach positions our system as a novel method for improving LLM alignment in subjective and high-creativity domains, offering a scalable framework for building conversational agents that excel in humor and social interaction.

To achieve modularity and scalability, we will use a LoRA (Low-Rank Adaptation) approach for fine-tuning instruction-tuned models. LoRA enables parameter-efficient training by modifying only a small subset of low-rank layers within the base model, preserving the original architecture while enabling domain-specific customization. Our roadmap includes fine-tuning LoRA adapters on a hybrid dataset of conversational heuristics, multimodal examples, and humor-specific tasks, augmented with RLHF training to align the model with human preferences. These adapters are stackable, allowing developers to extend the base model for new domains or refine specific capabilities, such as tone modulation or multimodal reasoning.

By releasing the model as a LoRA-tuned LLM [ Llama3.1-70b-RoastyAI-instruct ] we will provide a modular framework that developers can easily integrate into existing systems or adapt for novel use cases, and share with developers to build customized conversational AI as a community-ready resource. The model will be accompanied by detailed documentation, including pretrained LoRA adapters, multimodal configurations, and usage instructions for integrating the system into real-world applications. By combining the grounding power of RAG, the adaptability of instruction tuning, the versatility of multimodal reasoning, and the humor alignment achieved through RLHF on real-world Twitter bot interactions, our framework lays the foundation for a new generation of conversational AI systems. These systems are capable of not only answering questions and generating text but also understanding and responding to complex, multimodal inputs in a user-aligned, contextually rich, and genuinely humorous manner. This roadmap represents a step forward in making advanced conversational AI accessible, extensible, and applicable to a wide range of creative and functional domains.

A Roadmap for Video-Based AI Roasting: Text-to-Audio, Viseme Mapping, and Lip-Synced Character Animation

We propose a technical roadmap for extending our conversational AI system into a video-based roasting pipeline. In this workflow, the text response from the model is converted into an audio clip, and the corresponding visemes (visual representations of phonemes) are generated. These visemes are then mapped to the RoastyAI visual character to synchronize lip movements with the speech. The output will be a fully-rendered, lipsynced video output of the roast delivered by the character, providing an engaging and dynamic user experience.

The workflow begins with the text-to-audio pipeline, where the AI-generated roast response is passed to a Text-to-Speech (TTS) module. Using advanced neural TTS models, the system synthesizes audio that conveys the intended tone and humor of the roast. Once the audio clip is generated, the system transitions to viseme generation, where the phonemes in the audio are analyzed and mapped to their corresponding visual representations. The mapping process breaks down the audio into segments, associating each phoneme with a viseme that defines the shape of the mouth during speech. These visemes are precisely time-aligned with the audio to ensure that lip movements match the speech in both timing and articulation.

The viseme sequence is then applied to facial landmarks of the RoastyAI visual character, driving the character's lip movements, and the output is composited into a video format. The system integrates both the audio and the viseme-driven animation into the output video.

This roadmap outlines a fully automated pipeline for video-based AI roasting, starting from a text response and culminating in a high-quality, lipsynced video output featuring the RoastyAI character. By combining text-to-speech synthesis, viseme generation, and character animation, this system delivers a novel immersive experience. The integration of phoneme-level precision, expressive animation, and comedic timing positions our approach as a cutting-edge solution for AI-generated video content, enabling applications in entertainment, social media, and personalized content creation.