AffectFace

👤 Participant Information

📋 Please provide your information to begin. A unique ID will be assigned to track your responses for research purposes.

Name / Pseudonym *

Age

Role

Notes (Optional)

I consent to anonymous data collection for research

🔄 Resume Session

Enter your 6-digit session code to resume a previous session

Session Code

🔬 How It Works - Technical Documentation

📋 System Overview

AffectFace is a multimodal emotion detection system that combines facial expression analysis, voice prosody, and large language models to create emotion-aware conversational AI. The system captures real-time emotional data and uses it to contextualize LLM responses.

🏗️ System Architecture

1. Video Capture

Tool: MediaDevices API
Resolution: 640×480px
Frame Rate: 30 FPS
Processing: Real-time canvas extraction

2. Facial Analysis

Tool: Hume AI Streaming API
Emotions: 53 distinct emotions
Accuracy: 95% (research-validated)
Latency: ~100-200ms per frame

3. Voice Analysis

Tool: Web Audio API + Meyda.js
Features: Pitch, Energy, Rate, Valence
Sampling: 44.1kHz, 16-bit
Buffer: Rolling 10-second window

4. LLM Integration

Provider: OpenAI API
Models: GPT-3.5, GPT-4, GPT-4o
Context: Emotion vector + metadata
Strategies: 12 prompt approaches

🧮 Emotional Cluster Calculation

The system classifies 53 Hume emotions into 5 psychological clusters based on the Circumplex Model of Affect (Russell, 1980):

                  Algorithm:

                  1. Extract top 3 emotions by score

                  2. Count matches in predefined categories

                  3. If 2+ matches in same category → Assign cluster

                  4. Otherwise → "Mixed Emotional State"

Stress Response
High arousal, negative valence
Anxiety, Fear, Distress

Positive Affect
High arousal, positive valence
Joy, Excitement, Amusement

Cognitive Engagement
Moderate arousal, neutral
Concentration, Interest

Negative Affect
Low arousal, negative valence
Sadness, Disappointment

Mixed State
No clear pattern
Complex emotions

🔄 Data Flow Pipeline

Capture: Video (30 FPS) + Audio (continuous)
Extract: Frame → Base64 → Hume API
Analyze: 53 emotions + scores (0-1 range)
Classify: Top 3 → Emotional cluster
Prosody: Pitch, energy, rate extraction
Congruence: Face-voice alignment check
Snapshot: Aggregate all modalities
Format: Rich emotion context (JSON)
Inject: Context → LLM system prompt
Query: User question + emotion → LLM
Response: Emotion-aware answer
Store: Session data (localStorage)

⚙️ Technical Specifications

Frontend
TypeScript, Vite, Vanilla JS
No framework dependencies

APIs
Hume AI (WebSocket)
OpenAI (REST)

Storage
localStorage (client-side)
JSON export capability

Privacy
No server storage
Consent-based analysis

📚 Research Foundation

Hume AI: Research-validated 53-emotion model (95% accuracy)
Circumplex Model: Russell (1980) - Arousal × Valence framework
Affective Computing: Picard (1997) - Emotion-aware systems
Multimodal Fusion: Face + Voice integration for accuracy
Prompt Engineering: 12 strategies from recent research (2020-2024)

⚠️ Known Limitations

Lighting: Facial analysis requires adequate lighting
Internet: Hume API and OpenAI require active connection
Browser: Chrome/Edge/Safari recommended (WebRTC support)
Cultural: Emotion expressions may vary across cultures
Privacy: Video/audio processed by third-party APIs
Latency: ~200-500ms total processing time

🎓 Research Use Cases - Quick Guide

✅ Use Case 1: Single Analysis

Start Analysis → Express emotion → Ask question → See emotion-aware response

🔬 Use Case 2: Multi-Model

Check "Compare 6 Models" → Ask → See GPT-3.5 vs GPT-4 vs GPT-4o evolution

📚 Use Case 3: Prompt Strategies

Try different strategies → Same emotion → Different responses → Click "Show Prompt"

⚠️ Use Case 4: Congruence

Smile + sad voice → System detects mismatch → LLM responds to incongruence

Analysis Status

Inactive

⚠️ If face does not react, click this button

🎬 Test Your Emotions

Watch these videos to trigger emotions and test the system! Click a button to load the video.

📚 Research-validated videos from UW-Madison designed to elicit smiles

💡 Tip: Start Analysis first, then play a video and watch your avatar react to your emotions!

🧠 Affective Prompt Strategy

📚 Use Case 3: Prompt Strategy Research

Select different strategies below to see how the SAME emotion data produces DIFFERENT responses. After asking, click "Show Full Prompt" to compare what was sent to the LLM!

How it works: Your captured emotion → Selected strategy → LLM response

Prompt Strategy

Select a prompt strategy to see its description

🎤 Voice Capture & Strategy Comparison

📚 Use Case 3: Prompt Strategy Research

Speak naturally → System captures your words + emotions → Sends to ALL 13 prompt strategies (including custom) → Compare responses side-by-side

📋 How it works:

1. Make sure "Start Analysis" is active (capturing emotions)
2. Click "🎤 Start Voice Capture" below
3. Speak your question naturally (e.g., "I'm worried about my exam tomorrow")
4. Click "⏹ Stop & Compare All Strategies"
5. See up to 13 different responses - same emotion data, different strategies!
6. Click "Show Prompt" on any response to see full emotion context

Ask LLM

Current Provider: OpenAI

Model: gpt-4

🔬 Use Case 2: Multi-Model Comparison

Enable: Compare 6 AI Models (GPT-3.5 → GPT-4 → GPT-4o)

When enabled, sends your question to 6 different models and displays responses side-by-side. Shows AI evolution from 2023-2024!

Your Question

⚡ Ready to Ask?

1. Make sure "Start Analysis" is active (emotion capturing)
2. Type or speak your question above
3. Click the blue button below to send to LLM

Multi-Model Comparison

Ask the same question with different models to compare responses

Score Response

Affect Acknowledgment (0-5)

Alignment (0-5)

Calibration (0-5)

Helpfulness (0-5)

Civility (0-5)

⚙️ Configuration

🎭 Choose Your Avatar

Emotion Provider

LLM Configuration

Saved Configurations

👤 Participant Information

🔄 Resume Session

🔬 How It Works - Technical Documentation

📋 System Overview

🏗️ System Architecture

1. Video Capture

2. Facial Analysis

3. Voice Analysis

4. LLM Integration

🧮 Emotional Cluster Calculation

🔄 Data Flow Pipeline

⚙️ Technical Specifications

📚 Research Foundation

⚠️ Known Limitations

🎓 Research Use Cases - Quick Guide

Analysis Status

🎬 Test Your Emotions

🧠 Affective Prompt Strategy

✏️ Custom Prompt Strategy

🎤 Voice Capture & Strategy Comparison

Ask LLM

Answer

Multi-Model Comparison

Model Comparison

Score Response

Score Response

Dashboard

Live Video Feed

Prompt Details

🎯 Emotion Calibration

Calibration Complete!

AffectFace

⚙️ Configuration

🎭 Choose Your Avatar

Emotion Provider

LLM Configuration

Saved Configurations

👤 Participant Information

🔄 Resume Session

🔬 How It Works - Technical Documentation

📋 System Overview

🏗️ System Architecture

1. Video Capture

2. Facial Analysis

3. Voice Analysis

4. LLM Integration

🧮 Emotional Cluster Calculation

🔄 Data Flow Pipeline

⚙️ Technical Specifications

📚 Research Foundation

⚠️ Known Limitations

🎓 Research Use Cases - Quick Guide

Analysis Status

🎬 Test Your Emotions

🧠 Affective Prompt Strategy

✏️ Custom Prompt Strategy

🎤 Voice Capture & Strategy Comparison

Ask LLM

Answer

Multi-Model Comparison

Model Comparison

Score Response

Score Response

Dashboard

Live Video Feed