AI-Powered Smart Home System

An intelligent IoT automation ecosystem processing natural language commands via Google Gemini, managed by a highly scalable Flask & MQTT backend architecture.

View Repository

Problem Statement

Traditional smart home systems rely heavily on rigid command structures or button-based interfaces. They struggle to parse contextual intent, particularly in multi-lingual or mixed-language (Hinglish) environments.

Objectives

  • Develop a flexible natural language interface.
  • Ensure sub-second latency for device state changes.
  • Establish a secure, decoupled communication layer between the edge hardware and the cloud backend.

System Architecture

[ Architecture Diagram / Flowchart Placeholder ]

Hardware Architecture

At the edge, an ESP32 microcontroller acts as the central IoT hub, executing firmware written in C++. It maintains a persistent WebSocket connection to the MQTT broker, listening for topics related to device states (e.g., relays, lights).

Software Architecture

The backend is a RESTful API built with Flask (Python). It acts as the orchestrator—receiving user input, interfacing with the AI model for intent parsing, and publishing the resulting JSON payload to the MQTT broker (HiveMQ).

Technology Stack

  • Python / Flask
  • ESP32 (C++)
  • MQTT (HiveMQ)
  • Google Gemini API

Communication Flow & AI Workflow

1. Input: User submits a natural language command (e.g., "Bhaiya, hall ki light band kardo").

2. AI Processing: The Flask backend forwards the string to the Google Gemini API with a strict system prompt constraint to return only a standardized JSON state object representing the user's intent.

3. Message Brokering: The Flask app parses the JSON and publishes the explicit command to a secure MQTT topic via HiveMQ.

4. Edge Execution: The ESP32, subscribed to the topic, receives the payload in near real-time, toggles the corresponding GPIO pins, and publishes an acknowledgment back to the server.

Challenges & Solutions

Latency in AI processing: Standard LLM responses are slow.
Solution: Enforced strict schema generation and temperature controls on the Gemini API to reduce generation time by 60%.

Connection Drops: ESP32 disconnecting from WiFi/MQTT.
Solution: Implemented a robust reconnect loop in the firmware with exponential backoff.

Future Improvements

  • Migrate from HTTP REST to WebSockets for the frontend-backend link.
  • Implement Local LLM processing (Edge AI) to reduce cloud dependency.
  • Add physical sensor arrays (Temperature, Motion) for automated trigger conditions.