Skip to content

README.md

Latest commit

 

History

History
390 lines (268 loc) · 13.2 KB

README.md

File metadata and controls

390 lines (268 loc) · 13.2 KB

FixMyCity – SIH 2025

An AI‑powered civic issue management platform that helps citizens report problems quickly and helps city officials act faster and more transparently.

FixMyCity combines computer vision, voice + WhatsApp automation, a smart department classifier, and a citizen chatbot to create a unified pipeline from issue detection → triage → resolution → citizen acknowledgement.

1. Problem & Solution Overview

  • Problem: Civic issues like potholes, broken streetlights, water leakage, overflowing garbage, etc. are often:

    • Reported late (or not at all),
    • Sent to the wrong department,
    • Poorly tracked, with no feedback to citizens.

  • Solution: FixMyCity provides:

    • Multi‑channel reporting: automated AI detection, SOS hardware buttons, voice calls, chatbot, and certificates over WhatsApp.
    • AI‑based triage: classifies issues into the right civic department using CNN models and LLMs.
    • Central backend: unified APIs that aggregate all issues on a live map/dashboard.
    • Citizen feedback: auto‑generated resolution certificates shared digitally.

2. High‑Level Architecture

The project is organized as several focused micro‑services:

  • backend_python/ – Flask backend for:

    • Receiving SOS alerts from physical boxes / simulators.
    • Receiving AI‑detected issues (e.g. pothole detection).
    • Storing issues in memory and exposing them via APIs for the frontend/map.
    • Managing vehicle live locations (for field teams / response vehicles).

  • agentic-chatbot-user/ – FastAPI + Gemini based citizen service chatbot for Mumbai:

    • Answers questions about local civic services.
    • Uses a structured citizen_services_geo.json lookup.
    • Falls back to an LLM (Gemini) when data is not in the JSON.

  • ml-dept-classifier/ – Civic department classifier:

    • CNN‑based PyTorch model (models/best_model.pth) to classify issue images into:
      • roads, sanitation, electricity, water, other.
    • FastAPI service (app.py) for programmatic classification.
    • Streamlit UI (src/streamlit_app.py) to upload images and view predictions.

  • user-certificate/ – Issue resolution certificate generator:

    • Generates personalised certificates using Pillow on a PNG template.
    • Prepares links for WhatsApp delivery via Twilio (utility code provided).

  • backend_python/calling_agent.py – Twilio Voice + Whisper + Gemini agent:

    • Citizens can call a number and verbally describe issues.
    • Audio is recorded by Twilio, transcribed via Whisper, and summarized/replied via Gemini/OpenAI.

  • Root Node package (package.json):

    • Provides supporting Node utilities (WebSocket, Twilio, Google GenAI integrations, i18n, etc.).
    • Can be used by a frontend or gateway service if needed.

3. Key Features

  • AI‑powered detection

    • YOLOv8 based pothole detection training script (backend_python/train_model.py).
    • Designed to be extended to other visual civic issues.

  • SOS hardware integration

    • /report_sos_issue endpoint ingests alerts from physical SOS buttons or simulators.
    • Issues are stored and served to a dashboard via /get_sos_alerts and /get_issues.

  • Voice‑based citizen assistant

    • Twilio Voice webhook (/voice, /outbound_voice) handles Hindi speech.
    • Whisper converts speech → text; Gemini/OpenAI generates empathetic responses.

  • Department classifier for routing

    • Given an image URL, the classifier predicts the responsible department.
    • Used to route issues to roads, sanitation, electricity, water, or other.

  • Citizen services chatbot

    • FastAPI + Gemini chatbot with CORS enabled for local and hosted frontends.
    • Uses citizen_services_geo.json for department + location lookups.

  • Digital resolution certificates

    • High‑quality PNG certificates generated from templates + TrueType fonts.
    • Designed to be sent over WhatsApp via Twilio using public URLs (e.g. ngrok).

4. Tech Stack

  • Languages: Python, JavaScript/TypeScript
  • Frameworks & Libraries:
    • Backend: FastAPI, Flask, Flask‑CORS
    • ML / CV: PyTorch, torchvision, ultralytics YOLOv8, scikit‑learn
    • UI: Streamlit, (Node package prepared for frontend integrations)
    • AI: LangChain, Gemini (langchain-google-genai), Whisper, OpenAI
    • Messaging/Telephony: Twilio (Voice + WhatsApp)
    • Others: requests, pandas, Pillow, python-dotenv, ws

5. Project Structure (Important Folders)

From FixMyCity-SIH-2025/SIH-2025f:

  • agentic-chatbot-user/

    • main.py – FastAPI chatbot service.
    • citizen_services_geo.json – department/location data.
    • requirements.txt – dependencies for this service.

  • backend_python/

    • app.py – core SOS + AI issue + vehicle location Flask API.
    • calling_agent.py – Twilio voice bot + Whisper + Gemini/OpenAI.
    • train_model.py – YOLOv8 training script for pothole detection.
    • yolov8n.pt – base YOLOv8 model (lightweight).
    • requirements.txt – dependencies for backend services.

  • ml-dept-classifier/

    • app.py – FastAPI model inference service.
    • models/best_model.pth – trained CNN weights.
    • src/model.py, src/inference.py, src/train.py – model definition and training pipeline.
    • src/streamlit_app.py – Streamlit frontend.

  • user-certificate/

    • templates/certificate_template.png – base certificate design.
    • fonts/ – custom fonts (Alice-Regular.ttf, ARIAL.TTF, Skrine.otf).
    • utils/certificate_generator.py – certificate rendering logic.
    • utils/whatsapp_sender.py – Twilio WhatsApp sender (currently commented/sample code).
    • main.py – orchestrates certificate generation and WhatsApp sending.

  • Root files

    • requirements.txt – consolidated Python dependencies (ML + Streamlit + Twilio + Flask, etc.).
    • package.json – Node dependencies (Google GenAI, Twilio, WebSocket, i18n, etc.).

6. Setup & Prerequisites

  • Python: 3.9+ recommended.
  • Node.js: 18+ (if you plan to use Node utilities / frontend).
  • pip / venv: for managing Python dependencies.
  • ffmpeg: required for Whisper (speech‑to‑text) used in calling_agent.py.
  • Twilio account: for Voice and WhatsApp flows.
  • Google Gemini / OpenAI API keys: for LLM‑based responses.

The services are intentionally decoupled. You can run only the parts you need (for example, just the chatbot + department classifier) or run everything for a full demo.

7. Running the Services

All commands assume you are in:

cd FixMyCity-SIH-2025/SIH-2025f

It is recommended to create separate virtual environments per service to keep dependencies clean.

7.1 Citizen Services Chatbot (agentic-chatbot-user)

Environment variables

Create a .env file in agentic-chatbot-user/ with:

GOOGLE_API_KEY=your_gemini_api_key_here

Install & run

cd agentic-chatbot-user
python -m venv .venv
# Windows
.venv\Scripts\activate
# Linux / macOS
# source .venv/bin/activate

pip install -r requirements.txt
uvicorn main:app --host 0.0.0.0 --port 8001 --reload

The service exposes:

  • POST /chat – body: { "query": "your question" } → { "reply": "answer" }

7.2 SOS + AI Issues + Vehicle Backend (backend_python/app.py)

Environment variables

Some endpoints do not require special env variables, but Twilio‑related features (in calling_agent.py) will.

Install & run

cd backend_python
python -m venv .venv
.venv\Scripts\activate  # or source .venv/bin/activate

pip install -r requirements.txt
python app.py

This starts the Flask app on http://0.0.0.0:5003 (see app.py).

Key endpoints (app.py)

  • POST /report_sos_issue – receive SOS alerts from hardware/simulators.
  • GET /get_sos_alerts – list active SOS alerts.
  • POST /report_issue – ingest AI‑detected issues (e.g. from CV models).
  • GET /get_issues – list all issues.
  • POST /resolve_issue – mark an issue as resolved and remove it.
  • POST /update_location – update a vehicle’s lat/lon.
  • GET /get_locations – fetch current vehicle locations.

7.3 Voice Agent with Twilio & Whisper (backend_python/calling_agent.py)

Environment variables (.env or system)

Set at least:

PUBLIC_BASE_URL=https://your-ngrok-or-public-domain
TWILIO_ACCOUNT_SID=...
TWILIO_AUTH_TOKEN=...
TWILIO_FROM_NUMBER=+1XXXXXXXXXX  # Twilio phone number in E.164 format
GEMINI_API_KEY=...               # recommended
OPENAI_API_KEY=...               # optional fallback

Install & run

Use the same virtual environment created in backend_python or a fresh one:

cd backend_python
python calling_agent.py

This starts a Flask app on http://0.0.0.0:5004.

Configure your Twilio Voice webhook to:

  • Incoming calls → POST {PUBLIC_BASE_URL}/voice

7.4 Department Classifier (ml-dept-classifier)

Install & run FastAPI service

cd ml-dept-classifier
python -m venv .venv
.venv\Scripts\activate  # or source .venv/bin/activate

pip install -r ../requirements.txt  # uses shared requirements
uvicorn app:app --host 0.0.0.0 --port 8000 --reload

Run Streamlit UI

In another terminal (with the same venv activated):

cd ml-dept-classifier
streamlit run src/streamlit_app.py

The FastAPI app (app.py) exposes:

  • GET / – health/info message.
  • POST /predict_url – { "image_url": "https://..." } → prediction JSON.

The Streamlit app (src/streamlit_app.py) lets you upload an image and view the predicted department + confidence.

7.5 Certificate Generator (user-certificate)

Environment variables

For full WhatsApp integration (when you enable the Twilio sender utility), you need:

TWILIO_ACCOUNT_SID=...
TWILIO_AUTH_TOKEN=...
TWILIO_WHATSAPP_NUMBER=whatsapp:+1XXXXXXXXXX

The current utils/whatsapp_sender.py is provided as a template and is commented out. You can adapt and enable it as needed.

Install & run

cd user-certificate
python -m venv .venv
.venv\Scripts\activate  # or source .venv/bin/activate

pip install -r ../requirements.txt
python main.py

main.py demonstrates trigger_certificate_process(...), which:

  1. Generates a PNG certificate using certificate_generator.py.
  2. Builds a public URL using a hard‑coded ngrok URL in the script.
  3. Sends the link via WhatsApp (when Twilio integration is enabled).

You can import trigger_certificate_process into other services (e.g. after an issue is marked resolved in backend_python/app.py) to automatically send certificates.

8. Training & Extending the ML Models

8.1 YOLOv8 Pothole Detector

The script backend_python/train_model.py shows how to fine‑tune YOLOv8:

  • Update dataset_yaml_path in the script to point to your custom dataset’s data.yaml.
  • Run:
cd backend_python
python train_model.py

After training, copy the generated best.pt from:

  • training_results/pothole_detector/weights/best.pt

into the appropriate backend folder and update any inference scripts you add.

8.2 Department Classifier

The CNN‑based classifier is defined and trained using:

  • ml-dept-classifier/src/model.py – model architecture.
  • ml-dept-classifier/src/train.py – training loop.
  • ml-dept-classifier/src/dataloader.py – dataset loading helpers.

You can:

  1. Prepare a labeled dataset of issue images.
  2. Adjust classes and transforms in these files.
  3. Retrain and save a new best_model.pth in models/.

9. Environment Variables Summary

Depending on which modules you run, you may need some or all of these:

  • LLM / AI

    • GOOGLE_API_KEY – for chatbot (Gemini via LangChain).
    • GEMINI_API_KEY – for voice agent’s Gemini calls.
    • OPENAI_API_KEY – optional fallback for voice agent.

  • Twilio (Voice & WhatsApp)

    • TWILIO_ACCOUNT_SID
    • TWILIO_AUTH_TOKEN
    • TWILIO_FROM_NUMBER – for voice calls.
    • TWILIO_WHATSAPP_NUMBER – for WhatsApp messages (when enabled).

  • Public URL

    • PUBLIC_BASE_URL – e.g. ngrok HTTPS URL that Twilio can reach.

Keep secrets in .env files and never commit them to git.

10. Suggestions for Demo Flow

A suggested end‑to‑end demo:

  1. Start SOS + AI backend (backend_python/app.py).
  2. Start department classifier (ml-dept-classifier/app.py) and Streamlit UI.
  3. Start the citizen chatbot (agentic-chatbot-user/main.py) and connect a simple web UI or test via tools like Postman.
  4. (Optional) Start the voice agent (calling_agent.py) and configure Twilio.
  5. (Optional) Start the certificate generator and wire it into the issue resolution flow.

This shows:

  • Issues coming from multiple channels (SOS button, AI CV module, chatbot, voice).
  • Smart routing to departments using ML.
  • Visibility into issues via APIs / dashboards.
  • Citizens receiving digital proof of resolution.

11. Notes

  • Parts of the system (especially Node utilities and frontend) are intentionally minimal or stubbed and can be further expanded.
  • Feel free to adapt port numbers, endpoints, and deployment strategies to your infrastructure.
  • Consider containerizing each service (Docker) for easier deployment in production or on cloud platforms.