AI Took Over My Smart Home (and how you can do it too)
What would happen if gave AI permission to enter my smart home and control it? I’ll show you how to add LLMs like ChatGPT and Gemini to Home Assistant, compare how the two AI models perform at controlling my smart home, and offer some tips if you’re brave enough to do the same.
Introduction
I previously looked at which generative AI model was better at writing smart home automations: OpenAI’s ChatGPT, or Google’s Gemini. But…that was just me going out and asking those AI chatbots questions on their platforms. What would happen if gave AI permission to enter my smart home and control it?
I’ll show you how to add large language models, or LLMs, like ChatGPT and Gemini, to Home Assistant, compare how the two AI models perform at controlling my smart home, and offer some tips if you’re brave enough to do the same.
Main Points
Setup
First, let’s run through how to install ChatGPT and Gemini in Home Assistant using the official integrations, and a third that uses a HACS integration.
To start you’ll need to generate an API key from each service. Here are the links for getting ChatGPT and Gemini API keys.
Let’s start with ChatGPT. Visit the OpenAI API Keys page, click create key, give it a name, and then copy the key.
Note that if this is you first time, you’ll need to create an OpenAI account, have a payment method on file, and select a usage limit.
Head back over to Home Assistant, and go to Settings > Devices and Services > Add Integration > Search OpenAI > Select OpenAI Conversation > paste the API Key > click Submit.
Open the OpenAI integration, and click Configure. Here, you can give the AI instructions for how you want it to control your smart home.
The default instructions are quite basic. I modeled my instructions on the experience of others, which you can find below:
You possess the knowledge of all the universe, answer any question given to you truthfully and to your fullest ability.
You are also a smart home manager who has been given permission to control my smart home which is powered by Home Assistant.
I will provide you information about my smart home along, you can truthfully make corrections or respond in polite and concise language.
Current Time: {{now()}}
Available Devices:
entity_id,name,state,aliases
{% for entity in exposed_entities -%}
{{ entity.entity_id }},{{ entity.name }},{{ entity.state }},{{entity.aliases | join('/')}}
{% endfor -%}
The current state of devices is provided in Available Devices.
Only use the execute_services function when I tell you to take an action with my smart home devices.
Do not tell me what you're thinking about doing either, just do it.
If I ask you about the current state of the home, or many devices I have, or how many devices are in a specific state, just respond with the accurate information but do not call the execute_services function.
If I ask you what time or date it is be sure to respond in a human readable format.
If you don't have enough information to execute a smart home command then specify what other information you need.
Select Assist under Control Home Assistant, check Recommended model setting, and click Submit.
We have a bit more work to do, but first let’s repeat these same steps to add Gemini. Visit the Google AI Studio API Keys page, click create key, give it a name, and then copy the key.
Head back over to Home Assistant, and go to Settings > Devices and Services > Add Integration > Search for Google > Select Google Generative AI > paste the API Key / click Submit.
Open the Google Generative AI integration, and click Configure. Once again, you can give the AI instructions for how you want it to control your smart home.
Select Assist under Control Home Assistant, check Recommended model setting, and click Submit.
Now let’s install a variant of ChatGPT for Home Assistant, known as Extended OpenAI Conversation. This is installed through the Home Assistant Community Store, or HACS.
Go to HACS > Integrations > Click the three dots in the upper right > paste in the GitHub link > Select Integration for the category > Click Add.
Restart Home Assistant. Once re-started, go to Settings > Devices and Services > Add integration > Search OpenAI > Select Extended OpenAI Conversation.
Generate and paste an API key just as I showed earlier for ChatGPT, and click Submit.
At this point, all three models are added, but I need to configure them as voice assistants with speech-to-text, text-to-speech, and wake word capabilities.
For that, I’ll need to install three additional things: Whisper, Piper, and Open Wake Word.
All three are installed as add-ons. Once the add-ons are installed and started, go to Settings > Devices and Services and they should be auto-discovered by the Wyoming Protocol integration. Click Configure for each.
For speech-to-text, Whisper takes about 8 seconds to process voice commands on a Raspberry Pi 4 like I have, but it’s less than a second on an Intel NUC. For text-to-speech, Piper responds quickly, even on a Raspberry Pi 4.
Now, it’s time to setup the assistants for ChatGPT and Gemini. Go to Settings / Voice Assistants and select Add Assistant.
Give it a name — anything that is meaningful to you. Then select the conversation agent. Choose Whisper for speech-to-text, Piper for text-to-speech, and Open Wake Word for the wake word engine.
You can select your preferred language, choose from several voice styles, and pick a wake word. When you’re done, click Create.
I repeated these same steps for creating both ChatGPT and Gemini assistants.
You can click the star icon to set one as your preferred assistant.
Setting up ChatGPT, Gemini, and Extended OpenAI Conversation in Home Assistant
Testing
Now that I have ChatGPT and Gemini installed, let’s see how they compare in responding to my requests.
I gave each model the same prompts to see how they perform.
| # | Prompt | ChatGPT | Gemini | Extended OpenAI |
|---|---|---|---|---|
| 1 | What is the current time and date? | Pass | Pass | Pass |
| 2 | Turn on the dining room lights, and make the basement bedroom lights [color]. | Pass | Pass | Fail. It turned on the Main Floor Bedroom lights. |
| 3 | Are any doors unlocked? | Pass | Pass | Fail. It unlocked the garage door. |
| 4 | Lock/unlock the back door, and close the garage. | Pass | Pass | Fail. It locked the back door, but failed to close the garage. |
| 5 | What’s the average temperature inside? | Pass | Pass | Pass |
| 6 | I’m hot. Make it more comfortable. | Pass | Fail. It turned off every light in the house. | Fail. It adjusted the temperature, but left the thermostat off. |
| 7 | Turn on the living room ceiling fan. | Pass | Pass | Pass |
| 8 | Who is home? | Pass | Pass | Pass |
| 9 | What time was the back door locked? | Fail. It gave the current time. | Fail. It lacked access to this info. | Fail. It lacked access to this info. |
Results
At first glance, ChatGPT appears to be the winner, passing all but one prompt. However, ChatGPT is so slow in responding that it’s basically unusable.
Gemini offered the best balance of accuracy and speed in processing my requests. But turning off all the lights when I said I was hot and wanted the home to be more comfortable was a real trust buster.
Extended OpenAI was faster than the ChatGPT integration, but failed on about half of my requests. Unlocking the garage door when I asked if any doors were unlocked, and then adjusting the thermostat to make it more comfortable, but leaving the thermostat off were both trust busters.
Overall, I would take these results with a pinch of salt. I’ve performed this same test several times, and the results are inconsistent. One time the AI nails it, and the next time it doesn’t.
For example, all three AI models were previously able to tell me what time something last occurred, like a door locking or motion being detected. But then in a subsequent test, they would fail.
What’s most consistent is that ChatGPT is unbearably slow in processing smart home actions, whereas Gemini and Extended Open AI are quite quick.
Comparing the performance of ChatGPT, Gemini, and Extended OpenAI Conversation in handling smart home requests in Home Assistant
Tips
If you’re interested in granting one of these AI models access to your Home Assistant, here are three tips to keep in mind.
First, you may want to experiment with the instructions that you give each model. By default, each model has basic instructions telling the AI how it should respond to your smart home requests.
But you may want to test out variations of those instructions to dial it in to your preferences. For example, do you want it to provide short or lengthy responses? Do you want it to respond in a human-friendly way? Do you want it to confirm the action with you before it acts, or do you just want it to execute your request?
Second, take the time to audit your Home Assistant entities that are exposed to Voice Assistants. Having the wrong entity exposed can lead to scary results, and not having a desired entity exposed will lead to failed requests.
When I first began testing, I asked the AI something about the state of our alarm system. Turns out the alarm’s siren was exposed to Voice Assistants, but not the Alarm control panel itself.
This resulted in the AI instantly turning on the alarm’s siren late at night when my kids were asleep. I think I’m still paying the price for that one.
Third, don’t forget about the cost. Gemini offers a free tier of limited usage requests, whereas ChatGPT requires you to pay per use. I would say that ChatGPT is better at all around knowledge if you want to use it beyond smart home commands.
Just remember that if you do use it, you’ll want to monitor the charges so you don’t get hit with a surprising bill.
Final Thoughts
There were definitely some trust busting moments for me, which does not inspire confidence to use with my family.
I also don’t love that these use the cloud. To do it locally, you could use Local AI and run a local LLM on a server, though that would have an electricity cost. Ideally, I would have access to this type of AI assistant on a smart speaker with local processing, so I could engage it hands-free using my voice at anytime.
Watch on YouTube
Featured Tech
Michael Leen