Amazingly, it is true that an ESP32 can run a LLM although Dave over at https://github.com/DaveBben/esp32-llm 2 years ago did not find it useful.
However, supplier Espressif have provided the guideline on the application of its more recent models https://www.espressif.com/en/ecosystem/largelanguagemodel .
The ESP32-S31NRV32 with its 32MB of PSRAM should be very capable and this guy ran a 28.9M parameter LLM on an ESP32-S3 with 512KB of SRAM, 8MB of PSRAM and 16MB of flash.
The bulk of the model parameters (embedding tables) are stored directly in the chip’s external flash memory or PSRAM pulling only tiny chunks into SRAM as needed. It’s still early days with this. Currently it might be called experimental but with all of the makers and tinkerers out there it won’t be long before practical applications will be developed around specific use cases.
For us the S-Series in a local voice application in Home Assistant may be a sweet spot we have been looking for. This very likely would involve two ESP32 devices in the stand-alone voice-box. One to handle the load of the LLM and the other to provide the comms and integration to Home Assistant.
Related to this, speech recognition without the internet now fits on a $10 Chip using a package called Whistle. This video gives some explanation of how the LLM functions when analysing speech.
https://www.youtube.com/watch?v=bavO_V80ohM
Highly capable local voice assistant without needing the cloud is coming within reach.
Rob B and Sean C.