To simplify development, we've made the Edge TPU compatible with the standard TensorFlow Lite API for inferencing—no additional APIs are required. However, TensorFlow Lite's default behavior is to execute each model on the CPU, so this page shows you how to make it run your model on the Edge TPU, using Python.
To execute a TensorFlow Lite model, you must run it through an "interpreter."
In the Python API, that's available with the
By default, TensorFlow Lite executes each model on the CPU. Of course, this fails if your model is compiled for the Edge TPU, so you must instruct the interpreter to run your model using the Edge TPU. You can do that by specifying a TensorFlow Lite delegate for the Edge TPU. Then, whenever the interpreter encounters a graph node that's compiled for the Edge TPU, it sends that operation to the Edge TPU.
Essentially, using the TensorFlow Lite API with the Edge TPU requires that you change just one
line of code: When you instantiate the
Interpreter, you need to specify
the Edge TPU runtime library as a delegate.
Load TensorFlow Lite and run an inference
To use TensorFlow Lite with the Edge TPU delegate, follow these steps:
First, be sure you've set up your device with the latest software.
Install the latest version of the TensorFlow Lite API.
Although you can access the TensorFlow Lite API from the full
tensorflowPython package, we recommend you instead use the
tflite_runtimepackage. This package includes only the
load_delegate()function, which is all that's required to run inference, saving you a lot of disk space.
To install the
tflite_runtimepackage, follow the TensorFlow Lite Python quickstart.
Open the Python file where you'll run inference with the
InterpreterAPI. (For an example, see label_image.py).
Instead of using
import tensorflow as tf, load the
tflite_runtimepackage like this:
import tflite_runtime.interpreter as tflite
Add the delegate when constructing the
For example, your TensorFlow Lite code will ordinarily have a line like this:
interpreter = tflite.Interpreter(model_path)
So change it to this:
interpreter = tflite.Interpreter(model_path, experimental_delegates=[tflite.load_delegate('libedgetpu.so.1')])
The file passed to
load_delegate()is the Edge TPU runtime library, and you installed it when you first set up your device. The filename you must use here depends on your host operating system, as follows:
Now when you run a model that's compiled for the Edge TPU, TensorFlow Lite delegates the compiled portions of the graph to the Edge TPU.
If you started with the label_image.py example, try passing it this model that's compiled for the Edge TPU. You can run that model using the same labels file and test image from the label_image README.
Checkout our other code examples using the TensorFlow Lite API.
If you have multiple Edge TPUs, read how to run multiple models with multiple Edge TPUs.
Or learn more about how to create models compatible with the Edge TPU.
Is this content helpful?