Building an ML Processor using CFU Playground (Part 1)
We have built a machine learning (ML) processor on an Arty A735T using CFU Playground. In Part 1, we have accelerated the inference of the Person Detection int8 model by 5.6 times.
Click here for related articles.
 Part 1: Person Detection int8 model (this article)
 Part 2: Keyword Spotting model
 Part 3: MobileNetV2 model
Note: Part 1 was updated on 8/21/2022 to reflect the results of Part 3.
CFU Playground
CFU Playground is a framework for tuning hardware (actually FPGA gateware) and software to accelerate the ML model of Google’s TensorFlow Lite for Microcontrollers (TFLM).
The figure below, quoting cfuplayground.readthedocs.io, gives an overview of the CFU Playground.
CFU Playground utilizes RISCV custom instructions to accelerate the inference of the ML model.
To add hardware for custom instructions, it uses a feature called Custom Function Unit (CFU) provided by the combination of RISCV 32bit soft CPU VexRiscv and SoC Builder LiteX.
For detail, it uses Amaranth HDL (previously nMigen) to generate cfu.v
from Python code and combine it with VexRiscv and LiteX Verilog files to create gateware. You can also create gateware by creating cfu.v
directly.
On the software side, #include "cfu.h"
allows the custom instruction cfu_op(funct3, funct7, rs1, rs2)
to be used within the TFLM.
In addition, there is a mechanism to measure the number of cycles of a specific process by using a performance counter. However, it seems that it may not be available on FPGA boards with limited resources.
Person Detection int8 Model
The Person Detection int8 model, called pdti8
in CFU Playground, is also featured in The StepbyStep Guide to Building an ML Accelerator (hereafter Guide).
This model is a total of 31 layers of CONV_2D
and DEPTHWISE_CONV_2D
(hereinafter dw_conv
) with 14 layers each and AVERAGE_POOL_2D
, RESHAPE
, and SOFTMAX
with 1 layer each.
The CONV_2D
in this model actually works as a 1×1 convolution (hereafter 1x1_conv
).
Building an ML Processor
In the Guide, the target of acceleration is only 1x1_conv
, but this time we also targeted dw_conv
.
Introducing the results first, we were able to speed up the total number of cycles of the pdti8
model from 215.3M to 38.4M, as shown in the featured image and the table below.
Person Detection int8 Model  Cycles  Speedup factor 


Before  After  
1x1_conv 
154.5M  23.2M  6.7 
dw_conv 
60.7M  15.1M  4.0 
Total  215.3M  38.4M  5.6 
In the Guide, the total number of cycles after acceleration is 86M, so it was possible to accelerate compared to this.
Also, when we previously benchmarked the GitHub proj/avg_pdti8
that is accelerating the pdti8
model, the total number of cycles was 67M, so the acceleration of dw_conv
contributed to the result.
Software Specialization
As described in the Guide, software specializations are effective in accelerating the ML model.
The ConvPerChannel
function of CONV_2D
looks like this:
for (int batch = 0; batch < batches; ++batch) { for (int out_y = 0; out_y < output_height; ++out_y) { const int in_y_origin = (out_y * stride_height)  pad_height; for (int out_x = 0; out_x < output_width; ++out_x) { const int in_x_origin = (out_x * stride_width)  pad_width; for (int out_channel = 0; out_channel < output_depth; ++out_channel) { int32_t acc = 0; for (int filter_y = 0; filter_y < filter_height; ++filter_y) { const int in_y = in_y_origin + dilation_height_factor * filter_y; for (int filter_x = 0; filter_x < filter_width; ++filter_x) { const int in_x = in_x_origin + dilation_width_factor * filter_x; // Zero padding by omitting the areas outside the image. const bool is_point_inside_image = (in_x >= 0) && (in_x < input_width) && (in_y >= 0) && (in_y < input_height); if (!is_point_inside_image) { continue; } for (int in_channel = 0; in_channel < input_depth; ++in_channel) { ...
In the ConvPerChannel1x1
function specialized for 1x1_conv
, the filter size is 1×1, so loops and variables can be omitted, and the calculation of is_point_inside_image
is not required, so the code is as follows.
for (int batch = 0; batch < batches; ++batch) { for (int y = 0; y < output_height; ++y) { for (int x = 0; x < output_width; ++x) { for (int out_channel = 0; out_channel < output_depth; ++out_channel) { int32_t acc = 0; for (int in_channel = 0; in_channel < input_depth; ++in_channel) { ...
CFU Optimization
As an example of CFU optimization, a SIMD multiplyaccumulate (MAC)
custom instruction introduced in the Guide has the following input/output.
int8_t int8_t int8_t int8_t +++++ in0 =  input_data[0]  input_data[1]  input_data[2]  input_data[3]  +++++ int8_t int8_t int8_t int8_t +++++ in1 =  filter_data[0]  filter_data[1]  filter_data[2]  filter_data[3]  +++++ int32_t ++ output =  output + (input_data[0, 1, 2, 3] + 128) * filter_data[0, 1, 2, 3]  ++
In addition, funct3
and funct7
are assigned to 2 and 1, respectively.
The inner loop of the original ConvPerChannel
function looks like this:
for (int in_channel = 0; in_channel < input_depth; ++in_channel) { int32_t input_val = input_data[Offset( input_shape, batch, in_y, in_x, in_channel)]; int32_t filter_val = filter_data[Offset( filter_shape, out_channel, filter_y, filter_x, in_channel)]; acc += filter_val * (input_val + input_offset); }
The inner loop of the ConvPerChannel1x1
function specialized for 1x1_conv
corresponding to the above is as follows.
for (int in_channel = 0; in_channel < input_depth; in_channel += 4) { uint32_t input_val_4 = *((uint32_t*)(input_data + Offset( input_shape, batch, y, x, in_channel))); uint32_t filter_val_4 = *((uint32_t*)(filter_data + Offset( filter_shape, out_channel, 0, 0, in_channel))); acc = cfu_op2(1, input_val_4, filter_val_4); }
In the case of 1x1_conv
, since TFLM is the channel last, input_data
and filter_data
of int8_t
can be accessed consecutively.
Summary
We have built an ML processor on an Arty A735T using CFU Playground. The ML processor can perform inference of the Person Detection int8 model in 38.4M cycles.