Skip to main content

Arduino Nano 33 BLE SENSE Gesture Mouse

Hi guys, are you an Arduino enthusiast?

The reason is that this is a very easy-to-use development board that can be programmed in C, C++, and even MicroPython programming languages. One of the main reasons I like it better is that in recent years they released the Nano Family, which has a set of embedded sensors such as temperature/humidity, pressure, gestures, microphone, etc. Today, I'd like to share a project using Arduino Nano 33 BLE Sense (192-7592) . If you are an Arduino lover, I believe you must have seen many articles about this development board before, and there are many projects using it to do some AI projects.

But in this project I want to keep it simple and let some beginners know about this board, I will only use the embedded sensors on the board. Hope you will like it!

Arduino Family

Arduino Nano 33 BLE SENSE

Arduino Nano 33 BLE SENSE I would say this is a powerful development board, it has a set of embedded sensors including 9 axis inertial sensor, humidity and temperature sensor, barometric pressure sensor, microphone, gesture, proximity, light colour, and light intensity sensor. I would say whether you are a beginner or an expert, there is at least one sensor that will interest you. If you really don't get interested in those sensors, this development also combines a tiny form factor, different environment sensors and the possibility to run AI using TinyML and TensorFlow™ Lite.

Arduino Nano 33 BLE Sense info

After introducing the content on the development board, we will enter today's project. For a gesture mouse, you can consider using a 9-axis inertial sensor. But this time I wanted to make it a little different, using a gesture detection sensor to detect the movement of my hand to control the cursor. And click the mouse by detecting the volume with the microphone.

Setup

void setup() {
  Serial.begin(9600);
  while (!Serial);

  if (!APDS.begin()) {
    Serial.println("Error initializing APDS-9960 sensor!");
  }

  // for setGestureSensitivity(..) a value between 1 and 100 is required.
  // Higher values make the gesture recognition more sensitive but less accurate
  // (a wrong gesture may be detected). Lower values makes the gesture recognition
  // more accurate but less sensitive (some gestures may be missed).
  // Default is 80
  //APDS.setGestureSensitivity(80);

  Serial.println("Detecting gestures ...");

  // configure the data receive callback
  PDM.onReceive(onPDMdata);

  // optionally set the gain, defaults to 20
  // PDM.setGain(30);

  // initialize PDM with:
  // - one channel (mono mode)
  // - a 16 kHz sample rate
  if (!PDM.begin(1, 16000)) {
    Serial.println("Failed to start PDM!");
    while (1);
  }
}

Mouse

Usually, when you make a mouse in Arduino, usually we use the mouse.h library. But since the library only allows 32u4 and SAMD-based boards (Leonardo, Esplora, Zero, Due, and MKR series), the Arduino Nano 33 BLE Sense is using nRF52840 MCU, so we have to choose another library.

#include "PluggableUSBHID.h"
#include "USBMouse.h"

USBMouse Mouse;

//Mouse moving distance
int d = 50;

USBMouse.h is for setting up any Arduino as a USB mouse. So, in this project, we can use this library.

Gesture Detection

Arduino Nano 33 BLE Sense has an embedded APDS9960 sensor, which is a proximity and gesture detection sensor. Together with the APDS9960 library, which can also control the built-in RGB LED through hand gestures.

#include <Arduino_APDS9960.h>

Through the Arduino_APDS9960.h library, it is possible to read gestures detected by the sensor. Simply put, a sensor can detect motion on the sensor, whether the motion is up, down, left, or right. The detected gesture can be one of the following values: GESTURE_UP: “up” gesture, GESTURE_DOWN: “down” gesture, GESTURE_LEFT: “left” gesture, GESTURE_RIGHT: “right” gesture, GESTURE_NONE: the gesture doesn’t match any of the above. 

To control the mouse cursor to move to different positions, we can use Mouse.move(int16_t x, int16_t y) to control the cursor movement. This means that the cursor has moved a certain distance on (x, y).

if (APDS.gestureAvailable()) {
    // a gesture was detected, read and print to Serial Monitor
    int gesture = APDS.readGesture();

    switch (gesture) {
      case GESTURE_UP:
        Serial.println("Detected UP gesture");
        Mouse.move(0,d);
        break;

      case GESTURE_DOWN:
        Serial.println("Detected DOWN gesture");
        Mouse.move(0,-d);
        break;

      case GESTURE_LEFT:
        Serial.println("Detected RIGHT gesture");
        Mouse.move(d,0);
        break;

      case GESTURE_RIGHT:
        Serial.println("Detected LEFT gesture");
        Mouse.move(-d,0);
        break;

      default:
        // ignore
        break;
    }
  }

Hand gesture controlling mouse pointer

Microphone

After we can control the movement of the mouse cursor, we also need to consider how to click the mouse button which is essential. Click the mouse button in this project, as I use this powerful development board, so I don't want to add any other components to it, I want to use the embedded sensor on the board. That's why I want to use the microphone as the left mouse button.

The development board is using the built-in omnidirectional digital microphone (MP34DT05) to capture. Use the PDM library to implement its functionalities in your projects.

#include <PDM.h>

// buffer to read samples into, each sample is 16-bits
short sampleBuffer[256];

// number of samples read
volatile int samplesRead;

To detect when I click a button on the microphone, we can trigger a mouse click by changing the volume. Also, since the Arduino Nano 33 BLE Sense has a built-in RGB LED, we can also use it as an indicator light to check if I clicked the button. To make the sound level different, we can simply tap the board, or make a loud sound to trigger it.

if (samplesRead) {
    // print samples to the serial monitor or plotter
    for (int i = 0; i < samplesRead; i++) {
      Serial.print(sampleBuffer[i]);
      // check if the sound value is higher than 500
      if (sampleBuffer[i]>=500){
        digitalWrite(LEDR,LOW);
        digitalWrite(LEDG,HIGH);
        digitalWrite(LEDB,HIGH);
        Mouse.click(MOUSE_LEFT);
        delay(100);
      }

      //check if the sound value is higher than 0 and lower than 250
      if (sampleBuffer[i]>=0 && sampleBuffer[i] < 250){
        digitalWrite(LEDG,LOW);
        digitalWrite(LEDR,HIGH);
        digitalWrite(LEDB,HIGH);
      }
    }

    // clear the read count
    samplesRead = 0;
  }

Note that since we don't want the mouse to double-click or even multiple-click at the same time, we need to delay it. But the delay time should not be too long, if you need to double-click, you still need to give enough time.

void onPDMdata() {
  // query the number of bytes available
  int bytesAvailable = PDM.available();

  // read into the sample buffer
  PDM.read(sampleBuffer, bytesAvailable);

  // 16-bit, 2 bytes per sample
  samplesRead = bytesAvailable / 2;
}

using gesture to control mouse on web page

Finally, I hope you enjoy this project. If you have any ideas on how to use the embedded sensor for right-clicking, or you have any comments welcome to leave a comment below.

JulianWong has not written a bio yet…
DesignSpark Electrical Logolinkedin