Soft Haptic Interfaces for Robot Control and VRFollow project
|1||STM32F767 high speed microcontroller board||123-1052|
|4||Lofelt linear actuators||L5|
|2||Adafruit DRV2605L haptic driver breakout boards||ADA2305|
|5||ADXL335BCPZ Analog Devices, 3-Axis Accelerometer, 16-Pin LFCSP||709-0148|
|1||RS PRO 156W Embedded Switch Mode Power Supply SMPS, 24V dc, Enclosed||161-8299|
|2||SMC 2/2 Pneumatic Solenoid Valve - Solenoid/Spring Push In 4 mm VDW Series 24V dc||892-9980|
|2||SMC Pneumatic Solenoid Valve -||893-0007|
|1||Formlabs Elastic 50A (silicone) resin||RS-F2-ELCL-01|
|2||Electronic vacuum regulator||ITV00903BS|
|2||Electronic air pressure regulator||ITV00303BS|
|1||Any vacuum pump||N/A|
|1||Any compressed air supply||N/A|
|1||Hammond 1598 Black Flame Retardant ABS Instrument Case, 249.7 x 159 x 75mm||119-8978|
|1||4mm x M5 pneumatic straight fitting||N/A|
|1||RS PRO Bulkhead Tube-to-Tube Adaptor to Push In 4 mm to Push In 4 mm 10 bar||916-0820|
|1||RS PRO Air Hose Clear Polyurethane 4mm x 30m CPU Series||917-2407|
|2||AD9833 I2C Function Generators||AD9833|
Soft haptics is a field of research at the intersection of soft robotics and haptics, primarily concerned with the design of soft, deformable haptic feedback devices. In recent years, the field has expanded rapidly as soft haptic technologies are ideally suited to the design of wearable haptic devices, which in turn have exciting and potentially lucrative applications in virtual reality entertainment. Soft haptic devices are however equally suitable as alternatives to more traditional haptic feedback devices like vibrating game controllers and force feedback joysticks, with additional functionality such as shape holding and changing as well as controllable softness. Vibrotactile feedback has generally not been explored in the context of soft devices, especially in tandem with other haptic effects. This project aims to investigate how vibrotactile feedback can be generated by a soft haptic device, and combined with the other sensations a soft haptic device can offer.
Particle jamming is used in several engineering fields to dynamically alter the shape and stiffness of small to medium-sized physical bodies, for example, to change the hardness or softness of a computer interface. This project proposes that a controllable stiffness fluid consisting of small particles can be used to control the uniform haptic vibrations produced by a motor. This effect can be achieved by reducing the air pressure in the particles’ container, causing the soft haptic pad to jam them together. This jamming effect causes the body of particles to stiffen, decreasing the motor’s capacity to displace itself within the fluid, and thus the amplitude of the vibrations transferred to the finger pad. By releasing the low pressure, the fluid returns to a free state, allowing the motor to vibrate strongly again.
This effect can then be combined with the inherent change in hardness of the particle fluid, and the change in shape achieved by inflating the device, to create a huge range of haptic effects that can be used in applications ranging from touching different materials in VR to feeling a remote robot driving across rough ground. The effect of this particle jamming system is described in more detail in the video below:
So far, I have designed and started building two interfaces based on the particle jamming system described above. The first of these is a small touchpad, which can vibrate, change its softness and inflate itself to create a domed shape for the user to feel. It also uses load cells to sense the position and force of a user's finger, so can be used as an input device to a computer, much like a laptop's trackpad.
The device consists of a CNC milled aluminium box that is filled with Quinoa seeds and covered on the top with a 3D printed silicone pad. Whilst sheet silicone could be used for this, 3D printing the pad opens up the possibility of creating textured surfaces, which will add yet another tactile sensation when using the device. The vibration is provided by a pair of Lofelt L5 linear resonant actuators (LRAs) and the softness and inflation are controlled by modulating air pressure, which is achieved by a small port to a pneumatic control system. More on that later...
The second device will be a haptic joystick, built by hacking a Logitech joystick apart and replacing the moulded plastic handle with a handle I am building based on the technologies I am researching. This will split the particle jamming effect into two separate pouches meaning that the two sides can be controlled independently and produce different sensations, ideal for driving a robot across rough ground.
The joystick construction is based on a 3D printed resin frame to which large 3D printed silicone pouches are fitted to hold the particle fluid. These also contain vibrating actuators which will be glued into aluminium 'antennae' to carry the vibrations from the base up to the much thinner handle. These are sealed with a machined aluminium clamping ring. The top of the frame is intentionally left flat for the joystick's original thumb controls to be mounted, as these are likely to be needed when controlling the robot. This setup allows for both sides of the joystick to inflate, harden and vibrate independently, giving the user a sense of the robot's roll, pitch, traction, distance to obstacles, interaction with air or water currents, or just about any other metric that a sensor can pick up. This is information that is very difficult to represent visually, and I am hoping that presenting it via haptics will reduce the risks of errors such as collisions or tipping when driving a mobile robot.
An essential part of my project is creating a control system for the various signals that need to be adjusted to create the effects I want users to be able to feel. These are currently:
- Vibration amplitude
- Vibration frequency
- Air pressure (positive to change shape, negative to change softness)
The vibrating actuators I am using are driven by sine waves of varying amplitude and frequency. This makes controlling these two properties quite easy and has the cool side effect of being able to connect them to an audio amp to vibrate in time to music!
The control system for the vibrating actuators is therefore very simple. First, generate a sine wave of the correct amplitude and frequency, then feed it into the haptic driver, where it will be amplified and sent to the vibrating motor. To generate the sine wave, I used the AD9833 function generator. This chip uses I2C to select a waveform (sine, square or sawtooth) and set its frequency, the chip does the rest and outputs a clean, high-frequency wave free of the quantization or latency you get with microcontroller sine generators. Frequency is only half the problem though, and as the AD9833 can't modulate amplitude, I had to find another way of controlling this programmatically. To achieve this, I opted (pun not intended) for a non-inverting amplifier circuit, with the gain selectable with a 'digipot' or digital potentiometer. This allows me to generate a sine wave of fixed amplitude, and then amplify (or reduce) it to the amplitude I need for my vibration signal. It is then a trivial task to send this to the DRV2605 haptic driver chip and set it to audio mode, where it will convert any incoming waveform to a vibration. This is all duplicated to support two vibrating actuators, meaning that different vibrations can be created on different sides of the interfaces.
The control system for the air pressure is more complex. The most complex of my devices, the joystick, requires two independently controlled pockets of the particle fluid, meaning two independent pressure control systems, each able to generate positive and negative pressure. Both sides of the system can use the same sources, which can be generated by any number of pumps and even a vacuum cleaner. After these pressure sources, the systems have to be independent. First, each supply is connected to two electronic regulators from SMC Pneumatics - either an ITV0030-3BS for positive pressure control (to inflate the touch surface) or an ITV0090-3BS for negative pressure control (to harden the particle fluid). Each of these is then fed to a solenoid valve to control whether each section of the interface is recieving positive or negative pressure, before being combined at the connection to the haptic interface. As the positive and negative regulators must never be connected at the same time, this requires 2 analog signals from a microcontroller or DAC to set the pressure on each side, and 4 digital signals via relays to control the solenoids.
A secondary purpose of the controller is to measure the vibrations on the touchpad or joystick in order to measure the effects of changing each control parameter. This is partially for calibration purposes but has also yielded interesting research into the physics of particle jamming. The controller, therefore, uses 5x 3.5mm headphone jacks to connect accelerometers, which can either be used individually or in an 'X' shape to measure the vibration from the interface. Measuring vibration is actually moderately challenging, and the microcontroller had to be upgraded early on in the project from an Arduino Mega to an ARM based STM32F767ZI CPU, in order to read the accelerometers fast enough to read the waveform.
The RS Student Project Fund has been a huge support in buying the components that I need to be able to make a self-contained control system for my haptic devices, especially when I lost access to the pneumatics equipment in my research lab during the COVID-19 lockdowns.
Teleoperation is a field in robotics concerned with the real-time control of remotely acting robots. Such systems may be mobile or fixed-base, aerial or marine, and sometimes incorporate semi-autonomous functionality to take over both mundane and highly complex tasks from the operator. Teleoperated robots offer users the maximum possible degree of control and oversight over pre-programmed and fully autonomous robots, but their overall effectiveness is limited by the attentiveness and capability of the operator. This negates one of the most significant benefits of modern robotics - the elimination of human error. This project aims to generate and evaluate a series of haptic feedback devices based on a novel particle jamming based technology to deliver task and state information that would not be possible with visual and auditory feedback alone. For example, hazards such as loose terrain may not be visually detectable, and with human perception of 720-degree video streams still an active area of research, blind spots are inevitable in real-world systems. These will then be evaluated through user studies in a telerobotics scenario designed to incorporate environmental hazards.
The increased controllability of the vibrotactile response that I hope to demonstrate also suggests that it could prove to be a valid actuation system in human-computer input devices such as joysticks and game controllers. Here, a combination of the particle jamming system and traditional electronic control of the vibrating source could offer a more nuanced control scheme for interaction designers.
With these considerations in mind, the intention of this project is to integrate particle jamming based haptic feedback into the control system for a real-time operated mobile robot. Haptic feedback can have many applications in this domain, from collision avoidance to shared-control, and a selection of these will be investigated via lab-based experiments.
In many ways, telerobotics has important interaction parallels with interactive media such as video games, especially in virtual reality. The mechanics of controlling a remote robot are no different to controlling a virtual game character. Haptic feedback in video games is not new, but the advent of virtual reality has brought about a new wave of efforts to mimic the physical sensations a player would be experiencing were they in the virtual world. As mentioned above, the use of soft materials and techniques in developing my technology positions it as a possible entrant in the growing field of wearable haptic devices being used in virual reality applications. Given the explosion of both commercial and academic research into wearable haptics for virtual reality, this is not a route I intend to pursue in this project.
As this is a PhD project, being written up for the RS Student Project Fund, there is a slight mismatch between the deadline for this article (31st May 2021) and my project (31st March 2023)! I hope this post gives you a flavour of what I am working on, what I hope to achieve and the incredible support the Student Project Fund has been in keeping my work on track through the latest COVID-19 lockdowns. I hope to be able to keep this as a 'living' article where I can add more about my work as I build more of the system and start running field trials with cool tank-style and underwater robots!