An RTL-SDR Based Smartwatch for Detecting Objects Touched by the Wearer

Disney Research have just released a paper describing an RTL-SDR based smart watch that they’ve developed a proof of concept for. The smart watch is unique in that it can be used to actually detect the exact object that the wearer is touching.

The prototype watch does this by using the RTL-SDR to detect the electromagnetic (EM) noise emitted by particular objects and compare it against a stored database. They call this technology EM-Sense. In the paper the authors summarize:

Most everyday electrical and electromechanical objects emit small amounts of electromagnetic (EM) noise during regular operation. When a user makes physical contact with such an object, this EM signal propagates through the user, owing to the conductivity of the human body. By modifying a small, low-cost, software-defined radio, we can detect and classify these signals in real-time, enabling robust on-touch object detection. Unlike prior work, our approach requires no instrumentation of objects or the environment; our sensor is self-contained and can be worn unobtrusively on the body. We call our technique EM-Sense and built a proof-of concept smartwatch implementation. Our studies show that discrimination between dozens of objects is feasible, independent of wearer, time and local environment.

The frequencies required for EM detection are around 0 – 1 MHz which falls outside the range of the RTL-SDR’s lowest frequency of 24 MHz. To get around this, they ran the RTL-SDR in direct sampling mode. The RTL-SDR is connected to the watch, but a Nexus 5 smartphone is used to handle the USB processing which streams the signal data over WiFi to a laptop that handles the signal processing and live classification. In the future they hope to use a more advanced SDR solution, but the RTL-SDR has given them the proof of concept needed at a very low cost.

An example use scenario of the watch that Disney suggests is as follows:

Home – At home, Julia wakes up and gets ready for another productive day at work. Her EM-Sense-capable smartwatch informs and augments her activities throughout the day. For instance, when Julia grabs her electric toothbrush, EMSense automatically starts a timer. When she steps on a scale, a scrollable history of her weight is displayed on her smartwatch automatically. Down in the kitchen, EM-Sense detects patterns of appliance touches, such as the refrigerator and the stove. From this and the time of day, EM-Sense infers that Julia is cooking breakfast and fetches the morning news, which can be played from her smartwatch.

Fixed Structures – When Julia arrives at the office, EMSense detects when she grasps the handle of her office door. She is then notified about imminent calendar events and waiting messages: “You have 12 messages and a meeting in 8 minutes”. Julia then leaves a reminder – tagged to the door handle – to be played at the end of the day: “Don’t forget to pick up milk on the way home.”

Workshop – In the workshop, EM-Sense assists Julia in her fabrication project. First, Julia checks the remaining time of a 3D print by touching anywhere on the print bed – “five minutes left” – perfect timing to finish a complementary wood base. Next, Julia uses a Dremel to cut a piece of wood. EM Sense detects the tool and displays its rotatory speed on the smartwatch screen. If it knows the task, it can even recommend the ideal speed. Similarly, as Julia uses other tools in the workshop, a tutorial displayed on the smartwatch automatically advances. Finally, the 3D print is done and the finished pieces are fitted together.

Office – Back at her desk, Julia continues work on her laptop. By simply touching the trackpad, EM-Sense automatically authenticates Julia without needing a password. Later in the day, Julia meets with a colleague to work on a collaborative task. They use a large multitouch screen to brainstorm ideas. Their EM-Sense-capable smartwatches make it possible to know when each user makes contact with the screen. This information is then transmitted to the large touchscreen, allowing it to differentiate their touch inputs. With this, both Julia and her colleague can use distinct tools (e.g., pens with different colors); their smartwatches provide personal color selection, tools, and settings.

Transportation – At the end of the day, Julia closes her office door and the reminder she left earlier is played back: “Don’t forget to pick up milk on the way home.” In the parking lot, Julia starts her motorcycle. EM-Sense detects her mode of transportation automatically (e.g., bus, car, bicycle) and provides her with a route overview: “You are 10 minutes from home, with light traffic”.

The EM-Sense watch detecting a door. The RTL-SDR dongle is the small square box under the watch.