Smartphone light sensor detects hands and movements - without any authorizations

Cameras on a smartphone or tablet can only be activated if authorization has been granted. Webcams can usually be blocked or sometimes even have a physical switch to interrupt the power supply.

The small light sensor leads a much freer life, with almost no controls or restrictions. It is designed to ensure that the display lights up brightly in daylight and shines only dimly at night under the comforter so that it is always perfectly legible. The smartphone does not need to be unlocked for this, nor does it require any authorization from the user.

In addition to smartphones, these sensors can also be found on tablets and modern TV sets, which offer sufficient points of attack, for example with Android TV as software.

The method was demonstrated with an unmodified Android tablet. In just a few steps, the researchers were able to obtain the outline of a hand from the measured values collected by the light sensor, in which individual fingers can also be easily recognized. To do this, the hand must be directly in front of the sensor so that it can determine different brightness values in different areas. Although several image processing steps are necessary, this should not be a major hurdle.

A second method is almost more impressive. While watching media content on the smartphone or tablet, the light sensor recognizes the brightly lit outlines of the face, not very accurately, but quite impressive for a first attempt. And the larger the display of the respective device, the better the results.

However, it was only possible to capture entire images at intervals of 200 seconds. Fluid movements or even certain gestures cannot be determined in this way. Apparently, the technical principles on which the environment sensor is based are too primitive.

Nonetheless, it shows how seemingly harmless helpers in technical devices could be converted for other purposes with manageable effort. The authors of the studies also make recommendations as to how the light sensor should not be designed: With a high frame rate, a usable resolution and aimed directly at the user, spying on them would be conceivable.

On the other hand, a small light sensor that can recognize the outline of a hand is hardly worth mentioning in some homes with active microphones in almost every room.

If you want to try it out for yourself and take your own pictures using the light sensor, you can find the code on Github.