How does Notebookcheck test laptops and smartphones? A behind-the-scenes look into our review process

Ever since Notebookcheck started about 20 years ago, we have measured and tested thousands of laptops and smartphones. For our reviews, we have developed specific criteria aimed at testing all aspects of the device that are relevant to the user.

Apart from traditional benchmarks, our reviews also encompass numerous hardware tests. Since our testing also covers smartphones, tablets, and convertibles in addition to laptops, the criteria for each device category change accordingly.

New, advanced technologies are being developed constantly. We endeavor to keep our testing at pace with the changing technological landscape.

Below, you will find an overview of all of our review criteria and details on each individual aspect of the devices that we test.

We keenly evaluate the chassis and build quality of the test devices we receive and evaluate them on the following aspects:

• Design (colors and design)
• Materials
• Feel (haptics)
• Dimensions
• Weight
• Workmanship (gap dimensions, paintwork, edges, accuracy of fit, accuracy of fit of the individual parts, etc.)
• Stability (behavior under pressure — with a finger press or hand, torsion resistance, etc.)
• Hinges (force required, rocking, opening angle)

The evaluation of individual points is also at the discretion of the respective editor and is carried out after consultation with the editorial team and the responsible senior editor. A comparison with other test devices of the class is also drawn up.

We evaluate the range of available ports, connectors, and their positioning, including any additional functions and communication features offered.

The available communication modules such as LAN, WLAN, Bluetooth, and mobile communications are assessed.

For smartphones, in addition to a practical test of the telephony functions, we also carry out a standardized Wi-Fi test connected to our reference Wi-Fi 6E router, the Asus ROG Rapture GT-AXE11000.

If the review device features an SD card reader, it is subjected to a test of expected transfer rates using the Angelbird AV Pro V60 128 GB microSD — our current fast reference SD card. 

The transfer rate is tested when copying large blocks of data (such as videos) from the SD card to the test device, by using the AS SSD Seq. Read test, as well as while copying many individual images.

250 images of around 5 MB are transferred three times from the SD card to the test device. The transfer time is automatically logged by a script developed in-house.

The values of some particularly fast or slow devices are shown below for comparison, including some devices that offer middling SD card reader transfer speeds.

We currently use the Asus ROG Rapture GT-AXE11000 tri-band router for our Wi-Fi performance tests. The review sample is placed at a distance of one meter from the router on the same level.

A server is connected to the router via 2.5 GbE LAN. iPerf3 with settings -i 1 -t 30 -w 4M -P 10 -O 3 is used to determine the transmission speed between the server and the test device in both directions.

The fastest transmission standard supported by the test device up to Wi-Fi 6E is used, as well as an older standard such as Wi-Fi 5. 30 data blocks are transmitted during the test. This helps us determine whether the connection with the router is stable or subject to fluctuations.

We test the front (webcam/selfie camera) and rear camera (main camera on smartphones) and evaluate the achievable image quality using standardized reference cards, as well as in direct practical comparison with current flagship devices.

Image sharpness, color reproduction, contrast, light sensitivity, and video function are evaluated among other parameters.

The following is an image comparison using the 2016 Samsung Galaxy. Three different scenes are used to compare various available camera lenses.

The main image can be navigated around to reveal details in comparison with other devices.

Devices should last as long as possible and be repairable in the event of a failure or emergency. That's why, wherever possible, we also take a look inside our review units and check what upgrade and repair options are available.

In terms of sustainability, we also check the packaging for the use of environmentally friendly materials, the spare parts supply offered by the manufacturer, software update plan, and hardware warranty.

We also try to provide information on the expected CO₂-equivalent emissions of a device, if provided by the manufacturer.

Whether laptops or smartphones, all devices come with different input devices. These include keyboards, touch or click pads, touchscreens, input pens or, in the case of gaming handhelds, controller elements.

We examine the following with respect to keyboards:

• Key layout (arrangement, size, grouping, overview, function keys, labeling)
• Typing feel (stroke distance, pressure point, stroke, background noise)
• Additional keys

For touchscreens and touchpads:

• Response behavior (surface, multi-touch, precision, smoothness)
• Mouse replacement buttons (operation, background noise)
• virtual keyboard (layout, feedback, responsiveness, key size)

Of course, we also look at sensors, digitizers, digital input pens and other input methods.

Pulse width modulation (PWM) or other technical limitations can cause screens to flicker. With PWM, the brightness of the display's backlight is not dimmed, but the perceived brightness is reduced by switching the lighting on and off at very short intervals.

Depending on the frequency used, this can have undesirable effects (fatigue, headaches, etc.) on sensitive people. As a general rule, the higher the frequency, the less likely it is to affect the user. A list of all tested devices, sorted by flicker frequency, can be found here.

Part of the analysis is also the evaluation of the device's display for outdoor legibility. Here, it is tested practically how the built-in display copes with particularly bright ambient light conditions (3,000-10,000 cd/m², overcast to sunny conditions) and whether the displayed content is still easy to read.

Decisive factors include type of display surface (matte panels prevent annoying reflections), any reflection-reducing coatings, brightness of the displayed image and, as a result, image contrast.

Depending on the device class in question and the expected area of application, we test the performance of the respective model using various benchmark tests. These are programs that test individual components or the entire system and rate them by the number of points achieved.

We also run several practical tests with various programs and games, as they can often be used to test the processor and graphics card to the limit. If the games do not offer a representative integrated benchmark, we use CapFrameX to record the frame rates.

Before the tests, the system is brought up to date with Windows updates. If actively suggested by the system, the latest graphics drivers are also installed for devices with a dedicated graphics solution. No changes are made to the clock rates on the test system unless this is explicitly stated for demonstration purposes.

A driver update after our tests and, of course, any modifications can often lead to improvements in gaming and benchmark performance. In our opinion, it is the responsibility of the manufacturer to equip its system with up-to-date and powerful software at the time of delivery and to make updates as convenient as possible for the user.

The following areas are tested in terms of performance:

• CPU (Cinebench, Blender, etc...)
• System (PCMark, CrossMark, etc...)
• DPC latencies (LatencyMon and standardized test routine)
• Mass storage (CrystalDiskMark, AS SSD, and DiskSpd read loop)
• GPU (3DMark, Geekbench, etc...)
• Gaming (a mix of current games and popular older titles).

Further information:

• Comparison of mobile graphics cards: Detailed information, technical data, and performance values of all available mobile graphics cards, sorted according to their performance
• Benchmark list of mobile graphics adapters: Sortable table in which all available mobile graphics cards can be sorted according to various benchmarks and technical data
• Smartphone graphics adapter benchmark list: Sortable table with all tested smartphone graphics solutions
• Benchmark list of mobile CPUs: Sortable table in which all available mobile processors can be sorted according to various benchmarks and technical data
• Smartphone processors benchmark list: Table with all tested processors primarily used in smartphones
• Game performance list: Which GPU can play current games smoothly and at which settings
• Laptop SSD and HDD benchmarks: Table with numerous benchmarks of solid state drives and hard disks
• DPC latency ranking: Results table and information on the topic of DPC latencies and real-time applications

The speakers are assessed in terms of their sound and in combination with the maximum possible volume. For our measurements, we again used the Earthworks Audio M23R microphone and the measurement software ARTA in accordance with the standardized test setup.

We use pink noise as the output file. Here, humans perceive all frequency ranges of the audible sound spectrum at approximately the same volume. Measurements are taken at maximum volume and, if necessary, also at lower levels if we detect clipping.

Different devices can be compared with each other in the frequency diagram provided. Individual frequency curves can be shown or hidden by selecting or deselecting the selection boxes. Ideally, all frequency ranges of the audible spectrum (125 Hz to 16 kHz) should be equally loud. The overall volume should be at least 80 dB(A).

We currently differentiate between four different battery run times in our tests:

• Minimum runtime: Determined during operation under load with the help of the BatteryEater software tool (Classic Test). The following settings are made on the test device: maximum display brightness, all communication modules switched on (WLAN, Bluetooth), Windows power profile set to Best performance.
  For Android-based devices, the Burnout-Benchmark app is used as a substitute.
  For iOS-based devices, we use PUBG in the lobby (Ultra HD details, max frame rate).
• Maximum runtime: Also determined with the BatteryEater tool (Reader's Test). The possible run time is measured with minimum display brightness, all power-saving functions activated, WLAN and Bluetooth deactivated, and Windows power profile set to Best power efficiency.
  For Android-based devices, a script that calls up pages of a text document in the browser is used as a substitute: http://www.notebookcheck.com/fileadmin/Notebooks/book1.htm
• WLAN operation: Achievable runtime in WLAN operation with adjusted display brightness (~150 cd/m²) and activated power-saving measures (power profile Balanced or similar). For the run time test, an automatic script (updated on 05.03.2015 to v1.3 with HTML5, javascript, and no flash) is executed, which calls up a mix of Internet pages that change constantly every 30s.
• Video playback: We perform our H.264 test with the freely available Big Buck Bunny Film in 1080p and H.264 encoding in a continuous loop. The power-saving measures are activated, the screen is set to a brightness of 150 cd/m², Wi-Fi and Bluetooth are switched off (flight mode), and the sound is set to silent.
• Gaming: Battery life while playing Cyberpunk 2077: Phantom Liberty in 1080p Ultra without upscaling.

With regard to the test results, it should be noted that our test samples are generally new devices and full battery performance can only be achieved after several complete charging and discharging cycles. The tests carried out are thus only indicative of the review device's battery life.

Every product tested by us as part of a detailed report is assessed in all the areas listed above and compared with competing options in the same class. The evaluation result is made up of a total of 12 points that define the number of points achieved as a percentage on a scale of 0-100 (the higher the better).

Finally, the reviewed device receives an overall evaluation in which the individual evaluation points are taken into account to varying degrees with regard to the device class in question. Except the chassis and input devices, all evaluation criteria are proposed automatically and based on a database using a special algorithm, whereby the collected measurement and benchmark data forms the basis.

You can find more information about our new rating system in this article along with our points weighting table. Our constantly updated Rating Table gives a summary of all rated devices in our reviews at a glance.