Notebookcheck Logo

Our Test Criteria

In this article, you will find information about the tests we run on our test models and how we process the results.

Introduction

Our editors have years of experience and have tested numerous notebooks from various manufacturers. We used this extensive experience to create a catalog of test criteria which cover all purchase relevant aspects of a device. Consumer's demands have changed over the years and technologies evolved. As such, we must also constantly update and add to our test criteria so as to give the reader the information they are looking for. As a reader, you can contribute greatly with your input. We are always happy to receive an email or a new post in our forum. Our editors and a long line of helpful moderators are available and will gladly take your tips.

The following are the main aspects of every review performed by Notebookcheck.net (explanations are included):

Case

The case of our test model is examined very closely. We judge the test model with the following criteria: design (colors, shape, material, feel, measurements, weight, etc.), build quality (gaps, finish, edges, precision, how secure every component sits, etc.), sturdiness (how the notebook reacts to pressure - at a single point, on the entire surface, torsion resistance, etc.), hinges (power, how well they hold the display, longevity, etc.), maintenance (possible upgrades which can be performed by the user, cleaning of the device/cooler fan, etc.). The respective editor rates the build quality according to his judgment by comparing it to previously tested models. The rating is discussed with the editorial team and the respective divisional director.

Connectivity

We judge the ports and interfaces available on the test model and their positioning in consideration of the device category.

SD-Card Reader

If a test model features an SD-card reader, we test it to estimate the transfer rates you can expect. For this purpose, we use our respective reference SD cards, which are currently a Toshiba Exceria Pro SDXC UHS-II for full-size readers and a Toshiba Exceria microSDHC UHS-I for devices with MicroSD slots. We test for the maximum transfer rate you can expect when copying big data blocks (e.g. videos) from the SD card to the test device by using the AS SSD Seq. Read test and for the expected transfer rate when copying many images (about 1 GB .jpg files; about 5 MB each, standardized test files), which is usually significantly lower than the maximum transfer rate. The following figures show the current minimum and maximum values as well as the average as available in our database (as of 07/2016).

SD Card Reader - maximum AS SSD Seq Read Test (1GB)
max. (Surface Book)
246 MB/s
average (Lenovo B50-10)
87.3 MB/s
min. (Acer E5-552G)
10.2 MB/s
SD Card Reader - average JPG Copy Test (av. of 3 runs)
max. (Precision 7710)
213 MB/s
average (Zenbook UX303UB)
64 MB/s
min. (Lenovo E31-70)
10.3 MB/s

Communication Features

We evaluate the communication features, such as LAN, Wi-Fi, Bluetooth, 3G, 4G, etc. In addition to a real-world test of the telephone features in smartphones, we also run a standardized Wi-Fi test.

Wi-Fi Performance

In a standardized test setup, we determine the maximum transfer rates (sending and receiving) when connected to our Asus ROG Rapture GT-AXE11000 (Wi-Fi 6E) (previously Netgear Nighthawk AX12 12-Stream Wi-Fi 6 Router RAX120 or Linksys EA8500) reference router (in 1 m distance) with the iperf3 (attributes: -i 1 -t 30 -w 4M -P 10 -O 3) software. This test uses the fastest transfer standard supported by the test model (up to Wi-Fi 6, additional Wi-Fi 6E tests done if supported). Below you can see three examples with the current min, avg. and max values from our tests (May 2017). The server is connected with 2.5 GBit wired connection from the router (e.g. using Intel 10Gbit NICs).

WiFi performance: receive

Networking - iperf3 receive AX12
max. (MBP13 2016)
Broadcom BCM15700A2 802.11ac
950 MBit/s
average (Nexoc G739)
Intel Dual Band Wireless-AC 8260
443 MBit/s
min. (Elephone S7)
802.11a/b/g/n
19.8 MBit/s

WiFi performance: transmit

Networking - iperf3 transmit AX12
max. (MBP15 2016)
Broadcom BCM15700A2 802.11ac (Klaus I211)
949 MBit/s
average (iPad 2017)
802.11 a/b/g/n/ac
389 MBit/s
min. (AGM A8)
802.11a/b/g
18.4 MBit/s

GPS

GPS test: in a building
GPS test: in a building
GPS test: outdoors
GPS test: outdoors

We run a real-world test on mobile devices which come with a GPS module. While the editor cycles along a route, he records it with the test model and our reference navigation device. A comparison of the data allows us to judge precision and reliability of the incorporated GPS module. In addition, we record the GPS signal inside and outside of buildings.

Test model: overview of the route
Test model: overview of the route
Test model: crossroads
Test model: crossroads
Test model: landmark
Test model: landmark
Garmin Edge 500 reference: overview of the route
Garmin Edge 500 reference: overview of the route
Garmin Edge 500: crossroads
Garmin Edge 500: crossroads
Garmin Edge 500: landmark
Garmin Edge 500: landmark

Camera

We test the front camera (webcam/selfie camera) and the rear camera (primary camera of smartphones) and evaluate the quality by comparing the image to standardized reference cards and the image quality of current flagship models. Among others we judge sharpness, colors, contrast, sensitivity to light and video features available.

Photo of the reference card
Photo of the reference card
Detail of the reference card
Detail of the reference card
ColorChecker: photo of the colors. The lower half of each patch shows the original colors.
ColorChecker: photo of the colors. The lower half of each patch shows the original colors.

The following example compares the image quality of the Samsung Galaxy J5 2016 to the reference camera's and other models'. The main image allows selecting a part for comparison with other devices by moving the mouse pointer over it.

Image Comparison

Choose a scene and navigate within the first image. One click changes the position on touchscreens. One click on the zoomed-in image opens the original in a new window. The first image shows the scaled photograph of the test device.

Scene 1Scene 2Scene 3
click to load images

Input Devices

The following criteria are taken into consideration: keyboard - keyboard layout (positioning, size, grouping, function keys, inscription, etc.), typing experience (key travel, pressure point, stroke, noise, etc.), additional keys if available; touchpad - response (surface, multi-touch, etc.), mouse keys (use, noise, etc.), touch display - response (precision, reaction time, etc.), virtual keyboard (layout, feedback, response, key size, etc.), sensors, digitizer if available (capacitive display).

Display

The following criteria are considered for the display rating based on the measurement results: resolution and format (pixel density, clarity of the display, ease of use with multiple windows, etc.), display brightness [cd/m²] (maximum, minimum, mains operation/battery mode, etc.), brightness distribution (dark areas, bleeding, etc.), contrast (max., black value, etc.), colors (DeltaE ColorChecker, Grayscale), covered color space (sRGB, AdobeRGB98), viewing angles, glare, PWM, response time.

Display brightness: measurement segments
Display brightness: measurement segments

Brightness, brightness distribution, and contrast

We use an X-Rite i1 Pro 2 photo spectrometer together with the latest version of the CalMAN Ultimate software for our display measurements. The measurement of the brightness is taken after the screen has stayed 100% white for a period of 10 minutes. Device settings, such as automatic adjustment of brightness, are deactivated and the color profile of the device has factory settings (not user defined).

The black value is also measured after a time period of 10 minutes, during which time the screen is 100% black (@max. brightness). The measurements are then taken from the central area of the screen while in a completely dark room. These measurements are used by us to calculate the maximum contrast of the display in the central area of the screen. We calculate the illumination by comparing the brightest segment of the screen to its darkest quadrant.

Outdoor use is part of the display test. We check to see how well the display can display content (legibility, reflections, etc.) in bright surroundings (3000-10000 cd/m² - cloudy to sunny). The list of criteria for this test is: type of display surface (matte panels prevent reflections), brightness of the picture, and contrast of the picture.

Colors: out-of-the-box vs. calibrated

The naked eye can hardly detect color deviations from its ideal (sRGB) if the DeltaE is smaller than 3. This is already very difficult for a DeltaE smaller than 5, while differences are increasingly notable at higher deviations (see following figure, actual - target). Hence, devices for editing graphics, images and videos should have a DeltaE smaller than 3 and support calibration well.

In addition, we calibrate devices meant for graphics work and measure the color precision again. You can download the created .icc profile from the review page.

CalMAN Grayscales
CalMAN Grayscales
CalMAN ColorChecker
CalMAN ColorChecker
CalMAN Saturation
CalMAN Saturation
CalMAN Grayscales calibrated
CalMAN Grayscales calibrated
CalMAN ColorChecker calibrated
CalMAN ColorChecker calibrated
CalMAN Saturation calibrated
CalMAN Saturation calibrated

Color Space

Apart from precise colors, the covered color space is important for professional graphics and image editing. We check the coverage of the sRGB and AdobeRGB98 color spaces. For this, we use the .icc file created by the iProfiler and the Argyll software.

sRGB color space 100%
sRGB color space 100%
AdobeRGB color space 74%
AdobeRGB color space 74%
ThorLabs PDA100A-EC or PDA100A2sensor to detect PWM and measure response times
ThorLabs PDA100A-EC or PDA100A2sensor to detect PWM and measure response times

Response Times

We measure how fast the display is able to change from white to black (0% to 100%) and gray to gray (50% to 80%) with an oscilloscope and a switchable gain, amplified silicon detector from ThorLabs (PDA100A-EC and PDA100A2). Fast response times are particularly important for displays designed for gaming. For example, such displays feature response times of only a few milliseconds.

Display Response Times

Display response times show how fast the screen is able to change from one color to the next. Slow response times can lead to afterimages and can cause moving objects to appear blurry (ghosting). Gamers of fast-paced 3D titles should pay special attention to fast response times.
       Response Time Black to White
17 ms ... rise ↗ and fall ↘ combined↗ 4 ms rise
↘ 13 ms fall
The screen shows good response rates in our tests, but may be too slow for competitive gamers.
In comparison, all tested devices range from 0.1 (minimum) to 240 (maximum) ms. » 33 % of all devices are better.
This means that the measured response time is better than the average of all tested devices (21.6 ms).
       Response Time 50% Grey to 80% Grey
25 ms ... rise ↗ and fall ↘ combined↗ 9 ms rise
↘ 16 ms fall
The screen shows relatively slow response rates in our tests and may be too slow for gamers.
In comparison, all tested devices range from 0.2 (minimum) to 636 (maximum) ms. » 29 % of all devices are better.
This means that the measured response time is better than the average of all tested devices (34 ms).

PWM - Pulswidth Modulation

As the response times, we also test for PWM regulation of the backlight using the PDA100A-EC probe. The display is set to 100% brightness and a white background. We then reduce the brightness and look for a PWM signal. Low PWM values (e.g. 200 Hz) can lead to sore eyes, headache and eye fatigue for some users. A list of all tested devices can be found here: PWM list. An explanation on what PWM is can be found here.

Screen Flickering / PWM (Pulse-Width Modulation)

To dim the screen, some notebooks will simply cycle the backlight on and off in rapid succession - a method called Pulse Width Modulation (PWM) . This cycling frequency should ideally be undetectable to the human eye. If said frequency is too low, users with sensitive eyes may experience strain or headaches or even notice the flickering altogether.
Screen flickering / PWM detected 200 Hz

The display backlight flickers at 200 Hz (worst case, e.g., utilizing PWM) .

The frequency of 200 Hz is relatively low, so sensitive users will likely notice flickering and experience eyestrain at the stated brightness setting and below.

In comparison: 53 % of all tested devices do not use PWM to dim the display. If PWM was detected, an average of 18170 (minimum: 5 - maximum: 3846000) Hz was measured.

Viewing Angles

Among others, different display technologies differ in viewing-angle stability. Currently, IPS displays, which allow very flat angles without image distortions, are wide-spread in higher-priced notebooks and particularly in tablets and smartphones. Cheap notebooks often use TN displays, which tend to be very dependent on the viewing angle, but they deliver better response times. We test the viewing-angle stability of the notebook subjectively (using the test model) and by turning the display to fixed angles (45 ° turns). For this test, the display is photographed at a fixed shutter speed and aperture inside a dark room.

Comparison: IPS display
Comparison: IPS display
vs. TN display
vs. TN display

Performance

Our performance tests vary depending on the device class and the expected use. This section includes a variety of benchmarks (which either test a single component or the entire system and present the result in points), and real-world tests with various programs and games, since these often stress the processor and graphics card extremely. If the game does not offer a representative integrated benchmark we usually use CapFrameX to record the framerate.

Prior to the tests, the system is brought up-to-date with Windows updates. In addition, the latest graphics drivers are installed on devices with dedicated graphics solutions if the system actively suggests doing so. We do not change the clock rates unless we actually mention that we did so for demo purposes. Updating the graphics drivers after our tests and possible modifications can certainly boost gaming and benchmark performance. In our opinion, it is the duty of the manufacturer to provide customers with the latest drivers out-of-the-box and make useful updates as easy as possible for the user.

We consider the following aspects for evaluating the performance: CPU (Cinebench, Turbo analysis, etc.), system (PCMark), storage device (HD Tune, CrystalDiskMark, AS SSD, etc.), GPU (3DMark, Unigine Heaven, etc.), gaming (a selection of current games and popular older games).

Additional information:

Emissions

Noise emissions: position of the gauge
Noise emissions: position of the gauge

System Noise

We use a noise level gauge (Audix TM1), the measurement software ARTA and a standard test setup to measure the emissions of a test model. The gauge is fixed 15 cm from the notebook and is secured against vibrations emanating from the test model. The measurements are taken in dB(A) (Decibel). The following are our test categories:

  • Idle
    Minimum: minimum noise emission while laptop is idle (Windows power plan: "Energy Saving")
    Medium: average noise emission recorded while laptop is idle (power plan: "Energy Saving")
    Maximum: highest noise emission measured while the laptop is idle (power plan: "High Performance")
  • Load
    Medium: average noise emission while the computer is running at high level of system use (3DMark06, power plan: "High Performance")
    Maximum: highest possible noise emission while the system is under heavy load (power plan: "High Performance", 100% CPU and GPU usage - thanks to Prime95 and Furmark)

The following may help the reader better understand the results: 

In a quiet room, the human ear can hear background noise, which should amount to around 28 dB. A conversation at a normal volume ranges at 60 dB. All these values are dependent on the distance from the source of the noise. This is why we fix our gauge into place at a constant distance from our test models. This allows us to get clear results which can be compared with each other. The measurements are presented graphically and can be judged subjectively (deviations caused by different frequencies are possible):

  • Under 30 dB: barely audible.
  • Up to 35 dB: audible but not distracting. Ideal level of noise emission for a laptop running office programs.
  • Up to 40 dB: clearly audible, and might be distracting after a while.
  • Up to 45 dB: might disturb the user if they are in a quiet room. Still acceptable while playing games. 
  • Over 50 dB: notebook emissions over this level are uncomfortably loud.

Noise Characteristics

In addition to our measurements, we record a frequency diagram of each fan level. This allows judging whether the perceived noise is rather low or high frequency. In our measurements, the audible range starts from about 125 Hz and reaches up to about 16000 Hz depending on the volume.

Three fan levels, brown, green, and red: You can see a bigger increase in the higher frequency range at higher rotation speeds here.
Three fan levels, brown, green, and red: You can see a bigger increase in the higher frequency range at higher rotation speeds here.
Comparison: audible range, music, and entertainment.
Comparison: audible range, music, and entertainment.
Measurement quadrants for the surface temperature
Measurement quadrants for the surface temperature

Temperature

The distribution of surface temperature (which can be felt by the user directly) is measured with an infrared thermometer (Raytek Raynger ST or similar) which never touches the test model. The top and bottom of the notebook is split into nine quadrants, and the maximum measurable temperature in each quadrant is recorded. To verify the measurement, we use a calibrated Fluke T3000 FC on hotspots and recalculate the result if necessary.

The measurements are taken after an idle period of 60 minutes and a stress period of 60 minutes (100% CPU and GPU usage - Prime95 and Furmark). 

In addition we also closely observe the GPU and CPU during the stress test with software (tools: HWInfo64, HWMonitor, GPUz, etc.) and note any significant variations in performance (drops due to throttling).

The following scale describes the categories we put our measurements in:

  • Less than 30 °C: Barely noticeable increase in temperature.
  • 30 - 40 °C: Temperature rises noticeably but is bearable.
  • 40 - 50 °C: Contact with the notebook over a long period of time at these temperatures will be uncomfortable.
  • Over 50 °C: Very hot. Problematic if using the notebook on the lap.

In addition to the mentioned maximum values we create thermographic images (Flir One), which depict the distribution of the surface temperatures in a continuous way.

Distribution of the surface temperature: idle
Distribution of the surface temperature: idle
Distribution of the surface temperature: load
Distribution of the surface temperature: load

Speakers

We rate the speakers based on sound quality and performance at maximum volume. Once again we use a measurement microphone (Audix TM1), the measurement software ARTA and a standardized setup for our measurements. We use Pink Noise as an output file, measured at maximum volume and we add measurements at lower volume when we detect overdrive.

Our frequency diagrams allow comparing different devices to each other and selecting / deselecting each frequency curve by means of checkboxes.

dB(A) 0102030405060708090Deep BassMiddle BassHigh BassLower RangeMidsHigher MidsLower HighsMid HighsUpper HighsSuper Highs2035.335.12532.931.83131.8324036.535.15032.428.9633328.98036.328.810048.32712561.52716052.924.820060.92425062.822.731563.32240069.521.250067.82163074.82080075.919.4100072.718.912507117.7160070.117.820006917.6250071.817.6315068.117.6400071.417.6500073.717.6630070.417.5800071.617.61000071.617.61250069.617.41600059.717.5SPL83.630.6N62.51.5median 69.6median 17.8Delta4.72.434.334.53432.630.833.334.534.633.635.132.232.634.329.136.228.348.128.841.525.954.724.650.423.857.223.468.222.569.621.871.220.968.719.876.519.372.71976.618.673.318.272.117.964.91867.417.870.517.869.91864.21858.817.959.517.952.617.983.731.257.11.6median 67.4median 198.22.1hearing rangehide median Pink NoiseApple MacBook 12 (Early 2016) 1.1 GHzDell XPS 13 2016 9350 (FHD, i7-6560U)

Power Consumption

Metrahit Energy Multimeter
Metrahit Energy Multimeter

Apart from the battery life, the power consumption of a notebook is measured in various scenarios (power adapter sided). The test settings for each scenario are as follows:

    • Idle: power consumption while the notebook is idle.
      Minimum: all additional modules are off (Wi-Fi, Bluetooth, etc.), minimum brightness, and Windows power plan is set to "Energy Saving".
      Medium: maximum brightness, additional modules off, Windows power plan: "Balanced".
      Maximum: maximum power consumption while notebook is idle. All modules are on (Wi-Fi, Bluetooth, etc.), maximum brightness and power plan set to "High Performance".
    • Load: notebook runs with maximum brightness, all modules on and power plan set to "High Performance".
      Medium: For this test we used 3DMark06 and record the average power consumption in the first part of the test. In Android and IOS we use Asphalt 8 the replay of the first level.
      Maximum: stress test with 100% CPU and GPU load using Prime95 and Furmark benchmarks. Maximum power consumption possible on the test model.
      In Android devices, we use the app "Stability Test" CPU+GPU for Maximum, Classic for Medium.
      In IOS systems we use the PUBG in the lobby (Ultra HD details, max framerate).

    Currently, we use the multimeter Metrahit Energy from Gossen Metrawatt as measurement device. It even makes e.g. measuring of standby power consumptions of smartphones possible thanks to simultaneous measurement of TRMS current and voltage and high precision.

    Battery Life

    We run our test models through 4 different tests:

    • Minimum runtime: we use the "Classic" test of Battery Eater Classic to measure the minimum runtime of the test model. For this test, the screen brightness is set to maximum and all communication modules, such as WLAN, Bluetooth, etc. are turned on. Additionally, the Windows power plan is set to "High Performance".
      For our Android-based test models we use the app "Stability Test" to judge the minimum runtime. If the app does not run on the device (due to compatibility issues) then we run a 3D game which simulates high load, thus, allowing us to measure the minimum runtime of the test model.
      For IOS-based devices we use the PUBG in the lobby (Ultra HD details, max framerate).
    • Maximum runtime: the "Reader's" test of the Battery Eater tool is used to measure the maximum runtime of the test model. The brightness is set to minimum and all power-saving options are turned on. The Windows power plan is set to "Power Saver" and WLAN and Bluetooth are switched off.
      Android-based devices are tested with a script which loads text pages from the site: http://www.notebookcheck.com/fileadmin/Notebooks/book1.htm
    • Wi-Fi mode: the possible battery life while surfing the Internet via Wi-Fi with medium brightness (~150 cd/m²) and power-saving options ("balanced" mode) switched on. We measure the runtime by letting the device run an automatic script (HTML 5, JavaScript, no Flash - update 03.05.2015 v1.3), which picks a mix of websites and switches between them every 30 seconds.
    • H.264 video playback: We test the video playback battery runtimes with the free Big Buck Bunny movie on loop at 150 cd in power saving mode. Wifi and Bluetooth is disabled (flight mode), sound is muted.
    • DVD playback: runtime while the laptop is playing a DVD with maximum brightness, WLAN and Bluetooth off, and power-saving options turned on (such as the Windows "Power Saver" or higher - whichever is necessary for fluid playback of the DVD).

    The reader should take into account the fact that our test models are usually new laptops. This means that the battery of the laptop will have to be emptied and recharged a few times before it can deliver its peak battery life. Furthermore, our tests provide results which are taken over a relatively brief period of time. More information on how to optimize the battery life of your laptop can be found here: FAQ article.

    Rating

    Each test model receives ratings for each section mentioned in this article and is also given a rating after being compared to other models of the same class. The final rating is influenced by 12 points and the influence each point has on the final rating varies from class to class: netbook, gaming notebook, etc. We present the rating on a scale of 0-100% (higher is better).

    Finally, a total rating is calculated with the different impact of each aspect depending on the device class (weighting).

    The various rating criteria (the case and input device ratings are excluded) are processed with a special algorithm, which uses the various measurements and benchmark data in our database to deliver the result.

    More information about the rating system can be found here.

    Please share our article, every link counts!
    J. Simon Leitner, Klaus Hinum, 2016-09-26 (Update: 2022-10- 3)