Notebookcheck

Our Test Criteria

J. Simon Leitner (translated by Vinay Pradhan), 09/26/2016

In this article, you will find information about the tests we run on our test models and how we process the results.

Working For Notebookcheck

Are you a loyal reader of notebookcheck? Are you a techie who knows how to write? Then join our Team!

Especially wanted:
Review Editor
 - Details here
News Editor - Details here

German-English-Translator - Details here

Introduction

Our editors have years of experience and have tested numerous notebooks from various manufacturers. We used this extensive experience to create a catalog of test criteria which cover all purchase relevant aspects of a device. Consumer's demands have changed over the years and technologies evolved. As such, we must also constantly update and add to our test criteria so as to give the reader the information they are looking for. As a reader, you can contribute greatly with your input. We are always happy to receive an email or a new post in our forum. Our editors and a long line of helpful moderators are available and will gladly take your tips.

The following are the main aspects of every review performed by Notebookcheck.net (explanations are included):

Case

The case of our test model is examined very closely. We judge the test model with the following criteria: design (colors, shape, material, feel, measurements, weight, etc.), build quality (gaps, finish, edges, precision, how secure every component sits, etc.), sturdiness (how the notebook reacts to pressure - at a single point, on the entire surface, torsion resistance, etc.), hinges (power, how well they hold the display, longevity, etc.), maintenance (possible upgrades which can be performed by the user, cleaning of the device/cooler fan, etc.). The respective editor rates the build quality according to his judgment by comparing it to previously tested models. The rating is discussed with the editorial team and the respective divisional director.

Connectivity

We judge the ports and interfaces available on the test model and their positioning in consideration of the device category.

SD-Card Reader

If a test model features an SD-card reader, we test it to estimate the transfer rates you can expect. For this purpose, we use our respective reference SD cards, which are currently a Toshiba Exceria Pro SDXC UHS-II for full-size readers and a Toshiba Exceria microSDHC UHS-I for devices with MicroSD slots. We test for the maximum transfer rate you can expect when copying big data blocks (e.g. videos) from the SD card to the test device by using the AS SSD Seq. Read test and for the expected transfer rate when copying many images (about 1 GB .jpg files; about 5 MB each, standardized test files), which is usually significantly lower than the maximum transfer rate. The following figures show the current minimum and maximum values as well as the average as available in our database (as of 07/2016).

SDCardreader Transfer Speed - maximum AS SSD Seq Read Test (1GB)
max. (Surface Book)
246 MB/s ∼100%
average (Lenovo B50-10)
87.3 MB/s ∼35%
min. (Acer E5-552G)
10.2 MB/s ∼4%
SDCardreader Transfer Speed - average JPG Copy Test (av. of 3 runs)
max. (Precision 7710)
213 MB/s ∼100%
average (Zenbook UX303UB)
64 MB/s ∼30%
min. (Lenovo E31-70) (Toshiba Exceria Pro SDXC 64 GB UHS-II)
10.3 MB/s ∼5%

Communication Features

We evaluate the communication features, such as LAN, Wi-Fi, Bluetooth, 3G, 4G, etc. In addition to a real-world test of the telephone features in smartphones, we also run a standardized Wi-Fi test.

Wi-Fi Performance

In a standardized test setup, we determine the maximum transfer rates (sending and receiving) when connected to our Linksys EA8500 reference router (in 1 m distance) with the Jperf software. This test uses the fastest transfer standard supported by the test model.

Networking - iperf Server (receive) TCP 1 m
max. (MBPr 13)
Broadcom 802.11ac
735 MBit/s ∼100%
average (Aspire S13)
Qualcomm Atheros QC61x4 Wireless Network Adapter
320 MBit/s ∼44%
min. (CX61)
Realtek RTL8723AE Wireless LAN 802.11n
24.3 MBit/s ∼3%
Networking - iperf Client (transmit) TCP 1 m
average (GL62)
Intel Dual Band Wireless-AC 3165
272 MBit/s ∼100%
min. (B510-10)
Realtek RTL8723BE Wireless LAN 802.11n PCI-E NIC
41.3 MBit/s ∼15%

GPS

GPS test: in a building
GPS test: in a building
GPS test: outdoors
GPS test: outdoors

We run a real-world test on mobile devices which come with a GPS module. While the editor cycles along a route, he records it with the test model and our reference navigation device. A comparison of the data allows us to judge precision and reliability of the incorporated GPS module. In addition, we record the GPS signal inside and outside of buildings.

Test model: overview of the route
Test model: overview of the route
Test model: crossroads
Test model: crossroads
Test model: landmark
Test model: landmark
Garmin Edge 500 reference: overview of the route
Garmin Edge 500 reference: overview of the route
Garmin Edge 500: crossroads
Garmin Edge 500: crossroads
Garmin Edge 500: landmark
Garmin Edge 500: landmark
Discussion

Camera

We test the front camera (webcam/selfie camera) and the rear camera (primary camera of smartphones) and evaluate the quality by comparing the image to standardized reference cards and the image quality of current flagship models. Among others we judge sharpness, colors, contrast, sensitivity to light and video features available.

Photo of the reference card
Photo of the reference card
Detail of the reference card
Detail of the reference card
ColorChecker: photo of the colors. The lower half of each patch shows the original colors.
ColorChecker: photo of the colors. The lower half of each patch shows the original colors.

The following example compares the image quality of the Samsung Galaxy J5 2016 to the reference camera's and other models'. The main image allows selecting a part for comparison with other devices by moving the mouse pointer over it.

Image Comparison

Choose a scene and navigate within the first image. One click changes the zoom step. One click on the zoomed-in image opens the original in a new window. The first image shows the scaled photograph of the test device.

Scene 1Scene 2Scene 3

Input Devices

The following criteria are taken into consideration: keyboard - keyboard layout (positioning, size, grouping, function keys, inscription, etc.), typing experience (key travel, pressure point, stroke, noise, etc.), additional keys if available; touchpad - response (surface, multi-touch, etc.), mouse keys (use, noise, etc.), touch display - response (precision, reaction time, etc.), virtual keyboard (layout, feedback, response, key size, etc.), sensors, digitizer if available (capacitive display).

Display

The following criteria are considered for the display rating based on the measurement results: resolution and format (pixel density, clarity of the display, ease of use with multiple windows, etc.), display brightness [cd/m²] (maximum, minimum, mains operation/battery mode, etc.), brightness distribution (dark areas, bleeding, etc.), contrast (max., black value, etc.), colors (DeltaE ColorChecker, Grayscale), covered color space (sRGB, AdobeRGB98), viewing angles, glare, PWM, response time.

Display brightness: measurement segments
Display brightness: measurement segments

Brightness, brightness distribution, and contrast

We use an X-Rite i1 Pro 2 photo spectrometer together with the latest version of the CalMAN Ultimate software for our display measurements. The measurement of the brightness is taken after the screen has stayed 100% white for a period of 10 minutes. Device settings, such as automatic adjustment of brightness, are deactivated and the color profile of the device has factory settings (not user defined).

The black value is also measured after a time period of 10 minutes, during which time the screen is 100% black (@max. brightness). The measurements are then taken from the central area of the screen while in a completely dark room. These measurements are used by us to calculate the maximum contrast of the display in the central area of the screen. We calculate the illumination by comparing the brightest segment of the screen to its darkest quadrant.

Outdoor use is part of the display test. We check to see how well the display can display content (legibility, reflections, etc.) in bright surroundings (3000-10000 cd/m² - cloudy to sunny). The list of criteria for this test is: type of display surface (matte panels prevent reflections), brightness of the picture, and contrast of the picture.

Colors: out-of-the-box vs. calibrated

The naked eye can hardly detect color deviations from its ideal (sRGB) if the DeltaE is smaller than 3. This is already very difficult for a DeltaE smaller than 5, while differences are increasingly notable at higher deviations (see following figure, actual - target). Hence, devices for editing graphics, images and videos should have a DeltaE smaller than 3 and support calibration well.

In addition, we calibrate devices meant for graphics work and measure the color precision again. You can download the created .icc profile from the review page.

CalMAN Grayscales
CalMAN Grayscales
CalMAN ColorChecker
CalMAN ColorChecker
CalMAN Saturation
CalMAN Saturation
CalMAN Grayscales calibrated
CalMAN Grayscales calibrated
CalMAN ColorChecker calibrated
CalMAN ColorChecker calibrated
CalMAN Saturation calibrated
CalMAN Saturation calibrated

Color Space

Apart from precise colors, the covered color space is important for professional graphics and image editing. We check the coverage of the sRGB and AdobeRGB98 color spaces. For this, we use the .icc file created by the iProfiler and the Argyll software.

sRGB color space 100%
sRGB color space 100%
AdobeRGB color space 74%
AdobeRGB color space 74%

Response Times

We measure how fast the display is able to change from white to black (0% to 100%) and gray to gray (50% to 80%) with an oscilloscope and a photodiode. Fast response times are particularly important for displays designed for gaming. For example, such displays feature response times of only a few milliseconds.

Display Response Times

Display response times show how fast the screen is able to change from one color to the next. Slow response times can lead to afterimages and can cause moving objects to appear blurry (ghosting). Gamers of fast-paced 3D titles should pay special attention to fast response times.
       Response Time Black to White
17 ms ... rise ↗ and fall ↘ combined↗ 4 ms rise
↘ 13 ms fall
The screen shows good response rates in our tests, but may be too slow for competitive gamers.
In comparison, all tested devices range from 0.8 (minimum) to 240 (maximum) ms. » 7 % of all devices are better.
This means that the measured response time is better than the average of all tested devices (27.4 ms).
       Response Time 50% Grey to 80% Grey
25 ms ... rise ↗ and fall ↘ combined↗ 9 ms rise
↘ 16 ms fall
The screen shows relatively slow response rates in our tests and may be too slow for gamers.
In comparison, all tested devices range from 0.9 (minimum) to 250 (maximum) ms. » 7 % of all devices are better.
This means that the measured response time is better than the average of all tested devices (42.4 ms).

Viewing Angles

Among others, different display technologies differ in viewing-angle stability. Currently, IPS displays, which allow very flat angles without image distortions, are wide-spread in higher-priced notebooks and particularly in tablets and smartphones. Cheap notebooks often use TN displays, which tend to be very dependent on the viewing angle, but they deliver better response times. We test the viewing-angle stability of the notebook subjectively (using the test model) and by turning the display to fixed angles (45 ° turns). For this test, the display is photographed at a fixed shutter speed and aperture inside a dark room.

Comparison: IPS display
Comparison: IPS display
vs. TN display
vs. TN display

Performance

Our performance tests vary depending on the device class and the expected use. This section includes a variety of benchmarks (which either test a single component or the entire system and present the result in points), and real-world tests with various programs and games, since these often stress the processor and graphics card extremely.

Prior to the tests, the system is brought up-to-date with Windows updates. In addition, the latest graphics drivers are installed on devices with dedicated graphics solutions if the system actively suggests doing so. We do not change the clock rates unless we actually mention that we did so for demo purposes. Updating the graphics drivers after our tests and possible modifications can certainly boost gaming and benchmark performance. In our opinion, it is the duty of the manufacturer to provide customers with the latest drivers out-of-the-box and make useful updates as easy as possible for the user.

We consider the following aspects for evaluating the performance: CPU (Cinebench, Turbo analysis, etc.), system (PCMark), storage device (HD Tune, CrystalDiskMark, AS SSD, etc.), GPU (3DMark, Unigine Heaven, etc.), gaming (a selection of current games and popular older games).

Additional information:

Emissions

Noise emissions: position of the gauge
Noise emissions: position of the gauge

System Noise

We use a noise level gauge (Audix TM1), the measurement software ARTA and a standard test setup to measure the emissions of a test model. The gauge is fixed 15 cm from the notebook and is secured against vibrations emanating from the test model. The measurements are taken in dB(A) (Decibel). The following are our test categories:

  • Idle
    Minimum: minimum noise emission while laptop is idle (Windows power plan: "Energy Saving")
    Medium: average noise emission recorded while laptop is idle (power plan: "Energy Saving")
    Maximum: highest noise emission measured while the laptop is idle (power plan: "High Performance")
  • Load
    Medium: average noise emission while the computer is running at high level of system use (3DMark06, power plan: "High Performance")
    Maximum: highest possible noise emission while the system is under heavy load (power plan: "High Performance", 100% CPU and GPU usage - thanks to Prime95 and Furmark)

The following may help the reader better understand the results: 

In a quiet room, the human ear can hear background noise, which should amount to around 28 dB. A conversation at a normal volume ranges at 60 dB. All these values are dependent on the distance from the source of the noise. This is why we fix our gauge into place at a constant distance from our test models. This allows us to get clear results which can be compared with each other. The measurements are presented graphically and can be judged subjectively (deviations caused by different frequencies are possible):

  • Under 30 dB: barely audible.
  • Up to 35 dB: audible but not distracting. Ideal level of noise emission for a laptop running office programs.
  • Up to 40 dB: clearly audible, and might be distracting after a while.
  • Up to 45 dB: might disturb the user if they are in a quiet room. Still acceptable while playing games. 
  • Over 50 dB: notebook emissions over this level are uncomfortably loud.

Noise Characteristics

In addition to our measurements, we record a frequency diagram of each fan level. This allows judging whether the perceived noise is rather low or high frequency. In our measurements, the audible range starts from about 125 Hz and reaches up to about 16000 Hz depending on the volume.

Three fan levels, brown, green, and red: You can see a bigger increase in the higher frequency range at higher rotation speeds here.
Three fan levels, brown, green, and red: You can see a bigger increase in the higher frequency range at higher rotation speeds here.
Comparison: audible range, music, and entertainment.
Comparison: audible range, music, and entertainment.
Measurement quadrants for the surface temperature
Measurement quadrants for the surface temperature

Temperature

The distribution of surface temperature (which can be felt by the user directly) is measured with an infrared thermometer (Raytek Raynger ST or similar) which never touches the test model. The top and bottom of the notebook is split into nine quadrants, and the maximum measurable temperature in each quadrant is recorded.

The measurements are taken after an idle period of 60 minutes and a stress period of 60 minutes (100% CPU and GPU usage - Prime95 and Furmark). 

In addition we also closely observe the GPU and CPU during the stress test with software (tools: HWInfo64, HWMonitor, GPUz, etc.) and note any significant variations in performance (drops due to throttling).

The following scale describes the categories we put our measurements in:

  • Less than 30 °C: Barely noticeable increase in temperature.
  • 30 - 40 °C: Temperature rises noticeably but is bearable.
  • 40 - 50 °C: Contact with the notebook over a long period of time at these temperatures will be uncomfortable.
  • Over 50 °C: Very hot. Problematic if using the notebook on the lap.

In addition to the mentioned maximum values we create thermographic images (Flir One), which depict the distribution of the surface temperatures in a continuous way.

Distribution of the surface temperature: idle
Distribution of the surface temperature: idle
Distribution of the surface temperature: load
Distribution of the surface temperature: load

Speakers

We rate the speakers based on sound quality and performance at maximum volume. Once again we use a measurement microphone (Audix TM1), the measurement software ARTA and a standardized setup for our measurements. We use Pink Noise as an output file, measured at maximum volume and we add measurements at lower volume when we detect overdrive.

Our frequency diagrams allow comparing different devices to each other and selecting / deselecting each frequency curve by means of checkboxes.

dB(A) 0102030405060708090Deep BassMiddle BassHigh BassLower RangeMidsHigher MidsLower HighsMid HighsUpper HighsSuper Highs2035.335.12532.931.83131.8324036.535.15032.428.9633328.98036.328.810048.32712561.52716052.924.820060.92425062.822.731563.32240069.521.250067.82163074.82080075.919.4100072.718.912507117.7160070.117.820006917.6250071.817.6315068.117.6400071.417.6500073.717.6630070.417.5800071.617.61000071.617.61250069.617.41600059.717.5SPL83.630.6N62.51.5median 69.6Apple MacBook 12 (Early 2016) 1.1 GHzmedian 17.8Delta4.62.434.334.53432.630.833.334.534.633.635.132.232.634.329.136.228.348.128.841.525.954.724.650.423.857.223.468.222.569.621.871.220.968.719.876.519.372.71976.618.673.318.272.117.964.91867.417.870.517.869.91864.21858.817.959.517.952.617.983.731.257.11.6median 67.4Dell XPS 13 2016 9350 (FHD, i7-6560U) median 198.23hearing rangehide median Pink Noise

Power Consumption

Metrahit Energy Multimeter
Metrahit Energy Multimeter

Apart from the battery life, the power consumption of a notebook is measured in various scenarios (power adapter sided). The test settings for each scenario are as follows:

    • Idle: power consumption while the notebook is idle.
      Minimum: all additional modules are off (Wi-Fi, Bluetooth, etc.), minimum brightness, and Windows power plan is set to "Energy Saving".
      Medium: maximum brightness, additional modules off, Windows power plan: "Balanced".
      Maximum: maximum power consumption while notebook is idle. All modules are on (Wi-Fi, Bluetooth, etc.), maximum brightness and power plan set to "High Performance".
    • Load: notebook runs with maximum brightness, all modules on and power plan set to "High Performance".
      Medium: For this test we used 3DMark06 and record the average power consumption in the first part of the test.
      Maximum: stress test with 100% CPU and GPU load using Prime95 and Furmark benchmarks. Maximum power consumption possible on the test model.
      In Android devices, we use the app "Stability Test" CPU+GPU for Maximum, Classic for Medium.
      In IOS systems we use the Epic Citadel app in demo mode.

    Currently, we use the multimeter Metrahit Energy from Gossen Metrawatt as measurement device. It even makes e.g. measuring of standby power consumptions of smartphones possible thanks to simultaneous measurement of TRMS current and voltage and high precision.

    Battery Life

    We run our test models through 4 different tests:

    • Minimum runtime: we use the "Classic" test of Battery Eater Classic to measure the minimum runtime of the test model. For this test, the screen brightness is set to maximum and all communication modules, such as WLAN, Bluetooth, etc. are turned on. Additionally, the Windows power plan is set to "High Performance".
      For our Android-based test models we use the app "Stability Test" to judge the minimum runtime. If the app does not run on the device (due to compatibility issues) then we run a 3D game which simulates high load, thus, allowing us to measure the minimum runtime of the test model.
      For IOS-based devices we use the app Epic Citadel in demo mode.
    • Maximum runtime: the "Reader's" test of the Battery Eater tool is used to measure the maximum runtime of the test model. The brightness is set to minimum and all power-saving options are turned on. The Windows power plan is set to "Power Saver" and WLAN and Bluetooth are switched off.
      Android-based devices are tested with a script which loads text pages from the site: http://www.notebookcheck.com/fileadmin/Notebooks/book1.htm
    • Wi-Fi mode: the possible battery life while surfing the Internet via Wi-Fi with medium brightness (~150 cd/m²) and power-saving options ("balanced" mode) switched on. We measure the runtime by letting the device run an automatic script (HTML 5, JavaScript, no Flash - update 03.05.2015 v1.3), which picks a mix of websites and switches between them every 30 seconds.
    • DVD playback: runtime while the laptop is playing a DVD with maximum brightness, WLAN and Bluetooth off, and power-saving options turned on (such as the Windows "Power Saver" or higher - whichever is necessary for fluid playback of the DVD).

    The reader should take into account the fact that our test models are usually new laptops. This means that the battery of the laptop will have to be emptied and recharged a few times before it can deliver its peak battery life. Furthermore, our tests provide results which are taken over a relatively brief period of time. More information on how to optimize the battery life of your laptop can be found here: FAQ article.

    Rating

    Each test model receives ratings for each section mentioned in this article and is also given a rating after being compared to other models of the same class. The final rating is influenced by 12 points and the influence each point has on the final rating varies from class to class: netbook, gaming notebook, etc. We present the rating on a scale of 0-100% (higher is better).

    Finally, a total rating is calculated with the different impact of each aspect depending on the device class (weighting).

    The various rating criteria (the case and input device ratings are excluded) are processed with a special algorithm, which uses the various measurements and benchmark data in our database to deliver the result.

    More information about the rating system can be found here.

    > Notebook / Laptop Reviews and News > Benchmarks / Tech > Our Test Criteria
    J. Simon Leitner, 2016-09-26 (Update: 2016-09-26)