I love Lighthouse. It's a priceless tool in keeping an eye on your websites performance, SEO, and best practices and I have used it ever since it shipped in Chrome. Recently, Google's PageSpeed Insights has also started using Lighthouse to record its metrics, which is also a massively popular tool used by developers and SEO analysts alike.
I received a message the other day, telling me that one of the websites I work on had a poor performance score. Concerned, I fired up my browser, ran a lighthouse audit, and awaited the results... 92/100, pretty good if you ask me. So why was it flagged as poor? I ran a few other tests using different services, confirmed the performance was up to scratch and then asked the question, "Where did you get that metric from?"
What!? We compared results, and sure enough, they didn't match up. But how could that be? I ran a lighthouse test on both machines, with the same settings, with a single tab open, and sure enough, there was a huge discrepancy. The website took 3 times as long to load, with no obvious explanation. After some playing about, I decided to try the test again, but in incognito mode. The tests both showed the exact same results. It appears that browser extensions interfere with the tests, and skew the results of the audit. The same happens on the PageSpeed Insights website, which implies the test runs client-side.
Here is a test running on the React website in a regular Chrome window:
And here is one in incognito mode:
As you can see, there's a massive hit to the performance metrics, and results in unreliable and inconsistent results. My recommendation, always run audits in incognito mode!
Editors note (25/01/2019) - It looks as though Chrome have now updated their Audits section to inform their users of this potential issue. PageSpeed Insights no longer requires being run in incognito to get accurate results.