Microsoft fixes 5-year-old Defender bug, reducing Firefox-related CPU use by 75%

Flags: needinfo?(mjaritz)

Flags: needinfo?(mjaritz)

Flags: needinfo?(mjaritz)

Flags: needinfo?(mjaritz)

Flags: needinfo?(mozilla)

Flags: needinfo?(nirbheek.chauhan)

Flags: needinfo?(nirbheek.chauhan)

Flags: needinfo?(mozilla) → needinfo?(jmathies)

Flags: needinfo?(jmathies)

Component: Storage → General

Product: Toolkit → Firefox

Component: General → Other

Product: Firefox → External Software Affecting Firefox

Flags: needinfo?(jmathies)

Flags: needinfo?(ke5trel)

Flags: needinfo?(ke5trel)

Flags: needinfo?(jmathies)

Flags: needinfo?(ke5trel)

Assignee: nobody → honzab.moz

Assignee: honzab.moz → nobody

Flags: needinfo?(ke5trel)

Flags: needinfo?(mcastelluccio)

Whiteboard: [qf] → [qf:p2:resource]

Flags: needinfo?(mcastelluccio)

Summary: Antimalware Service Executable very active when using Firefox → Antimalware Service Executable (Windows Defender) very active / high CPU when using Firefox

Flags: needinfo?(particlecore)

(In reply to Gian-Carlo Pascutto [:gcp] from comment #73)

Hi, can you follow the steps in https://bugzilla.mozilla.org/show_bug.cgi?id=1441918#c66 and report the feedback ID? We can try to nudge Microsoft to look into it perhaps.

There’s nothing actionable in Firefox itself here, unfortunately.

I already disabled the AntiMalware service entirely and Firefox continues to lock at 25% cpu usage after a while, randomly. This is on a fresh profile with no addons installed. I have to restart the browser to get this under control, otherwise it just chugs the process constantly for absolutely no reason whatsoever.

Is there any way we can inspect what Firefox is writing to during that time? This is f*ing annoying and is driving me nuts for months.

Flags: needinfo?(particlecore)

I have been able to reproduce the high CPU usage on my end consistently and it is always always always after Firefox updates itself while I am still using it and I do not restart it for a while.

This is how I am reproducing it (at least on Nightly, regardless of which version it is):

Use the browser for an entire day without restarting it once, keep it always open and use it as normal with as around 12 tabs or just 4, makes no difference

Firefox options menu will eventually show a green dot notification typically associated with the restart prompt, ignore it

Keep using the browser and, again, never restart it or close the browser once during this whole process

Eventually the CPU usage will start ramping up on its own, especially when you’re idle, and will stay locked at that CPU usage permanently. For me it has been lately locked at constant 25% usage

How do I know this is because of the update? Because I disabled the auto update option in the browser and now it has NEVER EVER DONE IT AGAIN until I tell it to download the update. Then it happens again, without exception.

This is the difference between keeping the browser open for 3 days, never closing it once, with the PC going into sleep mode at the end of the day and the CPU NEVER RAMPS UP on its own. It only does whenever it is doing demanding tasks, like watching high FPS videos on YouTube.

vs

Letting the browser update itself and never restarting it, guaranteed to start ramping up the CPU over time and staying completely locked at 25% CPU usage for no fng reason whatsoever. And I mean, I am away from the PC with nothing running and I start hearing the fans blowing constantly as a result of this. Every. Single. Time. It’s Firefox locked at 25% cpu usage. EVERY. SINGLE. TIME.

Performance Impact: — → P2

Whiteboard: [qf:p2:resource]

Severity: normal → S3

Performance Impact: medium → —

Some new reports of this popping up: https://www.reddit.com/r/firefox/comments/zyzsk6/high_cpu_usage_from_windows_defenderantimalware/

I’m personally running Win10 22H2 Build 19045.2604, all latest updates and I encounter this problem too.

Excluding the Appdata/Roaming/Mozilla and Appdata/Local/Mozilla folders alleviates the problem and brings the CPU usage down to what other browsers (Edge, Chrome) cause.

Things I’ve tried to alleviate

• Disable accessibility services
• Uninstall Firefox, manually getting rid of all Mozilla folders, and reinstalling, starting a new profile

I’ve started noticing it now because I have a new CPU (5900X) that gets a bit hotter when a thread spins up, resulting in a variation of my fan speed.

(In reply to Jeroen Baert from comment #77)

Some new reports of this popping up: https://www.reddit.com/r/firefox/comments/zyzsk6/high_cpu_usage_from_windows_defenderantimalware/

I’m personally running Win10 22H2 Build 19045.2604, all latest updates and I encounter this problem too.

Hello,

Thank you for sharing this. I think I can reproduce the problem, so I will provide some data that describes what happens on my machine in this comment. It would be helpful if you could confirm if the data looks similar on your machine, to be sure that I can use my own machine as a basis for investigating this issue. I will describe how to collect similar data by yourself in a second comment.

I recorded a session with Windows Performance Recorder, then opened it in Windows Performance Analyzer. I only had Firefox open when I did the test. What I did was launch youtube.com once, wait for 10 seconds, refresh it, and again (in total, 5 loads). I did the same in Chrome (after disabling Software Reporter Tool which was somehow having a lot of CPU activity on my machine, which could have affected the experiment). Attachments show the activity graph for the most active processes in each session, then the details of where MsMpEng.exe spent CPU time in both sessions. Firefox session is always shown at the top, and Chrome at the bottom.

MsMpEng.exe‘s CPU activity is shown with red bars, while the useful processes are shown with other colors. We can see that with both Firefox and Chrome, MsMpEng.exe‘s activity is correlated to the activity of the other processes: it is high when Firefox or Chrome processes themselves have high CPU activity. This is explained by the fact that MsMpEng.exe spends most of its CPU time in sechost.dll!ProcessTrace, in other words it is busy processing ETW events that result from the activity of other processes. However, the CPU time used overall by MsMpEng.exe seems indeed much higher (5x) with Firefox compared to Chrome. By trying to list the specific ETW events which MsMpEng.exe is listening to, we may identify that there are some events which Firefox’s current behavior generates a lot of, and in that case we could then try to reduce the amount that we generate. So, if the Firefox graph looks similar on the reporter’s machine, then I can try doing that.

Note: I see no disk activity from MsMpEng.exe, so for the moment this looks like legitimate MsMpEng.exe activity (although high) to me, unlike (my understanding of) the original bug.

To produce the data, first I recommend looking at the short video tutorial here from Microsoft to have an overview of the interface of Windows Performance Analyzer. Then, here are the steps to reproduce the graph and numbers from the first image:

1. Install Windows Performance Recorder and Windows Performance Analyzer from the Windows ADK for your version of Windows. For Windows 10 22H2 I think this one should work. You only need to select Windows Performance Toolkit as a feature after launching the ADK installer.
2. Close as many open applications as possible to avoid noise in collecting the data.
3. Open Windows Performance Recorder and configure it as shown in attachment. Click Start.
4. Open Firefox and do the steps for which you want to analyze performance (e.g. load YouTube 5 times separated by intervals of 10 seconds).
5. Back in Windows Performance Recorder, click Save, click Save, click Open in WPA.
6. If Windows Performance Analyzer fails to load with error 0x80070032, install Windows Performance Analyzer Preview from the Windows Store and load the saved file from there to avoid the error (I personally have to do this).
7. In Windows Performance Analyzer, unroll Computation, drag and drop CPU Usage (Precise) to the currently empty Analysis tab. A graph shows up.
8. Under Series, navigate to the top. The processes are sorted based on which were the most active (hopefully, some firefox.exe processes and MsMpEng.exe). By using shift-clicking or control-clicking, select the most active firefox.exe processes and MsMpEng.exe, but not Idle (which is not a process). Then, right-click, Disable, All but selection.
9. Near, Utilization by Process, Thread, Stack, there is a Select Chart Type button. Click it and select Stacked bars, then move the cursor towards Max.
10. (Optional) Select the relevant portion of the graph, right-click, Zoom.
11. Use Snipping Tool to save and share the graph.

Note: Please share only the graph, and not the raw file saved by Windows Performance Recorder which may contain private information.

Note: The graph and numbers should be enough assuming that it looks reasonably similar to mine. To reproduce the detailed activity of MsMpEng.exe, the steps are mostly the same, but you would need to set Detail level to Verbose when recording, then after all the previous steps in WPA you would click the Trace menu, click Load Symbols, wait a long time, and then progressively unroll the top-most entries under MsMpEng.exe like in the image I shared.

Hello,

Here is a short update on this issue:

• There is a serious performance issue in the current version of MsMpEng.exe, which I was able to pinpoint using the WPR profiles from comment 78 and some debugging. We have reported the details to Microsoft, and they have acknowledged it.
• This performance issue makes calls to VirtualProtect (among other things) unreasonably expensive on Windows platforms when Windows Defender’s Real-time Protection feature is active (the unreasonably expensive computations execute in the MsMpEng.exe process). Microsoft has applied a patch that will mitigate the issue with the upcoming March release (engine version 1.1.20200.2), which means that users will gradually get the fix within the next 4 weeks. People who would like to bypass the progressive rollout and get the patch sooner (i.e. next week) can use Set-MpPreference -EngineUpdatesChannel Preview.
• With a standard Firefox configuration, the amount of calls to VirtualProtect is currently very high, and that is what explains the high CPU usage with Firefox. The information that the most impactful event originates from calls to VirtualProtect was forwarded to us by Microsoft, and I confirm it. In Firefox, disabling JIT makes MsMpEng.exe behave much more reasonably, as JIT engines are the source of the vast majority of calls to VirtualProtect.
• On Firefox’s side, independently from the issue mentioned above, we should not consider that calls to VirtualProtect are cheap. We should look for opportunities to group multiple calls to VirtualProtect together, if possible. Even after the performance issue will be mitigated, each call to VirtualProtect will still trigger some amount of computation in MsMpEng.exe (or third-party AV software); the computation will just be more reasonably expensive.

I will provide more details with numbers later.

(In reply to Jeroen Baert from comment #77)

Some new reports of this popping up: https://www.reddit.com/r/firefox/comments/zyzsk6/high_cpu_usage_from_windows_defenderantimalware/

I’m personally running Win10 22H2 Build 19045.2604, all latest updates and I encounter this problem too.

Can you confirm if MsMpEng.exe behaves more reasonably after navigating to about:config in Firefox and changing the following preferences:

• set javascript.options.baselinejit to false;
• set javascript.options.wasm_baselinejit to false;
• set javascript.options.wasm_optimizingjit to false.

If that doesn’t solve the issue, you might be experiencing a different issue from the one mentioned in comment 82.

This image shows the impact of the preferences listed in comment 83 on my personal machine: top is with JIT, bottom is without JIT. We can see that for the specific case of loading youtube.com 5 times in Firefox:

• MsMpEng.exe total CPU time decreases from 16s to 6s by disabling JIT in Firefox (so 63% less CPU time);
• firefox.exe total CPU time summed for the most active processes is around 23.5s independently of whether or not we are using JIT in Firefox.

(In reply to SeriousHoax from comment #85)

I would also like to add that this high CPU usage issue while using Firefox is not exclusive to Microsoft Defender. It’s an issue for Norton’s AV products also and should be the same for Symantec Endpoint products too.


So, you should also test them.

Hello, if you have one of these AV products installed, can you confirm if the steps comment 83 impact the CPU usage of your AV processes? Can you share some CPU usage measurements like described in comment 81?

Flags: needinfo?(serioushoax)

(In reply to Yannis Juglaret from comment #82)

• There is a serious performance issue in the current version of MsMpEng.exe, which I was able to pinpoint using the WPR profiles from comment 78 and some debugging. We have reported the details to Microsoft, and they have acknowledged it.

Let me share more details on this, as this may be interesting for other AV vendors who would be watching this discussion. The TdhFormatProperty can be used to get information associated with ETW events. This function has the following prototype:

TDHSTATUS TdhFormatProperty(
  ...,
  [in, out]       PULONG            BufferSize,
  [out, optional] PWCHAR            Buffer,
  ...
);

This is a classic Windows API pattern: you can first call it with a NULL pointer as Buffer to know the required buffer size. Then you can call it a second time with a properly-sized buffer.

// First call to get the required buffer size
ULONG BufferSize = 0;
if (TdhFormatProperty(..., &BufferSize, NULL, ...) == ERROR_INSUFFICIENT_BUFFER && BufferSize > 0) {
    // BufferSize now holds the required buffer size: allocate a buffer and call again
    LPVOID Buffer = HeapAlloc(GetProcessHeap(), 0, BufferSize);
    if (TdhFormatProperty(..., &BufferSize, Buffer, ...) == ERROR_SUCCESS) {
         // All good, we can now use Buffer
    }
}

Except that if you use this classic pattern here, you will have very bad performance with the current implementation of TdhFormatProperty. The code above is, in the best case scenario, equivalent to the one below:

#define KB 1024
// First call to get the required buffer size
ULONG BufferSize = 64 * KB;
LPVOID Buffer = HeapAlloc(GetProcessHeap(), HEAP_ZERO_MEMORY, BufferSize);
if (TdhFormatProperty(..., &BufferSize, Buffer, ...) == ERROR_SUCCESS) {
    HeapFree(GetProcessHeap(), 0, Buffer);
    // BufferSize now holds the required buffer size: allocate a buffer and call again
    Buffer = HeapAlloc(GetProcessHeap(), 0, BufferSize);
    if (TdhFormatProperty(..., &BufferSize, Buffer, ...) == ERROR_SUCCESS) {
         // All good, we can now use Buffer
    }
}

In other words, with the current implementation of TdhFormatProperty, every call to TdhFormatProperty with a NULL buffer yields an allocation of a 64-KB zero-ed heap buffer. So, for example, because the ETW event for monitoring VirtualProtect has 18 properties, if as an AV vendor you are listening to this event and calling TdhFormatProperty on each of its property using the pattern described above, each call to VirtualProtect on the system will result in the allocation and zero-ing of 18 64-KB buffers in your AV process. If moreover the AV code considers that TdhFormatProperty is not expensive, and calls it multiple times for the same property, that makes the situation even worse.

As a result of using the pattern described above, MsMpEng.exe was spending 48% of its time allocating and zero-ing 64-KB buffers in the recording from comment 78. Microsoft coincidentally already had a new implementation for TdhFormatProperty on its way when we reported the issue; the new implementation is not there yet but it should enhance the performance of the pattern described above when it gets shipped. However the best fix is to not rely on the pattern from above, and instead reuse the same buffer all the time, and something along those lines will get shipped with Windows Defender engine version 1.1.20200.2.

(In reply to Yannis Juglaret from comment #83)

(In reply to Jeroen Baert from comment #77)

Some new reports of this popping up: https://www.reddit.com/r/firefox/comments/zyzsk6/high_cpu_usage_from_windows_defenderantimalware/

I’m personally running Win10 22H2 Build 19045.2604, all latest updates and I encounter this problem too.

Can you confirm if MsMpEng.exe behaves more reasonably after navigating to about:config in Firefox and changing the following preferences:

• set javascript.options.baselinejit to false;
• set javascript.options.wasm_baselinejit to false;
• set javascript.options.wasm_optimizingjit to false.

If that doesn’t solve the issue, you might be experiencing a different issue from the one mentioned in comment 82.

I can confirm that toggling these options does seem to have an impact, but haven’t reduced the CPU usage of MsMpEng.exe to the low levels of Chrome or Edge. I’m still seeing loads that are 3x that of loading the same website (I usually test with cnn.com, because it’s large and full of mixed content) in Chrome/Edge.

(In reply to Jeroen Baert from comment #89)

I can confirm that toggling these options does seem to have an impact, but haven’t reduced the CPU usage of MsMpEng.exe to the low levels of Chrome or Edge. I’m still seeing loads that are 3x that of loading the same website (I usually test with cnn.com, because it’s large and full of mixed content) in Chrome/Edge.

Thanks. Your observations align well with the numbers I have obtained experimenting with this. I counted how many events are generated on the Microsoft-Windows-Threat-Intelligence ETW provider when loading YouTube (didn’t see significant difference with CNN), and here are the numbers I obtained:

• Firefox with normal configuration: ~14000 events, 98% of which are PROTECTVM_LOCAL;
• Firefox with the preferences from comment 83: ~6500 events, 95% of which are PROTECTVM_LOCAL;
• Edge: ~2000 events, 91% of which are ALLOCVM_LOCAL;
• Chrome: ~300 events.

So indeed, even with switching the prefs, we still have a high number of calls to VirtualProtect. This problem has two sides: Microsoft was doing a lot of useless computations upon each event; and we are generating a lot of events. The combination is explosive. Now that Microsoft has done their part of the job (comment 82), we need to reduce our dependency to VirtualProtect. Bug 1822650 in particular will help with that.

(In reply to SeriousHoax from comment #85)

I would also like to add that this high CPU usage issue while using Firefox is not exclusive to Microsoft Defender. It’s an issue for Norton’s AV products also and should be the same for Symantec Endpoint products too.


So, you should also test them.

It is true that we should analyze the situation with other AV vendors, however, given the numbers shared above, and given how relevant it is to keep track of memory protection changes in order to detect malicious behavior, it is very likely that the explanation for Windows Defender also applies (at least in part) to other AV vendors.

I was able to install mpengine.dll version 1.1.20200.3, which should incorporate the fix for the performance issue mentioned in comment 82 and described in comment 87. I confirm that this performance issue has correctly been addressed. Attached is a comparison of a previous recording (top) with a new recording (bottom).

The numbers suggest a ~75% improvement in CPU usage from MsMpEng.exe when browsing with Firefox (the specific number applies to browsing youtube.com on my machine), bringing it close to where CPU usage was when browsing with Chrome back in comment 78 (of course, since the performance improvement benefits all programs installed on the system, the new numbers for Chrome should be lower if measured with version 1.1.20200.3 of mpengine.dll).

You can get mpengine.dll version 1.1.20200.3 (currently Beta) as follows:

• run a Powershell terminal as administrator;
• type Set-MpPreference -EngineUpdatesChannel Beta, press Enter;
• open Windows Update, browse updates, a new update should appear, install it;
• back in the terminal, you can type Set-MpPreference -EngineUpdatesChannel Broad and press Enter if you don’t want to stay in Beta for the future.

You can check your version of mpengine.dll as follows:

• navigate to C:ProgramDataMicrosoftWindows DefenderDefinition Updates;
• look for a folder named something like {7C9E9F14-9A1B-4833-AC74-DA15E21884D4}, open it;
• right click on mpengine.dll, Properties, Details, look for Product version.

Disclaimer: I have also received updates for my whole system and Firefox between the two recordings, which may also partially affect the numbers observed here.

I am closing this bug as per comment 91 the specific problem for Windows Defender seems fixed. I filed bug 1823634 for future work. Thank you!

Note: Feel free to reopen if the problem is not fixed on your own computer with a version of mpengine.dll higher than 1.1.20200.2.

Status: NEW → RESOLVED

Closed: 20 days ago

Resolution: — → FIXED