Monday, December 4, 2017

Excessive Modified Memory

Logged into my workstation today and start noticing "Out of memory" errors. Whaatt? This computer has 8GB of memory and doesn't run that much!

Resource monitor showed excessive "modified" pages like this (not my screenshot, but shows how it looked):

Well, thats odd.. lets fire up google search.... 30minutes later and I've hit the solution (that many others have before more):

https://what.thedailywtf.com/topic/17472/finally-nailed-my-windows-memory-leak-a-k-a-the-official-we-hate-karl-club?page=1

Cause is due a bad realtek application, "runsw.exe".

The final kicker was the 800,000 HANDLES the application had open, you can see them in task manager if you add the handles column.

The service appears to be some kind of realtek monitoring/watchdog program that talks to another service.

Ok then, handle leak, lets find out why. First, let fire up API monitor, a wonderful program I've used before to solve issues.

Well well, lots of calls to Process32Next. Lets get it loaded up IDA and have a look. Browse the import table, do a reverse cross reference, convert to pseudocode, and tada, we get a function:


The function seems to be looking for a process using CreateToolhelp32Snapshot, then Process32First and Process32Next, then exiting, on first glance, no issue, until you consider the CreateToolhelp32Snapshot call, from the documentation:

If the function succeeds, it returns an open handle to the specified snapshot.
To destroy the snapshot, use the CloseHandle function.
Well, i'm not seeing any CloseHandle function, and the return is only being stored in esi (the HANDLE v0l // esi@1 line ).

We have located there memory leak. Looking at what is calling this function, it appears to be called once a second, so it's leaking at least one handle a second. Over time, these handles build up and use up Windows memory until Windows is either restarted or dies with out of memory errors.

A poorly designed executable from RealTek. I've emailed them about it but I don't expect it to be fixed, the memory leak has been reported about for years with no resolution.