FastMM4 vs FastMM5 vs FastMM4-AVX

Configuration

In order to test the performance with our components, a new windows console application has been created, sgcBenchmark.exe, which will be used to measure the performance of every memory manager using our sgcWebSockets components.

The test is very simple, a client (or more than one client) connects to a server, sends a message to server and server replies with the same message to client. This is repeated 100.000 times. The tests are repeated changing the number of concurrent clients, first 1, then 10, 100... the measured time is the time elapsed between the first message sent by client and the last message received from server (so the time used to connect to server is not measured).

The benchmark will compare the performance using the Default Memory Manager that comes with Delphi 10.4.1, FastMM5 and FastMM4-AVX

Benchmark Indy WebSocket Server

In the first Benchmark, the Server used is the Indy WebSocket Server, this server is based on Indy TCP Server, so every connection creates 1 thread. 

The values are measured in milliseconds, so for example, the first test that is done with 1 client in Windows32 platforms, using the default memory manager takes 4135 milliseconds, using FastMM5 takes 4214 milliseconds and using FastMM4-AVX takes 4823 milliseconds. The percentage calculated is against the reference value, in this case against the Default memory manager that comes with delphi, as much lower is the percentage, better performance has.

The Benchmark has been done 3 times and the values showed are the sum of the benchmarks / 3.

For the benchmark, the server used was:

• Windows 2016 Server Datacenter
• 16 Virtual Processors
• 32 GB RAM
• 2.2 GHz

The Delphi version used was Delphi 10.4.1, and the latest FastMM5 and FastMM4-AVX versions from github servers.

Find below the result of the benchmark.

Benchmark HTTP.SYS Server

In the second Benchmark, the Server used is the HTTP.SYS WebSocket Server, this server is based on HTTP API Microsoft Framework and the connections are handled by a pool of threads.

The values are measured in milliseconds, so for example, the first test that is done with 1 client in Windows32 platforms, using the default memory manager takes 5364 milliseconds, using FastMM5 takes 5182 milliseconds and using FastMM4-AVX takes 5838 milliseconds. The percentage calculated is against the reference value, in this case against the Default memory manager that comes with delphi, as much lower is the percentage, better performance has.

The Benchmark has been done 3 times and the values showed are the sum of the benchmarks / 3.

For the benchmark, the server used was:

• Windows 2016 Server Datacenter
• 16 Virtual Processors
• 32 GB RAM
• 2.2 GHz

The Delphi version used was Delphi 10.4.1, and the latest FastMM5 and FastMM4-AVX versions from github servers.

Find below the result of the benchmark.

Comments about Benchmarks

• Using in single threaded application, there are no big differences in performance between FastMM4, FastMM5 and FasMM4-AVX.
• FastMM5 and FastMM4-AVX work much better in multithreaded applications.
• The differences between FastMM5 and FastMM4-AVX are small, at least doing these benchmarks.
• Windows 32 benchmarks performs better than Windows 64 tests. Using FastMM5 or FastMM4-AVX in a Windows 64 applications improves performance more than in Windows 32.

The final decision to choose one memory manager or another depends of the project, I think there is no single memory manager that works as the best in all conditions, so before choose one or another, test, test and test again to see which performance better for your needs.

Console Benchmarks  

Find below the compiled windows console applications used to do benchmarks, just execute one of them and 2 console applications will start, one for server and another for client, just let the client do the tests and when finish, both applications will be closed automatically. There is one console application for every platform and memory manager.

To execute all tests, I used a .bat file to call every benchmark and save in a text file the results.

call bin\Win32\sgcBenchmark.exe >> benchmark_win32.txt
call bin\Win32\sgcBenchmark_FMM5.exe >> benchmark_win32_FMM5.txt
call bin\Win32\sgcBenchmark_FMM4_AVX.exe >> benchmark_win32_FMM4_AVX.txt

call bin\Win64\sgcBenchmark.exe >> benchmark_win64.txt
call bin\Win64\sgcBenchmark_FMM5.exe >> benchmark_win64_FMM5.txt
call bin\Win64\sgcBenchmark_FMM4_AVX.exe >> benchmark_win64_FMM4_AVX.txt

call bin\Win32\sgcBenchmark.exe -t both.sys >> benchmark_sys_win32.txt
call bin\Win32\sgcBenchmark_FMM5.exe -t both.sys >> benchmark_sys_win32_FMM5.txt
call bin\Win32\sgcBenchmark_FMM4_AVX.exe -t both.sys >> benchmark_sys_win32_FMM4_AVX.txt

call bin\Win64\sgcBenchmark.exe -t both.sys >> benchmark_sys_win64.txt
call bin\Win64\sgcBenchmark_FMM5.exe -t both.sys >> benchmark_sys_win64_FMM5.txt
call bin\Win64\sgcBenchmark_FMM4_AVX.exe -t both.sys >> benchmark_sys_win64_FMM4_AVX.txt

exit 

Updated 16-May-2021 

After a request from Maxim, the developer of FastMM4-AVX, informing about the following improvements, find below his comments:

I have improved FastMM4-AVX to work better in multithreaded environments at the cost of single-thread performance.
The speed improvement primarily relates to 64 bits.
The essence of the improvement is the following. If the list of blocks of this size is locked on releasing a memory block, the FreeMem returns immediately, and the block will be released later, during the next FreeMem. As a result, two memory blocks will be released with a single memory lock.

FastMM5 uses a different technique: it has not a single list of blocks for each size, but several of them called "arenas", so the thread contention will occur less likely because each thread will probably get its own arena. However, this requires much more memory, and may need a much higher number of memory pages, which, in some scenarios, may be slower.

I've downloaded the latest versions from FastMM5 and FastMM4-AVX and repeated the benchmarks, find below the results.

Benchmark Indy WebSocket Server 

In the first Benchmark, the Server used is the Indy WebSocket Server, this server is based on Indy TCP Server, so every connection creates 1 thread.

The values are measured in milliseconds, so for example, the first test that is done with 1 client in Windows32 platforms, using the default memory manager takes 4109 milliseconds, using FastMM5 takes 4156 milliseconds and using FastMM4-AVX takes 4255 milliseconds. The percentage calculated is against the reference value, in this case against the Default memory manager that comes with delphi, as much lower is the percentage, better performance has.

The Benchmark has been done 3 times and the values showed are the sum of the benchmarks / 3.

For the benchmark, the server used was:

• Windows 2016 Server Datacenter
• 16 Virtual Processors
• 32 GB RAM
• 2.2 GHz

The Delphi version used was Delphi 10.4.2, and the latest FastMM5 and FastMM4-AVX versions from github servers.

Find below the result of the benchmark.

Benchmark HTTP.SYS Server 

In the second Benchmark, the Server used is the HTTP.SYS WebSocket Server, this server is based on HTTP API Microsoft Framework and the connections are handled by a pool of threads.

The values are measured in milliseconds, so for example, the first test that is done with 1 client in Windows32 platforms, using the default memory manager takes 5182 milliseconds, using FastMM5 takes 5255 milliseconds and using FastMM4-AVX takes 5057 milliseconds. The percentage calculated is against the reference value, in this case against the Default memory manager that comes with delphi, as much lower is the percentage, better performance has.

The Benchmark has been done 3 times and the values showed are the sum of the benchmarks / 3.

For the benchmark, the server used was:

• Windows 2016 Server Datacenter
• 16 Virtual Processors
• 32 GB RAM
• 2.2 GHz

The Delphi version used was Delphi 10.4.2, and the latest FastMM5 and FastMM4-AVX versions from github servers.

Find below the result of the benchmark.

Comments about Benchmarks 

As shown in the benchmark results, the FastMM4-AVX library has improved the performance under multi threaded environments. 

• The results are very similar using the Indy server, FastMM4-AVX performs a bit better than FastMM5, especially with high concurrency and 64bits.
• Using the HTTP.SYS server, FastMM5 performs better than FastMM4-AVX, but the differences are smaller than in previous benchmarks.

Console Benchmarks

Find below the compiled the updated windows console applications used to do benchmarks