Life in DllMain

I briefly mentioned that you should be careful what you do in DllMain in the windows hooks post. Since I would have had to expand on that in the next post, and sticking important information in the middle of posts on other topics isn't such a good idea, I decided to write this post; it should be fairly brief.

The key limitation of DllMain is that DllMain calls are serialized for every DLL In the process. It should be obvious that this rules out calling LoadLibrary or FreeLibrary inside DllMain. However, this also rules out calling any functions in any other DLLs that may or may not be loaded; if your DllMain is running, you can't be sure that another DLL's DllMain was run before yours. The exception to this rule is Kernel32.dll; as it is the first API DLL to get loaded in a new process and the last to be unloaded, it is guaranteed to be loaded when your DllMain gets called.

However, even for functions within Kernel32.dll, some restrictions apply. While there are some situations where it may be safe to do so, it is generally not a good idea to do any waiting (via WaitForSingleObject and kin) in DllMain.

Note also that it is possible to link to the C run-time library (CRT) as a DLL. When this is the case, you must not call any CRT functions from DllMain, as they would be calls to another DLL that you don't know is already initialized.

It is possible, however, to make an end run around these limitations, for a cost. Ironically, one of the things we can do safely in DllMain is creating new threads. This is somewhat counter-intuitive, as a new thread calls DllMain for DLL_THREAD_ATTACH notification when it begins executing. Yet that is exactly why this method works: the new thread will remain suspended until all DLLs have received their DLL_PROCESS_ATTACH notification, ensuring that all DLLs will be initialized by the time the thread gets run.

The cost of doing this, however, is that your initialization code no longer runs synchronously: you can't guarantee (in fact you can guarantee the opposite) that by the time your DLL finishes loading (DllMain returns) and the program continues running that your initialization will have completed. Creating an initialization thread in DllMain then waiting on it (or any similar scheme attempting to force your initialization thread to execute) would be guaranteed deadlock.