A fascinating article in the January issue of Scientific American summarizes recent evidence in maternity. To summarize some of the specific and general findings:
- Hormones during and following pregnancy cause drastic changes in the structure of female brains, effects which are long-lasting or permanent
- Mother rats are significantly better at processing sensory input, navigating mazes, and hunting prey than nonmaternal rats, indicating increased intelligence
- Mother rats show increased complexity in the hippocampus and proliferation of glial cells, as well as finding previously encountered rewards than nonmaternal rats, indicating improved memory
- Experiments with covered/uncovered ground indicate that mother rats experience less fear and tolerate stress better than nonmaternal rats
- Maternal rats demonstrate significantly slower intellectual decline in old age than nonmaternal rats
- Experiments involving multiple independent tasks or streams of input indicate that maternal rats are significantly better at multitasking than nonmaternal rats
- These benefits seem to increase with the number of children
Note that there has been little research of this kind done on human females to date. Note also that the title [Career Women Smarter than Housewives? Better think twice] was only meant to be provocative; it is not to imply that maternity and a place in the workforce are mutually exclusive (in fact, if these maternal traits also apply to human females, it's likely that they could be applied to the workplace).
Tuesday, January 24, 2006
& Science - Maternity
I usually try to keep non-programming stuff (especially non-computer stuff) off my blog, but there are just some news items so interesting that I have to post them. The summary I posted on Star Alliance:
Monday, January 23, 2006
Saturday, January 21, 2006
Q's Shiny Thing of the Day
I may yet write a post about my exploits for the last week. But suffice to say, for the purpose of this post, that I need to compare two copies of a LARGE amount of data (files), to ensure they're identical. Fortunately for me, the Windows Platform SDK comes with just such a program (originally I was worrying that I'd have to write one, myself) - WinDiff. If you have need of such a program, it comes in pretty handy.
By the way, I'm having a bit of trouble writing the post about writing multi-user friendly code. The reason is that there are three different but closely related topics I want to talk about, and I'm having trouble discussing all of them without the information becoming either redundant or chaotic.
By the way, I'm having a bit of trouble writing the post about writing multi-user friendly code. The reason is that there are three different but closely related topics I want to talk about, and I'm having trouble discussing all of them without the information becoming either redundant or chaotic.
Wednesday, January 18, 2006
Multi-User Compatibility
One of the things I listed on the recent wish-list of "taxes" features I want to support in LibQ is multi-user compatibility. What exactly is meant by that? Well, literally, it means the ability to share a single computer with multiple people, and not step on the toes of others.
In practice, it comes down to a game of permissions. In a typical multi-user system (unlike DOS, Windows 1.0-3.1, and Windows 9x, which were fundamentally single-user systems), disk resources are divided among the different users of the computer. Typically there will be one or more administrators, which have access to everything, and a number of individual users, who only have access to a portion of the disk.
In a strict multi-user system like Unix, individual users can only write to files and directories in their partition, called their home directory on Unix (while Windows NT doesn't really promote this idea by name, there is in fact a perfect analog to the home directory). Everything outside this home directory is likely to be read-only, and some of it (system directories, other users' home directories, etc.) may not even be readable to the individual users at all. Each user potentially has access to different directories than other users (this is always the case with home directories, but it may apply to things outside the home directories, as well).
To Unix users and programmers, this system is ancient, as Unix was designed (and operated) this way from day one. However, to Windows users, this is a fairly new concept. As mentioned, DOS, 16-bit Windows, and Windows 9x do not really have the concept of user isolation. Contrary to what most Slashdotters will swear, Windows NT was designed from the beginning to support user isolation; however, NT 3.1-4.0 (3.1 was the first version; it was called 3.1 because its interface was based on the Windows 3.1 shell) were primarily used (on home computers) by hardcore nerds, while Windows 2000 won over the casual geeks, and it wasn't until Windows XP (back in 2001) that the technophobic end user installed NT on their home computer. The end result of this is that, even today, programs that do not work on multi-user systems plague the NT platform, and this is the reason so many NT users run as administrator today.
Where I'm going with this topic is this: learn to write multi-user compatible programs; it is not a useless skill. If you plan to write Unix programs, you're going to have to learn; but even if you're just a Windows coder, it would save people like me a considerable amount of pain if you would still learn how to write programs that play nice.
Next time (whenever I write another technical post): how to write multi-user compatible programs on Windows and Unix. And coming soon: multi-user compatibility hall of shame.
In practice, it comes down to a game of permissions. In a typical multi-user system (unlike DOS, Windows 1.0-3.1, and Windows 9x, which were fundamentally single-user systems), disk resources are divided among the different users of the computer. Typically there will be one or more administrators, which have access to everything, and a number of individual users, who only have access to a portion of the disk.
In a strict multi-user system like Unix, individual users can only write to files and directories in their partition, called their home directory on Unix (while Windows NT doesn't really promote this idea by name, there is in fact a perfect analog to the home directory). Everything outside this home directory is likely to be read-only, and some of it (system directories, other users' home directories, etc.) may not even be readable to the individual users at all. Each user potentially has access to different directories than other users (this is always the case with home directories, but it may apply to things outside the home directories, as well).
To Unix users and programmers, this system is ancient, as Unix was designed (and operated) this way from day one. However, to Windows users, this is a fairly new concept. As mentioned, DOS, 16-bit Windows, and Windows 9x do not really have the concept of user isolation. Contrary to what most Slashdotters will swear, Windows NT was designed from the beginning to support user isolation; however, NT 3.1-4.0 (3.1 was the first version; it was called 3.1 because its interface was based on the Windows 3.1 shell) were primarily used (on home computers) by hardcore nerds, while Windows 2000 won over the casual geeks, and it wasn't until Windows XP (back in 2001) that the technophobic end user installed NT on their home computer. The end result of this is that, even today, programs that do not work on multi-user systems plague the NT platform, and this is the reason so many NT users run as administrator today.
Where I'm going with this topic is this: learn to write multi-user compatible programs; it is not a useless skill. If you plan to write Unix programs, you're going to have to learn; but even if you're just a Windows coder, it would save people like me a considerable amount of pain if you would still learn how to write programs that play nice.
Next time (whenever I write another technical post): how to write multi-user compatible programs on Windows and Unix. And coming soon: multi-user compatibility hall of shame.
Tuesday, January 17, 2006
Don't Buy Acer Laboratory Products
What has Q been doing for the last 5 days? Well, let's take a look at the e-mail he just sent to Acer Laboratories (ALi) customer service, and find out!
I've been having horrible problems with M1535D+ drivers since I started using my IDE drives on the motherboard IDE channels (previously several were on my Promise IDE controller, and my new Windows volume is brand new). I installed Windows XP SP1 fresh onto the new 250 GB drive. After installation, Windows could not use the drives in anything but PIO mode, despite all of them being set to "DMA if available".
I downloaded the beta ALi IDE controller drivers (the version I'm now using) from the Asus site. After that, the ALi IDE utility reported that the drive was running at UDMA100/133, but it was obvious from the slow performance and KERNRATE (from the Windows 2003 Resource Kit) data that it was still using PIO (when reading large amounts from the drive - I used a checksumming program on a 5 gig file to ensure that the drive would be read from, but no other drives would be accessed - more than 90% of the kernel time was spent servicing interrupts, and 83% of the time READ_PORT_BUFFER_USHORT was being used; both of these indicate PIO transfer is being used, and neither being the case when I used the same procedure on a drive connected to my Promise IDE controller).
Also, since installing the beta ALi drivers, my CD-ROM and DVD-ROM on the motherboard channels have not been working properly (reporting bad sectors on the CD/DVD when this can readily be proven false in another computer; both drives worked fine using the Windows XP default motherboard drivers).
Lastly, I am unable to uninstall the beta drivers. The ALI5INST utility that runs when I select the drivers in Add/Remove Programs simply opens a console window and remains there indefinitely (and it appears that this is a 16-bit program, as it runs in NTVDM). Any attempts to uninstall the drivers through device manager have resulted in the need to roll back the system state with system restore.
Oh, and did you know that the drivers page on your ALi site is a broken link?
Friday, January 13, 2006
Light at the End of the Tunnel
Well, after the seemingly bad situation yesterday, there's good news. First, I installed SuSE on VMWare to try it out, some. While there is no formal definition of the encoding of filenames (which are just null-terminated character arrays), common usage dictates that they be interpreted as UTF-8 (the Linux GUIs and some terminals do this). That means it would be moderately safe to translate Unicode filenames (and by that I mean UCS-2) to UTF-8 on Linux (and possibly POSIX at large).
Next, I was mistaken about the behavior of Windows NT on non-Unicode-capable drives. It does indeed appear to use UTF-8 encoding for filenames on such drives. The reason I thought it didn't was that it was appearing as unprintable characters (boxes, to be specific) in Explorer on the computer I tried it on (which was not my computer). Apparently it's been so long since I've used such a thing that I'd forgotten that Windows XP doesn't come with Asian fonts...
Lastly, while I could support (with effort) both the Unicode and ANSI versions of Windows file API functions, it's much easier to just use the Unicode version, and leave Win9x support (which doesn't support the Unicode versions) to the Microsoft Layer for Unicode.
So, in the end it looks like it will work fine to use all Unicode for filenames in LibQ.
On an unrelated note, I went and picked up the Hitachi 250 GB hard drive for $50 at Fry's Electronics. While I was in the checkout aisle, a couple of other things caught my eye, as well: the Quick Study Japanese Grammar and Latin Vocabulary sheets. So, I picked up those, as well.
Next, I was mistaken about the behavior of Windows NT on non-Unicode-capable drives. It does indeed appear to use UTF-8 encoding for filenames on such drives. The reason I thought it didn't was that it was appearing as unprintable characters (boxes, to be specific) in Explorer on the computer I tried it on (which was not my computer). Apparently it's been so long since I've used such a thing that I'd forgotten that Windows XP doesn't come with Asian fonts...
Lastly, while I could support (with effort) both the Unicode and ANSI versions of Windows file API functions, it's much easier to just use the Unicode version, and leave Win9x support (which doesn't support the Unicode versions) to the Microsoft Layer for Unicode.
So, in the end it looks like it will work fine to use all Unicode for filenames in LibQ.
On an unrelated note, I went and picked up the Hitachi 250 GB hard drive for $50 at Fry's Electronics. While I was in the checkout aisle, a couple of other things caught my eye, as well: the Quick Study Japanese Grammar and Latin Vocabulary sheets. So, I picked up those, as well.
Wednesday, January 11, 2006
Dilemma - Filenames
Somebody just kill me, now.
So, I've been considering supporting Unicode filenames in LibQ, but that brings up not nice problem. First and foremost, not all operating systems and/or file systems support Unicode filenames. Windows NT supports true Unicode (UCS-2) filenames on NTFS, but converts (and degrades) the filenames to ANSI (similar to ASCII, both not supporting foreign language characters) on file systems like FAT and FAT32 that don't support Unicode file names. I'm told some Linux file systems support Unicode file names, but I've yet to find an interface to manipulate such files.
I could get around this problem by automatically converting Unicode filenames on unsupporting OS or file systems to something like UTF-7 or UTF-8, which will fit in a char array, but this brings up its own problems. First, as previously mentioned, NT automatically converts Unicode filenames to ANSI code page on unsupporting file systems, replacing characters that can't be converted to some default character (and it's not feasible to detect file systems that don't support Unicode, to protect against this). Second, this would create an odd duality for files saved with UTF filenames. You could open them by the Unicode version of their name (as it would automatically convert the name to UTF before calling the OS functions), but the names could appear garbled in directory listings (as there'd be no good way to detect that a filename is UTF, and convert it back to Unicode).
Last but not least, I'm having difficulty figuring out the API to convert Unicode strings to UTF-7/8 on POSIX.
So, I've been considering supporting Unicode filenames in LibQ, but that brings up not nice problem. First and foremost, not all operating systems and/or file systems support Unicode filenames. Windows NT supports true Unicode (UCS-2) filenames on NTFS, but converts (and degrades) the filenames to ANSI (similar to ASCII, both not supporting foreign language characters) on file systems like FAT and FAT32 that don't support Unicode file names. I'm told some Linux file systems support Unicode file names, but I've yet to find an interface to manipulate such files.
I could get around this problem by automatically converting Unicode filenames on unsupporting OS or file systems to something like UTF-7 or UTF-8, which will fit in a char array, but this brings up its own problems. First, as previously mentioned, NT automatically converts Unicode filenames to ANSI code page on unsupporting file systems, replacing characters that can't be converted to some default character (and it's not feasible to detect file systems that don't support Unicode, to protect against this). Second, this would create an odd duality for files saved with UTF filenames. You could open them by the Unicode version of their name (as it would automatically convert the name to UTF before calling the OS functions), but the names could appear garbled in directory listings (as there'd be no good way to detect that a filename is UTF, and convert it back to Unicode).
Last but not least, I'm having difficulty figuring out the API to convert Unicode strings to UTF-7/8 on POSIX.
Tuesday, January 10, 2006
Tolkien Was an Anime Watcher?
Just watched The Two Towers with my grandpa (he's a big Lord of the Rings fan). For the first time it occurred to me that Lord of the Rings has a stereotypical anime girl that has a hopeless crush on one of the main characters, tries to win the guy's heart by cooking for him, but she sucks at cooking, and the plan goes terribly wrong. Or maybe that stereotype reaches further than anime, and I'm just too illiterate to know.
Real Life Fun - SBltROP3 Bug
Sometime after I woke up yesterday morning and started doing stuff online, Skywing messaged me. He asked me what ordinal (function number) 0x139 in Storm.dll (the custom run-time library used by Blizzard games) was. Now, the reason for asking me in particular this question was that I wrote a program called StormWatch, which logs all calls to Storm, and has a list of all the functions I've either positively identified (the names to go with their ordinals) or have reason to believe they are a given function. This list might possibly be the most complete list of Storm.dll functions held by anyone lacking access to Blizzard's source code.
The reason he wanted to know what ordinal 0x139 is was that he had installed Warcraft 2: Battle.net Edition on his computer, and it was crashing in ordinal 0x139. I knew this function - it was SBltROP3, a DirectDraw sprite-blitting function (the name is a pun based on the Windows SetROP2 - Set Raster Operation To - function, which sets what raster operation to use when drawing on a display context). I'd looked at this function some when Suicidal Insanity asked me to reverse engineer a variety of things in the graphics and sprite system of Starcraft. If memory serves, it generates dynamic blitting code on the fly for the various raster operations. This coincided with Skywing's observation that the function was generating code, and crashing when it attempted to execute the code. Goody. Debugging assembly generated by assembly code: good times.
Fortunately, the problem became readily apparent when Skywing supplied one more piece of information: his computer supported Data Execution Protection. Yup, that'd do it. Skywing and I have independently observed a number of programming blunders on Blizzard's part, and it was no surprise that they'd try to execute a function without marking it as code (although in their defense this code was written long before NX/DEP was available in hardware; but it was still supported in the Windows API).
To confirm this was the case, Skywing first examined the page that was allocated to store the rendering code. Sure enough, it wasn't marked executable. Next, he tried marking it as executable, and, sure enough, the crash evaporated.
So, now we know what and where the bug is, as well as how to fix it. The next question was how wide the problem had spread. We had positive confirmation that the bug existed in the version of Storm used for Warcraft 2: Battle.net Edition (Storm.dll version 1999.11.18.1), but Skywing doesn't have Diablo, Starcraft, or Diablo II (all of which can use DirectDraw, and thus may be subject to the bug. So, I guess that makes it a job for... me.
Unfortunately, I don't have a DEP-capable CPU to try it on; that leaves reverse-engineering. Proof of bug could be accomplished in a two-step process: first, use StormWatch to verify that they used SBltROP3. Second, verify that the bug exists in their version of Storm.
StormWatch reports (and debugging the games in WinDbg confirms) that Starcraft and Diablo both call SBltROP3; Diablo II does not. That just leaves the question of whether the two games have the bug in their version of Storm.dll.
Looking at the Diablo version of Storm.dll, it is version 1999.11.18.1 - the very same as the WC2:BNE version; FC (file compare) was used to confirm the files were byte-for-byte identical. Starcraft, however, uses a newer version - version 2003.1.1.0; that means that I'll have to get out a dissassembler and debugger to confirm that Starcraft has the same bug.
Stepping through SBltROP3, it's trivial to identify the place where it calls the generated code:
15007cf0 8b7d08 mov edi,[ebp+0x8]
15007cf3 8b750c mov esi,[ebp+0xc]
15007cf6 8b5db0 mov ebx,[ebp-0x50]
15007cf9 8b4d20 mov ecx,[ebp+0x20]
15007cfc 8b5514 mov edx,[ebp+0x14]
15007cff ffe3 jmp ebx {00cd0594}
So, there's the address of the function: 0xCD0594; now I needed to find out what the protection state was. There seems to be a way to make function calls on the command line of WinDbg, but as I'm not very proficient with WinDbg (I usually use the VC++ debugger for my reverse-engineering), I had to manually create a stack frame and set the instruction pointer to VirtualProtect (which will return the previous protection, which is what we're interested in).
Well, after ironing out a couple quirks in WinDbg, I got my result: 0x40. A quick look at winnt.h reveals the identity of this constant: PAGE_EXECUTE_READWRITE. Isn't that interesting? Looks like they fixed the bug in the Storm.dll shipped with the latest Starcraft patch.
Lastly, I performed the same process on Warcraft 2: Battle.net Edition, as a final confirmation. The result? 0x4 - PAGE_READWRITE. Sure enough, there was our bug. And so concludes the official investigation (now I can get back to working on truly evil things...).
& The Room
Remember that room we've been building in the back yard for quite a few months, now, and I haven't said anything about in ages? Well, my grandparents just arrived from Texas, today, and my parents let them use their room, while my parents are sleeping out in the new room. It isn't completely done, but the walls are all up and permanently set in place, the outside is painted, the insulation's in, the door and windows are in, the tool shed (right outside the room) is built, the rain gutters are in place, and some of the electrical works. All that's left is to put in and paint the drywall, put a second coat of paint on the outside, install the electrical outlets, switches, and air conditioner, and lay the floor down (it's still plain concrete, right now). So, making progress.
Monday, January 09, 2006
& Scheming
I'm contemplating doing something extremely evil. No, I won't say what it is, but I'll probably post about it, if I actually do it.
The Destruction Dilemma, Solved
The CThread class has been around (and written, to varying degrees) for quite some time, but I never felt like it was something I wanted to put into actual use, as some things are rather crude. In the spirit of all the other classes in LibQ so far, a CThread represents a thread. The thread is created on construct, and... well, what to do with destruction was the real question (particularly when the thread was still running), and one of the main things that made me hesitant to release CThread into the wild.
Obviously the CThread could not be destructed while the thread was still running, as that would lead to a crash when the thread tries to access its own CThread. At the time, I could think of three ways to handle destruction: kill the thread, wait for the thread to finish, and assert. Killing the thread seemed to be the worst of the options. Killing a running thread is an extremely risky proposition. In addition to the fact that the thread might be needed in the future, the bigger problem is that the thread may be in the middle of using data (possibly inside a mutex, or some other synchronization object), which would likely lead to either the program crashing or deadlocking.
The last two options were about the same, as far as desirability. The problem with both is that, if the thread were to deadlock or hang, the thread that was attempting to destruct the CThread would also hang (probably indefinitely). Ultimately, I decided to have it assert if the destructor should be called while the thread was still running; but I wasn't happy about it.
The problem became intolerable thanks to asynchronous I/O. Recall that one of the methods of asynchronous I/O completion notification was to queue an asynchronous procedure call (APC) to the thread that issued the I/O. In other words, a pointer to the CThread was being hidden away somewhere you couldn't see or get to. Once this occurred, you can pretty much forget about ensuring that the CThread was no longer being used; of course you could track all places you give a pointer to, but that would be uselessly cumbersome, and strongly opposed to the low-maintenance model I have for LibQ.
However, the asynchronous I/O problem hinted at the solution: reference counting - having the object itself track how many references exist to it, and destructing itself when that number reaches zero. Reference counting wasn't a new idea to me, but it's something I avoid as much as possible, as it requires that reference-counted objects be allocated on the heap. Memory allocations are fairly expensive, as they typically take hundreds of cycles. However, between necessity and the fact that creating a thread is expensive (if my memory serves, it takes more than 10,000 cycles on Windows), which makes the added cost of allocation marginal, this is a perfect place to use reference counting.
So, with one major hurdle down, CThread is that much closer to going public. And now that I'm thinking about reference counting, I'm considering making CSyncFile and CAsyncFile reference-counted, as well, for similar reasons (and in the process it will be possible to simplify their code, as well).
Obviously the CThread could not be destructed while the thread was still running, as that would lead to a crash when the thread tries to access its own CThread. At the time, I could think of three ways to handle destruction: kill the thread, wait for the thread to finish, and assert. Killing the thread seemed to be the worst of the options. Killing a running thread is an extremely risky proposition. In addition to the fact that the thread might be needed in the future, the bigger problem is that the thread may be in the middle of using data (possibly inside a mutex, or some other synchronization object), which would likely lead to either the program crashing or deadlocking.
The last two options were about the same, as far as desirability. The problem with both is that, if the thread were to deadlock or hang, the thread that was attempting to destruct the CThread would also hang (probably indefinitely). Ultimately, I decided to have it assert if the destructor should be called while the thread was still running; but I wasn't happy about it.
The problem became intolerable thanks to asynchronous I/O. Recall that one of the methods of asynchronous I/O completion notification was to queue an asynchronous procedure call (APC) to the thread that issued the I/O. In other words, a pointer to the CThread was being hidden away somewhere you couldn't see or get to. Once this occurred, you can pretty much forget about ensuring that the CThread was no longer being used; of course you could track all places you give a pointer to, but that would be uselessly cumbersome, and strongly opposed to the low-maintenance model I have for LibQ.
However, the asynchronous I/O problem hinted at the solution: reference counting - having the object itself track how many references exist to it, and destructing itself when that number reaches zero. Reference counting wasn't a new idea to me, but it's something I avoid as much as possible, as it requires that reference-counted objects be allocated on the heap. Memory allocations are fairly expensive, as they typically take hundreds of cycles. However, between necessity and the fact that creating a thread is expensive (if my memory serves, it takes more than 10,000 cycles on Windows), which makes the added cost of allocation marginal, this is a perfect place to use reference counting.
So, with one major hurdle down, CThread is that much closer to going public. And now that I'm thinking about reference counting, I'm considering making CSyncFile and CAsyncFile reference-counted, as well, for similar reasons (and in the process it will be possible to simplify their code, as well).
Friday, January 06, 2006
More To Do
- Pathfinding/multi-user interoperability. The ability to locate the appropriate directories for various things, such as machine-wide data, user-specific data (data that will be stored on the server if the user is on a domain), temporary files, shared programs, etc. I call this multi-user interoperability because it will allow (if used properly) programs to run as non-administrator/root (this is something most Unix people should know, but way too many Windows programmers don't), as well as work efficiently on a network.
- Power management. The ability to discern what state the computer and components in it are in, so that inappropriate operations (i.e. doing background stuff on the hard drive when the computer is on battery) can be avoided. I'd also stick Windows hierarchal storage management (the automatic archival of files to tape, and automatic recall on access) in this category, because it also suggests that things should not be touched even though they can be.
- Session information. The ability to tell when the system is shutting down, the application is going to the background, and if the application is running locally (on the computer the user is sitting at) or remotely (on a different computer, where the user's computer is just a terminal, and where bandwidth is often low, and latency high).
So yeah, welcome to the todo list, gang!
Thursday, January 05, 2006
I Have Scary Friends
Oh yeah, and did you know that NX/DEP (the feature that prevents execution of noncode - i.e. buffer overflows tactics - in Windows XP+) doesn't work? Just got this link from Skywing (one of my 'best' friends - that is, the ones I talk to the most), although it's back from October. Notice the names of the authors. After reading this, I submitted the following summary to Slashdot (we'll see if it actually gets accepted):
A pair of hackers (including a personal friend) have identified a flaw in the design of Windows Data Execution Protection (DEP/NX), which ordinarily protects the system against remote code execution and other exploits that involve the execution of data, on capable processors. This flaw, detailed in their article, allows a worm or other malicious program to remotely disable DEP via a buffer overflow procedure that does not inject code of its own (and is thus not prevented by DEP). This possibility is not due to a bug in Windows, but rather due to the design decision to allow programs (for compatibility purposes) to disable their own DEP. As such, it cannot be 'fixed' in the normal sense; however, some clever tricks of Windows coding can be used to thwart this attack.
Subscribe to:
Posts (Atom)
Posts
