From KB article
KB #230287:
When installing IIS 4.0 on a computer running Windows NT Server, the NetLogon and Computer Browser services must be running. If these services are not running, you will receive a dialog that says "Cannot detect OS type" and Setup will fail.
Also see the following KB articles:
KB #205328 Err Msg: Cannot Detect OS Type
KB #250262 NTOP Installation Error Message: Cannot Detect OS Type
Or, get the heck away from NT 4.0 altogether. Join the rest of us in the world of Windows 2000, Windows XP Professional, and Windows Server 2003, where IIS is an integral part of the operating system, instead of a half-baked afterthought.