Wednesday, October 31, 2007

ISA Server 2006 Supportability Update

I do a lot of work with different firewalls including Microsoft ISA Server. As with any firewall, there is always something to troubleshoot. User can't access particular website, another user didn't receive his e-mail etc. These are all reasons why we need a good and fast way to filter and analyze firewall logs.

In ISA Server 2004 and 2006 there are few new features that come with updates and allow firewall administrators to save existing queries in XML file and reuse them at a later time.


Click image to enlarge

For me this is really a time saver. Now I don't have to waste time and write same filters over and over again (e.g. excluding specific traffic and including other). I can simply carry most common queries on my USB drive and import them whenever I need them.

On ISA Server 2004, you get these new features when you install Service Pack 3 (SP3) for ISA Server.

On ISA Server 2006, you can get these features by installing ISA Server 2006 Supportability Update that you can download manually or you can use Microsoft Update Service.


Click image to enlarge

More information on ISA Server 2006 Supportability Update package can be found here "Description of the Internet Security and Acceleration (ISA) Server 2006 Supportability Update package"

Thursday, October 25, 2007

0x9C_IA32_GenuineIntel -- MACHINE_CHECK_EXCEPTION (9c)

New memory.dmp file, new challenge... :-)

After loading file to WinDBG and running a standard set of commands I am left with following relevant information.


BUSCONNERR - Bus and Interconnect Error BUS{LL}_{PP}_{RRRR}_{II}_{T}_err These errors match the format 0000 1PPT RRRR IILL

Concatenated Error Code: -------------------------- _VAL_UC_EN_ADDRV_PCC_BUSCONNERR_0

This error code can be reported back to the manufacturer. They may be able to provide additional information based upon this error. All questions regarding STOP 0x9C should be directed to the hardware manufacturer.

BUGCHECK_STR: 0x9C_IA32_GenuineIntel <---- Error 0x0000009C

DEFAULT_BUCKET_ID: DRIVER_FAULT

CURRENT_IRQL: 2

LAST_CONTROL_TRANSFER: from 80a84154 to 8087c480

STACK_TEXT:
808a0770 80a84154 0000009c 00000000 808a07a0 nt!KeBugCheckEx+0x1b
808a08a4 80a7b86f 80042000 00000000 00000000 hal!HalpMcaExceptionHandler+0x11e
808a08a4 f6932f36 80042000 00000000 00000000 hal!HalpMcaExceptionHandlerWrapper+0x77
808a3600 80839b02 00000000 0000000e 00000000 p3!AcpiC1Idle+0x12
808a3604 00000000 0000000e 00000000 00000000 nt!KiIdleLoop+0xa

STACK_COMMAND: kb

FOLLOWUP_IP:
p3!AcpiC1Idle+12
f6932f36 6a00 push 0x0

FAULTING_SOURCE_CODE:

SYMBOL_STACK_INDEX: 3

FOLLOWUP_NAME: MachineOwner

SYMBOL_NAME: p3!AcpiC1Idle+12

MODULE_NAME: p3

IMAGE_NAME: p3.sys <---- Here it looks like p3.sys driver caused the crash
DEBUG_FLR_IMAGE_TIMESTAMP: 45d6972c

FAILURE_BUCKET_ID: 0x9C_IA32_GenuineIntel_p3!AcpiC1Idle+12

BUCKET_ID: 0x9C_IA32_GenuineIntel_p3!AcpiC1Idle+12

Followup: MachineOwner---------

From stack it looks like p3.sys driver (Processor Device Driver) caused the crash of the server. One the other hand error 0x9C (0x0000009C) indicates hardware error.

Best article on the subject of 0x000000C (Understanding and troubleshooting the "Stop 0x0000009C" screen) states:

"The Pentium and Pentium Pro processors provide a mechanism to detect and to report hardware-related problems such as memory parity errors and cache errors. To signal a hardware error, the processor signals the detection of a machine check error by generating a machine check exception (Interrupt 18). Windows NT simply reports the fact that the error occurred and displays parameters that you can use to decode the exception. Contact your hardware vendor or processor manufacturer for information regarding the Machine Check Architecture or consult the Intel Pentium Pro Family Developer's Manual - Volume 3: Operating System Writer's Manual."

Above information is also displayed in dump file.

Full analysis of the dump file can be found here.

Possible resolutions:
Best recommendation suggested by above KB article is "contact your hardware vendor"

Problem caused by computer hardware

What you can do on your own?
  • Test your hardware (memory, processor, ...)
  • Check hardware connections
  • Think about recent hardware changes (incompatible components)
  • Think about recent configuration changes (e.g. enable or disable ACPI)
  • Update BIOS and other hardware (firmware)
  • Stress test your hardware (best done before going into production with the server)

Wednesday, October 24, 2007

Exchange Outlook Web Access (OWA) and red X


Click image to enlarge

This problem ocures due to changes in Windows Vista and Internet Explorer 7 where dynamic HTML Editing ActiveX control was removed from Internet Explorer.

To solve this problem you need to update your Exchange servers with "Update for Exchange 2003 (KB 911829)". Note that you have to install Exchange 2003 SP2 before you can install this update.

Related KB article: You receive an error message when you try to perform any editing tasks, or you must click to enable the compose frame in Outlook Web Access

Important: Remember to always update your front-end servers first!
If you update your back-end server first and there is change in OWA functionality, clients will most likely not be able to use OWA. In the past these errors presented themselves as “Loading” text in OWA that never finished loading.
If you update your front-end server first, server will know about changes in functionality and will serve clients with working OWA. Now you can take your time and update all your back-end servers when you find time ;-).

Wednesday, October 10, 2007

ARCast.TV - Security Chat from Slovenia

This ARCast was made May 2007 at Microsoft’s NT conference in Portorož, Slovenia.

There are two Mihas on the stage. I am one of them. I can't tell you which due to security reasons. ;-)

Saturday, October 6, 2007

502 Proxy Error and ISA Server 2004

It took me a bit to figure out why this ISA server was serving users with this error when they didn't ask for it:
  • Error Code: 502 Proxy Error. Cannot complete this function. (1003)
  • IP Address: 207.46.250.101
  • Date: 21.9.2007 5:39:07
  • Server: isaserver
  • Source: proxy

This is what the users got in their browsers when they tried to surf to their website of choice that morning.


Basically there were three reasons this took me more than 5 minutes:
  • The problem was just too basic
  • Error didn't give any clue to the real problem
  • It was 7:30 a.m. and I was still half asleep :-)

I check the usual stuff for any hint what might be causing this. I checked the disk space and there was plenty of it. Nothing unusual in the Event Logs, ISA services were running fine and nothing unusual in ISA Server MMC.

Since there was almost nothing left to check, I check network cards. Here I noticed that external Network Interface Card (NIC) has been disabled. Once I enabled it, the problem went away and I was able to get another half an hour of sleep.

I wish I would get something like "Could not connect" instead of "Error Code 502 Proxy Error".

Note to self. Don't forget to check the basics. :-)

Monday, October 1, 2007

Antivirus and servers

I am not a big believer in file level antivirus software running on servers, specially when they are holding domain controllers role or are running exchange services. In my experience they tend to cause more problems than do good. While these problems are often related to misconfiguration of antivirus products they can lead to big problems with infrastructure (e.g. corrupted Active Directory or Exchange database, slow server responses and even Blue Screen of Death (BSOD)). No, I am not naming any vendor names, but I should still have some dump files laying around to prove it ;-). Anyway, they all have their fair share of "issues" :-)

Note: this post talks about file level antivirus, not an antivirus that is installed on e-mail servers to check inbound and outbound e-mails. I would recommend that every e-mail server has an antivirus installed to check arriving e-mails for viruses and other malicious code.

Let's take this idea step-by-step:

Virus
Property of the virus is that it can only infect a computer with some user interaction (e.g. user runs an infected file)

Worm
Worms on the other hand can infect a computer without any user action. They use vulnerabilities on systems that have not been updated (patched) to infect it. At the same time you cannot really rely on antivirus to protect the computer if the computer has not been updated (patched).
If we take a look at Blaster worm it used DCOM RPC Interface Buffer Overrun Vulnerability to infect the computer. Even up-to-date antivirus did not protect the system from infection, because the operating system itself was vulnerable. Once the operating system was updated, this computer was safe from the worm even without antivirus.

Side note: you could protect yourself from Blaster worm by enabling personal firewall on the computer (e.g. Windows Firewall)

Most common ways of infection
One of most common way of infection is by e-mail. I am yet to see a good reason to read an e-mail on a domain controller or on Exchange server itself.
Downloading infected file from the internet is another common way of infection. Just like with reading an e-mail, I can’t really see a scenario where administrator would need to browse the internet from domain controller. Patches can among other ways be deployed to computers using WSUS server. This is why I usually prohibit access to the internet from servers on the firewall. If I can I go even one step further and prohibit access to the internet for any users that are members of certain groups such as Domain Administrators group in domain.
I usually accomplish this by using Microsoft ISA Server, where you can configure who (user account or group) has access to the internet and who doesn't. You can also easily configure which web sites server has access to and discard all others. This way you can grant the server access to certain Microsoft websites (e.g. Windows Update) and deny access to all others by using URL addresses. Even if IP address of the destination web server changes access to the site will always work as long as URL address stays the same.
Other patches (e.g. driver patches) that are not available through Windows Update site, can be downloaded on the client PC where antivirus should be installed. Once the package was verified it can be copied to the server using USB memory sticks or even over the network.

If you decide to run antivirus software on your servers, make sure it is configured properly. Here are few articles that can help you with this:

Virus scanning recommendations for computers that are running Windows Server 2003, Windows 2000, or Windows XP

Overview of Exchange Server 2003 and antivirus software

The DHCP service does not start when you start a Windows Server 2003-based computer

Note: I have seen it few times now and this is the reason why I decided to post this. Certain antivirus product(s) forget the exclusions that you set under certain conditions which can cause unexpected problems. (as mentioned corrupted databases etc). If you are running antivirus software on your server, check on the exclusions every once in a while, specially if your server starts to behave oddly all of a sudden.