Exchange 2010 OWA Search Broken, 0x80041606

Customer with two Exchange 2010 servers, one Hub/CAS and one Mailbox server. Users started receiving an error when searching in OWA: "The action couldn’t be completed. Please try again". I was told that this happened after a power failure (and bad backup batteries), so it sounded like something was corrupted.

The event viewer was also logging an error:

Content Indexing function 'CISearch::EcGetRowsetAndAccessor' received an unusual and unexpected error code from MSSearch.
Mailbox Database: Operations Database 1 
Error Code: 0x80041606

Rebuild the Content Indexes

This was my first step, i took the easy route of stopping the Microsoft Exchange Search Indexer service and deleting the catalogs. These folders are located in every mailbox database folder and start with CatalogData. It is perfectly safe to delete these and then start the service again. After starting the service you will see event viewer entries indicating that it is rebuilding the indexes. You can also run the PowerShell command Get-MailboxDatabaseCopyStatus to see the status. The ContentIndexState column will indicate whether it's Crawling or Healthy.

Repair Symlinks

I saw plenty of articles indicating that the symlinks needed to be repaired, this did nothing.

Reinstall Filter Pack

Ultimately the solution was to reinstall the Microsoft Office Filter Pack on the Mailbox server. This did require a reboot (and yes, I tested it prior to rebooting). Afterwards the search was working as expected.

550 can't access file, 0xc0000427, and STATUS_FILE_SYSTEM_LIMITATION

The problem:

This is a problem i ran into when uploading large backup files (~1 TB) via FTP. I am running FileZilla FTP Server, and was also using the FileZilla FTP Client. Due to the connection speed these uploads were taking multiple days, and at certain points the transfers had to be resumed. After a certain point, on the largest files i started getting an error of 550 can't access file. This didn't make much sense to me, as there was no chance of a permissions issue or that something else was using the file. I restarted the FTP service and eventually even rebooted the server but the issue persisted.

Tracking down the cause:

I ended up running Procmon on the FTP Server service to see what it was doing that was causing the error. This actually gave me an error code on WriteFile which i was able to to follow.

What is actually wrong:

So now i have an error 0xc0000427 which translates to STATUS_FILE_SYSTEM_LIMITATION, an NTFS error. Unfortunately this is not as simple as filesystem damage or path length too long. I found this blog post from a Microsoft Engineer that, while it doesn't mention this issue, pointed me in the right direction. Basically when you upload a large file via FileZilla FTP Server, it doesn't allocate the space ahead of time, it just grows the file as needed. What this ends up doing is creating an extremely fragmented file. Growing it in this fashion will fill up the NTFS Attribute List for the file, eventually making it so no new child entries can be added. When this happens, any write attempts outside the allocated space will produce the 0xc0000427 error.

Fixing it:

So now I know what's wrong, how do i fix it? Well, Microsoft released a hotfix for Server 2008 R2 and earlier that patches ntfs.sys to support 4Kb Base Records instead of the current 1KB limit. The Base Record contains the Attribute List, so this allows it to grow larger to support... well... heavily fragmented files. The typical reason that people run into this limit are SQL database and VHD(x) files, since they by design have to grow at a steady pace. So that's great, but what do i do to fix it? Well, this hotfix supposedly is included as of Server 2012 R2, but even with that to leverage the 4KB Base Record size you have to format the entire volume with the /L switch. That's useless to me, as i'm not going to offload 20TB of data in order to do this. The article also mentions that running a defrag will not clean this up, as it will just move the fragments to be contiguous but not collapse the Attribute Lists.

TLDR:

The actual fix is so stupidly simple, yet i probably would not have thought to try it if I didn't understand the problem. Just create a copy of the file on the server, delete the old one, rename the copied version. There, that's it. Since it already knows the size of the file, it will allocate it at the full size, bypassing most fragmentation.

Fortigate Management - Filtering by IP

For most people the solution to restricting admin access to a Fortigate firewall is to use Trusted Hosts for the admin logins. This works fine, however any user (or bot) on the internet can still see the login prompt. This can also potentially cause alarms in security scans (such as for PCI compliance). The easy solution is to not allow access over the WAN interfaces however if you need to, here is a better way to do it using local-in policies.

For mine, i wanted to allow pings from anyone, admin access from my remote subnets, and then deny the rest. Here is the config, modify the interface names as needed, I used an Address Group for the allow rule.

config firewall local-in-policy
    edit 1
        set intf "wan1"
        set srcaddr "all"
        set dstaddr "all"
        set action accept
        set service "PING"
        set schedule "always"
    next
    edit 2
        set intf "wan1"
        set srcaddr "Admin Subnets"
        set dstaddr "all"
        set action accept
        set service "ALL"
        set schedule "always"
    next
    edit 3
        set intf "wan1"
        set srcaddr "all"
        set dstaddr "all"
        set service "ALL"
        set schedule "always"
    next
end

After implementing this you can remove your Trusted Hosts configuration as it has become pointless. I feel this solution is much more flexible, the only downside is that you can't see or administer it through the web interface (yes, even if you enable the Local-In Policy feature, it's still not there).

cPanel and Outlook Autodiscover

If you have a cPanel customer who also has an SSL certificate for their website, you may find that Outlook Autodiscover does not work properly (especially with older versions) as cPanel matches /Autodiscover/Autodiscover.xml and gives the client IMAP/SMTP settings instead of giving a 404 and letting it move on. You may even find that your requests are redirected to cpanelmaildiscovery.cpanel.net/autodiscover/autodiscover.xml which is obviously not desired. In this case, there is a quick fix, though it's really more of a hack. This assumes you're using Apache.

1. Edit /etc/httpd/conf/httpd.conf
2. Search for autodiscover, you should find a ScriptAlias line referencing it
3. Comment this line out or remove it completely
4. Restart apache (service httpd graceful)
5. Edit /usr/local/cpanel/APACHE_CONFIG and find the same line and remove it
6. At the top of your httpd.conf you should also have details on how to make cPanel retain the settings, i ran /usr/local/cpanel/bin/apache_conf_distiller --update to save the changes to the template.

Obviously a cPanel update could potentially overwrite these changes, but as far as i could tell this is the only way to get it to actually behave correctly.

Exchange Autodiscover after an On-Premise to Office 365 Migration

If for whatever reason you decided to do a cutover or similar migration from On-Premise Exchange to Office 365, you may find that Autodiscover can be a little sticky. The issue that I ran into is that for this environment the computers are domain joined, but we cannot immediately uninstall Exchange to remove the organization info from Active Directory. Due to this, when Outlook starts up and searches for Autodiscover, it will search for a SCP (Service Connection Point) in Active Directory, which will still be there and pointing to the wrong spot. Best case scenario this can make it take longer to start up, worst case it can end up either pointing to the wrong spot or certificate errors. The solution, though it's a bit of a hack, is to just change the connection point. I did this with ADSI Edit, note that I consider this a temporary workaround and not a permanent fix. Once you uninstall Exchange then all of this is removed from AD.

1. Open adsiedit.msc (typically from a Domain Controller)
2. Connect to the Configuration context
3. Navigate to Configuration > Services > Microsoft Exchange > (OrgName) > Administrative Groups > (Your Administrative Group Name) > Servers > (Server Name) > Protocols > Autodiscover
   
4. Edit the entry for your server and scroll down to ServiceBindingInformation
   
5. Adjust the value to reference the proper URL for your Office 365 environment. If you don't know, you can use the Autodiscover test in testconnectivity.microsoft.com to verify what your URL should be.
6. Outlook should detect it properly now, you can also verify by using the connectivity tester built into Outlook by Ctrl+Clicking the Outlook system tray icon and selecting Test E-Mail AutoConfiguration

Windows Server Unable to Make Outbound TCP Connections

Ran into an issue with a customer server where various things were just not working. A reboot always fixed it, but it had to be done every few days which was not acceptable. The types of errors being received were things like "no logon servers are available" or being unable to open any Active Directory tools. DNS and network shares were working fine, i could RDP in without issues, restarting various services did not help, ICMP/UDP were fine. I attempted to telnet out but any outgoing connection was immediately aborted, wireshark did not see the connection attempt, all firewalls were off.

I ended up putting PuTTY on the server and tried to telnet from there, and received a more specific "no buffer space available" error. Searching for that, i determined it was a Winsock error related to not enough free handles being available. With that knowledge, i opened the task manager and added the Handles column and sorted. There it was, an HP Plotter utility service with 6000+ open handles. I killed the process and everything immediately went back to normal.

Fortigate SSL Inspection with Custom CA Certificate

This was tested on a 30d, 60c, and 60d, so i assume it will be fairly accurate for most Fortigate units. When you enable SSL inspection in a policy, the unit will begin proxying all SSL traffic through itself so that it can see the actual traffic. The way that it does this is it has a CA certificate installed and upon visiting a site it will on the fly generate a new certificate for that website and sign it by itself. This works well because you still have the protection of the certificate not matching the site you're visiting, being expired, etc. The downside is that you have to install Fortigate's CA certificate into your trusted root certificate store on all client computers in order for the certificates to be accepted. For the very security conscious folk, you will note that this CA cert shares the same private key across all Fortigate units. So a nefarious person could potentially extract the private key and begin signing certificates with it for a targeted MITM attack. Unlikely, yes, however it's not hard to generate a new certificate to remove this potential issue.

Generate the certificate

To start you will need a computer that has openssl installed, i typically use a linux VM but anything will do. Start by generating a new CA private key, when prompted for a pass phrase enter something secure and that you will remember (you will only need it a few times).

[mike@mon1 ~ ]# openssl genrsa -aes256 -out ca.key 4096
Generating RSA private key, 4096 bit long modulus
............................++
...........++
e is 65537 (0x10001)
Enter pass phrase for ca.key:
Verifying - Enter pass phrase for ca.key:
[mike@mon1 ~ ]#

Next, sign the certificate. Fill in the information as you see fit, it is not really important for what we're trying to accomplish here. This is set for 3650 days, you can raise/lower this if you want.

[mike@mon1 ~ ]# openssl req -new -x509 -days 3650 -key ca.key -sha256 -extensions v3_ca -out ca.crt
Enter pass phrase for ca.key:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:California
Locality Name (eg, city) []:City
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Company
Organizational Unit Name (eg, section) []:
Common Name (eg, YOUR name) []:Company Fortigate CA
Email Address []:support@company.com
[mike@mon1 ~ ]#

Install the certificate

1. Transfer the ca.key and ca.crt files that you generated to your local computer. These are just text files, so you could potentially copy/paste the contents if that's easier.
2. On your fortigate you will need to enable the Certificate feature (Config - Features, or the Features panel on the Dashboard).
3. Navigate to System > Certificates > Local Certificates and select Import
4. Set the Type to Certificate, browse to the paths of the certificate file (ca.crt) and key file (ca.key) and enter the password you specified earlier.
5. Once the import is complete, navigate to Policy > SSL Inspection and select the new certificate and press Apply.
6. Enable SSL inspection on the policies of your choosing (typically this would be your internal to wan policies)
7. You can test that it's working by visiting an SSL website such as https://google.com and inspecting the certificate chain. You should see your CA at the top of the list and then the certificate for the site that was generated on the fly. Note that some browsers such as chrome will cache the certificate info, so you may need to force refresh a few times.
8. At this point in time you are probably still seeing certificate errors, so read the next sections to resolve that

Installing the root CA to PCs by hand

To install the CA certificate to your PC by hand, you just need to add the ca.crt file to the Trusted Root Certificate Store.

1. Browse to the path of the ca.crt file
2. Right click on the file and select Install Certificate
3. Select Local Machine as the store location
4. Select Place all certificates in the following store, press Browse, and select the Trusted Root Certification Authorities entry
5. Press Next and then Finish and you should be done
6. You can test by repeating the browser test, the certificates should now show up green

Installing the root CA to PCs via Group Policy

This obviously only applies to Active Directory environments, but it is a very easy way to get it pushed out.

1. Either edit an existing policy or create a new one, this will be applying to computers (not users)
2. Navigate to Computer Configuration > Policies > Windows Settings > Security Settings > Public Key Policies > Trusted Root Certification Authorities
3. Right click to import the ca.crt, press Next and Finish
4. A gpupdate /force will allow the computers to pull in the new certificate. Even if it prompts you to log off, you don't need to for it to work
5. You can test by repeating the browser test, the certificates should now show up green