Fortigate 40c and Redundant Internet Connections

Update: This article is pretty old at this point, it's only going to apply to older firmware and devices. That said, the concept is pretty much the same, just in different places.

If you're reading this, you're probably not very satisfied with how up to date Fortinet keeps their documentation for newer software versions. In previous versions, you could for the most part configure dual ISPs through the web interface, but on newer releases you have to use the CLI to do it.

The scenario:
ISP 1 is on wan1, your IP is 1.1.1.2, gateway is 1.1.1.1
ISP 2 is on wan2, your IP is 2.2.2.2, gateway is 2.2.2.1
This configuration will be set up in a primary/secondary failover mode, i will not be doing load balancing

1. Setup gateway detection on wan1

This section tells the firewall to ping an IP (in this case, our gateway) repeatedly to see if it's alive or not. For most cable and DSL type connections this will work fine, however if you are interfacing with another ISP's router that may continue responding to pings even if the circuit is down, then consider using an internet IP such as 4.2.2.2 or 8.8.8.8. You can do this on both wan interfaces if you want it to fail both directions.

config router gwdetect
    edit 1
        set interface "wan1"
        set server "1.1.1.1"
    next
end

On newer firmware, this is replaced with link monitoring:

config system link-monitor
    edit "wan1 check"
        set srcintf "wan1"
        set server "1.1.1.1"
        set gateway-ip 1.1.1.1
        set timeout 5
        set recoverytime 1
    next
end

2. Configure your interfaces

Most of this can be done in the web interface, the only critical piece here are the two fail-detect options on wan1. You can also turn on failure detection for wan2 (make sure you add a gwdetect for it as well), however i don't find it necessary here since if both wan1 and wan2 are down then i'm screwed anyways. Note that the fail-detect options are not necessary on newer firmware.

config system interface
    edit "wan1"
        set ip 1.1.1.2 255.255.255.248
        set fail-detect enable
        set fail-detect-option detectserver
    next
    edit "wan2"
        set ip 2.2.2.2 255.255.255.248
    next
end

3. Configure your static routes

These can be done in the web interface, the important piece is to set the distance on wan2 to a higher value than wan1 so that wan1 is always the preferred route

config router static
    edit 1
        set device "wan1"
        set gateway 1.1.1.1
    next
    edit 2
        set device "wan2"
        set distance 20
        set gateway 2.2.2.1
    next
end

4. Configure Policies

This doesn't need to be done in the CLI, you just need to have an internal to wan1 allow/NAT rule, as well as an internal to wan2 allow/NAT rule. It will choose the proper rule based on which port the traffic is being routed through (the active internet connection). In case you want to see it in CLI, here it is:

config firewall policy
    edit 1
        set srcintf "internal"
        set dstintf "wan1"
        set srcaddr "all"
        set dstaddr "all"
        set action accept
        set schedule "always"
        set service "ALL"
        set nat enable
    next
    edit 2
        set srcintf "internal"
        set dstintf "wan2"
        set srcaddr "all"
        set dstaddr "all"
        set action accept
        set schedule "always"
        set service "ALL"
        set nat enable
    next
end

5. Test and confirm

At this point you should be up and running. You can check the status of the gateway detection using get router info gwdetect. You can also check the routing table status using get router info routing-table all.

It is important that when you do your testing you simulate a logical failure of the circuit so your wan1 stays "up". The easy way to do this is to unplug the carrier side of the equipment (unplug the coax for cable, phone line for DSL, T1/loops for T1/EoC/EoDS1, etc.)

6. VPN setup

If you have IPSEC VPN tunnels and want them to also follow the redundancy here, then there is a little more work to do. This is assuming you already have the tunnel set up on wan1. You would just create the tunnel identically on wan2, selecting wan2 as the local interface. Make sure both tunnels have DPD (Dead Peer Detection) turned on.

After setting up the second tunnel, you will add an IPSEC policy from internal to wan2 using the new tunnel configuration. You will of course have to configure the secondary tunnel on the other end as well. After doing this, the VPN should follow the active interface

7. Using VIPs on backup links

One small quirk, if your default route points out wan1 for example, but you want to have servers using Virtual IPs on wan2 while wan1 is still active, you need to do this:

config system settings
  set asymroute enable
end

This is necessary because the Fortigate will think traffic coming in over wan2 is spoofed unless the default route is pointing there. It would be preferable to leave this feature disabled but this is really the only way to accomplish having both circuits live at the same time.

Exchange 2010 to 2013 Migration - Public folders not accessible to migrated users

This was a fun one, due to the ridiculous public folder migration process that is required to go from Exchange 2010 to 2013, you are basically forced to have the two versions coexist for a while to get everything migrated. In the interim, i ended up having to have my 2013 Client Access Server proxy all connections back to the public folder database on my 2010 server (combo Hub/CAS/Mailbox). What i noticed was that i had a few limitations and quirks:

• Users with mailboxes located on the 2013 Mailbox Server could not access public folders on the 2010 server, they received an error something along the lines of: Cannot expand the folder. Microsoft Exchange is not available. Either there are network problems or the Exchange server is down for maintenance.
• Users with mailboxes located on the 2010 Mailbox Server could not use RPC over HTTP (Outlook Anywhere) through the 2013 server.
• Users with mailboxes located on the 2013 Mailbox Server could use RPC over HTTP without a problem.
• All users could use OWA and Activesync

After quite a bit of troubleshooting, it was determined that the problem was the 2013 CAS was unable to proxy requests using RPC over HTTP to the 2010 CAS. There didn't seem to be a good reason for this, all certificates were good, internal/external URLs on all virtual directories specified properly, authentication methods configured properly, etc. Finally, this solution was found. I can't say for certain that this is the only fix required, however so far it has worked well for me.

The fix (run this on your 2010 CAS):

Set-OutlookProvider EXCH -CertPrincipalName msstd:mail.yourdomain.com
Set-OutlookProvider EXPR -CertPrincipalName msstd:mail.yourdomain.com

That's it. This is a setting that was always there on 2010 but was never needed before. Note that mail.yourdomain.com should be the CN (common name) of your certificate. You can get that, if you don't know it, by running Get-ExchangeCertificate -server 2010CAS and looking at the CN= section for the entry that has IP.WS. listed for services. The moment i did this, RPC over HTTP for all users started working, and public folder access was fixed.

ASA 8.4 and Hairpin NAT

I previously posted here about setting up Hairpin NAT on a Cisco ASA running version 8.2, but of course they changed the syntax yet again, so here is an updated version for 8.4. For those unaware, Hairpin NAT is what allows you to access a server by its public IP from inside the network. This is useful if you are not running split DNS or if you have needy users that want to be able to use the same IP wherever they are.

The procedure for this is actually a bit simpler, there are just a few commands required:

Allow the traffic to get turned around:

same-security-traffic permit intra-interface

Add the NAT rule for users without their own static mapping

object network obj_any_hairpin
 subnet 0.0.0.0 0.0.0.0

object network obj_any_hairpin
 nat (inside,inside) dynamic interface

Note that i separated these two blocks as that is how it usually appears in the config. The first block defines the object group, while the second one applies the NAT rule to it. This rule will match for any LAN users that do not have a static 1:1 mapping defined.

Add NAT rules for all static users

object network obj_10.10.10.10
 host 10.10.10.10
object network obj_10.10.10.10_hairpin
 host 10.10.10.10

object network obj_10.10.10.10
 nat (inside,outside) static 1.2.3.4
object network obj_10.10.10.10_hairpin
 nat (inside,inside) static 1.2.3.4

The object groups need to be separate so that you can apply different NAT rules to them. The first object group and the corresponding NAT rule is for the external access coming in, while the second set are the ones that allow the internal to internal traffic.

Setting up a setuid PHP binary

I'm not going to go into it too much, but i assure you i had a very good reason for doing this.

The problem: I needed to run PHP on the command line as root when invoked by a non privileged user. The simple solution is to make a copy of the php binary and them chmod +s it and chown it to root. For most apps, this will just cause it to run as root and everything will be solved. This part is still necessary:

# cd /path/to/my/project
# cp `which php` .
# chown root:root php
# chmod +s php

Unfortunately, when running this, you will find that PHP does not actually think it's running as root, and subsequently you can't run things like shutdown which require root privileges. You can see this by running a script like this:


test.php:

<?php
    echo shell_exec('id');
?>

Command:

# sudo -u www ./php test.php
uid=80(www) gid=80(www) groups=80(www)

Well that didn't work, and the PHP docs and google didn't seem to have anything about PHP dropping privileges. A little more googling and i saw that PHP does have a setuid function, so i decided to give that a try. The function is called posix_setuid() and is allegedly included by default. Well guess what, on my PHP installation it shows the configure string as including --enable-posix=shared which for whatever reason means "i don't want to use this."

So i recompiled PHP with just --enable-posix and the function was now available. At this point, i began getting the desired results when calling the script: test.php:

<?php
posix_setuid(0);
echo shell_exec(`id`);
?>

Command:

# sudo -u www ./php test.php
uid=0(root) gid=80(www) groups=0(root),80(www)

Exchange 2010 Couldn't connect to the source mailbox

So I was working on decomissioning a horribly broken and misconfigured SBS 2011 server today. Part of this was to export all of the mailboxes to PST and import on the new server (don't ask why but the customer had two SBS 2011 servers). When attempting to export a mailbox, i ran into this error:

Couldn't connect to the source mailbox.
    + CategoryInfo          : NotSpecified: (0:Int32) [New-MailboxExportRequest], RemotePermanentException
    + FullyQualifiedErrorId : CA71126,Microsoft.Exchange.Management.RecipientTasks.NewMailboxExportRequest

Well that's fun, so i re-ran the export command with -Verbose and ended up with:

New-MailboxExportRequest : Couldn't connect to the source mailbox. -->
MapiExceptionCallFailed: Unable to make connection to the server. (hr=0x80004005, ec=1249)
Diagnostic context:
    Lid: 59431   EMSMDB.EcDoConnectEx called [length=154]
    Lid: 34855   EMSMDB.EcDoConnectEx returned [ec=0x4E1][length=56][latency=0]
    Lid: 59505   StoreEc: 0x4E1
    Lid: 52465   StoreEc: 0x4E1
    Lid: 60065
    Lid: 33777   StoreEc: 0x4E1
    Lid: 59805
    Lid: 52209   StoreEc: 0x4E1
    Lid: 56583
    Lid: 52487   StoreEc: 0x4E1
    Lid: 19778
    Lid: 27970   StoreEc: 0x4E1
    Lid: 17730
    Lid: 25922   StoreEc: 0x4E1

After a bit of googling i did find someone else with a similar error, but their solution actually worked oppositely for me, so here is what i did to resolve:

Set-CASMailbox jsmith -MAPIBlockOutlookNonCachedMode:$false

After that, the mailbox exported correctly. I can't say for sure why this was blocked in the first place, but this did resolve it.

Drag and drop stops working in Windows 7 and 8

I've been struggling with this problem for a long time, and never really found a solution. Basically (and very randomly) drag and drop would stop working on my workstation. This includes dragging emails in Outlook (2007, 2010, 2013), dragging files in explorer, and even resizing columns in certain folders/views/apps. What threw me the most is that the issue persisted when i upgraded to a new workstation with a fresh OS, and also across Windows 7 and 8. I do have a lot of apps, so it's very possible one of them is to blame. A reboot would always fix it, but that's obviously annoying. Anyways, the fix: Go to explorer, click on a file, and press the escape key. That's it. I have yet to investigate why it works, but it works. The working theory is that somewhere a file is stuck in the drag state, and windows doesn't let you drag additional items. Hitting escape must clear that out and let you do it again.

Media Keyboard Shortcuts with AutoHotKey

Recently I switched to a mechanical keyboard, and though I love it, the one thing I missed were the media keys on my old keyboard. At work I frequently utilize the volume and play/pause controls when quickly answering a phone call, and stumbling to hit the volume icon in the system tray was just not quite fast enough. To solve this problem, I ended up installing AutoHotKey and setting up some very simple automation. The basic method is to create a new text file with an AHK extension (I called mine media.ahk) and add the hotkeys to the file. You can mix these up as much as you want, check the AHK docs for more info. This script uses the windows key for all of the hotkeys:

#PgUp::Send {Volume_Up 3}
#PgDn::Send {Volume_Down 3}
#End::Send {Media_Play_Pause}
#Home::Send {Volume_Mute}
#Insert::Send {Media_Next}

After making the file, you can just double click on it for AHK to pick it up. See their docs for making that happen at login. Since I'm on Windows 8, there is already an on-screen volume meter when you start changing the volume, so I did not need an OSD or anything fancy.