Office 365 Hybrid with Exchange 2010 Setup

Couldn't find a guide that actually gives you ALL of the steps required to setup a Hybrid organization with Exchange 2010 and Office 365, so here it is. This is not super technical, just the overall steps and things not mentioned/explained in other guides. This assumes you already have created a 365 tenant and added your domains to it.

Configure Azure AD Sync

Prep work: make sure you have added your primary email domain as an alternate UPN suffix (see Active Directory Domains and Trusts) and set all of your users UPNs to match their primary email addresses. This is not required but makes it cleaner

Next you need to synchronize your Active Directory to Office 365. Install the Azure AD Sync tool, and check the boxes for Password hash synchronization and Exchange hybrid deployment. You can check other boxes if you know what you're doing, otherwise best not to.

Important note: If you want to set this up with a group or OU based filtering, that's fine for testing. But be aware, any user NOT synced into 365 will be unroutable for sending email from 365 back to on-prem. So you really need to sync everyone.

Run the Hybrid Wizard

Grab it from https://aka.ms/HybridWizard or by logging into the 365 admin console, opening Exchange, and going to Hybrid. You should run this from an Exchange 2010 server.

When going through the settings, the defaults are typically going to be correct. If you need to adjust anything, go for it. The wizard will automatically create connectors for you and set up various things

Do the extra tasks they don't tell you about

1. In the 365 Exchange Admin Console (EAC), navigate to Mail Flow > Accepted Domains and set your email domains to Internal relay. This is what allows 365 to route mail back to your on-premise environment
2. If you haven't already, set your Azure AD Sync to push all of your Exchange users into Azure AD. You can verify this by going to Recipients > Contacts in the EAC, and you should see them all in there as type Mail user. If they are not, then stop and fix that. Again, this is for any user in Exchange, not just the ones you want to migrate.
3. If you have assigned any licenses to users which grant Exchange access, you need to disable that or remove it. Users cannot be licensed for Exchange and also be migrated.

Migrate

At this point you can go to Recipients > migration and start a batch. Use the Remote move migration option and select whoever you want to migrate. I suggest just letting them auto-complete, but you can manually complete them if you wish. This takes a while. Once it's done, they may get a message in Outlook asking them to restart, however as long as they're on a newer client it should auto-reconfigure for 365 and you're done.

Setting up iOS14 mail with Azure AD App Consent

If you locked down your Azure AD to make it so users can't consent for approval on apps, then you may end up with iPhone/iPad users trying to setup their devices with iOS 14. Well, they can't consent, and the app isn't listed in the gallery or anywhere else. The user gets a message Need admin approval for Apple Internet Accounts. Instead of turning user approval back on, or trying to get a hold of an iOS 14 device yourself, you can still do the approval.

Head over to your Azure AD tenant and grab your Tenant ID (from the Overview tab). Then modify the following URL to add your tenant ID in:

https://login.microsoftonline.com/TenantID/oauth2/authorize?client_id=f8d98a96-0999-43f5-8af3-69971c7bb423&response_type=code&prompt=admin_consent

Heading to that URL should prompt for consent, and then your users can get in. Note that i omitted the redirect_uri parameter which would send you to the next site, so it may seem to hang after you approve. Just give it 10 seconds and close it out and have the user try.

Domain Controller Demotion Error: The operations master roles held by this directory server could not transfer to the following remote directory server.

In demoting a Domain Controller, ran into the following error:

The operations master roles held by this directory server could not transfer to the following remote directory server. 

FSMO Role: CN=Infrastructure,DC=ForestDnsZones,DC=domain,DC=local

Apparently during the last demotion, the roles for the DNS Zones weren't transferred (both for Forest and Domain). There is an easy fix, but it wasn't very straight forward to google.

1. On the new Domain Controller (the one you're not demoting), open notepad and paste this into a file named fixfsmo.vbs:
   
   

   const ADS_NAME_INITTYPE_GC = 3
   const ADS_NAME_TYPE_1779 = 1
   const ADS_NAME_TYPE_CANONICAL = 2
   
   set inArgs = WScript.Arguments
   
   if (inArgs.Count = 1) then
       ' Assume the command line argument is the NDNC (in DN form) to use.
       NdncDN = inArgs(0)
   Else
       Wscript.StdOut.Write "usage: cscript fixfsmo.vbs NdncDN"
   End if
   
   if (NdncDN <> "") then
   
       ' Convert the DN form of the NDNC into DNS dotted form.
       Set objTranslator = CreateObject("NameTranslate")
       objTranslator.Init ADS_NAME_INITTYPE_GC, ""
       objTranslator.Set ADS_NAME_TYPE_1779, NdncDN
       strDomainDNS = objTranslator.Get(ADS_NAME_TYPE_CANONICAL)
       strDomainDNS = Left(strDomainDNS, len(strDomainDNS)-1)
       
       Wscript.Echo "DNS name: " & strDomainDNS
   
       ' Find a domain controller that hosts this NDNC and that is online.
       set objRootDSE = GetObject("LDAP://" & strDomainDNS & "/RootDSE")
       strDnsHostName = objRootDSE.Get("dnsHostName")
       strDsServiceName = objRootDSE.Get("dsServiceName")
       Wscript.Echo "Using DC " & strDnsHostName
   
       ' Get the current infrastructure fsmo.
       strInfraDN = "CN=Infrastructure," & NdncDN
       set objInfra = GetObject("LDAP://" & strInfraDN)
       Wscript.Echo "infra fsmo is " & objInfra.fsmoroleowner
   
       ' If the current fsmo holder is deleted, set the fsmo holder to this domain controller.
   
       if (InStr(objInfra.fsmoroleowner, "\0ADEL:") > 0) then
   
           ' Set the fsmo holder to this domain controller.
           objInfra.Put "fSMORoleOwner",  strDsServiceName
           objInfra.SetInfo
   
           ' Read the fsmo holder back.
           set objInfra = GetObject("LDAP://" & strInfraDN)
           Wscript.Echo "infra fsmo changed to:" & objInfra.fsmoroleowner
   
       End if
   
   End if
   

2. Open an administrator cmd prompt at the location of the vbs
3. Run it once each with ForestDnsZones and DomainDnsZones (or just specify the one you're getting in the error) like this:
   
   

   cscript fixfsmo.vbs DC=ForestDnsZones,DC=domain,DC=local

4. You will need to fully close and re-run dcpromo to do the demotion, but it should go through now

Credit to this blog for the script.

Setting up ConnectWise Manage with Duo Access Gateway via SAML

This will give you all the details on how to set up ConnectWise Manage with Duo Access Gateway via SAML. I could not find any documentation for the pairing of these two products, so there was a little bit of trial and error.

Pre-Requisites

• A Duo Access Gateway (DAG) server already setup and ready to go.
• A Duo account with Duo MFA or higher licensing. (the free version does not support this)
• You must have an admin account to both CW:M and Duo (duh)

Recommendations/Caveats

• The moment you save the SSO configuration in CW:M, it will remove all settings related to LDAP and existing 2FA with Google Authenticator. The CW documentation says it disables it, but no, it rips it out.
• I recommend you sync your training database and do the setup there first. This way you can do it live, and also avoid issues caused by the point above
• I am not going to train you on how to use the various products, this is just about the integration. I expect you already know how to create applications in Duo and add them to the access gateway.
• This example is going to be for the training database in CW:M, so all URLs etc will reference that. The fake CW:M URL i will be using is manage.domain.com, and your DAG URL will be dag.company.com

Duo Setup

1. Log into the Duo Admin Panel and create a new Application of type SAML - Service Provider
2. Enter the following settings:
   
   • Service Provider Name: ConnectWise Manage - Training
   • Entity ID: https://manage.company.com/v4_6_release/auth/training/metadata
   • Assertion Consumer Service: https://manage.company.com/v4_6_release/auth/training/Acs
   • Service Provider Login URL: https://manage.company.com
   • NameID Format: urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress
   • NameID attribute: userPrincipalName (this can also be mail, just make sure it's an email address format that your DAG is pulling in from your identity source
   • All other settings: default
3. Click Save Configuration
4. Scroll down and set the name of the Application (i recommend setting it to the same thing: ConnectWise Manage - Training)
5. Save and then click the link at the top to download the configuration file
6. Head over to your DAG admin panel and upload the downloaded json file
7. Download the certificate

CW:M Setup

1. Navigate to System > Authentication (or Setup Tables > SSO Configuration)
2. Set the following options:
   • Description: Duo Access Gateway (this can be whatever you want)
   • SSO Type: SAML
   • Login URL: copy and paste SSO URL from the DAG Admin
   • Identity Provider ID: copy and paste Entity ID from the DAG Admin
   • Identity Provider Certificate: upload the certificate you downloaded from the DAG Admin
   • Locations: check the box for the locations that this SSO applies to. In most cases it will just be All.
3. Save and you're ready to test
4. Use a different (incognito/private) browser window to test so you don't get locked out.
5. Upon login it will redirect you to the SSO login. It will also handle user enrollments here so if this is a brand new org it's a good way to force your users to enroll
6. Once you're done and tested, repeat the process for your production database. Replace training in the URLs with your company id. Create a new Duo Application for this, that way you will have it always working in both live and training.

Chrome, Remote Desktop Services, and Time Zone Redirection

If you're using RDS and have enabled the GPO for Time Zone Redirection, there is a small issue with Chrome that may cause some issues. For whatever reason, JavaScript date functions in Chrome will pull the system's time data instead of the user's. So if your server is in UTC and your user is in Eastern, Chrome's detection will come back as UTC. Firefox and Internet Explorer do not appear to have this flaw (i did not test with Edge).

To tell if you're affected, open the Chrome inspector (Ctrl+Shift+I) and go to the Console tab. If you just enter in new Date() you will see a preview of what the date is, including the time zone.

As far as i can tell, there is no workaround, if someone has one please let me know. I'll be setting up separate pools for users just for this.

Fortigate SSL VPN with LDAP User Authentication

The documentation was pretty inaccurate/unclear on a lot of this, so there was a bit of trial and error. This assumes you're already familiar with setting up SSL VPN and already have it working with local user accounts. Here's how you do it:

Pre-requisites

• You need your SSL VPN portal and settings configured already
• You should also have already created your SSL VPN policy (allowing from the SSL VPN interface to your LAN)
• The above requires you to add a user or group already, you can re-use that group for the items below if desired

Set up LDAP Server

I'm using Active Directory, but you can use any LDAP based directory service. The example below assumes your AD domain is domain.local.

1. Navigate to User & Device > LDAP Server
2. Add a new server and enter the settings:
   
   • Name: this is the friendly name, i usually just put the hostname in
   • Server IP: obvious
   • Server Port: leave this default unless you know what you're doing
   • Common Name Identifier: This defaults to CN, which means Common Name. This is dumb. Your users are not going to log in as "Bob Smith", they're going to log in as bsmith or bsmith@domain.com. Set this to userprincipalname if you want them to use their UPN (username@domain.com) or maybe samaccountname if you want them to just use their username.
   • Distinguished Name: You can't use the Browse button until you fill out the rest of this page, but this can just be DC=domain,DC=local. Note: if you choose to scope this further than the domain as listed, your group (in the next section) must be in scope.
   • Bind Type: Regular
   • Username/Password: You can start off with a domain admin for testing, but ultimately you should create an unprivileged service account to use here
   • Secure Connection/Protocol: I used secure and selected LDAPS. If you don't have LDAPS then don't use it.
   • At this point you should be able to Test Connectivity and get a success.

Set up your group

1. In Active Directory, create a group and add users to it. I called mine SSL VPN Users
2. In the Fortigate, navigate to User & Device > User Groups
3. Click on Create New
4. Name the group the same as you created in AD (this isn't important, just a friendly name)
5. Select Firewall as the type
6. Under the Remote Groups section, click Add, select your LDAP server, and then search/select your group.
7. Important: You have to right click on it and select Add Selected. After that, hit OK, not before.
8. Hit OK again to save the group
9. Configure your SSL VPN firewall policy to use this group for authentication

Testing

At this point you should be done, because you already set up your SSL VPN, right? You should be able to log in as the user now, you can go to Log & Report > VPN Events to see what the error is if you're not able to log in. You can also use the command diag test authserver ldap "YOUR LDAP SERVER NAME" youraccount@domain.com yourpassword to do a direct test.

Apple Open Directory - Options Greyed Out after IP Change

I recently had to move a few Apple servers from one site to another for a client, and after the move the Open Directory was half-working (opens, lets you view users, but can't add/remove/edit). There were a few no-brainer fixes that i had to do initially, but at the end of the day the final fix was pretty much a bandaid.

Overall obvious steps:

1. Make sure your hosts file (/etc/hosts) is referencing the right IPs, in case anything changed.
2. Make sure the system is set to the right IP(s) for DNS.
3. Make sure the DNS Server records are updated to the new/correct IPs for both the forward and reverse zones.

The bandaid fix for me

In my case, when authenticating within the Workgroup Manager, it would lag and then eventually log in. Nothing in the logs about what was wrong, but the New User button was greyed out, as well as all of the fields on the actual accounts. For whatever reason i could reset passwords, not sure why. The ultimate fix was to add an additional network interface to the server, and set the IP address to the old IP of the server. Yep.

Troubleshooting Steps

• First i fired up tcpdump and did a filter of "port 53" to see if any DNS requests were going out or coming back in incorrectly. Didn't find anything there.
• Next i did tcpdump "host 10.x.x.x" (with the old IP of the server) on a hunch, to see if it was trying to send any traffic there. And yep, it was, on port 389 (LDAP).
• Knowing that, i did a grep for the IP in /etc and a few other places but couldn't find it, assuming it's in a binary DB somewhere or something. So the only solution is to make the traffic actually connect.
• In Networking, i added a new interface under the existing Ethernet, gave it the same IP of the server before the move, but with subnet mask 255.255.255.255. This allows the IP to respond without interfering with any other IPs on the subnet.
• After making that change, the directory connected and started working instantly.