Migrating to Office 365 from BAE Systems / Silversky Hosted Exchange

This was a fun one... not an easy process to figure out so hopefully this helps someone. To migrate away from the BAE Systems aka Silversky Hosted Exchange platform, you need to get a few things:

1. Your Office 365 organization should already be set up and ready to go (obviously)
2. An admin account in Hosted Exchange that has Full Access to the mailboxes you wish to migrate, and is a licensed user (has an Exchange package)
3. Either an Outlook installation you can use, or the testconnectivity.microsoft.com website will also do.
4. Access to the BAE Systems Exchange Provisioning Console (it's a java app)

Office 365 Setup

At this point you should have your organization created, domain added, and at least one licensed user (admin account is ok). That part of the setup is far out of scope for this guide.

Getting Admin Access

If you have just a few mailboxes, then you can just do this yourself. I highly recommend you test this with one or two mailboxes first, as i have had the built-in cusrpt@ account not work correctly.

For each user:

1. Open the user account in the provisioning console
2. Go to Packages, and edit the Exchange package
3. Launch the Mailbox Rights Management window
4. Add your admin account and grant it Full Access
5. Save all the way out, rinse and repeat

If you have a ton of mailboxes, then you may need to call support. You can launch multiple instances of the provisioning console to pipeline the requests, but i just did 500 users, so that wasn't happening. In that case, you can call support and get them to run a Powershell command to grant you access. It may take a little work to get them to escalate it, and they also may want to charge you for it. I'm not going to go over the details of the command here, you can google that one.

Starting the Migration Batch

1. Open the Exchange admin center, and head over to Recipients/Migration
2. Create a new Migration batch, "Migrate to Exchange Online"
3. Set the batch type to Cutover
4. Enter the account credentials that you set up in the previous section (the one with access to all the mailboxes)
5. It will probably give you an error at this point, that's why this guide exists. You now need to enter the Exchange server and RPC proxy server. Here we go...
   The easy part first: set the RPC proxy to exchange.postoffice.net. Now the hard part. Use either the testconnectivity.microsoft.com website, or (for the security conscious) your Outlook's Autodiscover feature. To use the former, just go to the website and enter the info, doing an Autodiscover test. To use the latter, ctrl+click on your Outlook icon, go to Test E-mail Autoconfiguration, enter the credentials and un-check the Guessmart options.
   
   At this point, regardless of the option, you will end up with the Autodiscover XML output. In here, you will see a section that says:
   
   

   <Protocol>
     <Type>EXCH</Type>
     <Server>2bd60169-a9ac-4f15-bdd4-dea1a11129a7@yourdomain.com</Server>
   

   
   You need to copy/paste the contents of the Server tag (the part that looks like an email address, including the @yourdomain.com). This is what you use for the Exchange server field.
6. Once you've done that, you do need to use the advanced/more option and set Authentication to Basic and Mailbox Permission to Full Access and hit Next
7. Name your batch, select the notification email (aren't you glad i told you to have at least one licensed user?), and you can start.

At this point, you're migrating. You can monitor the batch status from the console like you normally would. If doing a large number of mailboxes, use Powershell.

Fortigate - Policy Routing to VPN Tunnels

I have found that FortiOS v6+ has slightly different/broken functionality when handling policy routes across VPN tunnels (phase1/2-interface). In v5 you could create a policy route pointing to the tunnel interface and leave the gateway address set to 0.0.0.0, and everything worked fine. In v6 however, it appears leaving it as 0.0.0.0 is Fortinet code for please don't use me, i'm stupid.

The fix is pretty simple, assign an IP on both sides on the tunnel interface (just a /30 is fine, i.e. 10.255.255.1/30 and 10.255.255.2/30). Then in the policy routes, reference that IP as the gateway, and it should start working.

Sample Config:

(Note that in this case, the tunnel was to route all internet bound traffic across the VPN, so yours may look different)

Firewall 1

config system interface
    edit "Corp"
        set vdom "root"
        set ip 10.153.153.2 255.255.255.255
        set type tunnel
        set remote-ip 10.3.153.1 255.255.255.252
        set interface "wan1"
    next
end

config router policy
    edit 1
        set input-device "internal"
        set src "10.2.53.0/255.255.255.0"
        set dst "0.0.0.0/0.0.0.0"
        set gateway 10.153.153.1
        set output-device "Corp"
    next
end

Firewall 2

config system interface
    edit "Remote"
        set vdom "root"
        set ip 10.153.153.1 255.255.255.255
        set type tunnel
        set remote-ip 10.3.153.2 255.255.255.252
        set interface "wan1"
    next
end

config router policy
    edit 1
        set input-device "internal"
        set src "0.0.0.0/0.0.0.0"
        set dst "10.2.53.0/255.255.255.0"
        set gateway 10.153.153.2
        set output-device "Remote"
    next
end

Automatically Rotating Group Policy Logon Messages

This is a pretty common task for various types of compliance, where you use the logon message via GPO to display security notices or other company facts. But you can't just have a stale message, it needs to be rotated to keep it fresh. I got tired of doing mine by hand, so i wrote a script to do it. Unfortunately the PowerShell Group Policy module has pretty poor documentation, and i'm not even confident it's capable of doing this. That said, some group policies use files in SYSVOL, while others store their data in AD. In this case, it's in a .inf file, so here is the powershell script:

$basepath="\\yourdomain.local\sysvol\yourdomain.local\policies\{CF267D2E-F5BE-46D9-85B3-58125FEFB1CF}\machine\microsoft\windows nt\secedit"
$tmpfile="$basepath\tmp.inf"
$tplfile="$basepath\GptTmpl.inf"
$bakfile="$basepath\GptTmpl_bak.txt"
$notices=@(
    "Your first notice.",
    "Your second notice.",
    "Etc."
)

$phrase=$notices[(get-random -Maximum ([array]$notices).count)]
$phrase=$phrase -replace ",","`",`""  #the .inf wraps commas in quotes when it's part of the string

new-item -path $tmpfile -ItemType file -Force | out-null
foreach ( $line in get-content $tplfile ) {
    if ( $line -match "LegalNoticeText" ) {
        "MACHINE\Software\Microsoft\Windows\CurrentVersion\Policies\System\LegalNoticeText=7,$phrase" | out-file -filepath $tmpfile -append
    } else {
        $line | out-file -filepath $tmpfile -append
    }
}
move-item $tplfile $bakfile -Force
move-item $tmpfile $tplfile

Just get the GUID of the GPO you're using from GPMC and replace it in the $basepath, and you're set. Run that as an account with permission to that path and it will automatically rotate it.

Exchange 2010 Installation Fails with Error "Provisioning layer initialization failed"

For a customer with an older environment, needed to re-build a DAG member and ran into this issue. Upon installation of the management tools, even with a fresh SP3 download, the install would error out. The guts of the error messages were this:

[04/24/2018 22:21:56.0372] [2] [ERROR] Provisioning layer initialization failed: '"Scripting Agent initialization failed: "File is not found: 'C:\Program Files\Microsoft\Exchange Server\V14\Bin\CmdletExtensionAgents\ScriptingAgentConfig.xml'.""'
[04/24/2018 22:21:56.0372] [2] [ERROR] "Scripting Agent initialization failed: "File is not found: 'C:\Program Files\Microsoft\Exchange Server\V14\Bin\CmdletExtensionAgents\ScriptingAgentConfig.xml'.""
[04/24/2018 22:21:56.0372] [2] [ERROR] "File is not found: 'C:\Program Files\Microsoft\Exchange Server\V14\Bin\CmdletExtensionAgents\ScriptingAgentConfig.xml'."
[04/24/2018 22:21:56.0388] [2] [ERROR] Provisioning layer initialization failed: '"Scripting Agent initialization failed: "File is not found: 'C:\Program Files\Microsoft\Exchange Server\V14\Bin\CmdletExtensionAgents\ScriptingAgentConfig.xml'.""'
[04/24/2018 22:21:56.0388] [2] [ERROR] "Scripting Agent initialization failed: "File is not found: 'C:\Program Files\Microsoft\Exchange Server\V14\Bin\CmdletExtensionAgents\ScriptingAgentConfig.xml'.""
[04/24/2018 22:21:56.0388] [2] [ERROR] "File is not found: 'C:\Program Files\Microsoft\Exchange Server\V14\Bin\CmdletExtensionAgents\ScriptingAgentConfig.xml'."

Turns out, this is related to the PowerShell cmdlet Extension called Scripting Agent, which is enabled by default. Many posts online say you can just go to the path listed above and rename the .sample file to fix it, but that doesn't actually fix the management tools. In actuality, you need to disable the agent, run the install, then (if desired) re-enable the agent.

1. Open a the EMS (Exchange Management Shell) on another exchange server in the org and run:
   

   Disable-CmdletExtensionAgent "Scripting Agent"

2. Fully install Exchange
3. Copy C:\Program Files\Microsoft\Exchange Server\V14\bin\CmdletExtensionAgents\ScriptingAgentConfig.xml from another exchange server to the same path on the new server
4. Re-enable the agent:
   

   Enable-CmdletExtensionAgent "Scripting Agent"

I suppose not much Microsoft could do to work around this without releasing a new service pack, but still frustrating.

ConnectWise Manage Report Writer - Repeaters and Subtotals

If you're reading this you're probably aware that the support/documentation provided by ConnectWise for more difficult tasks in Report Writer is a bit... lacking. I ran into a few issues on Saturday and while i figured them all out the same day, i did a CW chat and submitted a ticket and have yet to hear back (4 full business days later). Basically, if you want to use Repeaters and/or Subtotals, there are very specific constraints that you need to keep in mind. This post is not for someone just getting into report writer, you need to already know how to build queries, use joins, and more.

Repeaters

These are a great solution to not having to use a sub-report. That said, they're not very intuitive and are in fact pretty basic in nature. The gist of it is that once you start the repeater, the first column you reference and all subsequent ones (unless you hit a nested a repeater) are repeated until the initial column changes. The documentation flip flops between using [repeater], <!--[repeater]-->, and <repeater>. The latter is never correct. The first two can be used interchangeably, but if you're wrapping this around HTML content then the middle option is the correct one.

As an example, say you are making a report that lists a client's agreements with their additions. The data in table form may look like this:

Agreement Name	Anniversary	Addition Name	Addition Qty	Addition Price	Extended Price
Managed Services	1/1/2019	Antivirus	30	$5.00	$150.00
Managed Services	1/1/2019	Agent	30	$1.00	$30.00
Managed Services	1/1/2019	Spam Filtering	35	$3.00	$105.00
Telecom Support	2/1/2020	Handset Fee	20	$10.00	$200.00
Telecom Support	2/1/2020	PBX Maintenance	1	$100.00	$100.00

As you can see, the agreement name is duplicated for each time you have an addition on it, because this is how SQL works. So in Report Writer, on the Fields tab, you would click Design Form and then (if you're sane) edit the HTML directly. The code to put this together would look something like this:

[Company_Name]
<!--[repeater]-->
  <h3>[Agr_Name] - [Agr_Anniversary]</h3>
  <table>
    <tr><th>Item</th><th>Quantity</th><th>Cost</th><th>Ext Cost</th></tr>
    <!--[repeater]-->
      <tr>
        <td>[Line_Desc]</td>
        <td>[Qty]</td>
        <td>[Cost]</td>
        <td>[Ext]</td>
      </tr>
    <!--[/repeater]-->
  </table>
 <hr />
<!--[/repeater]-->

Note that there are nested repeaters. The first level is for the Agreement Name and Amount, the second level is for the items associated with each one. The resulting output looks something like this:

Company Name

Managed Services - 1/1/2019

Item	Quantity	Cost	Ext Cost
Antivirus	30	$5.00	$150.00
Agent	30	$1.00	$30.00
Spam Filtering	35	$3.00	$105.00

---

Telecom - 2/1/2020

Item	Quantity	Cost	Ext Cost
Handset Fee	20	$10.00	$200.00
PBX Maintenance	1	$100.00	$100.00

---

Subtotals

Now let's say that you want to add up all of the addition amounts on the agreement and show that total. Since you still want the itemized list, you need to use the Subtotal Decorator. The CW Manage documentation flip flops on whether this is @Subtotal or #Subtotal, don't ask me why. The correct usage is @Subtotal. There are however a few constraints to consider:

• Subtotal only makes sense in a repeater. If your data looks how you want it and you're not using a repeater, then this should be done in the query, not in the report designer.
• You must check the VG (visual group) box on every column not part of the repeater that will be subtotaled. This will force them to be sorted as well, but that's not a big deal.
• You must check the Add Subtotals box at the bottom of the Fields tab.
• CW will tell you that you can only use Subtotal on one field. This is incorrect, you can subtotal as many fields in a repeater group as you want.
• CW neglects to tell you that the column you are subtotaling cannot have spaces OR underscores in it.

Using the repeater example above, here is how you would add a subtotal of the overall ext costs of the additions:

[Company_Name]
<!--[repeater]-->
  <h3>[Agr_Name]</h3>
  <table>
    <tr><th>Item</th><th>Quantity</th><th>Cost</th><th>Ext Cost</th></tr>
    <!--[repeater]-->
      <tr>
        <td>[Line_Desc]</td>
        <td>[Qty]</td>
        <td>[Cost]</td>
        <td>[Ext]</td>
      </tr>
    <!--[/repeater]-->
    <tr><td colspan="3">Total:</td><td>[Ext@Subtotal]</td></tr>
  </table>
  <hr />
<!--[/repeater]-->

It's nice to note, the Subtotal can go above or below the repeater group. If you want to subtotal more than one column, just add that column with the @Subtotal decorator wherever you need.

Migrating VMkernel adapters on vSphere Distributed Virtual Switches

A chicken/egg scenario can sometimes occur when setting up DVS (distributed virtual switch) in a vCenter cluster. The issue being your vCenter is connected to your host over its' vmk0, so any network changes have to be confirmed or else they will be rolled back. In my case, i wanted to change the links to have my management network be a tagged VLAN instead of native. The problem here is that i would need to make the change in vCenter, then change the tagging on the switch ports, then hope that it picked up the changes. In almost all cases, it didn't, and reverted the settings. In a few cases i got lucky with the timing and it worked, but most of the time it didn't.

The solution is to set the Advanced vCenter setting of config.vpxd.network.rollback to false, then make the change. This causes it to not roll back the changes, so you better be sure that your configuration is correct. This should of course be set back to true when you're done.

Labtech and Corrupted Scripts

Update: The latest patch fixes this, or at least handles it more gracefully. v110.387 (Patch 15) is what is needed, otherwise see below.

Labtech (or Connectwise Automate, which i refuse to say outloud), has a current bug where you can quite easily corrupt a script. Since the scripts are not traditional text files, and are stored encoded in the database, there is no easy way to fix it if something does get corrupted. I'll outline how this happens, and how you can fix it, since their support has indicated that this is a low priority issue to them.

The Issue

For various reasons a script can get corrupted. When i say corrupted, it's all still there, but something is preventing it from getting loaded. For example, the other day my coworker added a new EDF and then asked me to take a look at a script. When i loaded it, the script functions did not show the EDF name but just showed the ID. Though i advised against it, we ended up making some changes and saving the script. After that, the script would no longer load with the error "Error loading script:Syntax error: Missing operand after '=' operator." (see below):

Now, you'd think that Labtech would still load the script and force you to fix it... NOPE. Any work you've put into that script is now lost, you need to restore from backups. This means either restoring your entire database, or grabbing the nightly .sql backup files, editing to grab the right data, and replacing via SQL. But what if you just put hours of work into the script and backups haven't run yet? Well, you're SOL in this case. Or are you?

The Fixes

Restoring from backup

If you just want to go to last night's backup, then the fix is not too difficult. You will need adequate knowledge of running SQL queries against the Labtech server, and the text editor of your choice.

1. Take a new backup, in case you mess something up.
2. Grab the backup that you want to restore. These are usually located in C:\Program Files\LabTech\Backup. You can go into the Tablebase folder or extract one of the zip files if you need an older copy. The file you need is lt_scripts.sql. Make a copy of this in a spot where you can edit it.
3. Determine the ID of the script that is corrupted. Easiest method here is to show IDs in the Labtech Client (Tools > Show ID's). The ScriptID is the number in parenthesis next to the script in the Navigation Tree. For the sake of this guide, we will be using ID 5986.
4. Edit the .sql file, i used notepad++. Search for the ID of the script, you should see something that looks like this. The first red square is what you're looking for with the script ID, the second is the actual content that you need to use. You want everything between the apostrophes, in my example starting with H4sI and ending in AAA===. I had to blur a lot of this for obvious reasons, but you should get the gist of it.
5. Open your preferred SQL editor (Labtech staff usually install SQLyog on the server, you can also use the CLI or MySQL workbench)
6. Craft a query that reads:
   UPDATE lt_scripts SET ScriptData='H4sI......AAA==' WHERE ScriptID=5986;
   (replace 5986 with your ScriptID, and paste the FULL content of what you selected into the ScriptData portion of the statement.)
7. Run the query, and it will replace the body of the script with your backup copy.
8. At this point you should be able to edit the script, no Refresh/Reload needed
9. If not, you may need to go to an earlier backup

Fixing the script in-place

This is definitely the more difficult route. If your backup from last night is good enough then i'd use the previous section to restore from backup. Otherwise, read on. This requires knowledge of running SQL queries, minor XML knowledge, as well as the ability to perform Base64 encodes/decodes and gzip compressions/decompressions. I did this on a linux server, however you can also do it with other tools. I will gloss over various tasks that are covered in the previous section for the sake of brevity.

1. Take a new backup, in case you mess something up.
2. Extract the ScriptData for this script using this query:
   SELECT ScriptData FROM lt_scripts WHERE ScriptId=5986;
3. Base64 Decode the result. You can use an online tool, or on a Linux shell:
   # base64 -d > output.gz
   After running that command, paste the data in and press Ctrl+D. However you do it, you need to end up with a .gz file containing the base64 decoded content.
4. Unzip the file. On a Linux shell you can do:
   # gzip -d output.gz
   or
   # gunzip output.gz
5. output is now an XML file containing the script. Edit this with a text editor (i use Notepad++) and find the section with the error. If you note, when you open the script editor and it errors out, it does load at least a few lines before dying. The last line it loads is where you want to start looking. If this has a variable name or something you can search for, use that. In my case of a missing EDF, i ended up with a section that looked like this:

   <ScriptSteps>
       <Action>2</Action>
       <FunctionId>103</FunctionId>
       <Param1 />
       <Param2>%computerid%</Param2>
       <Param3>DuoDefaults</Param3>
       <Param4 />
       <Param5 />
       <Sort>8</Sort>
       <Continue>1</Continue>
       <OsLimit>0</OsLimit>
       <Indentation>1</Indentation>
     </ScriptSteps>
     <ScriptSteps>
       <Action>2</Action>
       <FunctionId>103</FunctionId>
       <Param1>611</Param1>
       <Param2>%computerid%</Param2>
       <Param3>APIHostnameComputer</Param3>
       <Param4 />
       <Param5 />
       <Sort>9</Sort>
       <Continue>1</Continue>
       <OsLimit>0</OsLimit>
       <Indentation>1</Indentation>
     </ScriptSteps>
   

6. If you notice, the second ScriptSteps block has an ID specified in Param1, while the first one does not. This is where LT stripped out the non-existent EDF and shot itself in the foot. In my case, i just needed to find the EDF ID and replace <Param1 /> with <Param1>1234</Param1>. If you're not up to it, just delete the entire ScriptSteps block.
7. Once done, save the file, and gzip it back up again:
   # gzip output
8. Then, base64 encode the file:
   # base64 output.gz > fixed.txt
9. Finally, copy the contents of fixed.txt and run an update to replace the ScriptData with the fixed version:
   UPDATE lt_scripts SET ScriptData='H4sI......AAA==' WHERE ScriptID=5986;
   (replace 5986 with your ScriptID, and paste the FULL content of what you selected into the ScriptData portion of the statement.)
10. Run the query, and it will replace the body of the script with your backup copy.
11. At this point you should be able to edit the script, no Refresh/Reload needed

Final Notes

• Do not update the ScriptData of one script with the contents of another. The ScriptData has the GUID of the script encoded in it so it will still be broken.
• Don't have access to a linux server, or don't know linux? Here are some alternatives:
  • For Base64 encoding/decoding, you can use your programming/scripting language of choice. Powershell has the ability to do Base64 Encoding/Decoding, a quick google can tell you how to do that. You can't really use websites for this portion because you are dealing with binary data.
  • For gzip compression, you can install 7zip to take care of this for you.
• It's a good practice to export your scripts after making large changes. If you can store these in version control (such as git), that's even better.