Xerox Scan to Folder via SMB with Windows Server 2016

Setting up a new Server 2016 file server, unable to get a Xerox WorkCentre 7220 to scan to it via SMB. This took a bit of troubleshooting, the copier was reporting invalid credentials and nothing was working. I had tried all combinations of IP vs Hostname, forward/backslashes, DOMAIN\username and username@domain, etc. There was a time mismatch between the server and copier, but after resolving that it was still not working. Ultimately i ended up running Wireshark on the server and found that the copier was trying to establish an SMB connection first to the IP address of the server, then to SMBSERVER (which is not the hostname of my server...).

Some googling came up with a few options, one of which would be to add SMBSERVER as an alternate NetBIOS name for the server. This included reducing some security levels and enabling some other things in the registry that didn't sound like a good idea. Instead, on a hunch, i switched the port from 139 (default) to 445. After switching the port, it worked as expected. Below are the settings that ended up working:

Server: SERVERNAME.domain.local : 445
Share: TheShareName
Document Path: Documents/Scans (no leading slash, and i used forward slashes)
Login Name: DOMAIN\username (we use a dedicated scans user)

See here:

Microsoft Access and Outlook Integration - 48 Error in Loading DLL

A customer with a 3rd party developed Access database/application ran into this issue with some new computers. These were running 64 bit Windows 7 and 32 bit Office. The specific task was trying to open a new email in Outlook from Access.

While the error was 48 Error in loading DLL the problem had nothing to do with DLLs. It ended up being that because the computer was preloaded with Office 2016 (trial version, from Dell), when uninstalled it left some garbage behind in the registry. After running Procmon and finding out what it was trying to do, it wasn't too hard to find and fix.

The solution in the end was to delete the registry key (and subkeys):

HKEY_CLASSES_ROOT\TypeLib\{00062FFF-0000-0000-C000-000000000046}\9.6

Obviously make a backup of this key before deleting, but this did the trick for all affected computers.

PHP and imap_open with subfolders

This was tested against Exchange 2013, i assume similar environments will still apply. When using imap_open, if you're trying to access sub-folders there are a few caveats. Typically your connection string will look something like:

$imap=imap_open('{mail.server.com:143/imap/novalidate-cert}INBOX','username','password');

But there are exceptions to this:

• To connect to a subfolder of the Inbox, the end of the connection string should read INBOX.subfoldername, note the dot instead of a slash.
• To connect to a folder outside of the Inbox, the end should just be the folder name such as OtherFolder.
• To connect to a subfolder of a folder outside of the Inbox, the end should read OtherFolder/subfoldername. Note that instead of using a dot, you actually need the slash here.

cPanel high CPU usage on httpd

I had some repeated issues with multiple httpd processes sitting at 100% CPU for seemingly no good reason. A quick restart of the services would fix it for a bit, then it would come back. Running a ps axf showed that they frequently had defunct php children, but not often. For the sake of this article, let's say i was analyzing process ID 1234. I checked ls -l /proc/1234/fd and did not notice that it was any specific customer that was causing the lockups, so i knew it was not a code/site issue.

In order to dig further in, i ran strace -p 1234 to see what it was busy doing. What i saw was this:

read(517, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
read(517, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
read(517, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
read(517, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
read(517, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
read(517, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
read(517, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
read(517, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
read(517, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
read(517, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
read(517, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
read(517, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
read(517, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
read(517, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024

over... and over... and over. All other processes were doing the same thing. Based on that, i ran ls -l /proc/1234/fd/517 and found that it was pointing to /var/cpanel/secdatadir/ip.pag. Upon closer examination, this file (presumably the mod_security paging file for IP reputation or something like that, I don't really care) was 63 GB! That could explain a lot. I stopped the httpd services, killed any stragglers, and renamed this file and ip.dir as well. After starting httpd back up again, the files got rebuilt and everything is running smoothly.

DFS-R SYSVOL Replication - Performing an Authoritative Restore

Many articles have the correct procedure for this, however i have had a few of these cases come up recently so i figured it would be best to aggregate the fixes i've had to put in place.

Failure Reasons

• Unclean Shutdown - Microsoft changed it so that if DFS-R detects a dirty shutdown, it DOES NOT resume replication. This is obviously very bad if you don't regularly check your event logs.
• Last Contact Too Old - There is also a limit on how old the database can get without talking to another DFS-R peer. If you don't catch this in time then this can also prevent replication.

Helpful Commands

Please make sure you understand what you are doing before using these commands. These are tools to repair/workaround DFS-R issues, but can also introduce issues if used improperly.

• If you are getting eventlog entries indicating that you need to resume replication, it should already have the command in the log. It will look something like this (with a different volumeGuid):
  

  wmic /namespace:\\root\microsoftdfs path dfsrVolumeConfig where volumeGuid="B4A015E2-A116-11DE-89FB-806E6F6E6963" call ResumeReplication

• If your eventlog entries are giving ID 4012 saying that it's been disconnected too long, then you may want to (temporarily) raise the limit:
  

  wmic.exe /namespace:\\root\microsoftdfs path DfsrMachineConfig set MaxOfflineTimeInDays=100

• If you suffered from this due to a dirty shutdown, you can enable auto recovery to prevent future issues:
  

  wmic /namespace:\\root\microsoftdfs path dfsrmachineconfig set StopReplicationOnAutoRecovery=FALSE

Authoritative Restore Procedure

1. Manually backup your SYSVOL, typically it's in C:\WINDOWS\SYSVOL
2. Throughout this process, BE PATIENT. I have seen some of these steps take 10 minutes to take full effect. If you don't see the events right away do not try the process over again, just wait.
3. Always use an Administrator command prompt.
4. Stop the DFSR service on all domain controllers (net stop DFSR)
5. Run adsiedit.msc, connect to the Default Naming Context, and drill down to your Domain Controllers OU
6. Pick a domain controller that will be your authoritative restore source. This should be the one with the up-to-date copy of SYSVOL. Drill down to CN=Domain System Volume and then double click CN=SYSVOL Subscription.
7. Set msDSFR-Enabled to FALSE and msDFSR-Options to 1 and press OK.
8. On all other domain controllers, perform the same procedure except leave msDFSR-Options not set
9. On the primary domain controller:
   
   1. Force an AD replication (repadmin /syncall /AdP)
   2. Start DFSR (net start DFSR)
   3. Wait for event 4114 signaling that it has stopped replication
   4. Go back into adsiedit and only on the primary domain controller, set msDFSR-Enabled to TRUE
   5. Force an AD replication (repadmin /syncall /AdP)
   6. Run dfsrdiag pollad
   7. Wait for event 4602 signaling that it started replication, confirm with net share that SYSVOL and NETLOGON are being shared.
10. Now perform this task on every other domain controller:
    
    1. Start DFSR (net start DFSR)
    2. Wait for event 4114 signaling that it has stopped replication
    3. Go back into adsiedit and set msDFSR-Enabled to TRUE for this domain controller only
    4. Force an AD replication (repadmin /syncall /AdP)
    5. Run dfsrdiag pollad
    6. Wait for event 4602 signaling that it started replication, confirm with net share that SYSVOL and NETLOGON are being shared. Note that on this particular step i have had it take a bit of time, and have had to re-run the pollad command after 5-10 minutes for it to actually work.
11. You should now be fully replicated. If your issues were caused by unclean shutdowns then you might want to consider making it not stop replication on recovery

Port Forwarding on ADTRAN Netvanta Products with Redundant Uplinks

If you have to set up redundant connections on ADTRAN Netvanta products, there are a few different ways that this can be accomplished. In the event that you are using route maps to poilcy route traffic out different connections, you may run into issues with port forwarding. The issue comes up if the port forward comes in the interface that is not currently the default route. You will receive a message on the console similar to:

2015.07.12 13:30:27 FIREWALL id=firewall time="2015-07-12 13:30:27" fw=FW1 pri=1 proto=3389/tcp src=1.2.3.4 dst=4.5.6.7 msg="Spoofing detected, dropping packet Src 53668 Dst 3389 from ISP2 policy-class on interface vlan 10" agent=AdFirewall

This is because the unit does a URPF (Unicast Reverse Path Forwarding) check and sees that the packet came in on what it thinks is the wrong interface. On equipment from most other vendors, you have to disable this at a global level. For ADTRAN, you actually disable it at the policy-class level, which is counter-intuitive but gives you more granular control. To disable you simply issue this command on the policy class that's receiving the traffic:

FW1(config)#no ip policy-class ISP2 rpf-check

Where ISP2 is your policy class name. You will need to disable this on any policy class that may receive incoming traffic where that interface is not the default route.

Exchange 2010 OWA Search Broken, 0x80041606

Customer with two Exchange 2010 servers, one Hub/CAS and one Mailbox server. Users started receiving an error when searching in OWA: "The action couldn’t be completed. Please try again". I was told that this happened after a power failure (and bad backup batteries), so it sounded like something was corrupted.

The event viewer was also logging an error:

Content Indexing function 'CISearch::EcGetRowsetAndAccessor' received an unusual and unexpected error code from MSSearch.
Mailbox Database: Operations Database 1 
Error Code: 0x80041606

Rebuild the Content Indexes

This was my first step, i took the easy route of stopping the Microsoft Exchange Search Indexer service and deleting the catalogs. These folders are located in every mailbox database folder and start with CatalogData. It is perfectly safe to delete these and then start the service again. After starting the service you will see event viewer entries indicating that it is rebuilding the indexes. You can also run the PowerShell command Get-MailboxDatabaseCopyStatus to see the status. The ContentIndexState column will indicate whether it's Crawling or Healthy.

Repair Symlinks

I saw plenty of articles indicating that the symlinks needed to be repaired, this did nothing.

Reinstall Filter Pack

Ultimately the solution was to reinstall the Microsoft Office Filter Pack on the Mailbox server. This did require a reboot (and yes, I tested it prior to rebooting). Afterwards the search was working as expected.