Unicast NLB cluster generates large amount of broadcast traffic

When you set up a unicast Network Load Balancing (NLB) cluster, a large amount of broadcast network traffic will be generated on any switch to which a cluster node is connected. This is normal behavior for a unicast NLB cluster. You may not even notice this traffic unless you are running a packet capture from a machine connected to the same switch as the cluster nodes.

Normally, a switch builds a MAC address table by learning what ports a MAC address is communicating on. This automatic learning process only works if a given MAC address is unique across all the ports on a switch.

Because nodes in a unicast NLB cluster all share a common cluster MAC address, the network switch to which they are connected cannot learn which port the MAC address is tied to. Therefore it is never able to add the cluster MAC to its table. As a result, all traffic going to the cluster MAC is always broadcast out all switch ports.

This may or may not be a problem, depending on the amount of traffic going to your cluster and the amount of other traffic which is already being handled by the network switch. If it is a problem, there are several ways to resolve it.

1. Switch to a multicast or multicast IGMP NLB cluster. You will need to make sure your switches support multicast for this to work. Cisco switches with a relatively recent IOS should have this capability, but you should check first, to be sure.

2. Move the unicast NLB cluster nodes to a separate switch, where they are the only connected devices.

3. Set up a separate VLAN or network (dedicated router/firewall interface) just for the cluster, which will contain the broadcast traffic.

4. Add static MAC table entries on your switch to tell it which ports are being used by the cluster nodes. This way, traffic going to the cluster nodes would only be sent out the applicable ports. Each time you add another cluster node, you would also need to add an entry to the switch MAC table.

Option 4 is the easiest, and one that I have used in production on a small cluster.

All of these options will work; it’s really just your preference as to which one you use. As long as you document it, you’ll be in good shape in any case, right?

Here are some useful links regarding NLB:

http://technet.microsoft.com/en-us/library/bb742455.aspx

http://technet.microsoft.com/en-us/library/cc786562%28WS.10%29.aspx

http://www.cisco.com/en/US/products/hw/switches/ps708/products_configuration_example09186a0080a07203.shtml

How to use nslookup to test DNS servers

NSLookup is a very useful tool for testing specific DNS servers.  For instance, if you are having DNS resolution issues or if you are transferring your DNS records to different DNS servers, you can use nslookup to test the authoritative name servers for your domain.  This will enable you to ensure that each of the authoritative servers for your domain are serving all the domain DNS records correctly.  There is a command with a similar function in Linux, called ‘dig‘, however I will not cover use of that command in this post.

If you are planning on transferring your DNS records to different name servers, I would recommend having the DNS records created on the new servers first.  Then, you can use nslookup to verify that the new records are in place before you change the authoritative DNS servers for your domain in your registrar account.

NSLookup has been included in every recent version of Microsoft Windows.  It can be accessed by simply opening up a command prompt and typing the ‘nslookup’ command:

Now that you are running the nslookup program, you can select the server you want to use by simply typing “server <server IP address>”.  This sets the focus on the server that you want to test.  Any subsequent queries for DNS records will use this server until you select a different server.  For example:

By default, nslookup is set to query for ‘A’ records.  So, if you want find the A record for www.binarywar.com, you simply type that in, like so:

If you want to query for the MX records for a domain, you will first need to change the query type.  That is done using “set q=mx”, as shown here:

(you can also use set querytype=MX or set type=MX)

Then, type in the domain for which you would like to see the MX records:


Here are the various record types you can use with the “set q=” command:

  • A
  • ANY
  • CNAME
  • GID
  • HINFO
  • MG
  • MINFO
  • MR
  • MX
  • NS
  • PTR
  • SOA
  • TXT
  • UID
  • UINFO
  • WKS

You can find a description of these record types on this page:

http://technet.microsoft.com/en-us/library/bb490738.aspx

In addition, here is the command reference for the nslookup utility:

http://technet.microsoft.com/en-us/library/bb490721.aspx

Configure Resource Mailbox to AutoAccept in Exchange 2007

To configure a resource mailbox to automatically accept an appointment in Exchange 2007, you must use the Exchange management shell.

If there is already an appointment that overlaps at all with the appointment you are trying to schedule, you will get back a ‘Declined’ receipt from the resource mailbox.

Assuming you already have the mailbox created, and it is called “ResourceMailboxName”, here is the command you should run:

Set-MailboxCalendarSettings ResourceMailboxName -AutomateProcessing:AutoAccept

See these pages for more info on resource mailboxes:

http://msexchangeteam.com/archive/2009/02/26/450776.aspx

http://technet.microsoft.com/en-us/library/bb123495.aspx

Grant access to resource mailbox in Exchange 2007

To grant access to a resource mailbox, you must use the Exchange management shell.

Assuming you already have the mailbox created, and it is called “ResourceMailboxName”, here is the command you should run:

Add-MailboxPermission -AccessRights FullAccess -Identity ResourceMailboxName -User Username

If you need to grant a different level of permission, use a different option with the ‘AccessRights’ switch.  The other options are:

  • ChangePermission
  • ChangeOwner
  • DeleteItem
  • ExternalAccount
  • ReadPermission
  • SendAs

See these pages for more information on resource mailboxes:

http://msexchangeteam.com/archive/2009/02/26/450776.aspx

http://technet.microsoft.com/en-us/library/aa995916.aspx

Free/Busy info not available outside the network

If you are having problems with free/busy information in Outlook, it is most likely due to misconfiguration of the Exchange 2007 Autodiscover service.

The Autodiscover service provides info to the Availability service, such as the addresses (internal and external) that Outlook 2007 clients should use to connect to Exchange.  More info here.

I would recommend that you read this post at the Microsoft Exchange Team blog and the referenced whitepaper at the top.

Most likely, you do not have the internal/external addresses configured correctly in Exchange.  Double-check these.  In addition, with Outlook open, you can hold the ctrl key and right-click on the Outlook icon in your system tray to get the “Test Email Autoconfiguration” option.  Run this to see how Outlook is trying to connect to your Exchange server.  You may notice that Outlook first tries to connect to “domainname.com”, then to “autodiscover.domainname.com”.  These are the default addresses that Outlook tries.  You may need to create a CNAME for “autodiscover.domain.com” which points to your Exchange proxy address.  That is the address that you have in the Exchange proxy connection settings in Outlook, and is likely the same as your OWA address.

Once you have the Autodiscover/Availability services configured correctly, you will likely find that your free/busy info problems have been resolved.

Network Policy Server and Cisco RADIUS Authentication

Setting up RADIUS authentication between Cisco devices and Network Policy Server (NPS) in Windows Server 2008 is a bit different than in previous versions of Windows.

Here is a technet page with lots of good info on NPS:

http://technet.microsoft.com/en-us/network/bb629414.aspx

For now, I am just going to list the instructions needed to get up and going with NPS to allow your server to act as an authentication point for your Cisco switches/routers. This may work with other devices that can use radius authentication, but I have not tested it. YMMV.

1. Install the Network Policy Server service. It is a component under ‘Network Policy and Access Services’.

2. Open the Network Policy Server console from Administrative Tools.

3. Create a new radius client for the Cisco device. The process for this is very similar to the process in Server 2000/2003. You just need the device IP, choose the “radius standard” type, and make up a shared secret.

4. “Register server in Active Directory” by right-clicking on the “NPS (local)” item in the console. This will allow NPS to query AD when an authentication request comes in.

5.  Next, create a “Connection Request Policy”.  This is the step that is new to the process, and was not required before Server 2008.  Before, this was integrated into the remote access policy, as it was previously called.  The connection request policy doesn’t need to be anything complex.  The first step is to set the network access server type to “Unspecified”.

Next, add at least one condition to the policy.  I usually use the “day and time restrictions”, and then set it to ‘permitted’ 24×7.  Obviously, the condition(s) you choose should conform to your company’s security policy, so you may need something different here.

Finally, On the Settings tab, under Authentication, choose the radio button for “Authenticate requests on this server”.

6.  Create a Network Policy, formerly known as a remote access policy in previous versions of Windows Server.  On the Overview tab, configure the policy to use the network access server type of “Unspecified”.  In addition, set the access permission setting to “Grant Access”.

On the Conditions tab, add at least one condition.  Typically, this will be the Windows Group that is allowed to log in to the network devices.  As I said before, you may need to use different conditions than I show here due to your company security policy.

On the Constraints tab, the only change you should need to make is to enable the authentication method of “Unencrypted authentication (PAP, SPAP)”

Lastly, on the Settings tab, under Encryption, make sure that the “No Encryption” option is enabled.

7.  Point your network device(s) at this server for authentication.  The method for doing this varies depending on the make and model of your device.  With recent IOS images on Cisco switches, the commands will look something like this.

aaa new-model

aaa session-id common

aaa authentication login default group radius local

radius-server host 10.0.0.1 auth-port 1812 acct-port 1813 key putyoursecretkeyhere

8.  Finally, test it!

Top 10 Causes of Email Flow Problems

There are many, many things that can cause email issues.  Here are some of the most common issues I have encountered over the years. In no particular order…

1. DNS changes. Someone may have changed the MX records, or changed the authoritative DNS servers for your domain.

2. Firewall.  If the access or NAT rules for the mail server were changed, mail flow may be disrupted.  Make sure you have a system in place for tracking configuration changes on your firewall(s), so that you can look at the most recent change and decide if it created a problem with mail delivery.

3. Spam Filter.  After the firewall, the spam filter is probably the next hop for your mail after it passes through your firewall.  Whether it is a standalone device or software installed on your mail server, check it to see if your mail is getting hung up here.

4. Services/Processes not running on mail server.  If you have Microsoft Exchange, check the SMTP or Microsoft Exchange Transport services.

5. DNS resolution problems on mail server (for outbound mail).  If a server can’t resolve MX records, it’s not going to be able to deliver mail.  If you are running Microsoft Exchange, the primary and secondary DNS servers configured in the TCP/IP settings are likely to be two of your domain controllers.  The easiest way to see if you are having a DNS issue on the mail server is to see if you can browse the Internet from it.

6. Incorrect or misspelled email aliases or domain names.  This is normally a user mistake when sending a message.  However, sometimes it can be due to a misconfigured contact or distribution group in Active Directory.

7. Mailbox full, either on the sender or receiver side.  This will usually cause a bounceback message.  If the user with the full mailbox is on your server, you will need to have some kind of monitoring set up to alert you to the problem.

8. Smart host not responding, credentials incorrect, or otherwise misconfigured. Possible ‘daily limit’ reached on number of relays.

9. Reverse DNS issues.  See my previous post to make sure your reverse DNS is configured correctly.

10. Blocked by ISP.  Clearwire does not allow you to host a mail server by default. Others, such as AT&T, may blackhole your WAN IP if they detect virus-like or spambot activity coming from your address.  You should call your ISP to confirm if you suspect this is the problem, assuming they haven’t notified you already of the block.

Test Your DNS with Namebench

Namebench is a great utility to test your DNS performance against other publicly available DNS servers.  It will run a multitude of DNS-related tests and give you back a recommended DNS configuration for your computer, based on the connection you are using when you run the test.  The tool was created by a Google developer, but is not biased towards Google Public DNS.  In my case, it suggested I use ‘The Planet” (company in Dallas) as my primary DNS provider.

The original project page is here:

Namebench

It takes about 10 minutes to run, and can definitely be worth the trouble if you are having slow Internet browsing issues.

Troubleshooting Email Flow (Outbound)

Previously, I posted about troubleshooting inbound mail flow.  However, just as often (possibly more), you will be troubleshooting outbound mail flow.  Hopefully, this post will help with that.  As with inbound mail, there are many things which can cause problems for mail delivery going FROM one of your users to someone outside your organization.  You should not take the word of a non-technical person who is reporting the problem to you as gospel.  Verify the scope of the problem and ask questions such as these:

  • What is the scope of the problem?
  • How many people are affected?  Almost as importantly, is there anyone who seems UNaffected and can still receive mail?
  • Are users able to send mail between each other inside the company but not send to people outside?
  • When did it start?
  • Are there any error messages or common symptoms that the affected users are seeing in Outlook or other mail client?
  • Are users getting any kind of bounceback message when trying to send email out?  See if you can have a copy of one of these bouncebacks forwarded to you if at all possible.
  • What was changed?  Besides the obvious, that it was working and is now not, something may have been changed.  Ask anyone whom you know may have been working on the affected mail server or domain name within the last day or so.   Firewall rules?  Spam filtering device or spam filtering software on the server? etc.  A lot of the time, finding out what was changed will point you toward the cause of your problem.
  1. Check the outbound queue(s) on the mail server. If your company is having trouble sending out mail, there are probably messages piling up in an outbound queue.  If you find messages in the queue(s), are they addressed to many different domains or just one or two?  If just one, then there may just be a problem with the destination mail server.
  2. Send messages using webmail (e.g. outlook web access).  Send to several different domains (e.g. your personal Gmail, Yahoo, or other addresses) to see if they go through.
  3. Check services/processes.  Are the Microsoft Exchange services running, such as the Transport and/or SMTP services?  Or if using Sendmail or Postfix, are the processes running?  Sometimes, even if they are running, restarting the services/processes that deal with sending mail can correct a problem.
  4. Check logs in Windows/Linux for errors. For Exchange server itself, any diagnostically useful errors will be in the application log.  However, keep in mind that Exchange (and mail flow in general) relies heavily on DNS functioning properly.  So, you may have many errors that point to an Exchange problem, but it may just be a symptom of an underlying DNS or Active Directory issue.  Check the DNS and Directory Service logs as well.
  5. Check the firewall. Is it blocking outbound SMTP connections from your server IP address.  Use telnet to ensure that your mail server can connect outbound to other mail servers outside of your network on port 25.
  6. Check the remote firewall or spam filtering device. The IP address of your mail server may be blocked or blacklisted.  You have a limited number of ways to determine if this is the problem.   Test by initiating a telnet session to the destination server on port 25.  If there is no response, try the same thing from a computer on a different Internet connection, such as your home computer.  Your only other option is to get in touch with a network administrator for the destination server and see if he or she can help.
  7. Check DNS. Your mail server may simply be having trouble resolving DNS names to be able to deliver mail.  Look up the MX records for one of the domains to which you are having trouble sending mail.  Then, try to ping the DNS name for one of the MX records that was returned in the lookup.  Even if it doesn’t respond to ping (your firewall may block ping traffic), does it resolve to an IP address?
  8. Check your reverse DNS. Going back to the outbound queues on the mail server.  If there are many messages queued up, destined for various domain names, it could be a reverse DNS issue on your end.  See my previous posting about reverse PTR troubleshooting.
  9. Check your outbound spam filter, if you have one.  Some companies do, although it is rare.  Beyond your mail server queue, there is another queue on the spam filter that may be filling up.

There are many moving parts when it comes to mail delivery.  Answers to the pre-troubleshooting questions (top of this post) will likely help you arrive at a resolution more quickly than if you start from scratch.

Good luck!