[thelist] how can I establish a more reliable connection to my webserver?

[David Kaufman]

Joel D Canfield <joel at streamliine.com> wrote:
> All my client sites are hosted at Crystal Tech, who are gradually
> slipping in my esteem ... tracert and ping confirm that the issue is
> between here and Crystal Tech, not on their end. They have no useful
> suggestions to offer. I understand that Sprint's network reliability
> isn't their problem, but I was hoping for *anything* helpful.

pair.com has a network newsgroup where customers all over the U.S and
Europe post their traceroutes any time their connections seems dodgy.
Pair's engineers also post when one of their numerous Big Fat Pipe's is
getting clogged.  If they didn't already know, they usually advise the
customer to log into their pair server and perform the reverse 
traceroute
back to the customer's own IP.  Often this will take a different path 
and
indicate on who's network the fault really lies, the customer's ISP or
one pair's upstream providers.  In either case, if it's bad, and
especially if it's prolonged, they will usually open a ticket with the
problem network operations center because, not surprisingly, when your
ISP (or their ISP) gets a call from a big datacenter that traffic to or
from them is for shit, their response is much better than when one
of their own lowly customers complains...

> Wouldn't a hosting service physically closer to me be less subject to
> the vagaries of the internet?

Not necessarily.  Bad network engineers can make even a small local
network suck.  Take my cable company for instance...

> ...Since I have a static IP on my
> equipment, and have email and websites hosted here, I really don't
> want to get a new IP address, which was the only suggestion they had
> for prompting a new network route.

That's not a helpful hosting provider.  Sounds like you were talking to
a 1st-level tech, not a network engineer.  Even if the problem is not
"their fault", they can assist you in troubleshooting it (like doing the
reverse traceroute or compare your traceroutes with those from other
ISP's) and, like pair, even if the problem is not in their network or
under their direct control, they should (if they cared) help you
identify who to call, and escalate the issue with them.

Since you are their customer and you cannot access their network, their
NOC staff should be investigating because unless you're they're only
customers then most likely others are affected as well.

> For those of you responsible for multiple sites, how's your ftp
> reliability? Have you successfully overcome this issue in the past?
> Am I missing something?
>
> It took me nearly three hours to complete a three minute task
> yesterday. It's killing me.

These types of issues are incredibly frustrating, I know.  But even when
the answer is "we called them and they're working on it" it's a helluva
lot better to hear than "sorry. not my problem".  Call them back and ask
them to ask their boss how many days must pass while you and all your
clients can't use their service before they will decide that it is their
problem.

Or just switch to pair :-)

Here's a few Thank-God-Im-At-Pair moments that illustrate why:

http://pair.com/support/system_notices.html

[Apr 23, 2007, 9:14 AM] Network Problems
At approximately 8:45 AM Eastern, our Gigabit Ethernet circuit with
Level 3 went out of service. Traffic has shifted to our other providers
with no interruption. Level 3 has acknowledged the problem and is
working to resolve it. We will post another notice once the line has
returned to service.

[Apr 23, 2007, 10:23 AM] Network Problems Resolved
Our Gigabit Ethernet circuit from Level 3 has returned to service as of
approximately 10:05 AM Eastern. All traffic was carried by our other
providers during this outage.


[Apr 19, 2007, 6:47 PM] Network Problems
We are currently receiving widespread reports of difficulty reaching
sites hosted on our network from the Comcast network. We are attempting
to address the issue with Comcast, as there appears to be no problem on
this end. This is at least the third recurrence of this type of problem
this year; we remain hopeful that Comcast can resolve the difficulty.


Exploding Utility Poles:

[Mar 13, 2007, 6:09 PM] Utility Power Outage
Due to an apparent transformer failure a few blocks from our facility,
we have been operating without utility power since approximately 3:50 PM
Eastern time. Our generators and UPSes are maintaining power and we
expect no trouble from this incident. The estimated time to repair by
the power company is 8:00 PM Eastern time.

[Mar 13, 2007, 7:54 PM] Utility Power Restored
Utility power has been restored to our facility as of 7:45 PM Eastern.
Our power needs were handled seamlessly by our generators and UPSes
during this incident.


One of a couple of denial of services lately:

[Mar 12, 2007, 1:04 AM] Network Problems
At approximately 12:36 am, our network started to experience problems.
We are currently looking into the matter, and will post a notice when
more details are available. We do apologize for any inconvenience that
this may cause you.

[Mar 12, 2007, 1:26 AM] Network Problems Resolved
Beginning around 12:35 AM Eastern, we experienced an inbound
denial-of-service attack. Approximately one third of our servers were
affected. The attack was traced and blocked at around 1:05 AM, and
traffic has since returned to normal levels.


Another Big Fat Broken Pipe:

[Mar 4, 2007, 12:43 PM] Network Problems
At approximately 12:30 PM Eastern time, our Gigabit Ethernet circuit
with Global Crossing went out of service. Traffic has shifted to our
other providers with no interruption. Our vendors are working to resolve
this problem; we will post another notice when the line has returned to
service.

[Mar 4, 2007, 1:55 PM] Network Problems Resolved
Our Gigabit Ethernet circuit from Global Crossing has been returned to
service as of approximately 1:35 PM Eastern and traffic is flowing
normally. We continue to work with our vendors to isolate the cause of
the outage.


Finally, here's a post from pair in response to a customer complaint on 
the
newsgroup that included both forward and reverse traceroutes:

  This is a problem at hop 9 and/or 10 on your return path, inside
  the sbcglobal network.  Presumably they have resolved the problem?

  Remember that anytime you see a sudden jump at the very end of an
  inbound traceroute, it's either a saturated circuit here (not in this
  case, and highly unlikely in general), or a problem on the reverse
  path.

  I believe I saw a post on NANOG yesterday about someone who is
  seeing problems with the peering between SBC Global and Global
  Crossing.  So you're not alone.  I recommend complaining to your
  ISP; I will complain to Global Crossing.  I'm sure it will be
  resolved sooner or later, hopefully sooner.


If your host doesn't offer this level of transparency and proactive
network administration you really should ask yourself what it is
you pay them for!

-dave, shameless pair promoter and customer of (about to be) 10 years