This latency is caused by ____ ?

Starting a few days ago, nothing outside my house on the Net has been closer than about 300 miliseconds. Even UCSB.edu, which I can see from here, is usually no more than 30 ms away on a ping test. Here’s the latest:

PING ucsb.edu (128.111.24.40): 56 data bytes
64 bytes from 128.111.24.40: icmp_seq=0 ttl=52 time=357.023 ms
64 bytes from 128.111.24.40: icmp_seq=1 ttl=52 time=369.475 ms
64 bytes from 128.111.24.40: icmp_seq=2 ttl=52 time=389.372 ms
64 bytes from 128.111.24.40: icmp_seq=3 ttl=52 time=414.025 ms
64 bytes from 128.111.24.40: icmp_seq=4 ttl=52 time=428.384 ms
64 bytes from 128.111.24.40: icmp_seq=5 ttl=52 time=28.120 ms
64 bytes from 128.111.24.40: icmp_seq=6 ttl=52 time=164.643 ms
64 bytes from 128.111.24.40: icmp_seq=7 ttl=52 time=292.241 ms
64 bytes from 128.111.24.40: icmp_seq=8 ttl=52 time=332.866 ms
64 bytes from 128.111.24.40: icmp_seq=9 ttl=52 time=330.573 ms
64 bytes from 128.111.24.40: icmp_seq=10 ttl=52 time=369.385 ms
64 bytes from 128.111.24.40: icmp_seq=11 ttl=52 time=375.593 ms
64 bytes from 128.111.24.40: icmp_seq=12 ttl=52 time=405.028 ms
64 bytes from 128.111.24.40: icmp_seq=13 ttl=52 time=413.990 ms
64 bytes from 128.111.24.40: icmp_seq=14 ttl=52 time=437.124 ms

It’s been this way for days. I can’t get a human at Cox, our carrier, so I thought I’d ask the tech folks among ya’ll for a little diagnostic help.

Here is a traceroute:

traceroute to ucsb.edu (128.111.24.40), 64 hops max, 40 byte packets
1  ip68-6-68-81.sb.sd.cox.net (68.6.68.81)  5.828 ms  3.061 ms  2.840 ms
2  ip68-6-68-1.sb.sd.cox.net (68.6.68.1)  324.824 ms  352.686 ms  358.682 ms
3  68.6.13.121 (68.6.13.121)  359.635 ms  369.743 ms  372.376 ms
4  68.6.13.133 (68.6.13.133)  386.039 ms  389.809 ms  415.532 ms
5  paltbbrj01-ge600.0.r2.pt.cox.net (68.1.2.126)  430.554 ms  447.079 ms  423.461 ms
6  te4-1–4032.tr01-lsanca01.transitrail.ne… (137.164.129.15)  464.229 ms  453.908 ms  423.090 ms
7  calren46-cust.lsanca01.transitrail.net (137.164.131.246)  206.217 ms  251.298 ms  261.263 ms
8  dc-lax-core1–lax-peer1-ge.cenic.net (137.164.46.117)  264.824 ms  284.859 ms  285.808 ms
9  dc-lax-agg2–lax-core1-ge.cenic.net (137.164.46.110)  289.834 ms  307.450 ms  313.997 ms
10  dc-ucsb–dc-lax-dc2.cenic.net (137.164.23.3)  323.183 ms  331.668 ms  345.606 ms
11  r2–r1–1.commserv.ucsb.edu (128.111.252.169)  340.756 ms  377.695 ms  375.946 ms
12  128.111.4.234 (128.111.4.234)  365.500 ms  397.311 ms  393.919 ms

Looks to me like the problem shows up at the second hop. Any guesses as to what that is? Yes, I’ve rebooted the cable modem, many times. No difference.

I’m talking now over my Sprint data card. EVDO over the cell system. Latencies run around 70-90 ms. So the problem is clearly one with Cox, methinks.

I’m only home from the Live Oak Festival for a shower, so I’ll leaving again in a few minutes and won’t get around to dealing with this (or anything) until tomorrow. Just wanted to get the question out there to the LazyWeb in the meantime. If the problem really is Cox’s, I’d like to know what to tell them when I go down to their office. No use calling on the phone. Too many robots.

Happy solstice, everybody. And thanks!

Tags: , , ,

23 comments

  1. Richard Bennett’s avatar

    Most likely hop 2 is a Cox gateway node between your neighborhood and the rest of the Cox regional net. They have to fix it as it’s most likely messed up somehow.

  2. Richard Bennett’s avatar

    To clarify a bit, 68.6.68.81 is probably your cable modem, and 68.6.68.1 is first IP thing that your cable modem talks to inside the Cox network, from the address *.1 we can assume it’s some kind of gateway. It’s possible that your cable modem is broken and is getting lot of errors taking to the gateway, and it’s also possible the gateway is either broken or just severely overloaded. If the condition persists, that points to broken. So you’ll need to call Cox and harass their customer service.

  3. Shaun Walbridge’s avatar

    Richard’s comments above make good sense, it sounds as if it’s likely issues with the Cox gateway. Do you perhaps share service with a large number of students? This is the first week of summer, and that may be putting undue strain on your circuit (just a guess).

    I’m down at East Beach, and pings to UCSB look fine, with ~22ms response times. If you’re on OS X or Linux, try out mtr, which provides advanced statistics for a traceroute, perhaps that’ll give you a better idea of the nature of the problem

  4. rob friedman’s avatar

    Just to quickly eliminate the cable modem
    go to the HTTPd which is on every cablemodem (just about)
    http://192.168.100.1/
    and login
    Find the “Status” page
    It should show “OK”.

    Are you using a router? or Internet Sharing on a computer?

  5. Mike Warot’s avatar

    Here’s what I tell all of the people I support:

    Unplug the power from both your cable modem and router
    Wait at least 2 full minutes before
    plug the power back in and wait for everything to come back

    Sometimes the upstream configuration changes, and the cable modems don’t propagate the changes properly downstream… the two minutes gives the cable company a chance to notice you are gone, give you a new DHCP address, which forces everything else to start working.

  6. Russell Nelson’s avatar

    I vote for POTS modem backup.

    :)

  7. Doc Searls’s avatar

    Thanks, everybody.

    Rob, I’ve bypassed the router/hub/switch/whatever (forget what Netgear calls it, and I’m in the woods right now, literally), and get the same results.

    Mike, I’ve tried that, several times. It worked one time, then went to hell again. Makes me believe that Richard is right, and that the problem is most likely at a neighborhood connection.

    I’ll try it one more time after I get back tomorrow sometime; then if it’s still bad I’ll go down to the Cox office to take it up with a human. Their call center is robots all the way down, near as I can tell.

  8. Jan Searls’s avatar

    Remember how often you’ve helped me check the slowness here (Time Warner in NC)? I’ve also done Mike’s two-minute shut-down drill, which would help – for a while, but then everything reverted to sludge, again.

    It took a severe storm and half a tree falling on all the cables to the house to get TW to actually come out instead of checking from their end. Once I had the repair guy here restringing, I snagged him to check my speed. Although everything checked out, and the modem and router were fine, when he saw how slow I was he gave me a new modem.

    I’ve been greased lightening ever since. He said that any modem over 18 months old may check out as fully functioning, but is a very likely suspect in slowness. Since my puppy had been sitting there, blinking away, for 4 years ….

    So if you’re going to walk in to Cox today, take your modem and demand a replacement. Can’t hurt, might help.

    In a similar vein, when I awoke to internet and TV out last week, I called to report same. The robot system made me opt to report internet or TV, but not both. Human – eventually – had the same script and even insisted I do Mike’s drill – wonder what they thought that would do to help the TV? Next step was to send out a tech, but the first appointment they could give me was two days later.

    However, after a few hours of switching around my schedule to accommodate theirs, TW called and asked me to check my signals, and sure enough, everything was up and running. The human said it was a local outage. After some prodding questions, he admitted that their system requires at least 3 reports from the area before they look for an outage. He suggested next time I get at least the three reports in, even if I have to call once from my home phone, once from my cell, then “call for an elderly neighbor”

    Wouldn’t it be easier to actually listen to the first report?

  9. Dan Keyser’s avatar

    Cox is an iniquitous word in my way of thinking. Tho I have just recently joined this forum, I have been a SBMUG member in the past, but only have I recently, the past week, I subscribed to this forum.

    I came to SB back in the days before COX, and it was eons ago in my past, I can remember door-to-door sales men of COX trying to get me to buy their great service for only…. Which was less than $10 a month and I could get rid of those ugly rabbit ears.

    I do think that 99% of all cable companies run monopolies just like all utilities. But most utilities are regulated, not cable companies. And COX takes advantage of the lack of competition and regulation. Do you really think it costs COX 1/10th of what they bill their users? Look what competition has done to the telephone monopoly, Verizon! With Internet phone companies like Vonage, and the cell phone industry, gee whiz, Verizon had to be competitive or loose business.

    So, I went to the Loretta Plaza Cox office, yea the one with an ARMED SECURITY GUARD now standing at their door, and cancelled my cable TV.

    I had problems with my Internet connection in the past and sometimes it would take days for a cable repair man to show up at my door, but not then, he was there the next day, climbing up the pole, putting the appropriate filter making sure those electrons that send me cable TV would no longer reach my house.

    I keep track of the bandwidth that COX “gives” their internet users. And it has gone down since last November. I am sure there are other “smart” internet users that have noticed that.

    I wonder how long it’s going to be before big brother COX starts blocking companies like Netflix and other services like Vonage that provide better and less expensive services?

    Now I am putting my legal hat on. People, the wires that COX strings over all of Santa Barbara, ultimately becomes public domain, and they only have the right of ingress and egress over our property to maintain those wires. That’s it.

  10. Doc Searls’s avatar

    Dan, I’m on the phone with Cox right now. They re-provisioned my modem, and they are totally flummoxed. Or the person on the phone is.

    Here is what we see now:

    - High ping times to everywhere, but the shortest are to ucsd.edu in San Diego. They start with about 98ms, and then go up to >250ms, with occasional drops to values as low as 17ms. That happens about once every 50 pings. Same goes with pings to Google, Yahoo, Harvard and UCSB.

    - Traceroutes are now high to start with, then go down. Example:

    1 ip68-6-52-1 (68.6.52.1) 225.620 ms 250.442 ms 263.926 ms
    2 68.6.13.105 (68.6.13.105) 255.601 ms 259.117 ms 124.211 ms
    3 68.6.13.129 (68.6.13.129) 134.295 ms 143.251 ms 148.755 ms
    4  langbbr01-as0.r2.la.cox.net (68.1.0.230) 151.489 ms 156.048 ms 161.745 ms
    5  calren46-cust.lsanca01.transitrail.net (137.164.131.246) 204.833 ms 185.411 ms 192.036 ms
    6  dc-lax-core1–lax-peer1-ge.cenic.net (137.164.46.117) 202.864 ms 223.297 ms 228.114 ms
    7  dc-lax-agg2–lax-core1-ge.cenic.net (137.164.46.110) 229.751 ms 249.258 ms 251.020 ms
    8  dc-ucsb–dc-lax-dc2.cenic.net (137.164.23.3) 244.641 ms 144.884 ms 110.795 ms
    9  r2–r1–1.commserv.ucsb.edu (128.111.252.169) 17.579 ms 18.927 ms 17.427 ms
    10 128.111.4.234 (128.111.4.234) 18.873 ms 18.716 ms 18.260 ms

    That first ping is my cable modem, reprovisioned with a new IP address. (Which you can see by comparing with the posted one above.)

    I’m on hold now, waiting for Tier 2, or whatever they call it at Cox.

    Okay, just got off the phone with a guy named Michael, who checked from his end and found that ping times were terrible. So they need to send a tech out.

    On Wednesday afternoon. That’s two days from now.

    Meanwhile I at least have my Sprint datacard.

  11. Russell Nelson’s avatar

    They’re gonna come out and re-crimp every connector before they do anything else. Check to make sure they’re tight, and you just might fix the problem first. Wrench-tight, not just finger-tight.

  12. Doc Searls’s avatar

    Russ, the connectors I can reach — at the patch panel in the wiring closet, all are wrench-tight. As for what’s on the poles, it’s up to Cox. I guess we’ll find out tomorrow afternoon.

  13. Chip’s avatar

    Massive solar storms knocked out their systems at Cox … not

    http://spaceweather.com/

  14. Mike Warot’s avatar

    Your latency is caused by one of the three unsolved tipping points of the internet…. the lack of mesh networking. One node should not kill the internet, ever.

  15. Chip’s avatar

    Spent yesterday morning in meeting on broadband and Federal Funding for rural access

    Very interesting stuff pending – mix of fiber and wifi with totally open (IP structure) system, nice mesh and redundancy options.

  16. Doc Searls’s avatar

    I’d like to hear more, Chip. The key will be keeping service competitors in business and free to innovate, while also preventing monopolies and duopolies from coasting on their captives. I worry that ther Obama administration, to prevent the latter, may foreclose on the former.

    I also think the Net needs to be built out increasingly from the edges in, rather than the centers out. And that we stay open to the inevitable improvements in the Net itself.

    Absent a generally agreed-upon understanding of what the Net is, all the above will be hard to pull off.

  17. Rick DeNatale’s avatar

    I’m interested in this thread myself.

    For the past week or so, I’ve been seeing VERY intermittent problems similar to yours Doc.

    Several times a day, ping times to say google.com go up to 4000ms or higher, traceroutes never complete. Then things go back to normal. It’s so intermittent that I haven’t been able to rule out my router, but it does SOMETIMES happen with a direct laptop to model connection.

    In my case this is a DSL connection. I’m in Wake Forest, NC (Raleigh neighbor, not Winston-Salem which is where Wake Forest University moved in the 1950s), and my isp/telco is WindStream.

  18. Doc Searls’s avatar

    Hi Rick.

    I’m familiar with Wake Forest. My uncle, a big Wake fan, went to Wake when it was still in Wake.

    Has there been a speed increase lately from your service? I think these problems show up when the carrier upgrades something and it hoses downstream devices — including their own gateways and such. I think that’s the case here. Certainly my own old gear in the house didn’t help.

  19. Mac Bakewell’s avatar

    Hi Doc,

    Any resolution to this yet? I’m on the Mesa in Santa Barbara where there are a lot of City College rentals (i.e. lots of online gamers and video streamers).

    Because of the overload, in June of 08 Cox added a new node to this neighborhood and, for a while, things were very much better. The problem now is not speed but latency. Once locked into a site download speeds exceed 20 Mbps, but every so often, like at least twice an hour, both computers here go into a trance when trying to initiate contact with a new URL.

    My initial thought was DNS, so I flushed the DNS caches and switched DNS servers (from from 4.2.2.3 and 4.2.2.2 to Open DNS 208.67.220.220 and 208.67.222.222). That didn’t help so yesterday I bought a new modem from Cox (a Motorola SB5101 to replace the suspect SB5120).

    My hopes were up for the first hour, which went by without a hiccup, but then, and again all day today, we’re back to the same old tedious business. Sigh. I guess I’ll be calling Cox on Tuesday but meanwhile I’d be grateful to hear how this last thread worked out.

  20. Doc Searls’s avatar

    We worked it out by talking with higher-level techs than you’ll reach just by making calls. When you do make the call, navigate past the robots to a human, and then demand speaking to an expert who can run down the problem.

    In our case the problem as both a crimped line above our house and something with a gateway farther back within Cox. We found the problem mostly by running traceroutes.

    If you need more help, contact me directly at my firstname AT lastname dot com.

Comments are now closed.