Building The Net

Millions of LinkedIn, E-Harmony, and Last.fm password hashes posted on message board.

Well, if we have learned anything from the past, if it can go wrong, it will… Although this has been downplayed by the companies involved,  there is no doubt in my mind that many people will be effected by this compromise. Once again, public networking sites storing user data on the internet, have failed to protect that data, and worse, have tried to hide the importance of this compromise. This is sad, but certainly is nothing new. We can take some comfort in the fact that these companies at least used sha1 hashing when storing the password data. Thing is, we don’t know what other information was compromised besides the passwords.

Read more �

My God, can it all be the same?

Seems like most of my job now a days is looking at large systems and isolating problem areas. Things like performance problems, data corruption, or even failure analysis. Many of these systems have several independently managed processes, all tied together in a single forward facing application. Over the years, I’ve developed some methods of approaching system failures and problems that gives me a better chance of quickly evaluating and repairing the issues that plague these systems. I used to believe that these methods were only valid on larger system models, then, one day, a colleague of mine and I were sitting in a small coffee house discussing a problem they were having with one of the desktops they manage. While we exchanged ideas, I suddenly realized that I was using the same mental process on this little desktop as I did with the large cluster systems.

Read more �

The end of an Era…

Back in 1999, My company, ACT USA leased a small storefront in Thousand Oaks, CA. At the time, we did just about everything related to business computing. Not only did we service desktops back then, but we also served as a Corporate ISP to many of our customers, with T1s and Frame Relay connections terminating directly to that little facility.

A few months back, the CFO and I started to review our costs, and came to the conclusion, that we no longer needed to host any services locally, and that we could save a fair amount of money relocating to a business / industrial park. So, the hunt was on for a new facility. We managed to locate a new office only five miles from our old storefront that was perfect for our use.

Needless to say, these are exciting times for all of us at ACT USA. New bigger office in a beautiful complex, with a large shop area and room for a classroom! This move however, set into motion the dismantling of our NOC in that office.

See, the only way the move really made sense, was to completely eliminate all outward facing services from the Thousand Oaks NOC. This idea was nothing new, I had started this process almost 5 years ago, but found a few services / customers to be difficult and expensive to move, so I procrastinated. Well, that procrastination ended with the signing of a new lease, and all of a sudden I was faced with unraveling a location that I had personally kept operational 24/7 the last 12 years.

Today, November 18th, 2011, without fan fair or even a whimper… Yes, today, the NOC in Thousand Oaks fell silent. As I reached over and turned off the last router (border1), the sound of absolute silence over whelmed me. I had spent the last 12 years making sure this room stayed noisy, keeping this equipment running. Yes, the last 12 years… Responding to power failures and carrier issues all times of day and night. And now… Silence… How strange… How strange…

Thousand Oaks NOC

Brought On-Line: November 1, 1999

Decommissioned: November 18, 2011

— Stu

The End of IPv4… The Adoption of IPv6… “The King is Dead!, Long Live the King!”

At a ceremony today, February 3, 2011, the last five /8s were delegated to RIRs. For most people, this has little meaning, but to us that make our livings from the IPv4 protocol, and who have spent countless years learning the tricks of the trade, this marks an end of an era.

As for me, I’m ready for the ‘big switch’ to IPv6. But I know many of my friends and colleagues that have procrastinated, claiming this day would never come, or are waiting for a vendor to swoop in and save the day. Well, to those I say, WAKE UP! The companies you work for, and the customers you service will be greatly effected by the IPv4 shortage and the logical adoption of IPv6. The day is at hand, and vendors stand to make their money by just selling the upgrades to their equipment to handle IPv6, so I don’t think a magic bullet is in the cards. As of now, the best solution for your company to look at is dual stack. In as short a time as a year, you could have customers that are unable to reach your web based services, or only able to connect at modem speeds to them, due to overloaded proxies. I strongly suggest you start working on this now, especially if you have outward facing services such as a web server or email server.

That’s it for now, I’m busy preparing for the Southern California Linux Expo. This year it will be held at the Hilton LAX on February 25-27, 2011. Look forward to seeing you all there!

— Stu

SIP brute force attacks escalate over Halloween weekend.

Looks like the bad guys were up to no good again this weekend. SIP based PBX operators reported a huge increase in bogus registration attempts against their systems over the Halloween weekend. Our hosted PBX farm experienced this increase first hand. Logs showed an attack from a new and unique IP address about every minute. At the end of the weekend, over 1300 unique IP addresses were logged.

Read more �

Setting up IPv6 connectivity.

Back in December `09, my company ACT USA, began testing IPv6. These tests quickly advanced to our production environment. Over the last six months, I have been in the process of setting up native IPv6 connectivity for all our data centers. This connectivity is based on the dual stack model. This article attempts to cover the technology available, and the choices I made based on that technology.

Read more �

Building Redundant Networks in Data Centers.

I recently was asked to put together a brief web presentation on the different methods of creating redundant networks. I couldn’t think of a better place to put it, then right here on my blog. After all, I was overdue for a post anyways…

What do I mean by redundant networks?

A redundant network is two or more distinct paths for data to travel to and from an upstream network. In it’s simplest form, it can be a piece of equipment that can be manually placed into service easily upon a failure. More often though it is set up so that any single device or connection can fail, and without user intervention, a backup system or connection will automatically step in and take over the job of the failed device, or connection. A redundant network does not mean that no mater what happens, your data will still be reachable. There are many factors that need to be considered, ranging anywhere from your providers, to your applications, that can cause a failure.

Read more �

Amazon Brute Force SIP Attacks – Dave Michels Interviews Me.

Shortly After my “SIP Brute Force Attack Originating From Amazon EC2 Hosts” post, Dave Michels interviewed me for an article Dark Side of the Cloud. This is that interview:

Dave:   What do you believe the intent was of the attacks? Free long distance?

Stu: Certainly free long distance would be one reason… But there are many other reasons to hijack a SIP account. I’m sure that organized crime would pay for a block of active SIP logins. They could use them to circumvent surveillance, or possibly use them for fraudulent boiler room calls about extended warranties and such.

Remember, most folks still believe that the Telephone System is secure… They tend to believe someone who is calling them.

Read more �

SIP Brute Force Attack Originating From Amazon EC2 Hosts.

I woke up Saturday morning to find strangely high network activity on some of our inbound connections. After a quick review, it turned out that most of the traffic was going into several of our hosted PBX systems. After a little more digging, I discovered that several systems on the Amazon EC2 network were preforming brute force attacks, against our VoIP servers. They were attempting to guess user names and passwords for our SIP clients. I immediately blocked all traffic from the attacking IPs and examined the logs. Thankfully, I found that non of the attacks had succeeded in guessing passwords.

Read more �

A Microsoft Free Workplace in Six Months. Part 3

Well, I’ve been putting off updates on this for a few months, hoping that this project would begin moving forward again. It hasn’t, and for the life of me, I can not figure out why. The client has postponed the project due to a sudden surge in business. Their reasoning is that it would require too much retraining at a time where they don’t have time to retrain.

The confusing thing to me is that when asked what their findings were up to this point, they listed the following:

• Ubuntu systems operate faster then Windows systems.
• Ubuntu systems have better stability then Windows systems.
• Ubuntu is able to do all tasks we currently require to perform company related computing tasks.
• Ubuntu desktop can be tailored to incorporate shortcuts that end users can easily use.

The only negatives were:

• Ubuntu requires a rethinking regarding finding and installing software.
• Most end users will not be able to modify their own desktop without training

The second thing I would think would be a plus, but that’s the system administrator in me coming out.

So, for now, the project is stalled. So for now, so is this series of articles… 🙁

— Stu