This Message Will Self-Destruct in Five Seconds….
Written December 12th, 2007 by Stephen HalseyThis originally was not the first post I was going to write on Webiscope. My first post was going to be something exciting about integrating SEO, Advertising, PR and marketing campaigns to drive traffic to a site and increase customer conversion. Very interesting stuff, I thought.
So last week, as I was scribbling out some thoughts around that first post, I received word that one of our sites had been compromised by a hacker. Suddenly that topic I was originally writing about didn’t seem so interesting. Because of course, if you don’t have a functioning website to drive traffic to, it doesn’t really matter what kind of exciting campaign you are implementing.
Even more frustrating was that this was actually the second time in nine months this type of attack had occurred. Fortunately during both incidents, the damage was confined to web servers and didn’t make it inside the network or compromise patient safety systems. But still, damage was caused; weeks in productivity and many thousands of dollars were lost in dealing with either recovering sites and applications or retrofitting existing infrastructure into new security measures.
Up until that first incident nine months ago, the attitude toward web security was very embarrassingly laissez faire. Admittedly, even I was less forceful about ensuring that our sites were secure enough. In the three years that I was a part of the team as Lead Developer, I never really asked loudly enough whether we were secure. I thought, hey we have security people, server engineers, firewalls, even a DMZ! Surely, that group would say whether or not there was a concern. I thought, our web server is backed up weekly, we will be fine, nothing to worry about!
Even my manager at the time was not overly concerned (though he often said to have our sites go down was a potentially career ending disaster). Ironically, while not a result of that compromise, he left two weeks later and I was promoted to manage our sites.
So I inherited a bit of a mess but a lot of good has come out of that first incident. Overall, we are more secure. We have new network architecture with content replicating software on redundant backup systems. Our databases are more secure. All of our projects now go through production readiness processes where IS Security has the ultimate go-no go vote. And while we still don’t have the 99.9% uptime I want guaranteed, we are much closer.
So what happened last week if we now have everything in place that I just mentioned? Well, the site that was affected happened to be a site that was hosted by a third party web host, so it was missed in all of our shiny new production readiness processes. We didn’t host it, so again I assumed, it’s a professional hosting company, they know what they are doing, nothing to worry about again! Well, that turned out to be a bad assumption. (And actually, it was server level compromise that affected hundreds of other hosted sites). So obviously even they did not know what they were doing.
I have since accelerated plans to bring that site onto our own network where we can better manage and monitor it. And while web security is something that I know very little about technically, I understand that it’s a critical component that must be looked at closely. I know that it’s not easy to have a totally secure website, but I feel better knowing that we have brought resources and processes to bear to make them secure and that we try our best to make them so.
What have you done to ensure your site’s security? And what problems regarding security (hacking, certifications, etc.) have you had with your systems?
December 12th, 2007 at 6:41 am
Wow, that’s an absolutely fantastic point. It’s funny to think that not one person mentioned anything about this at the conference either.
I think regardless of where your site is hosted, or what your position is asking the question of “What would we do if…” is really necessary. Not just if it’s hacked, but also the disaster recovery plan? Do we have one? If not, what can we do to change that?
Great post Stephen, I’m glad that your site wasn’t too badly affected – looks like you got off with the right amount of a wake up call.
I know what I’m going to be doing today…
December 12th, 2007 at 1:42 pm
I think many of us (particularly if we’re small) rely on the assumption that if it’s hosted internally, it’s safe; if it’s hosted externally, again, we’re safe. But your situations have proven that might not be correct, and we have to do our due diligence to establish that the necessary resources and plans are in place in the case of hacker, disaster, or whatever else.
Great post that hasn’t been brought up before!
December 12th, 2007 at 3:17 pm
It’s pretty disturbing that a remote vendor would find themselves in that situation; I can understand a “DIY” shop not having the tallest fences around its wares – but part of selling a service is considering security, uptime and reliability as important as the service/product itself. That includes redundant servers, backbones, backups, and stringent security measures as the first line of defense.
The most common thing I run into here are script-kiddies trying to take advantage of any form they can find. Usually it’s just a spambot trying to push levitra/cialis/viagra (or some mis-spelling thereof), but some of those are looking to exploit the mailserver as a relayer too. They’re just ankle-biters though; the worrisome bunch are the ones trying to DoS and injection attacks on the databases. I built some common functions through which every form must get validated, which look for all the usual stuff (yay, regular expressions) – any form submission that fails skips right past the “real” processing. I want to expand upon this to record all the details of the offenders – IP addy, what they attempted and how – etc. It just helps me build my library of what to keep an eye out for. And that’s only part of the front end …
December 12th, 2007 at 3:58 pm
Agreed. That it was a remote vendor the second time around made it that much more disturbing.
Form validation, etc are all good things to look at (especially anything involving post calls). Hiring developers that have an understanding about things like form validation is helpful and have at least a familiarity of what constitutes the majority of issues (you summed it up nicely Cap’n) is also really good to have. Getting web security bumped high enough on security agendas so that backend processes are looked at it as well is really key. To be fair, we have had a changing of the guard here over the past year, and the new guard are much more demanding, which is great. Not just for me and my team of course but for our site’s users who correctly expect our websites to be up and running 24/7.
December 12th, 2007 at 4:15 pm
Nice work Cap’n. Not being an IT guy first and foremost, I wouldn’t know really where to begin solidifying my site. It’s a good thing to think about though, and I’ll definitely be doing more research on it in the near future.
I agree with Thomas, and just to reiterate, it was a nice refresher from some of the more regular topics that we see every day.
Also: Regular Expressions rule!
December 14th, 2007 at 1:14 pm
For those of us who understand (or not) what Cap’n is saying but couldn’t possibly do it ourselves … at what point do we trust the security assurances that we’ve been given by either our internal IT or, as in our case, by our development partners and hosting company? I mean, we pay a premium to be hosted at a data center that is independently audited and certified, has biometric entry, n+1 redundancies, dust-free air, etc. I always thought that seeking out this type of partner was enough; I can worry about other things. But now Stephen has me wondering …
December 14th, 2007 at 2:44 pm
Chris: sounds like it’s time for a field trip! I would think any place that advertises the seven-course meal ought to be able to show some proof – not necessarily with just service records, but hopefully documented fail-over test results, disaster recovery reports and security audit records. At a previous employer, they used to have disaster recovery practice once a year … and sometimes unannounced practice. Talk about keeping the provider on their toes. But if you’re curious about your provider’s security … what better way to find out than try & hack it yourself?
December 14th, 2007 at 4:30 pm
Not all of us are as technically proficient as you, Jake, haha. I’m long removed from my hacking and programming days, so I’m left with assuming (hopefully correctly) that their recovery plan is adequate. I think I’m in the same boat as many of us, where since we’re such a small team and maintain mostly communications and marketing efforts on the Web, we simply assume everything is fine since they’re the experts. So the question, then, is how can we who aren’t that technically proficient ensure that the system is adequate?
In our process of selecting a vendor we brought in a representative from IS who posed many questions that I didn’t have a clue what they were about. Certifications, tests, redundancy, etc.; all I care about is if the Web site is up and if I can add a streaming video to the front page.
Might be a good idea for the future to have someone such as yourself recommend a checklist of questions and adequate/satisfactory answers (that’s incredibly difficult, I know).
December 14th, 2007 at 4:58 pm
At some point you have to make the leap of faith that security assurances are accurate. I feel much better knowing that there are new standards and procedures in place that should reduce the percentage of these incidents occurring.
We now have a third party that does do unannounced security audits for the hospital as a whole. That opens up a lot of new issues as well though (they are testing out production systems which can cause issues when they find holes).
For remote hosts though, I highly recommend getting 3rd party verification of what they claim. After all, they are marketing their services as well, and the truth of those claims can’t be verified easily. But we shouldn’t have to settle for a buyer-beware mentality.