HR Geeks

After the recent Apple update, which included ’security fixes’ for Safari, Little Snitch popped up a warning message when I attempted to visit my banks website.  A process called ocspd wanted to visit “EVSecure-ocsp.verisign.com”.  Needless to say, I was instantly curious as to what in the world ocspd was, and why it was trying to talk to Verisign when I was visiting my banks webpage.

It turns out, ocspd is part of Apples new ’safe surfing’ update to Safari.  Online Certificate Status Protocol (OCSP) is the functional replacement for the old school PKI Certificate Revocation List (CRL).  It allows the Certificate Authority (CA) (in this case, Verisign) who signed the websites certificate, to authenticate the presented certificate in real time.  This is a much more ‘elegant’ solution than the old, crummy CRL, which had to be manually updated (or pushed down with OS patches, etc) and did not allow certificates to be rejected in anywhere near realtime if they were deemed fradulent.

Despite being a more elegant solution, it also creates a number of new problems.  

First, it places a big new load CAs, who went from being trusted certificate issuers to being real time certificate verifiers.

Secondly (and more importantly), it seriously breaches the privacy of every user using the service.

(continue reading…)

HRGeeks has a group started by Sean on LinkedIn (needs a logo!). For those un-aware (I hope everyone knows what LinkedIn is by now), LinkedIn is a social network site for business. Plaxo was first, but LinkedIn grew to great popularity recently, after activity has slowed for a year or two. Now Plaxo is making a comeback. If you did not know, www.linkedin.com and www.plaxo.com.

HRGeeks also has a group profile on the new local directory site Connect321. Go to www.connect321.com, search for HRgeeks group (or browse). Feel free to join the group there. Also feel free to add businesses or reviews! Take note that unlike LinkedIn and Plaxo, Connect321 is a local company so let’s show them some love!

In addition to this marketing, t-shirts are in the works (hopefully by Thanksgiving).

Job listings have been appearing in mass for local company Wasabi Systems. While I figured they would be gone by now, it appears they are indeed growing. Wasabi once employed a number of the people behind the awesome NetBSD project, not bad for a company in Norfolk Virginia! Wasabi makes iSCSI target & NAS solutions (Similar to the open source OpenFiler project, but a much smaller tighter package). Wasabi also performed coding work for a number of major companies in the embedded space. I’m not sure what the expansion is for, but if anyone knows it would be great to hear what is going on!

I have always been a fan of the underlying concepts of TOR. I frequently utilize it in my scripts to simulate traffic from all over the globe. From a programming prospective, I’ve used it for anything from testing a web service to rigging an online poll… OK, I’m just as guilty of using TOR for its unintended purposes as anyone

Recently I decided to try setting up a TOR hidden service. While “anonymous” browsing via TOR is fairly well known, the ability to setup anonymous servers is often overlooked. When you setup a hidden service, you can host applications on a server whose location is completely unknown. All traffic to and from the service is encrypted and comes “anonymously” over the TOR network. Sounds neat! What’s the catch!?

The Catch: If the server hosting your hidden service is ever compromised, so is the IP of the server and most likely its owner. If you setup a anti-scientology forum using PHPBB and it gets exploited (that never happens), any anonymity provided by the TOR hidden service can be stripped away. If this happens, expect to hear from their lawyers

I had the idea to setup an anonymous bittorrent tracker on the TOR network. With all the good trackers getting shut down, why not setup an “untrackable” tracker that couldn’t get shutdown.

VMWare is really handy for setting up a secure hidden service. Without VMWare you would need two dedicated workstations. The diagram below shows my take on setting up a secure hidden node.

The concept of this network is fairly straight forward. Your application server (in this example Apache) needs to be isolated from any networks that have internet connectivity. If your hidden service is exploited, the attacker must not be able to determine where your application server is located.

In this example, my application server (10.0.1.2) can only talk to the TOR proxy node on its secondary address on the 10.0.1.0/24 network. Firewalls running on both servers prevent any traffic that is not HTTP traffic requested from the TOR network. If someone exploits my tracker and runs unauthorized code, they only have access to the local box. They can not originate any traffic that was not requested by the TOR proxy running on 10.0.1.1.

The TOR node has a secondary interface on the SOHO LAN. This interface is used only for encrypted TOR traffic coming to/from the internet. This server should only be permitted to reach the TOR network over its encrypted connection and not have full NAT access to reach the internet. We’re putting a lot of trust in the TOR daemon. If it were exploited, you’re hidden service location would also be visible to the attacker.

I wasn’t super paranoid in my testing and used iptables on the hidden service and TOR node. If you’re really nervous, you’d need hardware firewalls to prevent the box from being exploited and having someone strip down your ruleset.

Is this a lot of work just to setup a hidden service? Yes, yes it is! For those wondering, my experiment was a total failure. While it worked technically, it was so slow no one would ever use it.

If you’re wondering, most hidden services are crap. The good ones are, well, hidden! The only real exception to the public hidden services is WikiLeaks. WikiLeaks is an excellent example of a site that was facing censorship and turned to the TOR network to ensure they could continue to make their information public.

So for those of you who may have had strange Cox outage issues in the HR area this fine Wednesday (04/02/08) morning, I have some good news. I think I know what was going on. At about 9:30 it would appear my network at work was on the painful end of a DoS attack from somewhere on the internet. At the peak we were getting about 40,000 packets/sec through one provider and 70,000 through the other, bringing our router to it’s knees (normal load for us is about 3000/sec). It would appear that this attack caused some pretty major problems for Cox as well. It just stopped all on it’s own at about 11:00 and neither provider has any reliable information about where it came from. It was still Apr 1 in China at 9:00 AM right?

This morning oreo and I were looking into a SSH issue with one of our cpanel servers (yes, yes I know) when we discovered a hacked web hosting account running a ventrillo server. When we went to kill the users’ other processes we noticed something else running that was a little more interesting…

/home/<removed>/public_html/drivers/CVC/src/ircd

Our hosting box was also running an IRC daemon optimized for controlling botnets. Neato!

After fiddling for a bit we now had the IRC connection password and the IRC OP login. When we logged in we were quite surprised to find 800 exploited computers happily connected and awaiting orders. We tried a few commands but couldn’t figure out exactly how to control the botnet.

That was until we discovered the brilliant botnet operator had turned on debug logging. All that work setting up a server designed to hide who was connected and what was going on, only to turn on debugging. Whoops!

Not only did we now have full logs of how to control the bots, we were able to lock the operator out of his network by changing all his passwords.

As you might expect, the botnet owner was pretty upset. We contacted him via IRC and received a friendly response…

if you not give me back my bots i’ll destorys you
remmber that
i got your computer and your box
and alot more
i known you

I’m sure the stress of losing all the nodes he worked so hard to exploit had him a little upset

I bought a 1TB Apple Time Capsule today. I plan on replacing my Linksys WRT54G and a Dell Linux Samba server I have.

As usual, Apple nails the out-of-box experience. Pop the CD in, run the utility, follow the directions presented, and the Time Capsule was up and running in about 5 minutes (3 of which were spent rearranging cables to accommodate the new machine).

The utility software picked up the un-configured Time Capsule and walked me through the configuration in just a few simple steps. After confirming that my cable modem used DHCP, entering a password for disk access, and entering a WPA2 password, everything was up and online.A nice little touch, the utility software that configured the Time Capsule’s wireless network automatically reconfigured my Airport card to connect to the WPA2 secured 802.11N network on the Time Capsule.

After getting online, I opened up the Time Machine configuration setting pane, selected ‘Change Disk’, picked the Time Capsule out of the list, and that was it – it’s now doing DHCP, Wifi, NAS, TimeMachine hosting (for both Macbooks), and routing/NAT’ing my cable modem, with a grand total of 5 minutes of configuration and maybe half a dozen clicks. Fairly impressive!

I attended a Win2k8 Server launch “party” a few weeks ago and one of the big new features is the ability to opt to run server core. I believe the quote was: Now here’s something you UNIX guys are really going to enjoy!

If you’re not up on your Windows tech, core is basically just a stripped down version of Windows Server 2008 w/no GUI. The screenshot above is all you get… just a command prompt. Everything is done from a command line. You can click all you want, but all it’s going to do is close the black window.

After demo’ing the stripped down OS to my peers it became clear why Microsoft has decided to offer the core version… sell more books and training classes. None of the other Windows only admins in my office get it. Not even a little. They think it’s great that they get a “more secure operating system,” but basic items like changing the IP address or installing a NIC driver have already resulted in endless scribbles on notepads and post-its on cube walls.

This next screenshot might be a little confusing to you. It’s server core running Hyper-V running server core running Notepad…

Normally on a GUI-less UNIX install I can’t bring up xemacs… I do love the Windows 3.11 style file browser. Good to see they’re reusing quality code

For the record, I am using it. It’s been great for the our VM boxes as it uses <300MB of RAM and has incredibly low overhead for Windows. Not core Linux install low, but decent. I’ve already come to the realization that anytime there are issues with these boxes I’m going to be the one getting paged =\

One of my big tasks at work is rolling out a completely new platform for our internal production and development networks. The goal is to move all of the servers that support these networks to VMs running in Hyper-V under Win2k8 (Yes I’ve sold out my UNIX roots).

As all of our applications are web based, I’m also been given the task of setting up a new SQL 2k5 database cluster that will utilize a portion of the new SAN. This DB cluster will be an active-active setup with a total of three servers.

I setup a couple VMs on my notebook to experiment with different SQL disaster and Win2k8 server failover scenarios. As I wanted my test network to completely standalone on my notebook, I needed to find a way to bring my SAN home with me. I found a really neat app from Rocket Division that would allow me to create a virtual iSCSI SAN on the host OS share virtual disks to my VMs.

For this experiment the 30 day demo works just dandy. There is also a free personal edition but it is limited to only one host connection. The app has an incredibly small footprint and runs quite happily on my notebook with my other VMs. I only have 3GB of RAM on my c2d notebook and am able to run 4 Windows VMs, the iSCSI emulator, and iTunes with really decent speeds.

For my disks, I choose to creates virtual image files on your host PC must like VMware. I’ve never needed more than about 1GB of storage for per LUN for my testing, but it will go all the way up to a terabyte. The options to use physical disks, RAID1 of image files, and snapshots makes for a pretty feature rich test environment.

There is an alternative iSCSI target application called WinTarget that’s worth checking out. I really dig Starwind because of it’s incredible ease of use and small footprint.