HR Geeks

llrighty!   Well my new machine just arrived about a week ago. With plans of myself jumping-ship to Australia, I have been also wanting to put a reliable distribution on my new lappy. Mid-life crisis? Well at 27 it still seems like it, as I have been a curmudgeon and have tended to get stuck in my ways. The big jump here is, well, choosing a new distribution. Like a new pair of undies, I want something that’s clean and inviting, it’s a big decision, and change is hard for me. As I mentioned, I’m kinda a curmudgeon, so something not venturing too far from my old distro would be well, my other personal machine runs Slackware (been a Slackware guy since 2003 or so).

Anyways, I’ll cut to the chase here. Like Slack, I want a distro that doesn’t blind me to the internals of Linux, and keeps things intuitive, isn’t that how we learn? Personally, I feel that using a ton of GUIs evades how things work. Sure, it’s not for everyone, but why not add another crinkle to the brain, like a wrinkle of knowledge. With that said, me changing my digital undies from Slack to someother distribution might add a few more wrinkles to the noggin. Also, Matt requirement #2 for distros is this: I don’t want to have to run the equivalent to a digital enema after install, lets keep the undies unscathed and free of stains. I like lean distros that do not imbibe the user with a ton of fluff; If I want it, I’ll install it. As being at a job that encourages develoeprs to use the same OS, or very similar, to what the customer runs, I have been babied with some reasoable package management. I used to be opposed to package management, as some of the earlier implementations (I’m not talking Slack here), seemed kinda confusing, but that was years ago, and a personal opinion. The newer stuff seems to me that package management is pretty up-to-par; however, I don’t really like being blinded to the build of the packages, it is nice to build what you want. I liked Slack packs, they were simple, but I always kinda just like building from scratch; however, the latter not always resolving dependancies and not always the fast thing to do if you just want to try something out quickly. The distro I have been using for work has frequent updates and such, and I can just dowload something rather fast if I need to see if a utility is for me. Quite simply, all I want is sleep and a minimalistic distro that provides simple, effective, frequet dependancy resolution package management, and has a strong developer base (I don’t want the distro to go moot in a few weeks).  Like undies (I wear boxers), a distro becomes something rather close to you, a friend.  They know all your personal details. I also want a non-corporate based distribution. I don’t care what the shareholders say, I care about what the poeple who develop for the project have to say!

Well, give it was Memorial Day weekend, I had some time to play around. As I was about to go insane with distro choices, hey take a look at distrowatch.org, I finally found my new buddy… Arch Linux. Having heard of them before, but never paying much attention, I was really hesitant, as I mentioned earlier, I want a well-established distribution that won’t disappear in a few weeks. Well, thanks to their website, Wikipedia, and Distrowatch.org, I must say this seemed like the golden pair-of-nuthuggers I had been looking for, quite possibly I could rest and get some sleep. Ok, well the golden pair of boxers (I don’t do the tidy-whitey-crunchers).

But how can I verify that this distribution is reputable? Sure, distrowatch.org had it ranked pretty high on popularity, and well Wikipedia and the Arch site mentioned the project was established in 2002, which is great, but how many hackers do we have that are currently running or trying to run it… lets check IRC. Yup, 800+ in freenode.irc.org #archlinux. I have arrived! By the way, another thing I wanted to make sure of was that there was some body of persons that make the overall decisions. And…. check, yep they do.  I didn’t delve into the politics so much, but I was told so in #archlinux, not that I really looked/googled/checked-their-wiki. Likewise, with such a simple question I was not even flamed in the IRC room! What, no flames? Ok, I like these guys! Arch away!

Arch uses pacman, which is an incredibly easy-to-use package management system that resolves dependancies. Other distros are leveraging this system as well, but I believe the roots of pacman are from the Arch project. Since the Arch philosophy is to evolve (after all, nature usually has things tested-out for us), there are no major releases of Arch, mainly evolving, constantly updating packages. And one can update their system, with a flick of one command to pacman. If one choses to use a ports-like system: download-source/build/install with package manager, then that can be accomplished as well.  Hey, it evolves, nature evolves, seems the logical approach to me.

It was encouraged that I write about my virgin experience installing Arch from one of the hackers in IRC. With that said, I did jot down a few notes. As this post is quickly approaching the TLDR lower-bound, I will make this quick. First thing that set off my awesomeness-meter was that I didn’t need to run syslinux or format my USB jump-drive to get their install iso to boot. Merely, a ‘dd‘ of iso to jump-drive was all that was needed. With that accomplished in just a few jiffies (yes a jiffy is a cycle of kernel time), I was ready to reboot to install. But before I even ran ‘dd’ I was thinking, should I ‘dd’ to /dev/sdX or the first partition, /dev/sdX1? As I was thinking such a question, I decided to look in their install guide, and low-and-behold it does mention this, as if it read my brain. By this point, I was utterly frightened, it new what I was thinking, and answered me!

The installer is well, in one word, elegant! I liked it, and install of the core-system was fast and pretty straight-forward. No GUI, just some ‘dialog‘ based menus to navigate, straight-forward.  Well, it looks like ‘dialog.’  The base-partitioner is cfdisk, which is not overly complicated. This, coupled with their installer, works pretty nicely, it took a few tries to see what was going on, but after a few minutes, I was ready to select which binaries I wanted to install for the core system.  I really like their beginners guide on the Arch wiki. Also, the hints/notes that the installer has per ‘dialog’ is helpful. But in all honesty, you really do not need a guide to install this, but it does help.

Another word of mega-positiveness towards Arch: Most devs seem to forgo or dislike documentation.  However, what also shines about Arch, aside from simplicity, elegance, and a friendly IRC room is their wiki documentation. It’s well written, in a manner that is guided to teach the reader as to why they run a certain command/edit-a-file. And it seems to have all the answers, if not most.

So my overall take here is that I am thoroughly impressed by this project, and it is now my distro of choice. My undies (boxers) are clean, and sleep can commence.

Up the Arch!

-Matt (enferex)

Sources:
My Brain
http://www.archlinux.org
http://www.distrowatch.org
http://en.wikipedia.org/wiki/Archlinux
http://en.wikipedia.org/wiki/Pacman_(package_manager)
#archlinux on irc.freenode.net

Years ago I thought it would be neat to try to predict something. Not in a psychic fashion per-se, but predict events based on current events, mainly trying to forecast stock prices. I had an interesting idea, sure I am aware that a hojillion or so other methods of modeling the stock market are out there now, and probably one identical to my initial concept. This concept being, trying to use a news service, such as Google News, to predict the stocks. In brief, take the headlines from a news RSS feed and then associate certain words in the headlines (e.g. killing, gun shots, hairy Hobbit feet) to values of certain stocks. Would certain current events, represented by news headlines, actually be a reasonable predictor of stock values? Well, I don’t really know.

So, I started working on an application that would help me do this, like I said, I am sure this is not a unique idea. Anyways, for one reason or another, I really do not recall, I decided to try to forecast the state of the world. Kinda like the Department of Homeland Security’s “Threat Level” thermometer. I care about the state of the world; however, predicting the stocks accurately would be mega-cool in my opinion. So, I came up with Slayercast. Being a fan of one of the greatest bands of all time, I figured, hey… they know what’s up! Let me see if associating Slayer lyrics to the news headlines would give me a “forecast” relating to the state of the world. Like the great Terrance McKenna’s 1970’s derived tool Timewave Zero, but with a nice metal-injection! After all, Slayer does sing about negative subjects, which are often portrayed by media. Why? I suppose we are parasitic beings that enjoy reading about scary-negative stories in the news, well at least the media seems to think that we enjoy reading the like (my assumption). The Slayerizer tool is rather simple, it combs the top ten headlines from Google News, and generates a ratio for each headline, a ratio of Slayer lyrics to non-Slayer lyrics. Thus, a value of 100% would mean that the Slayercast value is 100% for that headline. The ratio of Slayer-to-non-Slayer words is averaged for all ten headlines for a given hour. I cron’d the Slayerizer on users.757.org and captured a Slayercast for about each hour between the dates of August 19, 2009 till some time on November 12, 2009. I say “about” because for some hours, my utility returned early without providing data.

Caveats:

• This tool is not perfect and did not successfully grab all data all of the time.
• Not all hours were captured.
• Articles, like the words ‘a’ ‘an’ ‘the’ were not removed from the lyrics, hey if Slayer uses them, it’s fair game!
• This is not to be some thesis or great scientific endeavor, it’s just a goofy-project!

So what now? I sat on the data for a while, but never really did too much with it. Well, back to the stock concept. I never really linked the Slayercast to world events. But we can’t have data sitting around doing nothing! It must be exercised! Back to the stock world. I decided to see if Slayer knew what was up with the defense industry. I make the assumption that a good view of the defense industry can be represented by a stock exchange-traded fund (ETF). Basically, an ETF is just a bunch of stocks pertaining to a slice of the overall stock market, such as the technology sector, or in this case the defense industry sector. Since Slayer sings about death and stuff, and I hypothesize that the defense industry might provide a data relation, the Slyayercast value might be a reasonable tool to forecast the sector of the stock market pertaining to the defense industry.

The results? Can Slayer lyrics be used to forecast the stock values of the defense industry? I used the PowerShares Aerospace and Defense ETF to associate against the Slayercast value for a given day. I assume this is a reasonable “view” into the performance of the defense industry, from a market perspective. Hey, if the defense industry is rocking, what does that mean? That is a thought experiment for the reader.

With the data in hand I had to relate the Slayercast value against something! How about a stock value! Simply, I took the difference between the opening and closing values for the Slayercast for each day. 12AM Slayercast value subtracted from the 11PM Slayercast value. I did the same for the stock value of that given day, I took the difference between the opening and closing value for the ETF. One discrepancy is that the stock market open/close values are not 12AM to 11PM, but from 9AM to 4PM each day. The correlation between opening Slayercast values and opening stock market values is not terribly direct. Might I add, that some hours/days, when the Slayerizer borked (messed-up) or the market was closed, should not be in this graph. And, even for some headlines the Slayerizer might have messed-up, and that data might be reflected in this graph.

Now, interpret at your own jest:

Shall we sit down and have some story time? So here’s the deal. I have had this fascination about being able to predict future events given the current state of a system. Seemingly random events, I have a hard time believing are truly existent. Of course, I do favor the Heisenberg Uncertainty Principle, but Markov processes can be interesting. The latter being a process that is independent of its previous state. In other-words, you transition from your current state (present) to a future state because of a seemingly random event. In other, other, words, a stochastic process. Interpret the Slayercast values with your own discretion.

Data Sources:

• Slayerize Stuff: http://users.757.org/~enferex/slayerize/
• Slayer lyrics from (I think): Dark Lyrics
• Stock data from: Google Finance for PPA
• News data from: Google News

-Matt (enferex)

I learned something interesting today – you can get access to a bunch of logs from your iPhone, without jailbreaking it!  There are number of SQLite databases stored in
~/Library/Application Support/MobileSync/Backup/
on your OS X machine that the iPhone syncs with.
The filenames are SHA1 sums of their location on the iPhone [src]. Through trial and error, I’ve figured out the following files that should be common to every iPhone:

992df473bbb9e132f4b3b6e4d33f72171e97bc7a.mddata Voicemail list
ff1324e6b949111b2fb449ecddb50c89c3699a78.mddata Call log
3d0d7e5fb2ce288813306e4d4636395e047a3d28.mddata SMS Log
740b7eaf93d6ea5d305e88bb349c8e9643f48c3b.mddata Notes database
31bb7ba8914766d4ba40d6dfb6113c8b614be442.mddata Contact List

The schema for most of these can be found here: http://damon.durandfamily.org/archives/000487.html (although he references their on-phone location, and talks about jailbroken phones, these SQLite DB files are accessible on your desktop machine, and are updated in place every time you sync your phone.

In addition to these, a few interesting DBs I found that are specific to apps installed on my phone are:
6639cb6a02f32e0203851f25465ffb89ca8ae3fa.mddata Facebook friends list
970922f2258c5a5a6d449f85b186315a1b9614e9.mddata Flightstats
5ad81c93601ac423bc635c7936963ae13177147b.mddata Daily Burn food log


Each of these database can be accessed via the sqlite3 command line tool for interactive use. For bulk processing and playing with stuff in a spreadsheet or other DB, you can dump whole tables to CSV easily with sqlite3
sqlite3 -csv -separator , 3d0d7e5fb2ce288813306e4d4636395e047a3d28.mddata "select * from message" > smshistory.csv
you can dump your SMS history to a CSV file.

I’m writing a few scripts to generate ‘top talkers’ and some other statistics, and will post those later.

The local Hampton Roads Ruby/Cocoa Users Groups / Ken Collins is throwing an event friends, and it looks like it’s going to be good!

“On November 5th, 2009 Hampton Roads’ premier learning and network event for software developers, interactive agencies, and technology entrepreneurs will be held at Grow Interactive, Norfolk.”

Speakers:

Pragmatic Thinking and Learning: Refactor Your Wetware
by Andy Hunt
Author & Co-Founder Pragmatic Bookshelf

The Joy of Ruby
Clinton R. Nixon
Development Director at Viget Labs

iPhone Development: Touching Cocoa
Jamie Pinkham
Software Engineer at Mobelux

Sounds like a great event. It’s free, seating limited to 50. RSVP today!!

More information at www.757studio.org

libsheap is a simple shared library to aid runtime heap analysis of an application. This library wraps malloc and free calls. The allocated data returned by the memory allocator is output as well as the amount of CPU cycles between calls to the true, underlying malloc implementation. If glibc’s malloc, ptmalloc2, is used the actual size, including the header malloc attaches to each returned chunk is reported. The size of a returned chunk is typically more than that of what the user requests, as the extra size can help reduce fragmentation.

Special thanks to Google and stackoverflow.com where I learned about wrapping routines in the GNU/Linux system. Much of the wrapping code was paraphrased from Checker’s answer to a post over at stackoverflow.com Thanks Checkers!

This project is released under the GNU LGPLv3 license. So take to your shears!

-Matt (enferex)

I would like to announce the release of a 757Labs tool, mp3nema.

MP3nema is a tool aimed at analyzing and capturing data that is hidden between frames in an MP3 file or stream, otherwise noted as “out of band” data. This utility also supports adding data between frames, and capturing streaming audio.

This tool is under the GNU GPLv3 license. So have at it!

Thanks to Tele, skhisma, and the rest of the labs!

-Matt (enferex)

I would like to announce the initial release of the 757Labs’ PDFResurrect tool:

PDFResurrect is a tool aimed at analyzing PDF documents. The PDF format allows for previous document changes to be retained in a more recent version of the document, thereby creating a running history of changes for the document. This tool attempts to extract all previous versions while also producing a summary of changes between versions. This tool can also “scrub” or write data over the original instances of PDF objects that have been modified or deleted, in an effort to disguise information from previous versions that might not be intended for anyone else to read.

This tool and accompanying whitepaper can be found over at the labs:
http://www.757labs.com/projects/pdfresurrect

Special thanks goes out to all the hr-geekers, and 757labs. Many were consulted on this and provided suggestions. including Tele, Derez, Remad, Count, Sunpuke. Special thanks to Brent, not really part of the 757 crew, but aided in proofreading the paper. Thanks Guys!

-Matt (enferex)

While it’s not a new tool, it’s new to me!This weekend I was cleaning up my office and decided it was time to throw out a bunch of old hard disks and two old PCs I have been hanging on to for no good reason. I’m a lazy, lazy man, and I knew there had to be a semi-automated way to wipe all these disks without a writing a foreach loop

Say hello to my new friend Darik’s Boot and Nuke.

I installed all the disks in the two PCs and booted them both off a USB key with DBAN. It detected all the disks in the machines and prompted me with my wipe options. The nice part about this utility is that it wipes all disks simultaneously. If you’re in a hurry you can type ‘autonuke’ at the boot menu and it will start wiping all disks automatically. Good if the feds are knocking on your door…

It took just under 36hrs for DBAN to finish wiping all the disks with a 5x wipe. It will never replace my thermite, but it’s incredibly handy.

Okay, sure I could have purchased the 50-Piece resistor assortment pack from Radio shack for $6, but why bother when you can goto China and eBay and for $25 get 2500 resistors.

I will never go wanting again.

This place might join my list of favorite China direct companies Sun-Pec.

My others include:

LED Shoppe

Cell Phone Shop

Who do you use for China-direct stuff other than China-Mart, err.. Wal-Mart.

One of my big tasks at work is rolling out a completely new platform for our internal production and development networks. The goal is to move all of the servers that support these networks to VMs running in Hyper-V under Win2k8 (Yes I’ve sold out my UNIX roots).

As all of our applications are web based, I’m also been given the task of setting up a new SQL 2k5 database cluster that will utilize a portion of the new SAN. This DB cluster will be an active-active setup with a total of three servers.

I setup a couple VMs on my notebook to experiment with different SQL disaster and Win2k8 server failover scenarios. As I wanted my test network to completely standalone on my notebook, I needed to find a way to bring my SAN home with me. I found a really neat app from Rocket Division that would allow me to create a virtual iSCSI SAN on the host OS share virtual disks to my VMs.

For this experiment the 30 day demo works just dandy. There is also a free personal edition but it is limited to only one host connection. The app has an incredibly small footprint and runs quite happily on my notebook with my other VMs. I only have 3GB of RAM on my c2d notebook and am able to run 4 Windows VMs, the iSCSI emulator, and iTunes with really decent speeds.

For my disks, I choose to creates virtual image files on your host PC must like VMware. I’ve never needed more than about 1GB of storage for per LUN for my testing, but it will go all the way up to a terabyte. The options to use physical disks, RAID1 of image files, and snapshots makes for a pretty feature rich test environment.

There is an alternative iSCSI target application called WinTarget that’s worth checking out. I really dig Starwind because of it’s incredible ease of use and small footprint.