Vertica on a Stick

By cwardell • June 11th, 2009

Can you believe it? A bootable 16 Gb thumb-drive with Linux, GUI, Vertica AND room for data!?

Yes, I admit it. I am a geek. But how cool is this. Imagine developing, prototyping, and presenting a data warehouse without procuring a server, installing Linux, or commandeering your hard drive. Everything you need is on a small bootable USB thumb-drive.

Welcome VStick

There were a few challenges in creating the VStick, but the complications were mostly around finding the right Linux distro.

I had to find the right thumb-drive the could sustain decent read/write rates

Finding a distro that could be installed on the thumb-drive was easy, getting it to work well was a challenge

The distro had to be small enough to fit on a thumb-drive yet leave room for Vertica and some data.

I had to make sure that it had all of the features, GUI, Text editor, browser, etc.. that would help with the evaluation and development process.

I did a first pass in shutting down the unneeded services to free up some system resources

I went through a few load and query cycles to test it out.

Initially, I loaded some sample data and just about all the queries performed with sub-second response times. I am now in the process of cramming as much data on the stick that I possibly can. I am excited to see what the performance and compression would be like on the larger data set.

This little midnight engineering project really highlights the technology of Vertica.

The size of the entire Vertica installation is a miniscule 8 MB so it is very light on resources

The columnar approach and compression deployed on the database minimizes I/O substantially. So far, the impact of the slower flash drive has gone unnoticed during my tests.

If Vertica performs this well on a laptop with a thumb-drive, imagine what it can do on a rack of modern servers

Stay tuned, over the next few days, I will be loading and querying a larger data set and will let you know how it goes. Outside of evaluating and developing on Vertica, I was wondering if there was a market for a data warehouse on a stick? Perhaps the embedded market like Kiosks.

Let me know if you have any ideas?

Regards,

Charlie

Comments(4) Innovation, MPP

Comments

By Ramesh on July 10th, 2009 at 2:47 pm

Charlie,

Very intersting posts.. For the Vertica on USB drive, which linux distro did you use and how did you download vertica ?

Thanks,
Ramesh

By cwardell on July 11th, 2009 at 6:56 am

Ramesh,

I used quite a few linux distros. RedHat, Fedora, CentOs, Suse. Once I setup the distro the way I liked it, I downloaded vertica from Vertica V-Zone http://www.vertica.com/v-zone/download_vertica

I am getting ready to release an ISO image of the pre-configured and formated Vstick. I will be setting up a registration page and support forum. So please stay tuned for further details.

Regards,
Charlie

By Jerry Platz on August 27th, 2009 at 6:38 pm

Hi Charlie,
Nice hack. How’s this one: Similar concept only I put it on linksys wireless routers. This gives me a distributed dbms capable of running across hundreds of other routers (using the spare linux cycles that the routers possess). Talk about commodity hardware. (:-) Anyway, keep up the good work.
JP

By cwardell on August 27th, 2009 at 9:30 pm

Jerry,

Definitely an interesting concept. Have you already done this with Vertica? I assume you are talking about the WRT54G and would suspect that the challenge would be with RAM. I love the idea though and if you have not done so already, we should think about launching a project?