So let me start by sending a disclaimer. I have not had hands on with Aster Data so I can’t really comment on it as a product. However, the technology is interesting and I hope to eventually kick the tires.

Here is my take. It depends on what you want to do with your cluster. Aster is based on map-reduce so it opens up opportunities for many interesting things. With map-reduce, you have the ability to run application logic in parallel at the node level. However, it is a row-store database so the I/O is going to be higher than that of a columnar store like Vertica and I would expect that given the equal # of nodes, Vertica will out-perform Aster in query response.

Netezza has the ability to push algorithms down into the FPGA which made for lightning fast computations. With their twinfin product Netezza makes it relatively easy to move algorithms to the SPU level through user defined functions. This opens them up to a lot of opportunities to play in the HPC (high performance computing) space.

Vertica has done an awesome job with the compression combined with the Columnar approach to storing the data. I/O is really minimized and the query response times are incredibly fast. HPC was not their focus so running application logic at the node level was less of a priority. With that said, there is no reason why you could not have something like Hadoop running along side of your Vertica cluster. You would get map-reduce, incredible compression and the benefits of a column store. With the recent version 4.0 of Vertica, external user defined functions are possible, so they are closing that application logic gap.

So the question is what do you want to do? Query? HPC, both? Do you mind proprietary hardware? Do you want to grow your warehouse incrementally with the smallest possible initial capex? What does your data management staff look like? These are just a few of the many considerations you need to give when selecting the technology that suits your business. It will be easier to do a POC on Vertica or Aster, while Netezza will require quite a bit of co-ordination and qualification before they roll a Twinfin frame in your data center.

Netezza and Vertica have an INCREDIBLE client list. Aster is starting to make strides.

One last thing, the Patent office awarded Google a software method patent on MapReduce. What this means for hadoop I really can’t say, but make sure you investigate that path and ask the right questions.

Lot’s to consider as this is an area where you want to be very diligent. I hope this helps. Feel free to contact me if you would like to explore the options more formally. http://www.bcsolution.com/contact/

Best Regards,
Charlie

Can you throw some light on Asterdata as compared to Netezza and Vertica.
Which do you think would really break the ice.

When utilizing a columnar db for your data warehouse, you are typically abstracted from the underlying column store. To the client, the data appears to be stored in table, rows and columns. You simply create ansi standard sql as you would in any relational database. The the columnar db (Vertica in my case) takes care of the details behind the scenes.

Under the hood, the column store DB does not read a block of the table into memory but rather reads ONLY the columns needed to satisfy the query.

This alone minimizes the I/O. Vertica goes quite a few steps further with some advanced compression algorithms per column and a concept of bucket. Tremendously bringing down the cost of I/O.

Additionally, they have some more advanced topics utilizing their Flexstore technology that allows you to group columns for single disk reads allowing finer tuning when trying to minimize disk I/O

I believe that the Logical Data Models (Finance, Retail, Travel, etc) is a way for Teradata to sell “Solutions” as opposed to “Just Technology”. For example, you could very easily buy the NCR self service checkout registers as seen in Home-Depot knowing that it would feed a retail data model in Teradata.

Teradata has a pretty significant professional services group providing domain expertise not only for the Teradata database but for the vertical market as well.

Teradata handles transactions very well. So you can have your KPI’s sit on top of the transactional model. Depending on the size of data and the # of nodes, you may never need to aggregate your data. In some cases you would just let the parallel processing aggregate on the fly. This is what Teradata calls Real-Time data warehousing.

When you get into the MPP realm, many times it is not needed to create the KPIs as materialized tables.

I am pretty sure that Teradata will not sell there LDM without their appliance though. Let me know if you feel I addressed your questions.

Regards,
Charlie

Have you taken a look at VOLTDB. http://voltdb.com/
It can handle a huge volume of concurrent transactions and I suspect that there may be some vertica integration points on the horizon to handle the Analytics.

Can you please share some comparison table for VLDB and DW, which can handle this kind of data.

I did not forget about Sybase IQ, When I first had the opportunity to come across them, I did not see MPP as being the underlying architecture. At that time, they still had many components of their technology that was shared and could be considered a potential bottleneck. Things may have changed since I did my eval, but at that time, Column Store – Yes, Shared Nothing/MPP – No. As a result, I took them off my VLDB and MPP watch list.

Charlie

You ask some great questions. I am going to do a little research and see what I can find.

What I do know is that this effort involved three core technologies. DMExpress, HP Blade System C-Class, and Vertica. The DMExpress PR is here -> http://www.bcsolution.com/2009/05/syncsort-world-record/ – From what I understand, they were all extremely excited from the results.

I find Syncsort to be and extremely reputable brand. For over 30 years they have provided rock solid technology in the very large data space. I have only worked with them during the past 10 years but remember their advertising campaigns in the late 80′s.

So I believe you will find that Syncsort can stand behind their claims. They have some clients with enormous data deploys that I am sure have run through weekends.

I will see what I can find out.

Charlie

1. Who judges this world record and what are the qualifying criteria? Does it just depend on which “independent” analyst is looking at what solution at a particular point in time?

2. Who held this “world record” previously – does Vertica even know or are they guessing? In which case, do they really know they’ve beaten it?

3. The most obvious point. Nice performance over an hour in a controlled lab environment. But what sort of comparable stats can Vertica produce for a very large production environment (100 TB+) over a sustained period (i.e. weeks/months/years) – assuming they have such a production environment. And not the marketing-speak terms of peak performance, but TB/hr in continued, 24/7, mixed workload. There are systems out there loading 2TB/hr in real time into something as mainstream as Oracle 10gR2 (peaking at 5x that in busy hours), so is this really that remarkable or just more hollow marketing bluster from the niche players?

Thank you for your insight.
Charlie

I’m sorry for being picky about this but making inaccurate claims is a pet peeve of mine.

I reread your post and went back to the audit report to review section 4.2 a few more times. I believe your point is valid enough that the blog should be clarified. I have adjusted the blog above.

Thanks for pointing it out.
Charlie