Decooda Big Data Secret

Silent but busy! I have been pretty quiet in the world of data warehousing but for good reason. I have been in Decooda’s innovation lab building a Massively Parallel Processing platform for analytics and complex event processing. All I can say is that it is really quite amazing. We created an amazing infrastructure that scales incredibly well and takes advantage of multi-core processors in a very efficient way. (Yes, we avoid threading)

I can’t really say too much about the platform at this point in time, but it has the potential of being a MAJOR game changer because of its simplicity and performance. I am pretty confident that you will not find a simpler solution that scales as well as our platform.

As a use case, we will be setting up our dynamic real-time grid in a pretty beefy High Performance Computing center (HPC). We will be receiving over 250 Million documents per day and applying the majority of 100 different algorithms in real-time and in PARALLEL.

I can see this platform sitting along side of current investments like Netezza, Vertica, Greenplum, etc.. The potential role within data warehousing is to have this platform play as the stored procedure or UDF enablement engine along side of the existing warehouse. It’s just philosophically wrong to embed business logic processing within your database and this platform addresses that concern. Imagine having a 250 million++ record result passed on from Vertica that you would like to perform complex event processing on. Complex Algorithms, Calculation, notification, integration, scoring, tagging, AND linguistically analyzed NOW all made possible in PARALLEL without impacting the performance of your existing warehouse.

I am a co-founder and CTO at decooda.com and have been working very closely with David Johnson, CEO to bring some of his forward thinking concepts to the market. He is laser focused on Solutions and I am laser focused on technology around big-data. The blend of the two is extremely complimentary.

The Parallel platform will be launched by Decooda.com and is currently being used to support the market research applications. The first application to market is the analysis of open-end text in the form of survey responses and the same parallel processing platform is being used to analyze the social media and blogosphere.