Top Linux Links You Must Click On


Greenplum Claims Breakthrough in Data Loading
Company claims loading speeds of over 4TB an hour in production

Greenplum, the high-end open source database house and friend of Sun – so it’s gotta be nibbling on its nails wondering whether it’s lost a channel – has some sexy new technology to accelerate data loading for companies stressing under exponential data growth.

It’s called MPP Scatter/Gather Streaming or SG Streaming for short and it’s supposed to eliminate those darn bottlenecks usually associated with mainstream data loading.

Greenplum claims to have created the lightening-fast flow of data into its database for large-scale analytics and data warehousing. It says it’s getting loading speeds of over 4TB an hour in production with a negligible impact on concurrent database operations.

Let’s pause for a moment to consider that one terabyte is equal to a goose bump-provoking million books and that Greenplum’s claimed rates move the industry closer to real-time data warehousing.

Greenplum uses a shower head-like parallel-everywhere approach to loading in which the data flows from hundreds or thousands of parallel streams to every node of the database without any sequential choke points, quite different from the usual drip-drip-drip bulk-loading single-source stream like Oracle.

And the widgetry scales. The more nodes, the faster the loading rate so it can theoretically support better than 4TB an hour.

Data can be transformed and processed in-flight for extremely high-performance ELT (extract-load-transform) and ETLT (extract-transform-load-transform) loading pipelines.

The company says its precedent-setting approach avoids the need for a “loader” tier of servers (think MPP) that adds complexity and cost.

The widgetry supports both large batch and continuous near-real-time loading. Final gathering and storage of data takes place on all nodes simultaneously with data – compression is an option – automatically partitioned across nodes.

Greenplum says other parallel databases are as limited as traditional databases. Netezza, for instance, forces data to enter the system via a single node.

About Maureen O'Gara
Maureen O'Gara the most read technology reporter for the past 20 years, is the Cloud Computing and Virtualization News Desk editor of SYS-CON Media. She is the publisher of famous "Billygrams" and the editor-in-chief of "Client/Server News" for more than a decade. One of the most respected technology reporters in the business, Maureen can be reached by email at maureen(at)sys-con.com or paperboy(at)g2news.com, and by phone at 516 759-7025. Twitter: @MaureenOGara

In order to post a comment you need to be registered and logged in.

Register | Sign-in

Reader Feedback: Page 1 of 1

  Subscribe to our RSS feeds now and receive the next article instantly!
In It? Reprint It! Contact advertising(at)sys-con.com to order your reprints!
ADS BY GOOGLE
Subscribe to the World's Most Powerful Newsletters