Hadoop: And then there were three

Hortonworks, IBM, EMC Pivotal and others have announced a project called “Open Data Platform” to do … well, I’m not exactly sure what. Mainly, it sounds like:

• An attempt to minimize the importance of any technical advantages Cloudera or MapR might have.
• A face-saving way to admit that IBM’s and Pivotal’s insistence on having their own Hadoop distributions has been silly.
• An excuse for press releases.
• A source of an extra logo graphic to put on marketing slides.

Edit: Now there’s a press report saying explicitly that Hortonworks is taking over Pivotal’s Hadoop distro customers (which basically would mean taking over the support contracts and then working to migrate them to Hortonworks’ distro).

The claim is being made that this announcement solves some kind of problem about developing to multiple versions of the Hadoop platform, but to my knowledge that’s a problem rarely encountered in real life. When you already have a multi-enterprise open source community agreeing on APIs (Application Programming interfaces), what API inconsistency remains for a vendor consortium to painstakingly resolve?

Anyhow, it now seems clear that if you want to use a Hadoop distribution, there are three main choices:

• Cloudera’s flavor, whether as software (from Cloudera) or in an appliance (e.g. from Oracle).
• MapR’s flavor, as software from MapR.
• Hortonworks’ flavor, from a number of vendors, including Hortonworks, IBM, Pivotal, Teradata et al.

In saying that, I’m glossing over a few points, such as: Read more

MongoDB 3.0

Old joke:

• Question: Why do policemen work in pairs?
• Answer: One to read and one to write.

A lot has happened in MongoDB technology over the past year. For starters:

• The big news in MongoDB 3.0* is the WiredTiger storage engine. The top-level claims for that are that one should “typically” expect (individual cases can of course vary greatly):
  • 7-10X improvement in write performance.
  • No change in read performance (which however was boosted in MongoDB 2.6).
  • ~70% reduction in data size due to compression (disk only).
  • ~50% reduction in index size due to compression (disk and memory both).
• MongoDB has been adding administration modules.
  • A remote/cloud version came out with, if I understand correctly, MongoDB 2.6.
  • An on-premise version came out with 3.0.
  • They have similar features, but are expected to grow apart from each other over time. They have different names.

*Newly-released MongoDB 3.0 is what was previously going to be MongoDB 2.8. My clients at MongoDB finally decided to give a “bigger” release a new first-digit version number.

To forestall confusion, let me quickly add: Read more

Context for Cloudera

Hadoop World/Strata is this week, so of course my clients at Cloudera will have a bunch of announcements. Without front-running those, I think it might be interesting to review the current state of the Cloudera product line. Details may be found on the Cloudera product comparison page. Examining those details helps, I think, with understanding where Cloudera does and doesn’t place sales and marketing focus, which given Cloudera’s Hadoop market stature is in my opinion an interesting thing to analyze.

So far as I can tell (and there may be some errors in this, as Cloudera is not always accurate in explaining the fine details):

• CDH (Cloudera Distribution … Hadoop) contains a lot of Apache open source code.
• Cloudera has a much longer list of Apache projects that it thinks comprise “Core Hadoop” than, say, Hortonworks does.
  • Specifically, that list currently is: Hadoop, Flume, HCatalog, Hive, Hue, Mahout, Oozie, Pig, Sentry, Sqoop, Whirr, ZooKeeper.
  • In addition to those projects, CDH also includes HBase, Impala, Spark and Cloudera Search.
• Cloudera Manager is closed-source code, much of which is free to use. (I.e., “free like beer” but not “free like speech”.)
• Cloudera Navigator is closed-source code that you have to pay for (free trials and the like excepted).
• Cloudera Express is Cloudera’s favorite free subscription offering. It combines CDH with the free part of Cloudera Manager. Note: Cloudera Express was previously called Cloudera Standard, and that terminology is still reflected in parts of Cloudera’s website.
• Cloudera Enterprise is the umbrella name for Cloudera’s three favorite paid offerings.
• Cloudera Enterprise Basic Edition contains:
  • All the code in CDH and Cloudera Manager, and I guess Accumulo code as well.
  • Commercial licenses for all that code.
  • A license key to use the entirety of Cloudera Manager, not just the free part.
  • Support for the “Core Hadoop” part of CDH.
  • Support for Cloudera Manager. Note: Cloudera is lazy about saying this explicitly, but it seems obvious.
  • The code for Cloudera Navigator, but that’s moot, as the corresponding license key for Cloudera Navigator is not part of the package.
• Cloudera Enterprise Data Hub Edition contains:
  • Everything in Cloudera Basic Edition.
  • A license key for Cloudera Navigator.
  • Support for all of HBase, Accumulo, Impala, Spark, Cloudera Search and Cloudera Navigator.
• Cloudera Enterprise Flex Edition contains everything in Cloudera Basic Edition, plus support for one of the extras in Data Hub Edition.

In analyzing all this, I’m focused on two particular aspects:

• The “zero, one, many” system for defining the editions of Cloudera Enterprise.
• The use of “Data Hub” as a general marketing term.

Read more

21st Century DBMS success and failure

As part of my series on the keys to and likelihood of success, I outlined some examples from the DBMS industry. The list turned out too long for a single post, so I split it up by millennia. The part on 20th Century DBMS success and failure went up Friday; in this one I’ll cover more recent events, organized in line with the original overview post. Categories addressed will include analytic RDBMS (including data warehouse appliances), NoSQL/non-SQL short-request DBMS, MySQL, PostgreSQL, NewSQL and Hadoop.

DBMS rarely have trouble with the criterion “Is there an identifiable buying process?” If an enterprise is doing application development projects, a DBMS is generally chosen for each one. And so the organization will generally have a process in place for buying DBMS, or accepting them for free. Central IT, departments, and — at least in the case of free open source stuff — developers all commonly have the capacity for DBMS acquisition.

In particular, at many enterprises either departments have the ability to buy their own analytic technology, or else IT will willingly buy and administer things for a single department. This dynamic fueled much of the early rise of analytic RDBMS.

Buyer inertia is a greater concern.

• A significant minority of enterprises are highly committed to their enterprise DBMS standards.
• Another significant minority aren’t quite as committed, but set pretty high bars for new DBMS products to cross nonetheless.
• FUD (Fear, Uncertainty and Doubt) about new DBMS is often justifiable, about stability and consistent performance alike.

A particularly complex version of this dynamic has played out in the market for analytic RDBMS/appliances.

• First the newer products (from Netezza onwards) were sold to organizations who knew they wanted great performance or price/performance.
• Then it became more about selling “business value” to organizations who needed more convincing about the benefits of great price/performance.
• Then the behemoth vendors became more competitive, as Teradata introduced lower-price models, Oracle introduced Exadata, Sybase got more aggressive with Sybase IQ, IBM bought Netezza, EMC bought Greenplum, HP bought Vertica and so on. It is now hard for a non-behemoth analytic RDBMS vendor to make headway at large enterprise accounts.
• Meanwhile, Hadoop has emerged as serious competitor for at least some analytic data management, especially but not only at internet companies.

Otherwise I’d say:  Read more

Notes and comments, May 6, 2014

After visiting California recently, I made a flurry of posts, several of which generated considerable discussion.

• My claim that Spark will replace Hadoop MapReduce got much Twitter attention — including some high-profile endorsements — and also some responses here.
• My MemSQL post led to a vigorous comparison of MemSQL vs. VoltDB.
• My post on hardware and storage spawned a lively discussion of Hadoop hardware pricing; even Cloudera wound up disagreeing with what I reported Cloudera as having said. 🙂 Sadly, there was less response to the part about the partial (!) end of Moore’s Law.
• My Cloudera/SQL/Impala/Hive apparently was well-balanced, in that it got attacked from multiple sides via Twitter & email. Apparently, I was too hard on Impala, I was too hard on Hive, and I was too hard on boxes full of cardboard file cards as well.
• My post on the Intel/Cloudera deal garnered a comment reminding us Dell had pushed the Intel distro.
• My CitusDB post picked up a few clarifying comments.

Here is a catch-all post to complete the set.  Read more

Hortonworks business notes

Hortonworks did a business-oriented round of outreach, talking with at least Derrick Harris and me. Notes  from my call — for which Rob Bearden didn’t bother showing up — include, in no particular order:

• Hortonworks denies advanced acquisition discussions with either Microsoft and Intel. Of course, that doesn’t exactly contradict the widespread story of Intel having made an acquisition offer. Edit: I have subsequently heard, very credibly, that the denial was untrue.
• As vendors usually do, Hortonworks denies the extreme forms of Cloudera’s suggestion that Hortonworks competitive wins relate to price slashing. But Hortonworks does believe that its license fees often wind up being lower than Cloudera’s, due especially to Hortonworks offering few extra-charge items than Cloudera.
• Hortonworks used a figure of ~75 subscription customers. Edit: That figure turns out in retrospect to have been inflated. This does not include OEM sales through, for example, Teradata, Microsoft Azure, or Rackspace. However, that does include …
• … a small number of installations hosted in the cloud — e.g. ~2 on Amazon Web Services — or otherwise remotely. Also, testing in the cloud seems to be fairly frequent, and the cloud can also be a source of data ingested into Hadoop.
• Since Hortonworks a couple of times made it seem that Rackspace was an important partner, behind only Teradata and Microsoft, I finally asked why. Answers boiled down to a Rackspace Hadoop-as-a-service offering, plus joint work to improve Hadoop-on-OpenStack.
• Other Hortonworks reseller partners seem more important in terms of helping customers consume HDP (Hortonworks Data Platform), rather than for actually doing Hortonworks’ selling for it. (This is unsurprising — channel sales rarely are a path to success for a product that is also appropriately sold by a direct force.)
• Hortonworks listed its major industry sectors as:
  • Web and retailing, which it identifies as one thing.
  • Media.
  • Telecommunications.
  • Health care (various subsectors).
  • Financial services, which it called “competitive” in the kind of tone that usually signifies “we lose a lot more than we win, and would love to change that”.

In Hortonworks’ view, Hadoop adopters typically start with a specific use case around a new type of data, such as clickstream, sensor, server log, geolocation, or social.  Read more

Hortonworks, Hadoop, Stinger and Hive

I chatted yesterday with the Hortonworks gang. The main subject was Hortonworks’ approach to SQL-on-Hadoop — commonly called Stinger —  but at my request we cycled through a bunch of other topics as well. Company-specific notes include:

• Hortonworks founder J. Eric “Eric14” Baldeschwieler is no longer at Hortonworks, although I imagine he stays closely in touch. What he’s doing next is unspecified, except by the general phrase “his own thing”. (Derrick Harris has more on Eric’s departure.)
• John Kreisa still is at Hortonworks, just not as marketing VP. Think instead of partnerships and projects.
• ~250 employees.
• ~70-75 subscription customers.

Our deployment and use case discussions were a little confused, because a key part of Hortonworks’ strategy is to support and encourage the idea of combining use cases and workloads on a single cluster. But I did hear:

• 10ish nodes for a typical starting cluster.
• 100ish nodes for a typical “data lake” committed adoption.
• Teradata UDA (Unified Data Architecture)* customers sometimes (typically?) jumping straight to a data lake scenario.
• A few users in the 10s of 1000s of nodes. (Obviously Yahoo is one.)
• HBase used in >50% of installations.
• Hive probably even more than that.
• Hortonworks is seeing a fair amount of interest in Windows Hadoop deployments.

*By the way — Teradata seems serious about pushing the UDA as a core message.

Ecosystem notes, in Hortonworks’ perception, included:

• Cloudera is obviously Hortonworks’ biggest distro competitor. Next is IBM, presumably in its blue-forever installed base. MapR is barely on the radar screen; Pivotal’s likely rise hasn’t yet hit sales reports.
• Hortonworks evidently sees a lot of MicroStrategy and Tableau, and some Platfora and Datameer, the latter two at around the same level of interest.
• Accumulo is a big deal in the Federal government, and has gotten a few health care wins as well. Its success is all about security. (Note: That’s all consistent with what I hear elsewhere.)

I also asked specifically about OpenStack. Hortonworks is a member of the OpenStack project, contributes nontrivially to Swift and other subprojects, and sees Rackspace as an important partner. But despite all that, I think strong Hadoop/OpenStack integration is something for the indefinite future.

Hortonworks’ views about Hadoop 2.0 start from the premise that its goal is to support running a multitude of workloads on a single cluster. (See, for example, what I previously posted about Tez and YARN.) Timing notes for Hadoop 2.0 include:

• It’s been in preview/release candidate/commercial beta mode for weeks.
• Q3 is the goal; H2 is the emphatic goal.
• Yahoo’s been in production with YARN >8 months, and has no MapReduce 1 clusters left. (Yahoo has >35,000 Hadoop nodes.)
• The last months of delays have been mainly about sprucing up various APIs and protocols, which may need to serve for a similar multi-year period as Hadoop 1’s have. But there also was some YARN stabilization into May.

Frankly, I think Cloudera’s earlier and necessarily incremental Hadoop 2 rollout was a better choice than Hortonworks’ later big bang, even though the core-mission aspect of Hadoop 2.0 is what was least ready. HDFS (Hadoop Distributed File System) performance, NameNode failover and so on were well worth having, and it’s more than a year between Cloudera starting supporting them and when Hortonworks is offering Hadoop 2.0.

Hortonworks’ approach to doing SQL-on-Hadoop can be summarized simply as “Make Hive into as good an analytic RDBMS as possible, all in open source”. Key elements include:  Read more

WibiData and its Kiji technology

My clients at WibiData:

• Think they’re an application software company …
• … but actually are talking about what I call analytic application subsystems.
• Haven’t announced or shipped any of those either …
• … but will shortly.
• Have meanwhile shipped some cool enabling technology.
• Name their products after sushi restaurants.

Yeah, I like these guys. 🙂

If you’re building an application that “obviously” calls for a NoSQL database, and which has a strong predictive modeling aspect, then WibiData has thought more cleverly about what you need than most vendors I can think of. More precisely, WibiData has thought cleverly about your data management, movement, crunching, serving, and integration. For pure modeling sophistication, you should look elsewhere — but WibiData will gladly integrate with or execute those models for you.

WibiData’s enabling technology, now called Kiji, is a collection of modules, libraries, and so on — think Spring — running over Hadoop/HBase. Except for some newfound modularity, it is much like what I described at the time of WibiData’s launch or what WibiData further disclosed a few months later. Key aspects include:

• A way to define schemas in HBase, including ones that change as rapidly as consumer-interaction applications require.
• An analytic framework called “Produce/Gather”, which can execute at human real-time speeds (via its own execution engine) or with higher throughput in batch mode (by invoking Hadoop MapReduce).
• Enough load capabilities, Hive interaction, and so on to get data into the proper structure in Kiji in the first place.

Read more

Notes on TokuDB and GenieDB

Last week, I edited press releases back-to-back-to-back for three clients, all with announcements at this week’s Percona Live. The ones with embargoes ending today are Tokutek and GenieDB.

Tokutek’s news is that they’re open sourcing much of TokuDB, but holding back hot backup for their paid version. I approve of this strategy — “doesn’t lose data” is an important feature, and well worth paying for.

I kid, I kid. Any system has at least a bad way to do backups — e.g. one that involves slowing performance, or perhaps even requires taking applications offline altogether. So the real points of good backup technology are:

• To keep performance steady.
• To make the whole thing as easy to manage as possible.

GenieDB is announcing a Version 2, which is basically a performance release. So in lieu of pretending to have much article-worthy news, GenieDB is taking the opportunity to remind folks of its core marketing messages, with catchphrases such as “multi-regional self-healing MySQL”. Good choice; indeed, I wish more vendors would adopt that marketing tactic.

Along the way, I did learn a bit more about GenieDB. In particular:

• GenieDB is now just backed by a hacked version of InnoDB (no more Berkeley DB Java Edition).
• Why hacked? Because GenieDB appends a Lamport timestamp to every row, which somehow leads to a need to modify how indexes and caching work.
• Benefits of the chamge include performance and simpler (for the vendor) development.
• An arguable disadvantage of the switch is that GenieDB no longer can use Berkeley DB’s key-value interface — but MySQL now has one of those too.

I also picked up some GenieDB company stats I didn’t know before — 9 employees and 2 paying customers.

Related links

• Recent posts about TokuDB and GenieDB

Open source strategies

From time to time I advise a software vendor on how, whether, or to what extent it should offer its technology in open source. In summary, I believe:

• The formal differences between “open source” and “closed source” strategies are of secondary importance.
• The attitudinal and emotional differences between “open source” and “closed source” approaches can be large.
• A pure closed source strategy can make sense.
• A closed source strategy with important open source aspects can make sense.
• A pure open source strategy will only rarely win.

Here’s why.

An “open source software” business model and strategy might include:

• Software given away for free.
• Demand generation to encourage people to use the free version of the software.
• Subscription pricing for additional proprietary software and support.
• Direct sales, and further marketing, to encourage users of the free stuff to upgrade to a paid version.

A “closed source software” business model and strategy might include:

• Demand generation.
• Free-download versions of the software.
• Subscription pricing for software (increasingly common) and support (always).
• Direct sales, and associated marketing.

Those look pretty similar to me.

Of course, there can still be differences between open and closed source. In particular: Read more

← Previous Page — Next Page →

Subscribe to the Monash Research feed via RSS or email:

Login

Search our blogs and white papers

Monash Research blogs

• DBMS 2 covers database management, analytics, and related technologies.
• Text Technologies covers text mining, search, and social software.
• Strategic Messaging analyzes marketing and messaging strategy.
• The Monash Report examines technology and public policy issues.
• Software Memories recounts the history of the software industry.

User consulting

Building a short list? Refining your strategic plan? We can help.

Vendor advisory

We tell vendors what's happening -- and, more important, what they should do about it.

Monash Research highlights

Learn about white papers, webcasts, and blog highlights, by RSS or email.

• Recent posts

  • Political issues around big tech companies
  • How to beat “fake news”
  • New legal limits on surveillance in the US
  • Brittleness, Murphy’s Law, and single-impetus failures
  • Brittleness and incremental improvement

• Categories

  • About this blog
  • Analytic glossary
  • Analytic technologies
    • Business intelligence
    • Data mart outsourcing
    • Data warehousing
    • MOLAP
    • Predictive modeling and advanced analytics
  • Application areas
    • Games and virtual worlds
    • Health care
    • Investment research and trading
    • Log analysis
    • Scientific research
    • Telecommunications
    • Web analytics
  • Buying processes
    • Benchmarks and POCs
  • Companies and products
    • 1010data
    • Ab Initio Software
    • Actian and Ingres
    • Aerospike
    • Akiban
    • Aleri and Coral8
    • Algebraix
    • Alpha Five
    • Amazon and its cloud
    • ANTs Software
    • Aster Data
    • Ayasdi
    • Basho and Riak
    • Business Objects
    • Calpont
    • Cassandra
    • Cast Iron Systems
    • Cirro
    • Citus Data
    • ClearStory Data
    • Cloudant
    • Cloudera
    • Clustrix
    • Cogito and 7 Degrees
    • Cognos
    • Continuent
    • Couchbase
    • CouchDB
    • Databricks, Spark and BDAS
    • DATAllegro
    • Datameer
    • DataStax
    • Dataupia
    • dbShards and CodeFutures
    • Elastra
    • EMC
    • Endeca
    • EnterpriseDB and Postgres Plus
    • Exasol
    • Expressor
    • FileMaker
    • GenieDB
    • Gooddata
    • Google
    • Greenplum
    • Groovy Corporation
    • Hadapt
    • Hadoop
    • HBase
    • Hortonworks
    • HP and Neoview
    • IBM and DB2
      • pureXML
    • illuminate Solutions
    • Infobright
    • Informatica
    • Information Builders
    • Inforsense
    • Intel
    • Intersystems and Cache'
    • Jaspersoft
    • Kafka and Confluent
    • Kalido
    • Kaminario
    • Kickfire
    • Kognitio
    • KXEN
    • MapR
    • MarkLogic
    • McObject
    • memcached
    • MemSQL
    • Metamarkets and Druid
    • Microsoft and SQL*Server
    • MicroStrategy
    • MonetDB
    • MongoDB
    • MySQL
    • Neo Technology and Neo4j
    • Netezza
    • NuoDB
    • Nutonian
    • Objectivity and Infinite Graph
    • Oracle
      • Exadata
    • Oracle TimesTen
    • ParAccel
    • Pentaho
    • Pervasive Software
    • PivotLink
    • Platfora
    • PostgreSQL
    • Progress, Apama, and DataDirect
    • QlikTech and QlikView
    • Rainstor
    • Revolution Analytics
    • Rocana
    • salesforce.com
    • SAND Technology
    • SAP AG
    • SAS Institute
    • ScaleBase
    • ScaleDB
    • Schooner Information Technology
    • SciDB
    • SenSage
    • SequoiaDB
    • SnapLogic
    • Software AG
    • solidDB
    • Splunk
    • Starcounter
    • StreamBase
    • Sybase
    • Syncsort
    • Tableau Software
    • Talend
    • Teradata
    • Tokutek and TokuDB
    • Truviso
    • VectorWise
    • Vertica Systems
    • VoltDB and H-Store
    • WibiData
    • Workday
    • Xkoto
    • XtremeData
    • Yarcdata and Cray
    • Zettaset
    • Zoomdata
  • Data integration and middleware
    • Application servers
    • EAI, EII, ETL, ELT, ETLT
  • Data types
    • GIS and geospatial
    • Object
    • RDF and graphs
    • Structured documents
    • Text
  • DBMS product categories
    • Archiving and information preservation
    • Data warehouse appliances
    • Mid-range
    • NewSQL
    • OLTP
    • Open source
  • Emulation, transparency, portability
  • Fun stuff
    • Humor
  • Market share and customer counts
  • Memory-centric data management
    • Cache
    • In-memory DBMS
    • Streaming and complex event processing (CEP)
  • Michael Stonebraker
  • Parallelization
    • Clustering
    • MapReduce
    • Transparent sharding
  • Presentations
  • Pricing
  • Public policy
    • Surveillance and privacy
  • Software as a Service (SaaS)
    • Cloud computing
  • Specific users
    • eBay
    • Facebook
    • Fox and MySpace
    • JPMorgan Chase
    • TEOCO
    • Yahoo
    • Zynga
  • Storage
    • Solid-state memory
  • Theory and architecture
    • Columnar database management
    • Data models and architecture
    • Data pipelining
    • Database compression
    • Database diversity
    • Derived data
    • NoSQL
    • Petabyte-scale data management
    • Schema on need
    • SQL/Hadoop integration
    • Workload management
  • TransRelational
  • Uncategorized

• Date archives

  Select Month June 2019 June 2018 May 2018 February 2018 January 2018 December 2017 August 2017 June 2017 April 2017 March 2017 February 2017 December 2016 November 2016 October 2016 September 2016 August 2016 July 2016 June 2016 May 2016 February 2016 January 2016 December 2015 November 2015 October 2015 September 2015 August 2015 July 2015 June 2015 May 2015 April 2015 March 2015 February 2015 January 2015 December 2014 November 2014 October 2014 September 2014 August 2014 July 2014 June 2014 May 2014 April 2014 March 2014 February 2014 January 2014 December 2013 November 2013 October 2013 September 2013 August 2013 July 2013 June 2013 May 2013 April 2013 March 2013 February 2013 January 2013 December 2012 November 2012 October 2012 September 2012 August 2012 July 2012 June 2012 May 2012 April 2012 March 2012 February 2012 January 2012 November 2011 October 2011 September 2011 August 2011 July 2011 June 2011 May 2011 April 2011 March 2011 February 2011 January 2011 December 2010 November 2010 October 2010 September 2010 August 2010 July 2010 June 2010 May 2010 April 2010 March 2010 February 2010 January 2010 December 2009 November 2009 October 2009 September 2009 August 2009 July 2009 June 2009 May 2009 April 2009 March 2009 February 2009 January 2009 December 2008 November 2008 October 2008 September 2008 August 2008 July 2008 June 2008 May 2008 April 2008 March 2008 February 2008 January 2008 December 2007 November 2007 October 2007 September 2007 August 2007 July 2007 June 2007 May 2007 April 2007 March 2007 February 2007 January 2007 December 2006 November 2006 October 2006 September 2006 August 2006 July 2006 June 2006 May 2006 April 2006 March 2006 February 2006 January 2006 December 2005 November 2005 October 2005 September 2005 August 2005

• Links

  • Monash Research
  • White Papers

• Admin

  • Log in

• Home
• About
• Contact
• Feeds