Who Will Drive Data Driven Documents?

Check out this javascript library D3 for “Data Driven Documents” (D3 on Github).  At first D3 seems to be just yet another way to add graphs and charts to web pages, but if you spend a few minutes soaking it in you can see how it might change how people think about data and documents altogether.

Example data diagrams produced with D3 javascript library

Like anything that changes our mental paradigm it takes a bit of noodling to wrap your head around it. D3 is similar to basic jQuery with the twist that you can add and transform data attached to arbitrary DOM elements, then use that data to drive the visualization and behavior of the DOM dynamically.

D3.js is a small, free JavaScript library for manipulating documents based on data.

D3 allows you to bind arbitrary data to a Document Object Model (DOM), and then apply data-driven transformations to the document. As a trivial example, you can use D3 to generate a basic HTML table from an array of numbers. Or, use the same data to create an interactive SVG bar chart with smooth transitions and interaction.

Data Driven Functions

There are some clever things to be done in just a few lines of code when you use D3 to map what might normally be static attributes of your CSS/HTML/SVG (or other DOM elements) to data driven functions.  D3 provides: Continue reading

MongoDB – Storing Big Data Documents

If a Big Data set (or smaller data) is in the form of documents, then it’s difficult to store them in a traditional schema-defined row and column database. Sure, you can create large blob fields to hold large arbitrary chunks of data, serialize and encode the document in some way, or just store them in a filesystem, but those options aren’t much good for querying or analysis when the data gets big.

MongoDB Document Database

MongoDB is a great example of a document database. There are no predefined schemas for tables (the schema is considered “dynamic”). Rather you declare “collections” and insert or update documents directly into each collection.

A document in this case is basically JSON with some extensions (actually BSON – Binary encoded JSON). It supports nested arrays and other things that you wouldn’t find in a relational database. If you are object oriented, this is fundamentally an object store.

Documents added to a single collection can vary widely from each other in terms of content and composition/structure (although an application layer above could obviously enforce consistency as happens when MongoDB is used under Rails).

MongoDB’s list of key features is a fantastic mini-tutorial in itself: Continue reading

Documents Are Data

A graphical despiction of a very simple xhtml ...

Image via Wikipedia

If you think of an HTML page as a structured “marked up” document, it’s basically a form of data. The structure, in this case represented by HTML tags like <a> and <div>, identifies various document elements which can be interpreted as data fields. In fact, strict HTML is good XML (referred to as XHTML), a well-understood data format.

Think of “documents” as structured data where the structure is included in the document itself. The structure is free-form in the sense that the document author decides what data fields are included, how they are organized, ordered, nested, related, and so on. If you are object-oriented, you can also view each document as an object (technically for web pages this is referred to as the DOM – “Document Object Model“).

Big Data in a Small World

Google perhaps lead the way into this brave new world with their proprietary “Big Table” database architecture backing many of their services.  Apache’s Hadoop project (includes HDFS and HBase) is fundamentally based on Google’s open papers.

Most big data sets in our small world are going to be produced by numerous (countless?) authors and applications. Most big data is going to be in the form of documents rather than standardized (i.e. described by external schema) transactional data. Since most current data handling, storage, and analysis technology is aimed at transactional schema controlled data, I’m thrilled to explore today’s emerging market of new “big data” solutions.

Enhanced by Zemanta