Introduction
Welcome to Data Mining the City! The purpose of this class is to explore how new technologies such as Big Data, Data Mining, and Machine Learning could be applied to urban analysis to allow us to understand our cities better. The goal of this series of tutorials is to guide you through the creation of a minimal proof of concept that will show some of the possibilies of these new technologies, and provide a basic framework that you can expand for your own research.
From a technological standpoint, one of the goals of the class is to propose the next evolution of software for urban analysis. Currently most urban analysis is done within a handful of software packages known as Geographic Information Systems or GIS. These are, for the most part, fairly traditional software packages, released closed source by large corporations, and costing up to several thousands of dollars to use. One notable exception is a collection of new open-sourced tools such as QGIS, which can be used for free, and extended from the source code. However, ability to extend the code base requires knowledge in computer porgramming, and is out of reach for most urban analysis professionals.
Web 2.0 -> 3.0
While GIS software has changed relatively little in the last two decades, the growth of the internet has slowly revolutionized the way that most software is developed, acquired, and used. While the internet began as a series of static web pages that could be accessed from within a user’s browser, over time new technologies were developed that could offload some of the more computationaly expensive aspects of a website such as data storage and processing onto a ‘backend’ server existing remotely. Meanwhile, the user’s ‘frontend’ could be kept light, reducing the need for powerful computers on the user end. This separation between ‘frontend’ visualization and ‘backend’ data storage and processing was referred to as ‘Web 2.0’, and made possible a host of new web-based services such as social networks and online retail that we rely on today.
In recent years, however, this concept of a distributed web framework has been extended into applications previously handled by traditional software. For example, Google Docs can accomplish many of the same tasks as a traditional word processor such as Microsoft Word, but in a way that is much lighter from the user perspective. Because everything happens within a web browser, there is no software to install and no files that need to be stored locally. Meanwhile, all file storage and processing happens on Google’s own server computers, which are housed inside large remote warehouses and organized into huge clusters. This kind of distributed infrastructure is known as ‘Cloud Computing’, bacuse most of the computational work happens not on the user’s local computer but by a ‘cloud’ of distributed and interconnected computers elsewhere. This technology has introduced the concept of ‘Web 3.0’, which envisions that all computing tasks will one day be relegated to an external cloud, with the user requiring only a very basic terminal (such as a tablet or basic laptop) to access this wealth of resources. Besides minimizing the need for powerful computers on the user end, ‘cloud-based’ software can also take advantage of many useful features such as document storage, file sharing, and real-time collaboration that would be impossible with traditional software.
While these developments are effecting many types of software, they are also particularly important to the future of urban analysis software and GIS. Although some GIS packages such as ArcGIS have released versions that utilize cloud storage and a web-based interface, they have only adopted certain portions of the technology without embracing its full potential. There have also been a growing number of web-based mapping tools such as Mapbox and CartoDB, but these are mostly geared towards making maps and not robust enough to support real-world analysis workflows. Like their traditional counterparts, they are also closed source and limited to prescribed use cases, and cannot be extended into custom tools to support individual workflows.
The Web Stack
This class will study a wide range of the software and technologies that power Web applications today, and will make an initial attempt to appropriate them into a framework for urban analysis. Because of their distributed nature, modern Web applications do not rely on a single framework, but on a suite of technologies working together. This suite of technologies is commonly referred to as a ‘Web Stack’, and describes both the software that supports various parts of the system, as well as the communication that ties them together.
At it’s most basic a Web Stack is composed of:
- a database that stores data
- a back-end server that can query the database in realtime and perform data analysis and processing operations
- a front-end client accessed through a web browser which requests data from the server and presents it to the user
The basic technologies comprising the ‘Web Stack’ that will be developed during this course.
The following sequence of tutorials will provide a basic overview of three primary pieces of technology, each related to a component of a the Web Stack. For the database we will use OrientDB, a very modern and flexible database system utlizing the principles of NoSQL. For the back-end processing we will use Python, one of the most popular and flexible programming languages around. For the user interface we will use JavaScript, the undisputed standard of interactive Web design. Although mastery of these platforms is beyond the scope of this class, we will take a hands-on approach and learn just enough of the basics to build a fully functional Web Stack, specifically geared towards the needs of urban analysis. Although this won’t replicate all the functionalities of a full GIS tool, it will by it’s nature be open, extensible, and ‘born in the cloud’.