Working with Online Data

A chapter on data? Yes, this might seem strange at first. It makes us wonder how hard it can be to work with a TSV (tab-delimited values) or a CSV (comma-separated values) file format. Unfortunately, data files often do not present themselves so nicely. In our case, for our first dashboard, we use the government’s ATADS (Air Traffic Activity System) data, and it is accessed by going to a web interface, selecting various configuration options on different pages, and then deciding what format to download – between HTML, Word, and Excel. (Since their audience includes airline and airport managers, this is understandable.) Therefore, it is useful to know how to work with site navigation to grab online data (screen scraping). Even then, there are problems with the ATADS data – we would like to translate it into CSV format and be careful with embedded numeric commas, so “1,234” gets translated into “1234”; extra commas are not good in a CSV file.