Massachusetts Institute of Technology
Department of Urban Studies and Planning

Lab 8: Experimenting with Geospatial Services

This lab is optional

Lab 8 in-class notes

Objectives

1. To understand the layering of data, application tools, web mapping interoperability standards, and web service protocols involved in modern internet GIS applications.
2. To explore various online resources that may be a source of data for your individual project.
3. To compare alternative strategies for utilizing geospatial services, from spatial data search with GIS download to direct online use.
4. To examine a few geospatial services "under the covers" in order to understand the nature and purpose of the URL requests and XML responses between the clients and web services.
5. The lab Assignment involves creating a map in Part III-B and choosing ONE of the four options in Part IV

Overarching issues (when examining online GIS data repositories and web services)

• Audience: Whom does the system serve? Does it serve them well? How might you systematically assess this?
• Thematic Content: What are the primary types of data available? Is metadata available describing currentness, methods, scale, projection, etc?
• Background Content: What other incidental content is available? Can the background information be separated from the foreground?
• Basic Data Type: Is it a picture of a map? Unclassified imagery like an orthophoto? Classified grid data? Points, lines and polygon vectors?
• Focus and Flexibility: What other uses are supported? Can users download computable data? Can users upload or exchange data?
• Fitness for Use: Does the data/interface allow cartographic manipulation? Printing? Editing? Analysis?
• Price / Licensing: Is this data available free or at cost? Even if free, what are the licensing restrictions? How do these issues affect public planning use, private firm use, citizen use?

Tips and Tricks

Not all web services and web sites allow you to examine or determine what they are doing behind the scenes. However a significant number, especially those based on open standards, do allow this. Learn to parse URLs yourself (allows you to repurpose sites and services, debug errors)

Basic Rules for URLs and HTTP

1) Transactions are usually "stateless"

All of the information needed for the server to meet client's request is bundled into URL
(Exception: sometimes "cookies" are used to retain the client's 'state' based on a user ID or session ID)

2) Parsing URLs

The general URL format uses this sequence: protocol:[optional port number]://path/to/resource/scriptname?parameters
A question mark separates path/scriptname from parameters list

&parameter1=blah&parameter2=blah+again

The ampersand separates parameters and the equals sign sets the assigned value
Spaces are not allowed in URLS, so "+" symbol is substituted
To clarify and inspect, paste a URL into a text editor and substitute line breaks for question marks and ampersands

3) Example - using a standard Web Mapping Service (WMS) such as the MassGIS major road layer that we have been using during the semester:

http://giswebservices.massgis.state.ma.us/geoserver/wms?dpi=120&request=GetMap&layers=GISDATA.EOTMAJROADS_ARC&srs=EPSG:26986&bbox=233500,900000,236500,902500&width=1000&height=300&format=image/png&service=wms&transparent=true

Becomes

http://giswebservices.massgis.state.ma.us/geoserver/wms
dpi=120
request=GetMap
layers=GISDATA.EOTMAJROADS_ARC
srs=EPSG:26986
bbox=233500,900000,236500,902500
width=640
height=480
format=image/png
service=wms
transparent=true

Not self-explanatory, but a step in the right direction.

• dpi, width, and height specific the dots-per-inch on your screen and the width/height of your viewing window (in pixels)
• Note that the dpi, width/height, and bounding box (bbox) need to be in sync in order for the image on the screen to match the desired scale and aspect ratio
• In class, we used the European Petroleum Survey Group code 4326 for the World Geodesic Datum of 1984 (WGS84)
• In this case, we use the EPSG code 26986 for Mass State Plane coordinates (Mainland) NAD83
• Try this URL in a browser and try editing the bbox, width, and height values

For those who are interested, the full WMS description can be found here:
OpenGIS Web Map Service (WMS) Implementation Specification

 

Part I - Exploring Online Mapping Sites

A) Sites Designed Primarily for Online Use

Explore these sites briefly, noting the issues above and last week's lecture notes

EPA's Enviromapper:

• http://www.epa.gov/emefdata/em4ef.home
• Check out a city or zip code and examine various layers on this EPA site
• The 'toxic release inventory' sites in homework set #2 were obtained from these EPA databases
• Special permission required for access to underlying services (e.g., Oracle account)

EPA's Smart Location Database:

• https://www.epa.gov/smartgrowth/smart-location-mapping
• This database builds indicators for US census block groups that provide consistent measures of urban form, built environment, and residential population that may be helpful in understanding the relationships among land use, transportation access, and demographics.  
• Beware that the database is large and you will want to select a state or metro area before downloading.
• Check out the interactive map viewer (centered here on Washington DC):
  http://www.arcgis.com/home/webmap/viewer.html?webmap=137d4e512249480c980e00807562da10&extent=-77.2164,38.8286,-76.8799,39.0494

The Department of Housing and Urban Development's EZ/RC Locater:

• http://egis.hud.gov/ezrclocator/
• An interesting web map that allows users to determine whether an address is in an Empowerment Zone or a Renewal Community.

U.S. Census Bureau LED OnTheMap:

• http://lehdmap2.did.census.gov/
• OnTheMap is an online application to map Longitudinal Employer-Household Dynamics (LED) data. Using LED data, can create detailed maps showing where people work and where workers live with companion reports on worker ages, earnings, industry distribution, and local workforce indicators.

HealthyCity:

• http://www.healthycity.org/
• HealthCity offers a access to community resources, demographic, and health data for California. C

Census Data Mapper:

• http://www.census.gov/geo/maps-data/maps/datamapper.html
• "The Census Data Mapper is a web mapping application intended to provide users with a simple interface to view, save and print county-based demographic maps of the United States" using 2010 Census data.

U.S. Department of Transportation Federal Highway Administration

What is the annual average daily traffic volume on Mass. Ave?
http://hepgis.fhwa.dot.gov/hepgismaps11/ViewMap.aspx?map=Highway Information|2007 AADT#

Yes, there is a map on this site that color codes roads based on estimated traffic volume. However, the interface offers limited help capabilities and the right map is hard to find. Do not spend too long hunting for it. Just get a general sense of what the site offers and what you do/do-not like about the site capabilities and user interface.

Other Traffic info sites:

• 'Traffic' tab on Microsoft's Virtual Earth: http://local.live.com/
• NAVTEQ Traffic.com (real-time traffic congestion reporting): http://www.traffic.com/Chicago-Traffic/Chicago-Traffic-Reports.html

U.S. City Sites

Portland Maps (City of Portland, Oregon)

• http://www.portlandmaps.com/
• Extensive online mapping with detailed assessing and permitting information

Cambridge Maps (City of Cambridge, MA)

• http://www.cambridgema.gov/GIS/
• Check out the City Viewer (it may require Internet Explorer!):

Cartographic Modeling Lab: Neighborhood Information System (City of Philadelphia, PA)

• http://www.cml.upenn.edu/#!nis/c13cq
• Philadelphia NIS is a web-based property and social indicators information system used by city agencies and community based organizations throughout Philadelphia. NIS users research individual properties; run queries to locate comparables; plan, site and evaluate housing development programs; and study neighborhood conditions with user-defined maps, charts and reports.

B) Sites With Data Download Capabilities

The (US) National Map

http://nationalmap.gov/(cover page)
http://viewer.nationalmap.gov/viewer/ (direct link to mapping interface)
USGS-led effort at coherent, cross-agency national map layers

USGS

The National Map also provides access to selected international datasets such as the Global 90m Terrain Data that can be downloaded in a non-proprietary file format, SDTS, that most GIS packages will support.
(SDTS = Spatial Data Transfer Standard)

University of Maryland Global Land Cover Facility

http://glcf.umd.edu/ (overview)

Best place to download free Landsat imagery (warning: huge and unwieldy) and global 1km resolution land cover classification. U.S. Government data policy on Landsat has varied over the years. At the moment, access to ETM+ (enhanced thematic mapper plus) is for cost, but older TM and MSS data are available without charge.

Geospatial OneStop (Portal for Federal GIS data)

http://geo.data.gov

Latest version of the Federal Government's effort to provide one-stop-shopping for federal data.

NHGIS: National Historical Geographic Information System

https://www.nhgis.org/

"The National Historical Geographic Information System (NHGIS) provides, free of charge, aggregate census data and GIS-compatible boundary files for the United States between 1790 and 2013."

Social Explorer -

http://www.socialexplorer.com/

This site provides online maps and downloads of data from the Decennial Census and the American Community Survey.

CERLIS Superfund Database -

http://www.epa.gov/superfund/sites/phonefax/products.htm

This database contains information on all NPL and non-NPL superfund sites in the country. Don’t bother trying to geocode the non-NPL sites as the street data is horrible.

MassGIS

http://mass.gov/mgis
good state agency website for finding and disseminating Massachusetts geospatial data

won URISA ESIG award for web services part

Multiple services available:

online mapping: http://maps.massgis.state.ma.us/map_ol/oliver.php

downloadable datasets: http://www.mass.gov/anf/research-and-tech/it-serv-and-support/application-serv/office-of-geographic-information-massgis/datalayers/layerlist.html

web services: (discussed in next section)

ReferenceUSA

http://www.referenceusa.com/Home/Home

We have already mentioned ReferenceUSA in one of our lab assignments. It is a good source of point data for businesses across the country

MIT Libraries - GIS Home page:

http://libguides.mit.edu/gis/GISdata

We have already mentioned the MIT GeoData Repository supported by the MIT Libraries in collaboration with Harvard and Tufts. In addition to the GeoWeb website, the MIT Libraries also support an add-in tool for ArcMap that facilitate adding data layers in ArcMap.

• Here is the Library's online map viewer: http://web.mit.edu/geoweb/
• Click on the "Enter GeoWeb" button to start the viewer (MIT Web Certificates are reqiured)
• The viewer uses GeoServer, ArcSDE, and OpenLayers but the underlying web services (See Part III below) are not accessible to the user.

Stanford Geospatial Center - http://lib.stanford.edu/GIS/finddata

A comprehensive listing of GIS resources available on the web.

Part II - Accessing Geospatial Services from web browsers

A) Using APIs

Yahoo Developer tools and APIs: http://developer.yahoo.com/everything.html

Check out the 'APIs and Web Services' tools and, in particular, the GeoPlanet tool: http://developer.yahoo.com/geo/geoplanet/

Like many available APIs for geoprocessing and location based services, these are proprietary APIs that may make use of Open Geospatial Consortium protocols for interoperable geospatial services but offer special features that are proprietary and not standardized.

Google APIs: https://developers.google.com/apis-explorer/#p/

• Start here: Google Developers 'products' webpage: https://developers.google.com/products/?csw=1
• Google Maps APIs: https://developers.google.com/maps/
• Family of APIs to 'mashup' points, lines, and polygons on top of Google Maps and Google Earth
  • A free service - but read the 'fine print'
  • 'Terms of service': http://code.google.com/apis/maps/terms.html
    • Business use restrictions
  • What are risks to public agency that builds services with Google API?
• Google transit feed specification (GTFS) - the format for submission of transit routes and schedules to enable Google Maps to do From-A-to-B navigation on public transit: https://developers.google.com/transit/gtfs/
• KML (keyhole markup language)
  • This is the API we used in Lab #1 to put our map onto Google Earth
  • KML is a form of 'geographic markup language' (GML) for supporting dialog about geoprocessing service protocols
  
  

B) Custom Web User Interfaces backed by web services

United Nations Environment Programme
http://geodata.grid.unep.ch/

WMS Services / UMN Mapserver software behind the scenes. Most data at general national levels only, some regional data sets. Lots of drill-down required to navigate the interface and generate specific maps and data tables. A listing of their web services is at: http://geodata.grid.unep.ch/webservices/

C) Web Mapping - Leflet JS, TIleMill, Mapbox, CartoDB, etc.

DUSPViz provides a useful introduction to current and emerging web mapping libraries and applications.

Other sources of information that you might find useful include the Lefleat JS documentation, Mapbox, CartoDB, and TileMill.

Part III - Accessing ArcWeb and WMS Services (from ArcMap or ArcExplorer)

A) Finding WMS Services

Web-search approaches:

• Use government web sites such as the Fed's National Map or data.gov to find respositories
• Visit WMS client software sites such as the Index of OGC WMS Servers:
  • http://www.skylab-mobilesystems.com/en/wms_serverlist.html
  • As of Spring 2014, some 994 servers and 339254 data layers are identified on this website

MassGIS: http://mass.gov/mgis

• The MassGIS layers listed above can be accessed as OGC-compliant WMS and WFS web services as explained in last week's lecture notes and at this MassGIS website: https://wiki.state.ma.us/confluence/display/massgis/Home
• Sample URL requesting WMS image (street map) using MassGIS web services: (try pasting into a browser):
  • Here is the link used (above) to access the GISDATA_EOTMAJROADS_ARC layer of major roads.
  • http://giswebservices.massgis.state.ma.us/geoserver/wms?dpi=120&request=GetMap&layers=GISDATA.EOTMAJROADS_ARC&srs=EPSG:26986&bbox=233500,900000,236500,902500&width=800&height=300&format=image/png&service=wms&transparent=true

B) ESRI's ArcGIS Online:

Accessing ArcGIS Online services from ArcMap

• Home page information about ESRI's ArcGIS Online: http://www.esri.com/software/arcgis/arcgisonline/features/organizations.html

 

• During the semester, we have provided 'startup' ArcMap documents for some of the lab exercises that access both MassGIS web services and ESRI ArcGIS Online services
  • Copy 11.188_lab2_web.mxd from the class data locker to C:\temp\lab8
  • Copy the entire cambbgrp.shp and sales89.shp shapefiles from the data locker to C:\temp\lab8
  • Double-click on 11.188_lab2_web.mxd to launch ArcMap using this document

 

Any licensed ArcMap package can utilize ESRI ArcGIS Online services and WMS services as follows:

• Here are the steps I used to build the ArcMap document, 11.188_lab2_web.mxd, provided above

• Copy several familiar MA and Cambridge shapefiles from the class data locker (starting with the cambbgrp shapefile so the Data Frame is set to Mass State Plane coordinates) into C:\temp\lab8
• Start/All programs/ArcGIS/ArcMap 10.3.1 and add the shapefiles from C:\temp\lab8
• Choose File/map-document-properties and click the box next to 'Pathnames:' to store relative pathnames to data sources

Accessing ESRI ArcGIS Online services from inside ArcMap

• Click the 'plus-sign' button (Add Data)
• • Navigate to top level of Catalog (shown in 'Look in' box), under 'GIS Server' choose 'Add ArcGIS Server'.
  • Use 'GIS services'
  • Specify 'http://services.arcgisonline.com/arcgis/services' as the Server URL for the Internet-based ArcGIS Server Connection.
  • Click 'Finish' and then 'Add'
  • The service will show up as 'arcgis on services.arcgisonline.com'. Double click this service link
  • Both 'ESRI_Imagery_World_2D' and 'ESRI_StreetMap_World_2D' are background layers that you may find useful.
  • You may also want to navigate to 'Demographics' and select the 'USA Median Home Value'(or NGS_Topo_US_2D).
  • Play around with the map service and see what you can do with it.
• Note (lack of) responsiveness of layers due to server-side delay. Turn these layers off if you zoom in/out or pan frequently to avoid long waits for each image.
• Try the 'Info' button
• Recognize how source data stored in different coordinates have been converted and merged on screen
• Note that the free map viewer from ESRI, ArcGIS Explorer, can access ArcGIS Online Resource (but not OGC-compliant WMS services)

Accessing WMS services from inside ArcMap

• The lecture notes from last Wednesday contain further information relevant to this section
• Again, click the 'Plus-sign' button (to Add Data) in an ArcMap window with MA towns and Cambridge block groups and ArcWebService layer
• Under 'GIS Servers' choose 'Add WMS Servers'
  • This is the URL for the newer MassGIS WMS service running on GeoServer:
    • http://giswebservices.massgis.state.ma.us/geoserver/wms?
    • More info about the GeoServer service is here: https://wiki.state.ma.us/confluence/display/massgis/Home
• See instructions from last Wednesday's lecture to select desired MassGIS web service layers: http://mit.edu/11.188/www/lectures/lecture_webgis_intro.html
  • BEWARE: at the moment some of the MassGIS webservices will produce maps visible in ArcMap *only* if the Data Frame in ArcMap is set to lat/lon coordinates. There is a bug in the ArcMap handling of the MassGIS web services since the native projection for the MassGIS server is Mass State Plane, but only lat/lon views will be downloaded and displayed (for some of the MassGIS layers). If you encounter a problem, you can force the DataFrame to view the map in lat/lon coordinates by right-clicking on the Layers icon in the Table of Contents, choosing Properties/Coordinate-system, and then setting the 'pre-defined' coordinate system to be: Geographic-Coordinate-System/North-America/North-American-Datum-1983 (NAD 1983)
• We have already chosen a dozen useful MassGIS web services and loaded them for your use into the saved ArcMap document:
• Find and add two more layers: 'pre-kindergarten to high school buildings' and 'colleges and universities'
  • Here are some additional notes to help you zero in on the school layers.
  • Once the "MassGIS WMS" layer has been added into the ArcMap table of contents
    • Uncheck the layer so ArcMap does not try to draw it
    • Expand the layer (click the '+' symbol) so you see the sub-layers
      • Better yet, right-click on the 'Massachusetts Data from MassGIS (GeoServer)' layer and edit its properties
      • Use the 'layer' tab to select and move to the right side of the window those layers that you want listed in the table of contents.
    • Find the layer entitled: 'pre-kindergarten to high-school buildings' and the one entitled 'colleges and universities' (a little further down)
      • There are several hundred available layers and the list that appears in the layer property window available in ArcMap is not alphabetized.
      • To see an alphabetized list of all available MassGIS web services, you can use ArcCatalog to view the properties of "Massachusetts Data from MassGIS (GeoServer)". You can now 'drag and drop' layers from ArcCatalog into the table of contents of ArcMap. However, each "drag-and-drop" will generate a new instance of the "Massachusetts Data from MassGIS (GeoServer)" web service layer with the listing of the new layer visible by clicking the '+' sign next to "Massachusetts Data from MassGIS (GeoServer)"
    • Turn on these two layers and right-click each layer to choose 'add WMS legend to the map'
    • ArcMap should now try to get a WMS image for your school layers (plus the legends). A rotating globe will appear next to the coordinate information in the bottom 'status' row under the map. If no map appears, you can click the 'refresh view' button (the circular arrows next to the data-view and layout-view buttons below the Map). Note, there are several equally valid ways to select and edit the MassGIS layers you wish to view (that is, while adding the MassGIS site, or by editing the layer properties after adding the entire 'Massachusetts Data from MassGIS...' website. However you do the selection, we want you to end up with only the two school layers turned on: 'pre-kindergarten to high school buildings' and 'colleges and universities'
    • If a WMS image still does not appear, zoom in/out or pan the map to force a refresh.
    • If a WMS image still does not appear, check that the Data Frame is displaying in Geographic (lat/lon) coordinates (as explained in the previous "BEWARE" note). It may be necessary to set the Data Frame coordinate system to "Geographic Coordinate Systems" + World + "WGS 1984" in order for the schools to be displayed. (They are stored by MassGIS in state plane coordinates, but ArcMap version 9.2 has problems determining capabilities from the MassGIS server and is not reliable other than in lat/lon.)
  • Once you figure out how to get the School layer to be visible, try using the 'identify' button to find out information about particular schools. Notice that the 'identify' window wraps the data about the school in XML tags. If you were able to cut-and-paste the information into a browser or spreadsheet that understood the XML tags, the information would be displayed as a table.
  • Show the location of public and private schools and colleges in the Boston area based on these two MassGIS datasets. (Note, just zoom in to the Boston/Cambridge area so the individual school flags can be distinguished. With these WMS services, we do not have a way to explicitly select the schools within, say, the city of Boston.)
  • Because ArcGIS may not properly dialog with the MassGIS site, you may not be able to get the map legends from MassGIS. For your map, just add a text message box explaining the flags.
• Note that a grayed out check mark next to a layer means it can not be mapped (wrong zoom, coordinates, etc.)

Create a PDF-formatted map zoomed in to the Cambridge area showing the school location layer from MassGIS along with the Cambridge local roads and the Mass highway layer that you use in Lab 2. If you were successful in adding ESRI's GIS services, you can add the USA Median Home Value (or NGS_Topo_US_2D) layers from the ArcGIS Online section.

NOTE: this PDF-formatted map is the only result that you need to turn in from parts 1, 2, and 3 of this lab exercise!

• Optional: try some of the other OGC-compliant WMS services: http://www.skylab-mobilesystems.com/en/wms_serverlist.html

Part IV - "Mashups" and APIs

A) General Mapping Web Site (Mostly Commercial Advertising-Supported - all now with APIs)

Google Maps

http://maps.google.com/
Note "REST" interface (client side interface actions handled by Javascript, partial page refresh without a "submit" button)

https://developers.google.com/maps/

Microsoft Virtual Earth

http://www.bing.com/maps/

Note (with free account) ability to sketch, and share simple sketch (with labels but not attributes, non-standard format)

Yahoo Maps

http://maps.yahoo.com/beta/index.php

Note live traffic feed option, overview map leads full map for zoom interface preview. Also, mouse rollover on legend highlights search results dynamically.

B) Google Earth and Sketchup.

Optional: Try the Google Earth + Sketchup exercise explained in the optional part of last Wednesday's Internet GIS lecture notes. From Google Earth, you add a 3D model of the Eiffel Tower and place it, at scale, onto Briggs Field near the MIT dormatories. The Eiffel Tower model was built using Google Sketchup and downloaded from Google's 3D Warehouse of user contributed 3D models. From Google Earth, you can save a snippet of surface image into a format that Sketchup can read as your ground location. Then you can use Sketchup to build a 3D model which you can then add back into Google Earth at the appropriate location.

Part V: Lab Assignment Options

Pick any one of the 4 options below along with the PDF for the map in Part III-B

Option A: Build Your Own U.S. Case Study Using Downloaded GIS Data

1. Select a location in the U.S. which interests you. Using the national map or seamless.usgs.gov, find and download:

National Elevation Data (NED 1/9)
Land Use / Land Cover (NLCD)
Transportation (varies)
Hydrology (varies)

2. Note the interface or data problems you find, if any.

3. Turn in both a map illustrating your area, and a brief paragraph describing the user experience.

Option B: Build an International Case Study

1. Pick an international site of interest.
2. Using USGS's "seamless" web site, find and download the SDTS terrain for the area, and any other relevant data.
3. Using the University of Maryland's site, find which Landsat imagery is available for free for the area. Also, download the global 1km land use information for the area if available. (No need to actually download a full Landsat image at this time - just grab the preview image for now).
4. Using Google Maps or another online GIS, screen-grab imagery for your area.
5. Document what you found, what you didn't find, and your overall user experience in about a paragraph.

Option C: Build Your Own Map Using Web Maps (in Browser or in ArcGIS)

1. Same idea as in options above, different technical approach. Try using web mapping to create a map for particular area of interest. Turn in a screen capture of your map. You can capture the screen using the 'snipping tool' from Start/All-Program/Accessories/snipping-tool.
2. Try to find and use at least one of ESRI's ArcWeb Services, and one OGC Web Mapping Service (WMS).
3. Document at a paragraph level your user experience, including interface and data issues.

Option D: Critical Analysis

Web services can be evaluated from the point of view of a consumer, from that of a producer, or from that of a policymaker. Pick one of these points of view, and using the sites above as examples, describe what you find to be the "state of the art." Address what works, what are best practices, what doesn't work and what you think is required.

Write up approximately 1 - 2 pages of text, with illustrated examples

Please choose 1 of the 4 above-listed lab assignment options (in addition to the map detailed in Part III-B) and submit your assignment on Stellar. This assignment is optional.

Created October, 2006, by Michael Flaxman and Joseph Ferreira based on exercise developed by Joseph Ferreira for use in the URISA Internet GIS workshop taught by Zhong-Ren Peng and Joseph Ferreira.

Last modified 7 April 2016 [myc]

Back to the 11.188 Home Page.