Massachusetts Institute of Technology
Department of Urban Studies and Planning

Lab 8: Experimenting with Geospatial Services

Due by November 18, 2009

Objectives

1. To understand the layering of data, application tools, web mapping interoperability standards, and web service protocols involved in modern internet GIS applications.
2. To compare alternative strategies for utilizing geospatial services, from spatial data search with GIS download to direct online use.
3. To examine a few geospatial services "under the covers" in order to understand the nature and purpose of the URL requests and XML responses between the clients and web services.

Overarching issues

• Audience: Whom does the system serve? Does it serve them well? How might you systematically assess this?
• Thematic Content: What are the primary types of data available? Is metadata available describing currentness, methods, scale, projection, etc?
• Background Content: What other incidental content is available? Can the background information be separated from the foreground?
• Basic Data Type: Is it a picture of a map? Unclassified imagery like an orthophoto? Classified grid data? Points, lines and polygons?
• Focus and Flexibility: What other uses are supported? Can users download computable data? Can users upload or exchange data?
• Fitness for Use: Does the data/interface allow cartographic manipulation? Printing? Editing? Analysis?
• Price / Licensing: Is this data available free or at cost? Even if free, what are the licensing restrictions? How do these issues affect public planning use, private firm use, citizen use?

Tips and Tricks

Not all web services and web sites allow you to examine or determine what they are doing behind the scenes. However a significant number, especially those based on open standards, do allow this. Learn to parse URLs yourself (allows you to repurpose sites and services, debug errors)

Basic Rules for URLs and HTTP

1) Transactions are usually "stateless"

All of the information needed for the server to meet client's request is bundled into URL
(Exception: sometimes "cookies" are used to associate state based on a user ID or session ID)

2) Parsing URLs

Overall URL format is protocol:[optional port number]://path/to/resource/scriptname?parameters
A question mark separates path/scriptname from parameters list

&parameter1=blah&parameter2=blah+again

The ampersand separates parameters, and the equals sign sets the assigned value
Spaces are not allowed in URLS, so "+" symbol is substituted
To clarify, paste URL into a text editor and substitute line breaks for question mark and ampersands

3) Example (standard Web Mapping Service or WMS):

http://maps.massgis.state.ma.us/mapaccess/main.jsp?dpi=120&request=GetMap&layers=MHD+Roads&styles=Class&srs=EPSG:26986&bbox=233500,900000,236500,902500&width=640&height=480&format=image/png&service=wms

Becomes

http://maps.massgis.state.ma.us/mapaccess/main.jsp
dpi=120
request=GetMap
layers=MHD+Roads
styles=Class
srs=EPSG:26986
bbox=233500,900000,236500,902500
width=640
height=480
format=image/png
service=wms

Not self-explanatory, but a step in the right direction. Full WMS description can be found here for those interested:
OpenGIS Web Map Service (WMS) Implementation Specification

 

Part I - Exploring Online Mapping Sites

A) Sites Designed Primarily for Online Use

Explore these sites briefly, noting the issues above and last week's lecture notes

EPA's Enviromapper:

http://www.epa.gov/enviro/html/em
Special permission required for access to underlying services (e.g., Oracle account)

U.S. Department of Transportation Federal Highway Administration

What is the annual average daily traffic volume on Mass. Ave?
http://hepgis.fhwa.dot.gov/hepgis_v2/GeneralInfo/Map.aspx

Yes, there is a map on this site that color codes roads based on estimate traffic volume. However, the interface offers limited help capabilities and the right map is hard to find. Do not spend too long hunting for it. Just get a general sense of what the site offers and what you do/do-not like about the site capabilities and user interface.

Other Traffic info sites:

• 'Traffic' tab on Microsoft's Virtual Earth: http://local.live.com/
• NAVTEQ Traffic.com (real-time traffic congestion reporting): http://www.traffic.com/Chicago-Traffic/Chicago-Traffic-Reports.html

U.S. City Sites

Portland Maps (City of Portland, Oregon) http://www.portlandmaps.com/

• Extensive online mapping with detailed assessing and permitting information

Cambridge Maps (City of Cambridge, MA) http://www.cambridgema.gov/GIS/

• City Viewer (Internet Explorer only!): http://gis.cambridgema.gov/map/Viewer.aspx

B) Sites With Data Download Capabilities

The (US) National Map

http://nationalmap.gov/(cover page)
http://nmviewogc.cr.usgs.gov/viewer.htm (direct link to mapping interface)
USGS-led effort at coherent, cross-agency national map layers

USGS

http://seamless.usgs.gov/
Most important US government site for International Users because it provides free access to Global 90m Terrain Data that can be downloaded in a non-poprietary file format, SDTS, that most GIS packages will support.
(SDTS = a Spatial Data Transfer Standard)

University of Maryland Global Land Cover Facility

http://glcf.umiacs.umd.edu/data/ (overview)
http://glcfapp.umiacs.umd.edu:8080/esdi/index.jsp (direct access to mapping interface)

Best place to download free Landsat imagery (warning: huge and unweildy) and global 1km resolution land cover classification. U.S. Government data policy on Landsat has varied over the years. At the moment, access to ETM+ (enhanced thematic mapper plus) is for cost, but older TM and MSS data are available without charge.

Geospatial OneStop (Portal for Federal GIS data)

http://gos2.geodata.gov/wps/portal/gos
Latest version is ESRI-powered under the covers. "Drinking from a fire hose?"

Canadian Geospatial Data Infrastructure (CGDI): http://www.geoconnections.org/en/index.html

'Discovery Portal': http://geodiscover.cgdi.ca/gdp/search?action=searchForm&entryType=productCollection&formType=advanced&language=en

Atlas of Canada: http://atlas.nrcan.gc.ca/site/english/index.html

Other national spatial data infrastructure (SDI) sites

Indian National Data Infrastructure (viewer may be down or slow): http://gisserver.nic.in/nsdiportal/gotogos.jsp

MassGIS

http://mass.gov/mgis
good state agency website for finding and disseminating geospatial data

won recent URISA ESIG award for web services part

Accessible at several levels:

online mapping: http://mass.gov/mgis/mapping.htm
downloadable datasets: http://mass.gov/mgis/database.htm
web services: (discussed in next section)

MIT Libraries - GIS Home page: http://libraries.mit.edu/gis/index.html

We have already used the MIT GeoData Repository tool from inside of ArcMap

• Here is the Library's online map viewer: http://web.mit.edu/geoweb/
• Click on the "Enter GeoWeb" button to start the viewer (MIT Web Certificates are reqiured)
• The viewer uses GeoServer, ArcSDE, and OpenLayers but the underlying web services (See Part III below) are not accessible to the user.
  

Part II - Accessing Geospatial Services from web browsers

A) Using APIs

Yahoo Traffic API using "REST" web URLS (see REST description below)
http://developer.yahoo.com/traffic/rest/V1/index.html

What is the current traffic on Mass Ave at MIT?

http://local.yahooapis.com/MapsService/V1/trafficData?appid=YdnDemo&street=77+Massachusetts+Ave&city=Cambridge&state=MA&include_map=1
Your browser will not display the results on a map but will just list the XML that is returned. This is an example of a spatial query that return a non-spatial result. The results contain XML-wrapped text and X,Y locations explaining traffic events near the address sent in the URL. Examine the tags and content. With the 'include_map' flag set to one, the URL for a map of the area is included in the results. Cut-and-paste the URL for the map into another browser window.

B) Custom Web User Interfaces backed by web services

United Nations Environment Programme
http://geodata.grid.unep.ch/

Notes. WMS Services / UMN Mapserver software behind the scenes. Most data at general national levels only, some regional data sets. Lots of drill-down required to navigate the interface and generate specific maps and data tables. A listing of their web services is at: http://geodata.grid.unep.ch/webservices/

C) ESRI ArcGIS Server

http://resources.esri.com/help/9.3/arcgisserver/apis/flex/samples/index.html

What it does: General web client interface backed by ArcGIS Server. Also note "map types" and layer-level user control exposed.

How it works: web Services inside Flash-based Client inside browser

Which data are available and how much do they cost? (Some public services, other per hit)

Part III - Accessing ArcWeb and WMS Services (from ArcMap or ArcExplorer)

A) Finding WMS Services

Web-search approaches:

Use government web sites such as GOS above, or WMS client software sites

Index of OGC WMS Servers

http://www.skylab-mobilesystems.com/en/wms_serverlist.html

MassGIS: http://mass.gov/mgis

• The MassGIS layers listed above can be accessed as OGC-compliant WMS and WFS web services as explained at: http://lyceum.massgis.state.ma.us/wiki/doku.php
• Example URL requesting WMS image (street map) using earlier web services: (try pasting into a browser) http://maps.massgis.state.ma.us/mapaccess/main.jsp?dpi=120&request=GetMap&layers=MHD+Roads&styles=Class&srs=EPSG:26986&bbox=233500,900000,236500,902500&width=640&height=480&format=image/png&service=wms
• Equivalent URL requesting WMS image (street map) using newer MassGIS web services: (try pasting into a browser): http://giswebservices.massgis.state.ma.us/geoserver/wms?request=GetMap&layers=massgis:GISDATA.EOTMAJROADS_ARC&srs=EPSG:26986&bbox=233500,900000,236500,902500&width=640&height=480&format=image/png&service=wms

B) ESRI's ArcGIS Online:

ESRI's ArcGIS Online: http://www.esri.com/software/arcgis/arcgisonline/map-services.html

Access from ArcMap

• Start/ArcGIS/ArcMap\
• Open some familiar MA and Cambridge shapefiles from the class data locker
• Click the 'plus-sign' button (Add Data)
• Navigate to top level of Catalog (shown in 'Look in' box), under 'GIS Server' choose 'Add ArcGIS Server'.
• Use 'GIS services'
• Specify 'http://services.arcgisonline.com/arcgis/services' as the Server URL for the Internet-based ArcGIS Server Connection.
• Click 'OK' and then 'Add'
• The service will show up as 'arcgis on services.arcgisonline.com'. Double click this service link
• Both 'ESRI_Imagery_World_2D' and 'ESRI_StreetMap_World_2D' are background layers that you may find useful.
• You may also want to navigate to 'Demographics' and select the 'USA Median Home Value'(or NGS_Topo_US_2D).
• Play around with the map service and see what you can do with it.
• Note (lack of) responsiveness of layers due to server-side delay. Turn these layers off if you zoom in/out or pan frequently to avoid long waits for each image.
• Try the 'Info' button
• Recognize how source data in different coordinates have been merged

The free map viewer from ESRI, ArcExplorer v. 9.2, can access ArcGIS Online Resource (but not OGC-compliant WMS services)

Accessing WMS services from ArcMap

• Again, click the 'Plus-sign' button (to Add Data) in an ArcMap window with MA towns and Cambridge block groups and ArcWebService layer
• Under 'GIS Servers' choose 'Add WMS Servers'
  • This is the soon-to-be-retired URL for the MassGIS WMS service running on ArcIMS
    • http://maps.massgis.state.ma.us/mapaccess/main.jsp?
    • Click 'OK' and then 'Add'
      • If you click 'Get Layers' it will take 1+ minutes to get the long list; just click 'ok' after entering the URL
      • If you are using ArcGIS 9.3, change the protocol to be 1.1.x (i.e., some flavor of version 1.1 not 1.3) instead of the 'default' protocol which will not work.
  • This is the URL for the newer MassGIS WMS service running on GeoServer:
    • http://giswebservices.massgis.state.ma.us/geoserver/wms
    • You can see the available layers using this URL http://giswebservices.massgis.state.ma.us/geoserver/mapPreview.do
    • Beware that this GeoServer service may not ArcMap v. 9.2 (service pack 3)
    • More info about the GeoServer service is here: http://lyceum.massgis.state.ma.us
• Once back in ArcMap, the layer may still not show if zoomed in/out too far
  • BEWARE: at the moment this MassGIS webservice will produce maps visible in ArcMap *only* if the Data Frame in ArcMap is set to lat/lon coordinates. There is a bug in the ArcMap handling of the MassGIS web services since the native projection for the MassGIS server is Mass State Plane, but only lat/lon views will be downloaded and displayed (for some of the MassGIS layers). You can force the DataFrame to view the map in lat/lon coordinates by right-clicking on the Layers icon in the Table of Contents, choosing Properties/Coordinate-system, and then setting the 'pre-defined' coordinate system to be: Geographic-Coordinate-System/North-America/North-American-Datum-1983
• Find 'schools' under 'Infrastructure'
  • Since ArcGIS 9.2 does not handle the MassGIS services well, here are some additional notes to help you zero in on the school layer.
  • Once the "MassGIS WMS 1.1.13" layer has been added into the ArcMap table of contents
    • Uncheck the layer so ArcMap does not try to draw it
    • Expand the layer (click the '+' symbol) so you see the sublayers
    • Collapse and turn off the first few layers (e.g., Theme Maps, Status, Regulated Areas, and MEPA) until you get to "Infrastructure'
    • Collapse and turn off the first few layers under 'Infrastructure' until you get to 'Schools'
    • Turn on "Schools" and the school and college sub-layers
    • Now, turn on the 'Infrastructure' and 'MassGIS WMS 1.1.13' layers
    • ArcMap should now try to get a WMS image with the school layer. A rotating globe will appear next to the coordinate information in the bottom 'status' row under the map. If no map appears, you can click the 'refresh view' button (the circular arrows next to the data-view and layout-view buttons below the Map).
    • If a WMS image still does not appear, zoom in/out or pan the map to force a refresh.
    • If a WMS image still does not appear, check that the Data Frame is displaying in Geographic (lat/lon) coordinates (as explained in the previous "BEWARE" note). It may be necessary to set the Data Frame coordinate system to "Geographic Coordinate Systems" + World + "WGS 1984" in order for the schools to be displayed. (They are stored by MassGIS in state plane coordinates, but ArcMap version 9.2 has problems determining capabilities from the MassGIS server and is not reliable other than in lat/lon.)
  • Once you figure out how to get the School layer to be visible, try using the 'identify' button to find out information about particular schools. Notice that the 'identify' window wraps the data about the school in XML tags. If you were able to cut-and-paste the information into a browser or spreadsheet that understood the XML tags, the information would be displayed as a table.
  • Because ArcGIS 9.2 does not properly dialogue with the MassGIS site, you cannot get the map legends from MassGIS. For your map, just add a text message box explaining the flags and caps show the location of public and private schools and colleges based on MassGIS data
• Note that a grayed out check mark next to a layer means it can not be mapped (wrong zoom, coordinates,etc.)
• Cambridge, Flood (or NGS-Topo), and MassGIS layers won't mashup well unless ArcMap knows (and handles) the projection of each layer
• Create a PDF-formatted map zoomed in to the Cambridge area showing the school location layer from MassGIS along with the Cambridge local roads and the Mass highway layer that you use in Lab 2. If you registered successfully for ArcGIS Online, you can add the USA Median Home Value (or NGS_Topo_US_2D) layers from the ArcGIS Online section.
• Optional: try some of the other OGC-compliant WMS services: http://www.skylab-mobilesystems.com/en/wms_serverlist.html

Part IV - "Mashups" and APIs

A) General Mapping Web Site (Mostly Commercial Advertising-Supported - all now with APIs)

Google Maps

http://maps.google.com/
Note "REST" interface (client side interface actions handled by Javascript, partial page refresh without a "submit" button)

http://www.google.com/apis/maps/

Microsoft Virtual Earth

http://www.bing.com/maps/

Note (with free account) ability to sketch, and share simple sketch (with labels but not attributes, non-standard format)

Yahoo Maps (Beta)

http://maps.yahoo.com/beta/index.php

Note live traffic feed option, overview map leads full map for zoom interface preview. Also, mouse rollover on legend highlights search results dynamically.

B) "Mashups" (standard 2d maps + Javascript programming for interfaces or dynamic overlays)

Boston.com election contribution website

http://www.boston.com/news/special/bigarticles/campaign_finance/page2.html?p1=email_to_a_friend

Live 'mashup' webpage (for Back Bay):

http://boston.faneuilmedia2.com/gov/detail_map.html?from=top&zip=02116&z=13

Note: you can 'pan' the map or click on dots for further info

How was it done socially/technically? Overlay/identify entries in local election contribution database maintained by consultant (Faneuil Media, Inc.)

Looking under the covers

"View source" in your browser. Look for call to google with machine/directory-specific Key:

http://maps.google.com/maps?file=api&v=2.60&key=AB...

"REST" is a programming technique which uses client side javascript to interact with the server without full page refreshed. All mapping APIs are moving towards this, pressured by Google maps success. REST stands for Representational State Transfer and involves ways of exchanging information between client and server regarding the state of the client so that every exchange between client and server does not require a complete reconstruction of the client-viewed webpage. AJAX programming is one strategy for enabling complex client-side actions using new information from the server without having to rewrite all of the client webpage. AJAX stands for Asymmetric Javascript and Xml and uses javascript to send URL requests, parse the XML-tagged responses, and use the returned information to rewrite parts of the client-side browser window.

Optional Note: hard to get beyond "push pins" user interfaces

Moving Past Push Pins

Part V: Lab Assignment Options

Pick any one of the 4 options below along with the PDF for the map in Part III-B

Option A: Build Your Own U.S. Case Study Using Downloaded GIS Data

1. Select a location in the U.S. which interests you. Using the national map or seemless.usgs.gov, find and download:

National Elevation Data (NED 1/9)
Land Use / Land Cover (NLCD)
Transportation (varies)
Hydrology (varies)

2. Note the interface or data problems you find, if any.

3. Turn in both a map illustrating your area, and a brief paragraph describing the user experience. (Those of you actually wanting to use case study data later in the term should save it somewhere for later.)

Option B: Build an International Case Study

1. Pick an international site of interest.
2. Using USGS's "seamless" web site, find and download the SDTS terrain for the area, and any other relevant data.
3. Using the University of Maryland's site, find which Landsat imagery is available for free for the area. Also, download the global 1km land use information for the area if available. (No need to actually download a full Landsat image at this time - just grab the preview image for now).
4. Using Google Maps or another online GIS, screen-grab imagery for your area.
5. Document what you found, what you didn't find, and your overall user experience in about a paragraph. (Those of you actually wanting to use case study data later in the term should save it somewhere for later.

Option C: Build Your Own Map Using Web Maps (in Browser or in ArcGIS)

1. Same idea as in options above, different technical approach. Try using web mapping to create a map for particular area of interest. Depending on the interfaces used, you may actually need to do a screen capture to turn in, rather than a print.
2. Try to find and use at least one of ESRI's ArcWeb Services, and one OGC Web Mapping Service (WMS).
3. Document at a paragraph level your user experience, including interface and data issues.

Option D: Critical Analysis

Web services can be evaluated from the point of view of a consumer, from that of a producer, or from that of a policymaker. Pick one of these points of view, and using the sites above as examples, describe what you find to be the "state of the art." Address what works, what are best practices, what doesn't work and what you think is required.

Write up approximately 1 - 2 pages of text, with illustrated examples

Please do any one of the 4 Lab Assignment Options and submit them along with the PDF-formatted map to Stella. The assignment is due on Monday, November 18, 2009.

Created October, 2006, by Michael Flaxman and Joseph Ferreira based on exercise developed by Joseph Ferreira for use in the URISA Internet GIS workshop taught by Zhong-Ren Peng and Joseph Ferreira.

Last modified 8 December 2009 [jf]

Back to the 11.520 Home Page.