| 11.520: A Workshop on Geographic Information Systems |
| 11.188: Urban Planning and Social Science Laboratory |
Overarching issues (when examining online GIS data repositories and web services)
Not all web services and web sites allow you to examine or determine what they are doing behind the scenes. However a significant number, especially those based on open standards, do allow this. Learn to parse URLs yourself (allows you to repurpose sites and services, debug errors)
1) Transactions are usually "stateless"
All of the information needed for the server to meet client's request is bundled into URL
(Exception: sometimes "cookies" are used to retain the client's 'state' based on a user ID or session ID)2) Parsing URLs
The general URL format uses this sequence: protocol:[optional port number]://path/to/resource/scriptname?parameters
A question mark separates path/scriptname from parameters list¶meter1=blah¶meter2=blah+again
The ampersand separates parameters, and the equals sign sets the assigned value
Spaces are not allowed in URLS, so "+" symbol is substituted
To clarify and inspect, paste URL into a text editor and substitute line breaks for question mark and ampersands3) Example - using standard Web Mapping Service (WMS) similar to the example in the last lecture
Becomes
http://giswebservices.massgis.state.ma.us/geoserver/wms
dpi=120
request=GetMap
layers=GISDATA.EOTMAJROADS_ARC
srs=EPSG:26986
bbox=233500,900000,236500,902500
width=640
height=480
format=image/png
service=wms
transparent=trueNot self-explanatory, but a step in the right direction.
- dpi, width, and height specific the dots-per-inch on your screen and the width/height of your viewing window (in pixels)
- In class, we used the European Petroleum Survey Group code 4326 for the World Geodesic Datum of 1984 (WGS84)
- In this case, we use the EPSG code 26986 for Mass State Plane coordinates (Mainland) NAD83
- Try this URL in a browser
For those who are interested, the full WMS description can be found here:
OpenGIS Web Map Service (WMS) Implementation Specification
Explore these sites briefly, noting the issues above and last week's lecture notes
EPA's Enviromapper:
- http://www.epa.gov/emefdata/em4ef.home
- Check out a city or zip code and examine various layers on this EPA site
- The 'toxic release inventory' sites in homework set #2 were obtained from these EPA databases
- Special permission required for access to underlying services (e.g., Oracle account)
U.S. Department of Transportation Federal Highway Administration
What is the annual average daily traffic volume on Mass. Ave?
http://hepgis.fhwa.dot.gov/hepgis_v2/GeneralInfo/Map.aspxYes, there is a map on this site that color codes roads based on estimate traffic volume. However, the interface offers limited help capabilities and the right map is hard to find. Do not spend too long hunting for it. Just get a general sense of what the site offers and what you do/do-not like about the site capabilities and user interface.
Other Traffic info sites:
- 'Traffic' tab on Microsoft's Virtual Earth: http://local.live.com/
- NAVTEQ Traffic.com (real-time traffic congestion reporting): http://www.traffic.com/Chicago-Traffic/Chicago-Traffic-Reports.html
U.S. City Sites
Portland Maps (City of Portland, Oregon) http://www.portlandmaps.com/
- Extensive online mapping with detailed assessing and permitting information
Cambridge Maps (City of Cambridge, MA) http://www.cambridgema.gov/GIS/
B) Sites With Data Download Capabilities
The (US) National Map
http://nationalmap.gov/(cover page)
http://nmviewogc.cr.usgs.gov/viewer.htm (direct link to mapping interface)
USGS-led effort at coherent, cross-agency national map layersUSGS
http://seamless.usgs.gov/
Most important US government site for International Users because it provides free access to Global 90m Terrain Data that can be downloaded in a non-poprietary file format, SDTS, that most GIS packages will support.
(SDTS = a Spatial Data Transfer Standard)University of Maryland Global Land Cover Facility
http://glcf.umiacs.umd.edu/data/ (overview)
Best place to download free Landsat imagery (warning: huge and unweildy) and global 1km resolution land cover classification. U.S. Government data policy on Landsat has varied over the years. At the moment, access to ETM+ (enhanced thematic mapper plus) is for cost, but older TM and MSS data are available without charge.
Geospatial OneStop (Portal for Federal GIS data)
http://gos2.geodata.gov/wps/portal/gos
Latest version is ESRI-powered under the covers. "Drinking from a fire hose?"
Canadian Geospatial Data Infrastructure (CGDI): http://www.geoconnections.org/en/index.html
'Discovery Portal': http://geodiscover.cgdi.ca/gdp/search?action=searchForm&entryType=productCollection&formType=advanced&language=en
Atlas of Canada: http://atlas.nrcan.gc.ca/site/english/index.html
MassGIS
http://mass.gov/mgis
good state agency website for finding and disseminating geospatial datawon recent URISA ESIG award for web services part
Accessible at several levels:
online mapping: http://mass.gov/mgis/mapping.htm
downloadable datasets: http://mass.gov/mgis/database.htm
web services: (discussed in next section)MIT Libraries - GIS Home page: http://libraries.mit.edu/gis/index.html
We have already used the MIT GeoData Repository tool from inside of ArcMap
- Here is the Library's online map viewer: http://web.mit.edu/geoweb/
- Click on the "Enter GeoWeb" button to start the viewer (MIT Web Certificates are reqiured)
- The viewer uses GeoServer, ArcSDE, and OpenLayers but the underlying web services (See Part III below) are not accessible to the user.
Yahoo Traffic API using "REST" web URLS (see REST description below)
http://developer.yahoo.com/traffic/rest/V1/index.htmlWhat is the current traffic on Mass Ave at MIT?
http://local.yahooapis.com/MapsService/V1/trafficData?appid=YdnDemo&street=77+Massachusetts+Ave&city=Cambridge&state=MA&include_map=1
Your browser will not display the results on a map but will just list the XML that is returned. This is an example of a spatial query that return a non-spatial result. The results contain XML-wrapped text and X,Y locations explaining traffic events near the address sent in the URL. Examine the tags and content. With the 'include_map' flag set to one, the URL for a map of the area is included in the results. Cut-and-paste the URL for the map into another browser window.
United Nations Environment Programme
http://geodata.grid.unep.ch/Notes. WMS Services / UMN Mapserver software behind the scenes. Most data at general national levels only, some regional data sets. Lots of drill-down required to navigate the interface and generate specific maps and data tables. A listing of their web services is at: http://geodata.grid.unep.ch/webservices/
http://resources.esri.com/help/9.3/arcgisserver/apis/flex/samples/index.html
What it does: General web client interface backed by ArcGIS Server. Also note "map types" and layer-level user control exposed.
How it works: web Services inside Flash-based Client inside browser
Which data are available and how much do they cost? (Some public services, other per hit)
Web-search approaches:
Use government web sites such as GOS above, or WMS client software sites
Index of OGC WMS Servers
MassGIS: http://mass.gov/mgis
- The MassGIS layers listed above can be accessed as OGC-compliant WMS and WFS web services as explained at: http://lyceum.massgis.state.ma.us/wiki/doku.php
- Sample URL requesting WMS image (street map) using MassGIS web services: (try pasting into a browser):
- Earlier (above) we accessed the GISDATA_EOTMAJROADS_ARC layer of major roads. This time get the local streets:
- http://giswebservices.massgis.state.ma.us/geoserver/wms?dpi=120&request=GetMap&layers=GISDATA.NAVTEQRDS_ARC&srs=EPSG:26986&bbox=233500,900000,236500,902500&width=640&height=480&format=image/png&service=wms&transparent=true
ESRI's ArcGIS Online: http://www.esri.com/software/arcgis/arcgisonline/map-services.html
Access from ArcMap
- Start/ArcGIS/ArcMap\
- Open some familiar MA and Cambridge shapefiles from the class data locker (starting with the cambbgrp shapefile so the Data Frame is set to Mass State Plane coordinates)
- Click the 'plus-sign' button (Add Data)
- Navigate to top level of Catalog (shown in 'Look in' box), under 'GIS Server' choose 'Add ArcGIS Server'.
- Use 'GIS services'
- Specify 'http://services.arcgisonline.com/arcgis/services' as the Server URL for the Internet-based ArcGIS Server Connection.
- Click 'OK' and then 'Add'
- The service will show up as 'arcgis on services.arcgisonline.com'. Double click this service link
- Both 'ESRI_Imagery_World_2D' and 'ESRI_StreetMap_World_2D' are background layers that you may find useful.
- You may also want to navigate to 'Demographics' and select the 'USA Median Home Value'(or NGS_Topo_US_2D).
- Play around with the map service and see what you can do with it.
- Note (lack of) responsiveness of layers due to server-side delay. Turn these layers off if you zoom in/out or pan frequently to avoid long waits for each image.
- Try the 'Info' button
- Recognize how source data in different coordinates have been merged
The free map viewer from ESRI, ArcExplorer v. 9.2, can access ArcGIS Online Resource (but not OGC-compliant WMS services)
Accessing WMS services from ArcMap
- The lecture notes from last Wednesday contain further information relevant to this section
- Again, click the 'Plus-sign' button (to Add Data) in an ArcMap window with MA towns and Cambridge block groups and ArcWebService layer
- Under 'GIS Servers' choose 'Add WMS Servers'
- This is the URL for the newer MassGIS WMS service running on GeoServer:
- http://giswebservices.massgis.state.ma.us/geoserver/wms
- You can see the available layers using this URL http://giswebservices.massgis.state.ma.us/geoserver/web/?wicket:bookmarkablePage=:org.geoserver.web.demo.MapPreviewPage
- More info about the GeoServer service is here: http://lyceum.massgis.state.ma.us
- Once back in ArcMap, the layer may still not show if zoomed in/out too far
- BEWARE: at the moment this MassGIS webservice will produce maps visible in ArcMap *only* if the Data Frame in ArcMap is set to lat/lon coordinates. There is a bug in the ArcMap handling of the MassGIS web services since the native projection for the MassGIS server is Mass State Plane, but only lat/lon views will be downloaded and displayed (for some of the MassGIS layers). You can force the DataFrame to view the map in lat/lon coordinates by right-clicking on the Layers icon in the Table of Contents, choosing Properties/Coordinate-system, and then setting the 'pre-defined' coordinate system to be: Geographic-Coordinate-System/North-America/North-American-Datum-1983
- Find 'schools' under 'Infrastructure'
- Here are some additional notes to help you zero in on the school layer.
- Once the "MassGIS WMS" layer has been added into the ArcMap table of contents
- Uncheck the layer so ArcMap does not try to draw it
- Expand the layer (click the '+' symbol) so you see the sublayers
- Collapse and turn off the first few layers (e.g., Theme Maps, Status, Regulated Areas, and MEPA) until you get to the "Infrastructure" section. That is, find the vector layers that are listed under 'infrastructure' on the MassGIS Data layer listing: http://www.mass.gov/mgis/laylist.htm
- Find the layer entitled: 'pre-kindergarten to high-school buildings' and the one entitled 'colleges and universities' (a little further down)
- Turn on these two "School" layers and right-click each layer to choose 'add WMS legend to the map'
- ArcMap should now try to get a WMS image for your school layers (plus the legends). A rotating globe will appear next to the coordinate information in the bottom 'status' row under the map. If no map appears, you can click the 'refresh view' button (the circular arrows next to the data-view and layout-view buttons below the Map). Note, there are several equally valid ways to select and edit the MassGIS layers you wish to view (that is, while adding the MassGIS site, or by editing the layer properties after adding the entire 'Massachusetts Data from MassGIS...' website. However, you do the selection, we want you to end up with only the two school layers turned on: 'pre-kindergarten to high school buildings' and 'colleges and universities'
- If a WMS image still does not appear, zoom in/out or pan the map to force a refresh.
- If a WMS image still does not appear, check that the Data Frame is displaying in Geographic (lat/lon) coordinates (as explained in the previous "BEWARE" note). It may be necessary to set the Data Frame coordinate system to "Geographic Coordinate Systems" + World + "WGS 1984" in order for the schools to be displayed. (They are stored by MassGIS in state plane coordinates, but ArcMap version 9.2 has problems determining capabilities from the MassGIS server and is not reliable other than in lat/lon.)
- Once you figure out how to get the School layer to be visible, try using the 'identify' button to find out information about particular schools. Notice that the 'identify' window wraps the data about the school in XML tags. If you were able to cut-and-paste the information into a browser or spreadsheet that understood the XML tags, the information would be displayed as a table.
- Show the location of public and private schools and colleges in the Boston area based on these two MassGIS datasets. (Note, just zoom in to the Boston/Cambridge area so the individual school flags can be distinguished. With these WMS servcies, we do not have a way to explicitly select the schools within, say, the city of Boston.)
- Because ArcGIS may not properly dialogue with the MassGIS site, you may not be able to get the map legends from MassGIS. For your map, just add a text message box explaining the flags.
- Note that a grayed out check mark next to a layer means it can not be mapped (wrong zoom, coordinates,etc.)
- Cambridge, Flood (or NGS-Topo), and MassGIS layers won't mashup well unless ArcMap knows (and handles) the projection of each layer
- Create a PDF-formatted map zoomed in to the Cambridge area showing the school location layer from MassGIS along with the Cambridge local roads and the Mass highway layer that you use in Lab 2. If you were successful in adding ESRI's gis services, you can add the USA Median Home Value (or NGS_Topo_US_2D) layers from the ArcGIS Online section.
- Optional: try some of the other OGC-compliant WMS services: http://www.skylab-mobilesystems.com/en/wms_serverlist.html
Google Mapshttp://maps.google.com/
Note "REST" interface (client side interface actions handled by Javascript, partial page refresh without a "submit" button)Microsoft Virtual Earth
Note (with free account) ability to sketch, and share simple sketch (with labels but not attributes, non-standard format)
Yahoo Maps (Beta)
http://maps.yahoo.com/beta/index.php
Note live traffic feed option, overview map leads full map for zoom interface preview. Also, mouse rollover on legend highlights search results dynamically.
Optional: Try the Google Earth + Sketchup exercise explained in the optional part of last Wednesday's Internet GIS lecture notes. From Google Earth, you add a 3D model of the Eiffel Tower and place it, at scale, onto Briggs Field near the MIT dormatories. The Eiffel Tower model was built using Google Sketchup and downloaded from Google's 3D Warehouse of user contributed 3D models. From Google Earth, you can save a snippet of surface image into a format that Sketchup can read as your ground location. Then you can use Sketchup to build a 3D model which you can then add back into Google Earth at the appropriate location.
1. Select a location in the U.S. which interests you. Using the national map or seamless.usgs.gov, find and download:
National Elevation Data (NED 1/9)
Land Use / Land Cover (NLCD)
Transportation (varies)
Hydrology (varies)2. Note the interface or data problems you find, if any.
3. Turn in both a map illustrating your area, and a brief paragraph describing the user experience. (Those of you actually wanting to use case study data later in the term should save it somewhere for later.)
Web services can be evaluated from the point of view of a consumer, from that of a producer, or from that of a policymaker. Pick one of these points of view, and using the sites above as examples, describe what you find to be the "state of the art." Address what works, what are best practices, what doesn't work and what you think is required.
Write up approximately 1 - 2 pages of text, with illustrated examples
Please do any one of the 4 Lab Assignment Options and submit them along with the PDF-formatted map to Stella. The assignment is due on Wednesday, November 8, 2010.
Created October, 2006, by Michael Flaxman and Joseph Ferreira based on exercise developed by Joseph Ferreira for use in the URISA Internet GIS workshop taught by Zhong-Ren Peng and Joseph Ferreira.
Last modified 1 November 2010 [jf]
Back to the 11.520 Home Page.