11.520: A Workshop on Geographic Information Systems |
11.188: Urban Planning and Social Science Laboratory |
Not all web services and web sites allow you to examine or determine what they are doing behind the scenes. However a significant number, especially those based on open standards, do allow this. Learn to parse URLs yourself (allows you to repurpose sites and services, debug errors)
1) Transactions are usually "stateless"
All of the information needed for the server to meet client's request is bundled into URL
(Exception: sometimes "cookies" are used to associate state based on a user ID or session ID)2) Parsing URLs
Overall URL format is protocol:[optional port number]://path/to/resource/scriptname?parameters
A question mark separates path/scriptname from parameters list¶meter1=blah¶meter2=blah+again
The ampersand separates parameters, and the equals sign its assigned value
Spaces are not allowed in URLS, so "+" symbol is substituted
To clarify, paste URL into a text editor and substitute line breaks for question mark and ampersands3) Example (standard Web Mapping Service or WMS):
http://maps.massgis.state.ma.us/mapaccess/main.jsp?dpi=120&request=GetMap&layers=MHD+Roads&styles=Class&srs=EPSG:26986&bbox=233500,900000,236500,902500&width=640&height=480&format=image/png&service=wms
Becomes
http://maps.massgis.state.ma.us/mapaccess/main.jsp
dpi=120
request=GetMap
layers=MHD+Roads
styles=Class
srs=EPSG:26986
bbox=233500,900000,236500,902500
width=640
height=480
format=image/png
service=wmsNot self-explanatory, but a step in the right direction. Full WMS description can be found here for those interested:
OpenGIS Web Map Service (WMS) Implementation Specification
Explore these sites briefly, noting the issues above and last week's lecture notes
EPA's Enviromapper:
http://www.epa.gov/enviro/html/em
Special permission required for access to underlying services (e.g., Oracle account)
U.S. Department of Transportation Federal Highway Administration
What is the annual average daily traffic volume on Mass. Ave?
http://hepgis.fhwa.dot.gov/hepgis_v2/GeneralInfo/Map.aspxYes, there is a map on this site that color codes roads based on estimate traffic volume. However, the interface offers limited help capabilities and the right map is hard to find. Do not spend too long hunting for it. Just get a general sense of what the site offers and what you do/do-not like about the site capabilities and user interface.
Other Traffic info sites:
- 'Traffic' tab on Microsoft's Virtual Earth: http://local.live.com/
- Washington DC traffic (Maptuit/SmartRoute/Westwood-One site via NBC4): http://www.nbc4.com/trafficmaps/index.html
U.S. City Site
Portland Maps (City of Portland, Oregon)
B) Sites With Data Download Capabilities
The (US) National Map
http://nationalmap.gov/(cover page)
http://nmviewogc.cr.usgs.gov/viewer.htm (direct link to mapping interface)
USGS-led effort at coherent, cross-agency national map layersUSGS
http://seamless.usgs.gov/
Most important US government site for International Users because provides free access to Global 90m Terrain Data that can be downloaded in a non-poprietary file format, SDTS, that most GIS packages will support.
(SDTS = a Spatial Data Transfer Standard)University of Maryland Global Land Cover Facility
http://glcf.umiacs.umd.edu/data/ (overview)
http://glcfapp.umiacs.umd.edu:8080/esdi/index.jsp (direct access to mapping interface)Best place to download free Landsat imagery (warning: huge and unweildy) and global 1km resolution land cover classification. U.S. Government data policy on Landsat has varied over the years. At the moment, access to ETM+ (enhanced thematic mapper plus) is for cost, but older TM and MSS data are available without charge.
Geospatial OneStop (Portal for Federal GIS data)
http://gos2.geodata.gov/wps/portal/gos
Latest version is ESRI-powered under the covers. "Drinking from a fire hose?"
Canadian Geospatial Data Infrastructure (CGDI)
http://www.geoconnections.org/CGDI.cfm/fuseaction/maps.welcome/gcs.cfm
Other national spatial data infrastructure (SDI) sites
Indian National Data Infrastructure (viewer may be down or slow)
MassGIS
http://mass.gov/mgis
good state agency website for finding and disseminating geospatial datawon URISA ESIG award last year (for web services part)
Accessible at several levels:
online mapping: http://mass.gov/mgis/mapping.htm
downloadable datasets: http://mass.gov/mgis/database.htm
web services: (discussed in next section)
http://ortho.mit.edu
Note: 12 year old website, but was designed with a level of open access still not present in many other sites (why?)'Seamless' interface: http://ortho.mit.edu/nsdi/seamless6.cgi
Example of HTML-wrapped image: (try other values)
http://ortho.mit.edu/nsdi/seamless6.cgi?zoom=8&x0=237000&y0=902000&action=pan&pwidth=640&pheight=480&x=123&y=169Same ortho snippet but without HTML wrapper: (try other values)
http://ortho.mit.edu/nsdi/seamless8.cgi?zoom=8&x0=237000&y0=902000&action=pan&pwidth=640&pheight=480Parameters are now a little different in the WMS 1.2 'get map' standard
From browser, but not in browser (actually downloads heavy but very capable Java GIS client):|
Oliver http://maps.massgis.state.ma.us/massgis_viewer/index.htmDescription of Issues in creating/generating this service.
http://ogcuser.opengeospatial.org/node/90
C) Using APIs
Yahoo Traffic API using "REST" web URLS (see REST description below)
http://developer.yahoo.com/traffic/rest/V1/index.htmlWhat is the current traffic on Mass Ave at MIT?
http://local.yahooapis.com/MapsService/V1/trafficData?appid=YahooDemo&street=77+Massachusetts+Ave&city=Cambridge&state=MA&include_map=1
Your browser will not display the results on a map but will just list the XML that is returned. This is an example of a spatial query that return a non-spatial result. The results contain XML-wrapped text and X,Y locations explaining traffic events near the address sent in the URL. Examine the tags and content. With the 'include_map' flag set to one, the URL for a map of the area is included in the results. Cut-and-paste the URL for the map into another browser window.
United Nations Environment Programme
http://geodata.grid.unep.ch/Notes. WMS Services / UMN Mapserver software behind the scenes. Most data at general national levels only, some regional data sets. Lots of drill-down required to navigate the interface and generate specific maps and data tables.
http://www.arcwebservices.com/awx/index.jsp
What it does: General web client interface backed by ArcWeb services. Note custom geographic reporting capability. Also note "map types" but no layer-level user control exposed.
How it works: web Services inside Flash-based Client inside browser
Which data are available and how much do they cost? (Some public services, other per hit)
http://www1.arcwebservices.com/v2006/help/awx/datasources.htm
Web-search approaches:
Use government web sites such as GOS above, or WMS client software sites
Index of OGC WMS Servers
MassGIS: http://mass.gov/mgis
- The MassGIS layers listed above can be accessed as OGC-compliant WMS and WFS web services as explained at: http://lyceum.massgis.state.ma.us/wiki/doku.php
- Example URL requesting WMS image (street map) using earlier web services: (try pasting into a browser) http://maps.massgis.state.ma.us/mapaccess/main.jsp?dpi=120&request=GetMap&layers=MHD+Roads&styles=Class&srs=EPSG:26986&bbox=233500,900000,236500,902500&width=640&height=480&format=image/png&service=wms
- Equivalent URL requesting WMS image (street map) using newer MassGIS web services: (try pasting into a browser): http://giswebservices.massgis.state.ma.us/geoserver/wms?request=GetMap&layers=massgis:GISDATA.EOTMAJROADS_ARC&srs=EPSG:26986&bbox=233500,900000,236500,902500&width=640&height=480&format=image/png&service=wms
ESRI's ArcWeb Services: http://www.arcwebservices.com
- You need to register with ESRI to use their ArcWeb Services
- Sign up for your own 'evaluation' account, or a public/educational account
Access from ArcMap
- Start/ArcGIS/ArcMap\
- Open some familiar MA and Cambridge shapefiles from the class data locker
- Next, Sign in to ArcWeb Services:
- Tools/Online-Services/Sign-in-to-ArcWeb services
- Login using your own userid/password for arcwebservices
- Click the ‘Plus-sign’ button (Add Data)
- Navigate to top level of Catalog (shown in 'Look in' box) and select 'ArcWeb Services'
- Select 'FEMA Flood' (paying services or 90-day evaluation) or ‘NGS_Topo’ (public/educational)
- If a dialog box opens with a warning that the new layers use different coordinate systems from your data frame and reprojection may not be exact, ignore and click 'Close'.
- Note (lack of) responsiveness of layers due to server-side delay
- Try the ‘info’ button
- Recognize how source data in different coordinates have been merged
The free map viewer from ESRI, ArcExplorer v. 9.2, can access ArcWebServices (but not OGC-compliant WMS services)
Accessing WMS services from ArcMap
- Again, click the 'Plus-sign' button (to Add Data) in an ArcMap window with MA towns and Cambridge block groups and ArcWebService layer
- Under 'GIS Servers' choose 'WMS Servers'
- Enter this URL for the MassGIS WMS service
- http://maps.massgis.state.ma.us/mapaccess/main.jsp?
- Click 'OK' and then 'Add'
- If you click 'Get Layers' it will take 1+ minutes to get the long list; just click 'ok' after entering the URL
- Beware that the L for the newer MassGIS web service site (http://giswebservices.massgis.state.ma.us/geoserver/wms?) will not yet work properly with ArcMap v. 9.2 (service pack 3)
- Once back in ArcMap, the layer may still not show if zoomed in/out too far
- BEWARE: at the moment this MassGIS webservice will produce maps visible in ArcMap *only* if the Data Frame in ArcMap is set to lat/lon coordinates. There is a bug in the ArcMap handling of the MassGIS web services since the native projection for the MassGIS server is Mass State Plane, but only lat/lon views will be downloaded and displayed. You can force the DataFrame to view the map in lat/lon coordinates by right-clicking on the Layers icon in the Table of Contents, choosing Properties/Coordinate-system, and then setting the 'pre-defined' coordinate system
to be: Geographic-Coordinate-System/North-America/North-American-Datum-1983- Find 'schools' under 'Infrastructure'
- Note that a grayed out check mark next to a layer means it can’t be mapped (wrong zoom, coordinates,…)
- Cambridge, Flood (or NGS-Topo), and MassGIS layers won’t mashup well unless ArcMap knows (and handles) the projection of each layer
- Create a PDF-formatted map zoomed in to the Cambridge area showing the school location layer from MassGIS and the flood plane (or NGS_Topo) layers from ESRI ArcWebServices along with the Cambridge local roads and the Mass highway layer that you use in Lab 2.
Google Mapshttp://maps.google.com/
Note "REST" interface (client side interface actions handled by Javascript, partial page refresh without a "submit" button)Microsoft Virtual Earth
Note (with free account) ability to sketch, and share simple sketch (with labels but not attributes, non-standard format)
Yahoo Maps (Beta)
http://maps.yahoo.com/beta/index.php
Note live traffic feed option, overview map leads full map for zoom interface preview. Also, mouse rollover on legend highlights search results dynamically.
Boston.com election contribution website
http://www.boston.com/news/special/bigarticles/campaign_finance/page2.html?p1=email_to_a_friend
Live 'mashup' webpage (for Back Bay):
http://boston.faneuilmedia2.com/gov/detail_map.html?from=top&zip=02116&z=13
Note: you can 'pan' the map or click on dots for further info
How was it done socially/technically? Overlay/identify entries in local election contribution database maintained by consultant (Faneuil Media, Inc.)
Looking under the covers
"View source" in your browser. Look for call to google with machine/directory-specific Key:
http://maps.google.com/maps?file=api&v=2.60&key=AB...
"REST" is a programming technique which uses client side javascript to interact with the server without full page refreshed. All mapping APIs are moving towards this, pressured by Google maps success. REST stands for Representational State Transfer and involves ways of exchanging information between client and server regarding the state of the client so that every exchange between client and server does not require a complete reconstruction of the client-viewed webpage. AJAX programming is one strategy for enabling complex client-side actions using new information from the server without having to rewrite all of the client webpage. AJAX stands for Asymmetric Javascript and Xml and uses javascript to send URL requests, parse the XML-tagged responses, and use the returned information to rewrite parts of the client-side browser window.
Optional Note: hard to get beyond "push pins" user interfaces
In last Wednesday's lecture, we did not have time to get to this portion of the powerpoint slides - i.e., to a discussion of web services involving mapping where the web service renders reports and maps whose content and symbolization are filtered and customized by 'intelligent middleware' applications that can be owned and shared among other users. We may have time to discuss some of these examples and technical strategies later in the semester. We have included a few references and sample URLs, but we suggest that you wait to try them out until after you have finished Part V - the lab assignment.
Main page for Middleware tools: http://uis.mit.edu/umi/auth
Sample webpage for DSNI properties illustrating how a call to the middleware web services can be embedded in an plain HTML page using iFrame:
Development in the Dudley NeighborhoodView Source and lookfor URLs in IFRAME tags
Copy URLs for table and map into Firefox browser address bar
Edit URL for report to remove style sheet parameter
Wikka documentation of various Middleware web services: http://uis.mit.edu/wikkaTry other Web Services – top-10 owners in Dudley Triangle: http://uis.mit.edu/umi/service/report.php?Request=QueryReport&name=r_top10_lotsize&tablename=parcels_dsni3&studyarea=triangle&xsl=http://uis.mit.edu/umi/xl/report.xsl
Try entering this URL into Excel as Web Query
1. Select a location in the U.S. which interests you. Using the national map or seemless.usgs.gov, find and download:
National Elevation Data (NED 1/9)
Land Use / Land Cover (NLCD)
Transportation (varies)
Hydrology (varies)2. Note the interface or data problems you find, if any.
3. Turn in both a map illustrating your area, and a brief paragraph describing the user experience. (Those of you actually wanting to use case study data later in the term should save it somewhere for later.)
Web services can be evaluated from the point of view of a consumer, from that of a producer, or from that of a policymaker. Pick one of these points of view, and using the sites above as examples, describe what you find to be the "state of the art." Address what works, what are best practices, what doesn't work and what you think is required.
Write up approximately 1 - 2 pages of text, with illustrated examples
Please do any one of the 4 Lab Assignment Options and submit them along with the PDF-formatted map to Stella. The assignment is due on Monday, Oct 29th, 2007.
Created October, 2006, by Michael Flaxman and Joseph Ferreira based on exercise developed by Joseph Ferreira for use in the URISA Internet GIS workshop taught by Zhong-Ren Peng and Joseph Ferreira.
Last modified 21 October 2007.
Back to the 11.520 Home Page.