The public consultation official "standard" process consisted in the distribution of the Non-Technical Summary (NTS) of the EIA to newspapers, community groups, churches and NGO’s, together with a leaflet describing the essential of the process and where citizens could consult the EIA and obtain more information. Although not required by law in this case, the institute in charge of public consultation (IPAMB) also scheduled two public hearings. In this particular case, other instruments of consultation were also made available, including the ones set by my experiment. The period lasted 30 business days (27 May - 10 July), coinciding practically with the last of the 120 business days of the EIA review. In this chapter I describe the public consultation process and the experiment's components of it, such as an opinion survey and the use of IMS (both prototype and web), with brief references to observations made also after its end.

Besides of the IMS project initiative, other resources were made available for the first time for the public consultation. Altogether, there were components on the Internet, and others on Macintosh microcomputers.

The IMS prototype was installed and available at: IPAMB (Environmental Ministry Institute in charge of public consultation), DGA, DRARN-LVT {Ministry of Environment's Agencies}; Secretary of State for Environment; FCT-UNL {Faculty of Science and Technology, New University of Lisbon}, LPN {Environmental Protection League}; GEOTA {Environmental and Land Use Planning Study Group}; and CITIDEP {Research Center on Information Technologies and Participatory Democracy}.

The IMS prototype requirements were: Quadra or PowerPC (Macintosh), minimum 8 Mb (preferably 16 Mb or more), system 7.5 or more recent; disk space, minimum 30 Mb, preferably 300 Mb. Desirable: CD-ROM drive (the prototype was later distributed on CD, with 650 Mb of data). Ideal (but not required): modem for Internet connection. Required software: Hypercard (2.3 or more recent). In the scope of this project, whoever requested it, had at their disposition the following software: Hypercard 2.3 (Mac), Eudora (Mac or Windows), Netscape (Mac or Windows). There was also the possibility to facilitate Internet PPP access when needed.

It is important to note that because of the constraints and difficulties, described in the previous chapters and discussed later on, the IMS data was only fully available near the end of the public consultation period. It was also not widely advertised: the press conference at the Ministry of Environment to publicize the public consultation and the new multimedia tools supporting the consultation was held only a few days before its end. This had an obvious impact on its use.

The actual number of users and "visitors" to both complementary systems during the "legal" period of the EIA public consultation was low (29 recorded users for IMS, 184 web trails visitors counted, total of 213), and roughly equivalent to the number of participants in the public audiences. Considering that several of those page hits could be from recurrent visitors, the probable number of persons that visited the web trails was even less. However, the number of informal, non-recorded users of the IMS prototype was probably at least the double, bringing the total, in my estimate, to near 90 IMS users.

The number of emails received by IPAMB was almost insignificant (around a dozen), with only 3 opinions published on IPAMB web page. Although here other factors may have an influence, as already referred and discussed later.

Still, the few IMS users during the consultation period provided anecdotal evidence indicating strong user interest and no major difference between "blue-collar" workers and citizens or students with higher education, in what concerns difficulty in use. In fact, the separating lines were clearly along the variable "motivated" vs. "less motivated" citizens, and not "expert" vs. "lay" citizens.

One very interesting phenomenon is that long after the EIA review was over, the Web site with the EIA FAQ went on receiving visitors, inclusive with higher daily visit rates than before. A few of them kept sending questions and comments by email; and their accumulated number was 13304 by April 13, 2002. Even discounting the casual "web surfers" brought by the continuous increase of Internet access, this phenomenon remains worthy of further discussion.

The use of the IMS prototype by citizens with lower level schooling, as well as their way of handling the system, was not much different from other users, like students or experts: both asked some support and guidance about how to launch and operate the system, not very differently. None seemed troubled with the English user interface. One interesting observation is that citizens would come in groups and usually one of them was more at ease with the computer. This one handled the system, while the others watched and guided him on what they wanted to see and what information they wanted to get.

Another important consequence of "real-world" conditions, was the interesting feedback concerning IMS prototype design, leading me to program on-the-fly some of the requested improvements.

The more visible one was to allow users to accede to information in the "Virtual Office" module not just by content (either keyword or selecting questions from the FAQ list), but also by author. I quickly programmed and added that user interface feature, as shown in Fig. 5.12.3.-1

The expressed rationale was that they were interested in knowing what X or Y had to say about the subject, sometimes because they did not have any specific question in mind, or did not know very well where to start. Checking opinions of known people or of people with institutional responsibilities was the best way to begin; then, they could go on from there using the FAQ list for further inquiry.

Later, after the legal period of the public consultation, it was when some actors found the time to test and try more leisurely the content: with DRARN-LVT and with a few ENGO’s, in particular LPN, who wrote a feedback report (Moreno 1996). The essential of the LPN’s feedback was the following:

a) LPN representative had an issue with presenting side by side Valorsul’s position and other’s, like them. This could convey the message that there was a peaceful coexistence on their stands, or even worse, convey the impression that LPN was condoning Valorsul stands. Also, people could get confused at the end of a system session, mixing what was Valorsul’s opinions and what was LPN’s (or other’s) opinions, not remembering anymore who said what.

Curiously, this is exactly the same kind of concern expressed by some public participation decision makers, as presented in previous chapters. However, neither the anecdotal evidence gathered from users nor the opinion surveys (presented later in this chapter) seem to support these concerns.

b) LPN representative did not understand (and did not agree) why should the facility promoter, Valorsul, have a "double representation" and therefore double space within the system as compared to all others: Valorsul "office" and EIA "office". In their view, these two represented exactly the same entity, Valorsul, since they paid for the EIA.

This identification between Valorsul and their EIA consultants was a frequent accusation during the process, but from my own observation, as described in the previous chapters, there was some differences between the two voices, at least in some degree.

Besides other comments, LPN representative made an assessment of the perceived advantages of a system like the IMS:

LPN presented also corresponding suggestions to improve the user interface:

a) To reserve a fixed screen area for each actor, like one for Valorsul, other for ENGOs, etc. The assumption is that this would minimize the confusion on who said what;

b) To limit the volume of information dedicated to each actor, or originated by each actor, in order to equalize entities with very unequal resources. Otherwise, some have time and money to insert a lot of answers and documents and others much less, creating an unbalance in the system’s content.

These are interesting suggestions, and particularly the second addresses a real problem. Curiously again, not very far from the concerns expressed by public administration decision-makers, on system content bias, although the origin (and orientation) of this bias was seen differently. Whether it is feasible or not to impose such restrictions, it remains to find out with further research. By then, it was too late to include other changes in the experiment.

Citizens using the IMS prototype during this period expressed, without exception, that they found it interesting and that they got out of a session with it useful information, with a more detailed view of the two alternative solutions for the "garbage problem" (as they typically called the solid urban waste), presented by Valorsul and the ENGOs. In particular, they all considered that the FAQ covered the majority of the issues they were interested to query about.

Two citizens from S. João da Talha inserted (audio) recorded answers to 3 questions, from public hearing interventions, for the following questions :

The large majority of the IMS users during the legal period of public consultation came to IPAMB. That provided several opportunities for demonstrating the fully loaded system to senior staff at IPAMB. Their feedback was very positive with supportive observations in general, such as how good this would be for instance for newspaper reporters covering the EIA review, but raising also pertinent issues on the difficulties we would face in rural areas for this kind of system.

Less accessible for the public in general, the other locations with versions installed on desktop computers were used almost only by a few people working at the respective institution. For instance, at DRARN-LVT, senior staff used it, exploring all components of the system. Curiously, the Secretary of State for Environment, in the short time he had to test the IMS, favored the above mentioned new feature, checking who answered questions in the system and then checking a few of the authors’ answers.

Videos were, without doubt, the most requested type of document files among all media formats.

As mentioned in the experiment models chapter (on user models), I included a "trace" function that recorded the steps each user took while using and navigating through the IMS.

This function allows to reconstitute the user interaction with the system, and eventually detect some behavior pattern. As expressed then, there was no precise expectations on this regard, and while the information collected is interesting, it does not allow to conclude in favor of some kind of "user types".

To process the "trace" data, I developed a simple software tool that "parses" each line record, calculates and keeps track of time periods spend on each step, or object, or user interface environment, or even more specifically on each question asked by the user, and the order of those steps. Table 5.12.3. - 1 shows a small sample of the kind of data collected through this object-oriented trace function.

Fig. 5.12.3.-3 shows the IMS Module used to perform trace analysis and generate this information.

Each record (card) in the tool represents one session by one IMS prototype user. One single user may have several sessions with the system. The tool identifies automatically these sessions, inferring from factors such as new user identification, startup commands, or time lag between two steps much larger than reasonable. The session duration is calculated, and the software parses all steps. From there, the tool can generate lists of meaningful sub-sets, like all IMS prototype "buildings" (user interface environments) visited, or all questions asked, or all user interface objects used (buttons, links), etc. From these lists, the tool can produce charts and graphics.

Fig. 5.12.3.-3 shows the different kind of information that can be extracted from this trace data, with an example from a single user. The order of steps performed by the user is kept, from top (beginning) to bottom (end).

Users show different patterns, but the "Virtual Office" ("Experts") was clearly the dominant feature used, as shown in Figs 5.12.3.-4 with data from 3 of them:

The column on the left of Fig. 5.12.3.-4 visualizes all the user interface commands used by citizens C. F. and A. From them we can see that they stayed more or less time exploring the cover "entry" screen, and then all went to the "Reception Lobby" to identify themselves ("insert information" command). One chose to set some preferences different from the default, and from there their paths diverge. Two of them made considerable use of the "Help" feature (in Portuguese).

The column on the right visualizes the order of visit and time spent in the major user interface environments. All of the three users took a look at the "raw" file data base ("Archives"), but clearly the "Virtual Office" ("Experts") was their focus of attention. This was the only common pattern an all the recorded users.

Users also made use of the feature allowing them to generate, in real-time, a multimedia booklet around a chosen theme (IMS Trails), through a combination of keywords. Fig. 5.12.3.-5 and 5.12.3.-6 show two of these trails, generated respectively for keywords incineration and garbage (lixo):

Curiosity was also on the IMS team itself. One of the trails generated was about it. Fig. 5.12.3.-7 shows one of the pages of the respective multimedia booklet.

IPAMB organized two public hearings, the first in S. João da Talha, proposed site for the incinerator, and the second in Lisbon.

In these hearings, the stage is set with two separated tables. In one of the tables was Valorsul, with some of their EIA consultants that were presenting the EIA; in the other table were representatives of the Review Committee, which included at least one representative from IPAMB, chairing the session. As explained always at the opening of a session, the two separated tables was a deliberate setting to emphasize that these two actors were independent from each other, with different roles in the EIA review process.

The official meeting minutes were included in the EIA Review final report. I tape-recorded both. In here I present my own observations, with a few examples of events that help to describe the essential and are evidence of the difficulties faced by the traditional public consultation framework (only non-technical summaries for the public at large, and - not always - these hearings).

The public hearings at S. João da Talha took place first, with around 150 persons present at beginning. This hearing began at 8.30 PM and lasted near 6 hours, although with fewer people present near the end. This gives already an indication on how strongly many of the present felt about having their say on the microphones, and the careful handling of the meeting by both tables, that did not use the late hours as an excuse to close the session before all the inscribed could speak.

Valorsul and their consultants presented the EIA conclusions, in a very professional manner, with plenty of slides and diagrams. One of the first notorious reactions came when one expert was presenting the survey results on the perception of risk by the population of S. João da Talha. She concluded by saying that the significant percentage of perception of high risk, even if not founded, was in itself a negative impact, because of the stress and related aspects it caused. A woman sitting in the audience spoke loud: "So we will get sick not because of the incinerator, but because we are dumb? Is this what you are saying?". Many in the audience laughed. The expert was clearly taken aback by the reaction, since she thought she was making a statement that was actually negative towards the incinerator’s impacts.

The overall tone of citizen’s interventions was more of expressing their fears and discontent, sometimes outright rage, on the prospect of having to live with an incinerator on their backyard, rather than ask questions and seek information. One citizen began his intervention by saying "we are all going to die, this is going to become a desert", other said he could not even sell his house and leave, because the property values fell drastically and his family was stuck there; etc.

The feeling shown by many citizens of dismay, of being abandoned by society and institutions, of betrayal, was highlighted when the Mayor of Loures came in. Some citizens yelled at him, "you betrayed us", other cried he had seared his party membership card. The Mayor faced without flinching all the reactions and made a brief intervention, re-instating his view of the advantages of the CTRSU for Loures and the country. Some remarked that despite everything, the Mayor had the guts to show up and say his word, contrary to other actors (besides the Review Committee members, no other major politician or public officer was visible).

But there was several interesting questions asked, even if sometimes rhetorically. One citizen explained he had no schooling, he was just a car engine mechanic, but he understood a lot about engines. He said the best Rolls-Royce engine will always break up and need to be fixed at some point; the incinerator sure had plenty of engines and machinery, what was going to happen when they would malfunction? The EIA provided a view of the impacts if all was going to function as expected, but what about the impacts of malfunctions?

Other citizen asked: "in page X of book Y of the EIA, it says there will be Z amount of Dioxins produced". He did not know exactly what was a Dioxin, but page XX said it was a dangerous poison, so it was bad news. But "was this Z amount going to be produced by month, by year, or what?", that was certainly important and (according to him) the EIA did not say, therefore "it was not a good EIA".

These and other questions alike were a clear demonstration that the non-technical summary (NTS) alone was not an answer to their concerns. One citizen actually ridiculed the NTS: "According to this (NTS), the only thing is left to say is that we should walk with our babies on the grass near the incinerator, so good is it going to be. This (NTS) is written like if we were little children. We may not have college degrees, but we are not stupid".

Fig. 5.12.4-1 shows one view of this public hearing, from a generated IMS Trail.

The other public hearing, a few days later, took place in Lisbon, at LNEC (a National Laboratory on Civil Engineering). This meeting had less people attending (around 55 when it began, at 5.30 PM, by my estimate) and lasted about half the time of the other.

While some citizens of S. João da Talha came also to this hearing, and expressed negative views, the overall tone was less dramatic, with more technical discussion among consultants, experts from academia and experts from environmental NGOs (ENGOs). For instance, there was no applause or "buus" at the end or during some interventions.

Maybe because of the climate of the public hearing at S. João da Talha, interventions from ENGOs there, were strong and thorough but sometimes inconsistent with their own general proclaimed views. For instance, at some point one representative of one of the ENGOs asked the citizens of S. João da Talha "since Loures does not have an acute problem with solid urban waste, like Lisbon has, why should citizens of Loures put up with the incinerator?".

By contrast, at the LNEC public hearing ENGO’s experts made several sober presentations, for instance recognizing some problems with past experience with composting, etc. and presenting systematic, carefully argued alternatives. National leaders of some ENGOs intervened also on the strategic issues in question, referred in previous chapters.

The thesis experiment included a small opinion survey, about the incinerator issue and Valorsul’s proposal. This survey was distributed during the two public hearings described above and published on the web. The number of distributed printed copies of the survey was relatively small (around 100). There was no pretension of obtaining a statistically meaningful sample, only another indicator of the "climate" of the public hearings, to complement my own direct observations. In this sense, the survey results were indeed useful. Later, it was also distributed to two groups of undergraduate students (psychology and environmental engineering), having in mind a controlled experiment, described next chapter.

The questions in this survey, shown in table 5.12.5.-1, were discussed with IMS Expert Panel members, in order to distinguish between overall concerns with environment, general opinions on urban solid waste handling and concrete opinions about Valorsul’s proposal. Special care was taken with the options offered and their symmetrical scaling. It was also suggested that each person should respond twice, one before the hearing, the other after. In fact, except for the case of the controlled experiment with students, only very few filled two surveys. For comparative purposes, in the next pages, I present also a summary of the results of the student survey (corresponding to "before event" surveys).

Fig. 5.12.5.-1 and 5.12.5.-2 show the summary results of the surveys at the two public hearings. For comparative purposes, Fig. 5.12.5.-3 and 5.12.5.-4 show the surveys collected from students at the controlled experiment described next chapter. The background data for each survey is in table 5.12.5.-2.

With due caution against trying to read more than an indication of anecdotal evidence, it is nevertheless interesting to note the following:

• The large majority of the surveys collected at S. João da Talha show high levels of disapproval of Valorsul’s proposal, as compared for instance with the other hearing audience. This is not surprising.

• Among the very few surveys that were indeed filled in twice as requested, only one person changed opinion after the meeting, but still that change was from negative to very negative. On the other hand, many surveys (29) had extensive comments (some were actually only comments, with no answer to survey questions), and a few wrote several suggestions for improving Valorsul proposal. I transcribe below a few of these comments.

• The visible high percentage of blue collars in the S. João da Talha hearing, with ages above 40 (where very little schooling predominates), may have been a factor preventing some of the recipients of the survey from filling it, even if they were willing to participate.

• There is no coinciding opinion between the universe of people favoring Valorsul’s proposal, and the universe of people considering the environmental impact of Valorsul’s proposed incinerator as positive. This is interesting, in the sense that may originate either in giving more value to other factors in question than environment, or in considering Valorsul’s proposal the "least of the evils", as suggested by some of the written comments.

Here are some of the comments written in the survey forms:

Finally, while the Internet survey was not very visible (only 30 visitors, with 5 surveys sent), compared with the hearings, the response rate at the public hearings was much higher (near 40%).

"NIMBY" concept was itself a mystification precisely because it pretended to present citizens of areas in the neighborhood of problematic facilities as mindless selfish people when in fact they had all reasons to be the ones more concerned, as compared to others more distant to it.

In the view of some members of the IMS Expert Panel, the true NIMBY concept should be defined as "I don’t pay attention unless it is in my backyard". Whatever the interpretation, my own surveys are consistent with the influence of the citizens’ area of residence in the general trend of their opinions. Fig. 5.12.6.-2 show the results from all IMS opinion surveys (public hearings and students), by area of residence.

Soon after the public consultation ended, it ended also the EIA review .

The NGOs position, as mentioned in the institutional response chapter was delivered in a joint document, signed by the 3 major ENGOs: Quercus, GEOTA, LPN.

In this well articulated, 8 page document (Quercus, GEOTA & LPN 1996), the ENGOs present their strategic views and opposition to the planning process, as well as their specific criticisms to the incinerator as proposed by Valorsul. For them, the solution was: "predetermined by the past", "not integrated", "not sustainable" and relegating to a insignificant role composting and recycling. They point to the risks of the incineration and the contradictions in the EIA: between different parts of it but more in particular between the data presented and the conclusions extracted. They criticize the non-technical summary (NTS), as omitting "all CTRSU problems and all unfavorable arguments". They conclude by recommending that a proper planning process should be applied, and that the incinerator’s proposal should not be approved before the completion of such planning. They state that it is important that a strategic plan and integrated plans for regional areas should not assume already the option of incineration, on the contrary, should give priority to the 3R policy. (Reduce, Recycle, Re-utilize). Finally, they consider that given the weight of Valorsul in the context of Portugal (15% of the population, 20% of the waste), this consortium has special responsibilities in creating the right incentives towards consumer habits and economic agent’s behavior in line with the 3R policy.

The EIA Review Committee integrated in its final report a report on the public consultation, including the written opinions of a few citizens and the ENGOs document. Their final report contained several recommendations concerning deficient studies in some areas and how to minimize impacts in others..

This process was concluded with a favorable decision by the Environmental Minister in August 5 of 1996, on condition that the following measures where satisfied:

(Ferraz de Abreu and Chito 1997)

Therefore, the process ended with the Environmental Ministry basically deciding to impose on Valorsul these further studies and changes towards minimizing impacts, but giving green light to the incinerator.

I also tested the use of IMS with students from undergraduate programs (5th year Environmental Engineering at the New University of Lisbon and 3rd year Psychology at the University of Lisbon), as part of their course work. I conducted a controlled experiment, for better evaluating the role of the new information technology introduced in the EIA review process, in what concerns reducing the gap between experts and lay citizens, when dealing with technical information to form an opinion.

This experiment included an opinion survey -- the same done during the public consultation -- and a knowledge test. This chapter describes this controlled experiment, the knowledge test content and the results both from the survey and the test. The discussion is left for the next section.

After consulting with the IMS Expert Panel, I prepared a "knowledge test" about the EIA for the S. João da Talha incinerator (CTRSU). The objective of the test was to be one of the elements of a controlled experiment on the use of the IMS and help to evaluate how the use of the system allowed non-expert users to improve their understanding of technical information.

The questions were conceived to differentiate between the user area of expertise, user motivation on the subject, user degree of familiarity with the case, user ability to distinguish the different stands from the actors involved and their proposals, and user ability to grasp more in-depth understanding of the concepts in discussion. Fig. 5.13.2.-1 show the "knowledge test" tool, with respective questions.

The questions had obvious different levels of difficulty and depth. For instance, we offered for consideration 3 different solution sets in question 6:

we then asked the user to identify for each of the 3 solutions its best advantage and its worse disadvantage, from a given set of parameters: cost, energy, water quality, air quality, soil quality, volume reduction, land use / soil occupation. The probability of answering with some sense this question just by chance was considerably lower than with the others.

One of the questions (area of expertise of the user) was introduced just as a calibrating factor. As it happens, given the results of the test it was not necessary to use any calibration, as presented next.

The test counted with the participation of 35 students from undergraduate programs (10 from a 5th year Environmental Engineering at the New University of Lisbon and 25, from a 3rd year Psychology at the University of Lisbon). Two members of the IMS Expert Panel, faculty at these Universities, proposed to their students to collaborate in the experiment, as part of course work. The controlled experiment took place almost a year after the EIA review period, and was organized the following way:

For each user,

1) Fill the opinion survey on Valorsul proposal;

2) Answer the Knowledge Test, in 15 minutes;

3) Use the IMS prototype, during 20 minutes;

4) Answer again the same Knowledge Test, in 15 minutes;

5) Fill again the same opinion survey on Valorsul proposal.

The whole procedure was briefly explained to them, so they were aware they were expected to repeat the exact same test and survey.

The opinion survey was the same distributed during the public hearings and published as an on-line form (web).

The use of the IMS was preceded by a short demonstration and presentation, just like it was done during the public consultation sessions at IPAMB and elsewhere.

It was suggested to the subjects to imagine themselves in one of the following roles:

a) Just appointed to an EIA Review Committee, preparing for its first meeting;

b) Just designated by their neighborhood club, or their family, to form an opinion on the EIA in review and report back for group discussion;

c) Dropping by the public consultation office, to give their input for the EIA review.

and, in all cases, with only a few minutes to spare. Naturally, it was the subject’s choice to ignore any of this suggestions, since they could use the system as they wished.

The knowledge tests were graded according to a previously defined scale (considering the different degree of complexity and difficulty of the questions).

While the number of students involved is too small (35) for any significant statistical evidence, the results are interesting and indicate a pattern. Table 5.13.5.-1 show the grade results:

The fact that the worse grade average of environmental students is still better than the best grade average of psychology students is a good indicator of the adequacy and credibility of the test, since it was to be expected that environmental students, in their last year of the course, would know more about the issue in question than their (younger) colleagues of psychology.

What is more relevant is that the grade gap between environmental and psychology students was reduced after using the system. This suggests that technology like IMS can be helpful in reducing the "knowledge gap" between lay citizens and experts, in what concerns their contribution in public consultation involving technical matters.

Naturally, this experiment alone does not allow to conclude this will be always the case. But the experiment results are positive (and suggestive) evidence that such outcome is possible.

The results of the opinion survey among the students, before using the IMS, was shown already in the previous chapter (5.12.5.-3 and 5.12.5.-4). Their general trend is within expectations. A higher percentage of students of environmental engineering think incineration is bad for the environment, compared with their colleagues of psychology; they also show lower rate of "no opinions". When evaluating Valorsul’s proposal, the percentage of "No opinion" increase considerably (up to 50% or more), even among students of environmental engineering, making it reasonable to assume that many did not know the proposal, or did not remember enough details to form an opinion.

This is why in here (Fig. 5.13.5.-1), I include these opinion surveys but comparing their results before and after using the IMS prototype during the assigned 20 minutes of the experiment. These results are more interesting and show some unexpected patterns.

The first observation is the significant decrease of the "No opinions" on all questions, by more than 50%. This is a fair indication that the system provides intelligible information, even in such a short period of time, since at least many felt they had acquired sufficient information to form an opinion. That this opinion was well founded in general, is reflected by the knowledge test results.

The second observation is the curious trend of the change of opinions. While a good number of students moved from considering the incineration as a good alternative or good for the environment, to consider it, after using the IMS, a bad alternative and bad for the environment, a different pattern occurs in the evaluation of Valorsul’s proposal. In the last case, both approval and disapproval percentages show a clear increase (nearly the double), with the new formed opinions dividing evenly between favorable and unfavorable views concerning Valorsul’s proposal.

This denotes reasoned opinions, beyond simplistic "taking sides" for ENGOs or Valorsul. Clearly ENGO’s arguments cautioning on incineration’s negative impacts and the existence of better alternatives found an echo; but apparently so

did Valorsul’s argument that theirs was the best realistic solution for the current constraints, and better than prolonging the "open sky" garbage dump sites.

Besides all possible interpretations, what is noticeable is that the changes of opinion after using the IMS were more in the sense of moving from a "No Opinion" to some opinion, than from one opinion to a different one. This suggests that the IMS, at least as it was presented for public consultation, did not induce any bias favoring one actor versus another.

Again, the sample is too limited to allow any generalization of such conclusion to all IMS use, or IMS-like information technology. But it surely is a solid indicator, founded in experimental evidence.

It was also interesting that most students filling the opinion surveys used the "free comment" area to give feedback not on the subject of the survey -- their opinions on Valorsul’s proposal and related issues -- but on the IMS prototype. Given the insight they bring, I transcribe here a few of those comments:

"Even in the short time I was given for consultation, I learned and got some useful information. For instance, I had almost no idea of what was composting. Now I know. This is a useful system and easy to operate (even for those who dislike computers)"

With this user feedback, a proper register for such rich combination of new technologies, collaborative efforts and interesting institutional responses, it ended the IMS thesis experiment. In the next chapter I summarize its findings.

The thesis experiment was a complex project, in a controversial case with many different actors and interests at stake. Its stated goal was to test the introduction of new IT in a decision-making process, observing the IT performance and the institutional response, at every step. The experiment findings are a rich set of information about that performance and responses, but also complex, with multiple levels of observations and evidence. Just as it was needed a structure to design the experiment, it is useful to follow some equivalent structure to present its findings.

In the chapter characterizing the actors in this case, I summarized their expectations as to the role and performance of the new IT. In the chapter describing the experiment’s models of expectations, I delineated more specific tests for each facet of the experiment: EIA decision-making, public participation, knowledge representation; knowledge acquisition, Information system user. However, corresponding specific hypotheses (in all models) were defined in the context of an unchanged decision-making institutional framework, and intended to serve also as a test whether this current framework allowed improvements brought by the new IT. All these aspects provide a good structure to describe the experiment findings.

In this chapter I present a brief summary of the experiment findings, after comparing them with the modeled expectations. The discussion is left for next section

I wanted to test the feasibility of certain improvements in the decision-making model, enabled by the new IT introduced:

5.14.2.1. Concerning the EIA structure and presentation:

The experiment proved that it was feasible in both aspects (structure and media), with the successful achievement of a dual taxonomy and corresponding FAQ intelligent multimedia format. The main finding here is precisely the importance of considering not one single "domain" taxonomy, but also a related "Issue" taxonomy, as described in this section. The other is the importance of choosing an adequate representation model, suitable for the kind of data and knowledge in question, through a process of knowledge engineering, as described. For this particular test, the FAQ model proved adequate.

The prove arises from the fact that it was done and from the feedback obtained from all actors, in favor of the improvements. However, the late availability of the fully loaded system, a direct consequence of institutional and regulatory constraints, limited the reach and generalization of this feedback.

5.14.2.2. Concerning the nature of the non-technical summary and its relationship with the overall EIA:

The experiment proved the feasibility of this improvement, in the same sense of the previous paragraph.

5.14.2.3. Concerning the "modus operandi" of the EIA Review Committee, in particular the work division between thematic areas (health, air, soil, etc.), the articulation between the technical review and the public consultation, and the evaluation of the public consultation itself:

In part due to the imposed institutional regulatory constraints (on timely access to the EIA and on the access to Internet) and in part due to the lack of familiarization and lack of spread use of the new IT (specially the Internet), the experiment was not conclusive in this aspect, although the feedback from most senior and junior members of the EIA Review Committee shows they were convinced of this potential of support from tools such as the IMS and Internet.

I wanted to test the feasibility of certain improvements in the public participation model, enabled by the new IT introduced:

The introduction of the IMS and the FAQ Web trails expanded the access, but the number of users during the legal period of public consultation was relatively small, and on the same order of magnitude of the numbers of people present in "traditional" Fora (such as attending the public hearings or consulting the printed volumes at a public office). The IMS and Web users probably added to the numbers of the "traditional" resources’ users, although there is no direct evidence of this. Finally, a CD-based version was made only very later in the process, for the reasons described, and its distribution was in limited numbers, among the actors and a few journalists.

Again, there was the clear presence of institutional and regulatory constraints (such as the impossibility to accept email as a legal input, with an obvious dissuasion effect). It is not possible to assert whether an earlier availability would bring larger numbers of citizens into the process; although this late availability had also a direct bearing with institutional constraints. However, there is the evidence that the Web site with the EIA FAQ continued to accumulate visitors, in numbers that with time surpassed the number of citizens involved in the public consultation.

In the end, there is no doubt that, even discounting the casual "web surfers", the total number of people who read at least parts of the EIA through the FAQ on the Web is considerable larger than the number of people that consulted the EIA through the "traditional" process (going to a public office to consult the printed volumes). Of course the question is if this can occur during the limited period assigned for public consultation. In my view, if we consider that nowadays the number of people using the World Wide Web is vastly superior, it is reasonable to extrapolate that this will increase significantly the audience of an EIA public consultation process, even in such a brief 30 day period.

IMS users during the public consultation period claimed they had profited from the system to obtain useful information and understand better the different alternatives presented by Valorsul and the ENGOs. Evidence in the same direction was gathered from the controlled experiment with students.

While IMS may have been at the origin of some pertinent questions, with technical nature, from citizens with self-proclaimed little schooling in S. João da Talha (dioxin, frequency of filter clean-up, problems with break up periods, etc.), there is no evidence of that, and in any event it was a minor part in the middle of so many discussions that marked the public consultation.

In this regard, it is very likely that traditional media (TV, newspapers, etc.) dominated, showing more impact on the top of the moment, like the several references made by citizens and ENGOs to a TV advertisement from Valorsul. Again, late availability of the system is at the origin of the lack of conclusive evidence. It is suggestive, though, that people kept sending emails with pertinent questions after visiting the web site with the FAQ trails, long after the EIA review was over. This may be an indication that Web media has a cumulative, memory effect, where citizens get information also on past cases to enrich their understanding of current ones. An example of this are the renovated concerns on the incineration of hazardous waste, an issue that surfaced again after the CTRSU case and that may be the motivation behind some of the web traffic peaks.

I hypothesized that this "Intelligent Multimedia FAQ" model would be able to:

This was one of the more conclusive findings of the experiment. Indeed it was proven that it was possible to anticipate the FAQ, since many questions that arose during either the technical review or the public hearings were present, in one formulation or another, in the FAQ list. At the same time, the experiment provided valuable information on the requirements of a knowledge representation process. It is interesting that, even with the experiment’s more conclusive evidence, despite all institutional context constraints, this context was very much present: not only did it shape the way the FAQ was compiled but also acted as a catalyst factor to better reveal the nature of the institutional imprints, in different planning paradigms, as I will argue in the discussion section.

The feedback from the public consultation show that citizens made use of features like the multimedia book generation (IMS Trails) and clearly found attention-grabbing the use of sounds, photos and videos (specially videos), what may also be attributed to the novelty factor. This was confirmed by the students’ statements in the controlled experiment and confirmed overall by the log of file calls, where video files dominated by large.

As for the impact in the form of shaping questions from citizens, no conclusive evidence was gathered. The above comments about the public participation model tests apply equally here.

This model has some built-in assumptions that I wanted to test:

The experiment findings are better formulated in the following way: it was proved that it is possible to agree on an acceptable and functional structure, like the dual taxonomies developed. However, the experiment also shows that there is no unique, common "standard" structure or taxonomy for this planning knowledge, and that many other acceptable ways of organizing and structuring the knowledge may very well exist. Finally, agreement on a taxonomy was possible but not easy. The experiment shows the importance of a guided effort in that direction, with the clear goal of reaching a practical solution, accepting trade-off’s.

The experiment clearly shows that is not the case. Without a deliberate, planned effort, involving significant resources, the natural evolution is towards a unbalanced FAQ; either predominating the motivation factor, as at the beginning with the predominance of the critical views, or dominating the resource factor, as at the end, with the massive involvement of Valorsul in the FAQ, turning the bias in the other direction.

The experiment also provided rich insights concerning the institutional response to the knowledge acquisition process and the perceived bias in the intermediate - and final - stages. It is of particular interest to note that this concern was not exclusive to public administration decision makers, but shared, although with different or opposite interpretations and evaluation of the bias, with almost all actors with a direct stake in the outcome. Besides the mentioned concerns from administrative decision makers, the evidence of this is on the ENGO (LPN) feedback and Valorsul reactions, described in the previous chapters.

The experiment certainly proves two things in this case: first, that it is feasible, strictly from the point of view of the information technology’s ability to perform, in particular with knowledge representation and acquisition technologies, since it was done; second, that it is not feasible, under the current institutional and regulatory constraints, to do it on time (to be of real use). The experiment findings also point to the direction in which these constraints must be changed: either the EIA study is released long before the period of public consultation, or this period must be extended considerably. Ideally, both.

I proposed to test the feasibility of a IT/IS user model, shaped the following way:

The experiment proved this aspect is feasible.

The experiment proved that this is feasible, from a strictly IT point of view, since it was done. However, it also has shown that institutional and legal constraints (like the non recognition of email as valid input) are sufficient condition to make it not viable.

An interesting note: Portugal had to wait until 1998, to see a law establishing equivalence of emails to printed letters, at the same time it defined the legal context for electronic signatures.

Again, there are mixed results. The experiment proved it is possible to build a workable and functional model such as this (questions and answers with different technical depth), but the example of the sudden changes made by the ISP without advanced warning, eliminating all counter functionality, together with the lack of resources to fix the problem, show that nothing can be taken for granted in this matter. It is interesting however that again the problem comes from an institutional setting - this time, the monopoly of this ISP in the market of Internet Service Providers (a former state monopoly in the telephone and telecommunication services), whose behavior is not unrelated with its monopoly characteristics.

Also, the experiment shows it is not easy to identify many different "logical" sequences of questions (trails) with uniform technical depth level. This suggests that further refinement is required in the design of the knowledge acquisition process, in order to make sure that FAQ sources pay attention to this aspect.

The experiment proved it is feasible to build such a model, including different user interface paradigms (function or metaphor-based), and acquire interesting information on user behavior, providing thus a path for incremental improvement of the user interface of these systems.

In fact, it was not possible to detect classes of users, given the small sample available for any meaningful analysis. But it was interesting to detect that some components of the system, in this case the IMS "Virtual Office", were clearly favored by all users in this experiment.

In table 5.14.7.-1, I present a summary of the findings according to each role the new IT was expected to perform, by the different actors in this case. Naturally, I include myself as one of the actors, in all components of this expectation.

Among the experiment findings, some were predictable (like the Internet and web advantages, the positive reactions of citizens towards IMS innovation for the public consultation, etc.). Others were more or less predictable but not in the final shape they took, like the implementation problems and the need for more tool development and fine tuning during knowledge acquisition and web publishing phases; or like some institutional responses raising obstacles on Internet access and use. Some were not predictable although expected, like the IMS ability to contribute to reduce the gap between people with different degrees of expert knowledge and to form reasoned opinions, as well as not inducing a built-in bias in those new formed opinions. Some were not predictable but somehow contradicted partial expectations, like the difficulty to use rule-based knowledge representation and the enormous comparative success of the FAQ model.

However, some were not only unpredictable but also unexpected, sometimes even surprising. Among them, are: the unbalance between gathering questions versus gathering answers for the FAQ; the institutional response to this FAQ list; the different-than-expected behavior of some actors concerning their degree of proximity and involvement with the IMS project, for instance public administration staff and environmental NGOs; and the sustained increase of visitors to the EIA FAQ Web trails, long after the end of the EIA review process.

While all findings are important indicators towards proving or disproving my hypothesis, the unexpected are usually the more interesting and some of them require further analysis. In some cases they also required further investigation to test interpretations, which I did through documentation research and in particular with a small series of interviews with some of the actors.

At first sight, the general direction of the evidence and indicators gathered through the thesis experiment point to information technology performing close to expectations but hindered by several institutional and regulatory impediments. This is consistent with my hypothesis. But some elements are not so straightforward. Given the multiple aspects derived from the immediate findings of the experiment, in particular from the unexpected elements among them, the next section is dedicated entirely to its discussion.