Refineries produce a large proportion of the harmful waste products which later have to be disposed of in very specific ways in order to avoid harming the environment even more. In addition to that, conventional refineries create their own toxic wastes that must also be accounted for.
Description of Problem
The refining processes that are typically found paired with hardrock mining are just as harmful to the environment as the mining itself; perhaps even more so. Although mining exposes all of the contaminants, refining isolates and concentrates waste, in order to isolate and concentrate the desired final product. Refining processes also often use harsh acids and bases to separate the desired metals from the wastes, which can harm the environment by themselves when they change soil and water chemistry. For example, some fish can only live in water within a narrow pH range, and some plants only flower within a certain pH range.This can have far-reaching effects on agriculture and livestock, as well as on wilderness areas. 
See comments button for this one's comment. The Chinese have also used sulfuric acid refining techniques that generate 9,600 to 12,000 cubic meters of gas "laden" with flue dust concentrate, hydrofluoric acid, sulfur dioxide, and sulfuric acid to produce one ton of rare earth elements. Chinese refining processes also produce large amounts of liquid and solid waste; they estimate that after refining one ton of rare earth elements, approximately 75 cubic meters of acidic waste water and about one ton of radioactive waste residue are produced. The Institute for the Analysis of Global Security (IAGS) has reported that China produced over 130,000 metric tons of rare earth elements in 2008. Combining this figure with the figure for waste generation estimates that production yields very large amounts of waste - 1.2 billion to 1.6 billion cubic meters of waste gases per year, and 9.8 million cubic meters of acidic waste water. China has little environmental regulation, and there are many stories of pollution near areas with rare earth element production facilities, indicating that rare earth element production facilities can produce great amounts of waste Is there a percentage of stories of pollution near these areas or some indication that there's an actual causation factor between the facilities and the pollution?. 
Rare earth element ores tend to contain a range of different metals in their structure, including aluminum, arsenic, barium, beryllium, cadmium, cobalt, copper, lead, manganese, and zinc. These are the most harmful metals that are found in rare earth mines, and are separated from the ores during the refining process, as well as allowed into the open air in mines. Metals can be found in the air, the water, and the soil, which makes them concerning contaminants, since they are able to exist in all three environmental mediums. Since metals can only react into different compounds in the environment, not be destroyed, they can stay for a very long time in an ecosystem, which can allow them to accumulate to dangerous levels. The state that the metals are in dictates the severity of the toxicological effects in organisms, since the molecular state of a metal determines how easily it can be absorbed - smaller particles that are not bonded together are absorbed more easily. The three most dangerous heavy metals in the environment are lead, cadmium, and mercury, but all can be dangerous in high concentrations.  The amount of metal that the organism takes in is important as well. In humans specifically, many heavy metals are carcinogens. Some, like aluminum, can cause birth defects, while others, like arsenic, are simply toxic, shutting down vital organ systems. 
In addition to being found with the ores naturally, most mining operations use metals, reagents, or other compounds to process valuable minerals. Some reagents or heavy metals, such as cyanide and mercury, are valuable for use - mercury is used for its state as a metallic liquid to form amalgams with rare earth metals in order to extract the amalgams from the other impurities which are not metallic  and cyanide is used in a similar way as an extraction reagent . Metals can also be released into the environment in waste products. Small amounts of heavy metals, like those found when mining for rare earth elements or other ores, are essential for the survival of many organisms, but large amounts are toxic. Most terrestrial and aquatic species are ill-adapted to the presence of large quantities of heavy metals, especially some varieties of fish; for example, salmon are very sensitive to increased copper content, with juvenile fish being affected to a greater extent.  Mining and processing can also emit dangerous byproducts into the air if appropriate preventative measures are not taken. Major concerns with air emissions include radioactive particles and dusts with heavy metals, both of which can be inhaled. Refining in particular can cause even more damage with air emissions, which can include SO2, which is a component that assists in the production of acid rain and particulates in the air, and HCl, which becomes a strong acid when exposed to water, such as the water in the atmosphere, along with the aforementioned radioactive substances and metals, if no preventative measures are taken.   
Mining and refining can also consume great quantities of water, which can create a significant problem for ecosystems surrounding a mine or refinery, especially in an already arid climate, because then the operation drains the ecosystem of water, thereby unbalancing it. High CO2 emissions may also occur during the refining process, which could contribute to further climate change, since CO2 is a greenhouse gas. 
Many rare earth refining facilities in China do not have sufficient treatment systems for the harmful products produced in their refineries, despite the huge amount of waste produced. Some of the smaller rare earth smelting separation facilities do not have any kind of environmental protection at all. These are generally the small, illegal refineries and mines. 
In China, saponification with ammonia is still used in rare earth refining, so a large amount of waste water is still produced. Based on the production of 103,900 tons of rare earth oxide in 2005, it is estimated that approximately 20,000-25,000 tons of waste water are produced every year, though with new regulations, this number may be decreasing. To separate the ore of one ton of rare earth concentrate that has a rare earth element content of 92 percent rare earth oxide, 1-1.2 tons of ammonium bicarbonate are needed. The waste water then contains an ammonia and ammonium (NH3 and NH4+ respectively) content from 300 mg/litre to 5000 mg/litre, which exceeds the limit set by the Chinese government by more than ten to 200 times respectively.