Tag Archives: data science

November 2016 in Review

Sometimes you look back on a month and feel like nothing very important happened. But November 2016 was obviously not one of those months! I am not going to make any attempt to be apolitical here. I was once a registered independent and still do not consider myself a strong partisan. However, I like to think of myself as being on the side of facts, logic, problem solving, morality and basic goodness. Besides, this blog is about the future of our human civilization and human race. I can’t pretend our chances didn’t just take a turn for the worse.

3 most frightening stories

  • Is there really any doubt what the most frightening story of November 2016 was? The United Nations Environment Program says we are on a track for 3 degrees C over pre-industrial temperatures, not the “less than 2” almost all serious people (a category that excludes 46% of U.S. voters, apparently) agree is needed. This story was released before the U.S. elected an immoral science denier as its leader. One theory is that our culture has lost all ability to separate fact from fiction. Perhaps states could take on more of a leadership role if the federal government is going to be immoral? Washington State voters considered a carbon tax that could have been a model for other states, and voted it down, in part because environmental groups didn’t like that it was revenue neutral. Adding insult to injury, WWF released its 2016 Living Planet Report, which along with more fun climate change info includes fun facts like 58% of all wild animals have disappeared. There is a 70-99% chance of a U.S. Southwest “mega-drought” lasting 35 years or longer this century. But don’t worry, this is only “if emissions of greenhouse gases remain unchecked”. Oh, and climate change is going to begin to strain the food supply worldwide, which is already strained by population, demand growth, and water resources depletion even without it.
  • Technological unemployment may be starting to take hold, and might be an underlying reason behind some of the resentment directed at mainstream politicians. If you want a really clear and concise explanation of this issue, you could ask a smart person like, say, Barack Obama.
  • According to left wing sources like Forbes, an explosion of debt-financed spending on conventional and nuclear weapons is an expected consequence of the election. Please, Mr. Trump, prove them wrong!

3 most hopeful stories

3 most interesting stories

Nate Silver and college football

I thought Nate Silver only looked at professional sports. I was wrong – here is a cool interactive web page he has put together for college football. The numbers don’t always give you the answers you want to hear though – even if my beloved Gators somehow win all the rest of their games, which would include beating Alabama in the conference championship game, he gives them only a 13% chance of winning the national championship. Another nice thing about Nate Silver – he always explains his methodology.

We’ll be updating the numbers twice weekly: first, on Sunday morning (or very late Saturday evening) after the week’s games are complete; and second, on Tuesday evening after the new committee rankings come out. In addition to a probabilistic estimate of each team’s chances of winning its conference, making the playoff, and winning the national championship, we’ll also list three inputs to the model: their current committee ranking, FPI, and Elo. Let me explain the role that each of these play…

FPI is ESPN’s Football Power Index. We consider it the best predictor of future college games so that’s the role it plays in the model: if we say Team A has a 72 percent chance of beating Team B, that prediction is derived from FPI. Technically speaking, we’re using a simplified version of FPI that accounts for only each team’s current rating and home field advantage; the FPI-based predictons you see on ESPN.com may differ slightly because they also account for travel distance and days of rest…

Our college football Elo ratings are a little different, however. Instead of being designed to maximize predictive accuracy — we have FPI for that — they’re designed to mimic how humans rank the teams instead.4 Their parameters are set so as to place a lot of emphasis on strength of schedule and especially on recent “big wins,” because that’s what human voters have historically done too. They aren’t very forgiving of losses, conversely, even if they came by a narrow margin under tough circumstances. And they assume that, instead of everyone starting with a truly blank slate, human beings look a little bit at how a team fared in previous seasons. Alabama is more likely to get the benefit of the doubt than Vanderbilt, for example, other factors held equal.

R code to read Nate Silver’s data

Thanks to Nate Silver for posting all his polling data in a convenient text file that anyone can read! It’s a nice thing to do. Even though not many of us can do as interesting things with it as Nate Silver, it is a fun data set to play and practice with. Here is an R-bloggers post with some ideas on how to play with it.

 

weather forecasting history

I recently wrote about earthquake forecasting and how many scientists think it is essentially impossible. But it is interesting to compare that with the state of weather forecasting in the 1800s:

Before the Royal Charter storm, FitzRoy had been agitating in London for government funding for collection of weather data. He and other Victorian men of meteorology knew that the more they could parse what the weather had done in the past, the better they could warn what it might do in the future. FitzRoy called the concept “forecasting.” To show just how ludicrous that idea seemed at the time, Moore unearths a telling 1854 Commons debate. When a scientifically enthusiastic member of Parliament suggested that amassing weather observations from sea and land could someday mean “we might know in this metropolis the condition of the weather 24 hours beforehand,” laughter broke out raucously enough to stop the proceeding.

earthquake prediction

Here’s an interesting article on earthquake prediction. Basically, it has eluded scientists so far and will probably continue to do so.

Since the early 20th century, scientists have known that large quakes often cluster in time and space: 99 percent of them occur along well-mapped boundaries between plates in Earth’s crust and, in geological time, repeat almost like clockwork. But after decades of failed experiments, most seismologists came to believe that forecasting earthquakes in human time—on the scale of dropping the kids off at school or planning a vacation—was about as scientific as astrology. By the early 1990s, prediction research had disappeared as a line item in the USGS’s budget. “We got burned enough back in the 70s and 80s that nobody wants to be too optimistic about the possibility now,” says Terry Tullis, a career seismologist and chair of the National Earthquake Prediction Evaluation Council (NEPEC), which advises the USGS.

Defying the skeptics, however, a small cadre of researchers have held onto the faith that, with the right detectors and computational tools, it will be possible to predict earthquakes with the same precision and confidence we do just about any other extreme natural event, including floods, hurricanes, and tornadoes. The USGS may have simply given up too soon. After all, the believers point out, advances in sensor design and data analysis could allow for the detection of subtle precursors that seismologists working a few decades ago might have missed.

And the stakes couldn’t be higher. The three biggest natural disasters in human history, measured in dollars and cents, have all been earthquakes, and there’s a good chance the next one will be too. According to the USGS, a magnitude 7.8 quake along Southern California’s volatile San Andreas fault would result in 1,800 deaths and a clean-up bill of more than $210 billion—tens of billions of dollars more than the cost of Hurricane Katrina and the Deepwater Horizon oil spill combined.

 

Nate Silver weighs in

Nate Silver has launched his general election forecast page. He gives Hillary about an 80-20 chance of winning. He has a long discussion post about it here. I found this last paragraph interesting, where he relates a 20% chance of winning to a baseball game:

A 20 percent or 25 percent chance of Trump winning is an awfully long way from 2 percent, or 0.02 percent. It’s a real chance: about the same chancethat the visiting team has when it trails by a run in the top of the eighth inning in a Major League Baseball game. If you’ve been following politics or sports over the past couple of years, I hope it’s been imprinted onto your brain that those purported long shots — sometimes much longer shots than Trump — sometimes come through.

It’s an interesting way of thinking about risk. Let’s say your favorite team is in game 7 of the World Series, down by a run in the top of the eighth. The game is insanely late on the east coast as they always are, and you have to do something important early the next morning, like interview for a job or operate heavy machinery. Do you turn the TV off? No, of course not, you stay tuned.

June 2016 in Review

3 most frightening stories

  • Coral reefs are in pretty sad shape, perhaps the first natural ecosystem type to be devastated beyond repair by climate change.
  • Echoes of the Cold War are rearing their ugly heads in Western Europe.
  • Trump may very well have organized crime links. And Moody’s says that if he gets elected and manages to do the things he says, it could crash the economy.

3 most hopeful stories

  • China has a new(ish) sustainability plan called “ecological civilization” that weaves together urban and regional planning, environmental quality, sustainable agriculture, habitat and biodiversity concepts. This is good because a rapidly developing country the size of China has the ability to sink the rest of civilization if they let their ecological footprint explode, regardless of what the rest of us do. Maybe they can set a good example for the rest of the developing world to follow.
  • Genetic technology is appearing to provide some hope of real breakthroughs in cancer treatment.
  • There is still some hope for a technology-driven pick-up in productivity growth.

3 most interesting stories

index of redevelopment potential

This is an index of redevelopment potential for individual properties in Philadelphia and other cities. The application to real estate is obvious, but I can also see a number of applications to public policy. For example, changes to codes and ordinances can improve the overall health, safety, and environmental impact of a city. But these get implemented slowly and incrementally, especially in older cities with fixed boundaries, where most development is redevelopment. If you had a reasonable prediction of where and when redevelopment is likely to occur, you would know which areas to sit back and be patient, and which areas of the city to intervene more directly if you want to see change on any reasonable time frame.

It’s a little bit of a shame the person is not sharing their methodology. I’ve had a number of urban planners and economists tell me over the years this is very hard to do, and seen a few try and give up. So this is either brilliant, or it is little more than a guess. If it’s brilliant it could be very valuable indeed, so I guess I can see the financial incentive not to publish the details. But there is no way to know the difference without knowing how it is done. The author could at least publish a white paper showing some back testing of the algorithm against historic data.