Nate Silver’s Iowa Caucus Predictions

Political season is data science season! Here is some more on Nate Silver’s forecasting methods. If you are reading this in real time (Sunday January 31), by tomorrow night we will find out what actually happens. I will reproduce some graphics here – these are all from the FiveThirtyEight site, so please thank me for the free advertising and don’t send me to copyright jail.

For Clinton vs. Sanders, here is Nate’s average of polls as of today. He gives more recent polls greater weighting, and also adjusts somehow for bias shown in the same polls in the past.

Average of polls: Clinton 48.0% vs. Sanders 42.7%

Now, this is within the 4-6% “margin of error” reported by most polls. (I find this easier to find on the RealClearPolitics site, although curiously it lists margins of error for Democratic polls but not Republican ones. RealClearPolitics does a straight-up poll average without all the corrections that today is Clinton 47.3% vs. Sanders 44%. So all the corrections don’t make an enormous difference.) I can’t easily and quickly find information on whether the “margin of error” is a standard error or a confidence interval or what, but generally when the polls are within the margin of error the media tends to report it as a “statistical tie” or dead heat. And that is exactly what they are saying in this case.

Nate Silver does a set of simulations – it sounds very complicated, but in essence I assume he takes his adjusted poll average for each candidate, some measure of spread like standard error, then runs a whole bunch of simulations. Which leads to results like this:

Clinton-Sanders Simulation

http://projects.fivethirtyeight.com/election-2016/primary-forecast/iowa-democratic/

Based on this, Nate Silver gives Clinton an 80% chance of winning Iowa and Sanders only a 20% chance.

So what’s interesting is that you have the average of polls (48-43 or 47-44 depending on source), which everyone says is a statistical tie. You have Silver’s predicted result (50-43) based on a large number of simulations, and then you have the resulting odds considering both the predicted result and the spread in the predictions (80-20). In other words, the computer is generating random numbers and 80% of simulations end up favoring Clinton. Of course in real life the dice get rolled only once, but these odds seem pretty good for Clinton.

Meanwhile, the Trump-Cruz contest is similarly close in the polls (30-25 in favor of Trump), but the predicted result (26-25 in favor of Trump) and odds (48-41 in favor of Trump) are much closer. From a quick glance, this appears to be because the spreads are much wider. I don’t know why that would be the case – presence of more viable candidates on the Republican side? Or maybe there is just more variability in the polls and nobody actually knows why.

Republican Iowa Caucus simulation

http://projects.fivethirtyeight.com/election-2016/primary-forecast/iowa-republican/

 

 

fixing sidewalks at point of sale

You can require homeowners (actually home sellers) to fix sidewalks when they sell their property (“point of sale”). Los Angeles does this, which Donald Shoup says is a great idea. Philadelphia has a similar legal framework where the homeowner technically owns and is responsible for the sidewalk, but does not require point of sale repairs. The city can write you a ticket for a broken sidewalk if they want to, just like any other code violation, but that is not too often done because the public outcry would be pretty severe. Also, it’s the transportation agency that is supposed to do this and not the code enforcement agency, so may not fit clearly within their mission and priorities. One more interesting thing the article mentions is that if sidewalks were made more clearly a code violation, this would show up in real estate records, and there would then be some pressure from mortgage and insurance companies to fix them.

Another option, of course, would be for the city to take over the sidewalk, along with street trees and infrastructure up to the house line. This could be part of a grand vision for how to implement 21st century urban infrastructure – complete streets, better materials than tired old concrete and asphalt, urban forests, public spaces, modernized water, sewer, gas, electric, and communications infrastructure. I for one would be indifferent to paying slightly higher taxes rather than being ticketed for code violations for I have very little control over, and this would also be more equitable. But it’s easier for the politicians to pin in on the private owner and say it has “zero cost”.

more things you don’t want to be true

Actually I’m not sure if I want this to be true or not, but

John Kasich has reasonable views on immigration.

Why wouldn’t I want this to be true? I’m not sure. I suppose because I don’t want to like any conservative governor after the awful one we just survived in Pennsylvania. In four short years he destroyed public education across the state, especially poorer urban areas, allowed gas companies to gut the environment, and did nothing to fix our state’s pension woes or entrenched poverty (either of which gas royalties could have at least made a dent in when energy prices were so high, but it’s a little late now.) Trading natural capital for financial capital, and restoring fiscal responsibility, are things a true conservative would do. About all he gave us was an entertaining pornography scandal. But I digress – if Kasich has a reasonable view on one issue, you have to admit it’s at least possible he might have reasonable views on other issues too. And I’m about ready to settle for reasonable with this crop of candidates.

what is a blizzard?

According to Five Thirty Eight,

Three factors are required for a storm to be classified as a blizzard at a particular place, besides falling or blowing snow:

1. Sustained winds or frequent wind gusts of 35 mph or greater

2. Visibility under a quarter-mile

3. These conditions must persist for three hours.

This definition is the same whether you’ve got 1 inch or 40 on the ground.

what to eat?

New U.S. government nutrition guidelines are out. And predictably, they are being criticized by nutritionists.

This year, for example, there were notable differences between the Advisory Committee’s recommendations and the final guidelines. According to scientific evidence, individuals should reduce their consumption of red and processed meats and sugar-sweetened beverages, such as soda, to prevent chronic diseases. The scientific evidence for those two recommendations is “so clear, so strong,” says Willett — yet neither recommendation was included in the final guidelines…

The final guidelines also don’t include the Advisory Committee’s emphasis on sustainability of the food supply, including the need to reduce portions of beef, cited as “the single food with the greatest projected impact on the environment.”

“This is virtual proof that the USDA is not allowed to say anything negative about red meat,” says Willett. “The basic censorship of the report from the Advisory Committee is deeply troublesome.”

The report from the advisory committee is not exactly censored. It’s here, readily accessible for people who know how to find stuff on the internet.

An R package for modelling actual, potential and reference evapotranspiration

I and some people I know will be very excited about this. Which means I know some very weird people.

An R package for modelling actual, potential and reference evapotranspiration

Evapotranspiration (ET) is a vital component of the hydrological cycle and there are a large number of alternative models for representing ET processes. However, implementing ET models in a consistent manner is difficult due to the significant diversity in process representations, assumptions, nomenclature, terminology, units and data requirements. An R package is therefore introduced to estimate actual, potential and reference ET using 17 well-known models. Data input is flexible, and customized data checking and pre-processing methods are provided. Results are presented as summary text and plots. Comparisons of alternative ET estimates can be visualized for multiple models, and alternative input data sets. The ET estimates also can be exported for further analysis, and used as input to rainfall-runoff models.

the fourth industrial revolution

Reporting fro Davos…er…Philadelphia – the theme of this World Economic Forum is “the fourth industrial revolution”.

The First Industrial Revolution used water and steam power to mechanize production. The Second used electric power to create mass production. The Third used electronics and information technology to automate production. Now a Fourth Industrial Revolution is building on the Third, the digital revolution that has been occurring since the middle of the last century. It is characterized by a fusion of technologies that is blurring the lines between the physical, digital, and biological spheres.

There are three reasons why today’s transformations represent not merely a prolongation of the Third Industrial Revolution but rather the arrival of a Fourth and distinct one: velocity, scope, and systems impact. The speed of current breakthroughs has no historical precedent. When compared with previous industrial revolutions, the Fourth is evolving at an exponential rather than a linear pace. Moreover, it is disrupting almost every industry in every country. And the breadth and depth of these changes herald the transformation of entire systems of production, management, and governance.

The possibilities of billions of people connected by mobile devices, with unprecedented processing power, storage capacity, and access to knowledge, are unlimited. And these possibilities will be multiplied by emerging technology breakthroughs in fields such as artificial intelligence, robotics, the Internet of Things, autonomous vehicles, 3-D printing, nanotechnology, biotechnology, materials science, energy storage, and quantum computing.

SOTU

I’ll pull out a few quotes from Obama’s State of the Union that are relevant to the theme of this blog.

First, automation and globalization:

Today, technology doesn’t just replace jobs on the assembly line, but any job where work can be automated. Companies in a global economy can locate anywhere, and they face tougher competition. As a result, workers have less leverage for a raise. Companies have less loyalty to their communities. And more and more wealth and income is concentrated at the very top.

What automation and globalization have in common is that if you are a relatively low-skilled worker in a relatively high-income country like the U.S., there is a risk your job could be replaced either by a computer (automation) or a low-skilled worker in a low-income country (globalization). Where they differ is that automation is starting to squeeze those low-skilled workers in the low-income countries too, and gradually it will also start to squeeze the higher-skilled workers in the higher-income countries. Obama’s solutions to all this – education and training, unemployment and wage insurance, healthcare and childcare benefits to make employment more flexible, lowering barriers to entrepreneurship, are the obvious ones, but we’ve been tinkering with these things for a long time with only slow progress, and the trends are only going to accelerate.

Second, biotechnology and genetics:

Last year, Vice President Biden said that with a new moonshot,America can cure cancer. Last month, he worked with this Congress to give scientists at the National Institutes of Health the strongest resources that they’ve had in over a decade. (Applause.) So tonight, I’m announcing a new national effort to get it done…let’s make America the country that cures cancer once and for all.

This seems to be a nod to biomedical research and biotech more generally, which I am convinced is the next big technology revolution akin to the information revolution we have been going through over the past few decades.

Next, climate change and fossil fuels:

Now we’ve got to accelerate the transition away from old, dirtier energy sources. Rather than subsidize the past, we should invest in the future — especially in communities that rely on fossil fuels. We do them no favor when we don’t show them where the trends are going. That’s why I’m going to push to change the way we manage our oil and coal resources, so that they better reflect the costs they impose on taxpayers and our planet. And that way, we put money back into those communities, and put tens of thousands of Americans to work building a 21st century transportation system.

Actually, I am not sure what he is talking about here. I would support a revenue-neutral carbon tax, with the proceeds invested in education, training, research and/or infrastructure. But I’m only speculating. If there was some initiative announced along these lines I missed it.

Finally, corruption in U.S. politics:

We have to reduce the influence of money in our politics, so that a handful of families or hidden interests can’t bankroll our elections. (Applause.) And if our existing approach to campaign finance reform can’t pass muster in the courts, we need to work together to find a real solution — because it’s a problem… Those with money and power will gain greater control over the decisions that could send a young soldier to war, or allow another economic disaster, or roll back the equal rights and voting rights that generations of Americans have fought, even died, to secure.

This is pretty vague. I would support a constitutional amendment to clarify that a person is a human being and a human being is a person. Human beings should have the right to free political speech, but corporations and other special interest legal entities should not. The law can be written to preserve the important rights corporations do have that create a fair and predictable playing field for businesses to compete – equal protection under the law, access to the courts, protection from arbitrary seizure of property, and so forth. But the richest and most powerful shouldn’t be able to buy politicians and write the rules of the game unfairly in their favor.

On the possibility of those right-wing self-interested corporate entities joining forces with right-wing grass roots impulses, resulting in something truly ugly:

But if we give up now, then we forsake a better future.  And then, as frustration grows, there will be voices urging us to fall back into our respective tribes, to scapegoat fellow citizens who don’t look like us, or pray like us, or vote like we do, or share the same background.

I wouldn’t have believed that was likely a year ago, but here we are approaching the official beginning of an election season that is turning out to be very surprising, with Obama riding off into the sunset.

Strong AI

Here’s an article about “Strong AI“:

This is how computers interact with us. They don’t understand English or whatever language we speak, but they do understand binary code. A human programmer tells the computer how to respond to you based on your input. The same thing goes for chess. The computer does not at all understand that it’s playing chess or any other game – it doesn’t even know what a game is. It only knows that, if you make a move in chess, that equates to some machine code to which it should respond. It then references its giant chess book – put there by humans and written in machine code – to decide how to respond to you.

Over the years, these tomes have become enormous. Using the chess example again, it would take many years for a human to sift through one of these tomes, but a computer can do it in seconds. As a result, you now have computers that simulate the total sum of human knowledge with regards to chess and yet they don’t understand a lick of it. What they cando, however, is obey the instructions put there by a human to beat you, handily…

Once we have the technology to make an ambulatory, perceptive robot using pre-written instructions (i.e. the ‘tome’), the challenge will then be to ‘birth’ one that has no pre-written instructions and no prior knowledge of the world, forcing it to learn like a human child. This sort of self-learning robot is an example of what Searle calls ‘Strong AI[10]’.