Here’s an interesting article on predictive policing from Motherboard. People are concerned that if a particular area has been overpoliced in the past, that is where the algorithms are going to predict crime in the future and they will continue to be overpoliced. Others just don’t like the idea of proprietary algorithms. I think any of these concerns could be badly depending on how it is implemented, but I don’t see why the tool itself could not be implemented in a fair way. In fact, I don’t see why measures to prevent discrimination couldn’t be built into the algorithms themselves. If the algorithms say people in a particular area or in a particular demographic group are being arrested at higher rates, it could help the search for route causes and preventive measures to help a particular group revert back to the mean. Transparency seems good in principle, maybe publishing some generalized statistics and maps, but of course if it is too predictable exactly where the police are going to be and when, people could take advantage of that. You could try to get around this by balancing random and targeted patterns within the algorithm.
Tag Archives: data science
August 2018 in Review
Most frightening stories:
- In certain provinces with insurgent activity, the Chinese government is reportedly combining surveillance and social media technologies to score people and send those with low scores to re-education camps, from which it is unclear if anyone returns.
- Noam Chomsky doesn’t love Trump, but points out that climate change and/or nuclear weapons are still existential threats and that more mainstream leaders and media outlets have failed just as miserably to address them as Trump has. In related news, the climate may be headed for a catastrophic tipping point and while attention is mostly elsewhere, a fundamentalist takeover of Pakistan’s nuclear arsenal is still one of the more serious risks out there.
- The U.S. government is apparently very worried about a severe cyber attack. Also, a talented 11-year-old can hack a voting machine.
Most hopeful stories:
- There are some new ideas for adjusting GDP to account for natural capital and ecosystem services. There are also ideas to better account for “intangible products” like software in GDP. And R&D is a good investment that the U.S. could do more of.
- While the U.S. uses a few less straws and pats itself on the back, there are serious ideas in other countries for tackling the root problem of packaging.
- Vancouver has successfully combined green street and complete street concepts. The American Society of Landscape Architects has also compiled some helpful resources on this topic.
Most interesting stories, that were not particularly frightening or hopeful, or perhaps were a mixture of both:
- Google Lens can identify a plant or animal from its picture, and the subway body scanners from Total Recall are now real.
- There are some neat high-tech camp stoves out there that can burn almost anything with very little smoke and even charge electronic devices.
- I found a good article about making box plots in R.
box plots!
One of my nerdy interests in box plots. And no, you can’t make great ones in Excel. Here is a blog about making fantastic ones in the R package ggplot2.
March 2018 in Review
Most frightening stories:
- One reason the U.S. blunders into war repeatedly is that it does not do a good job of analyzing the motives of its adversaries.
- International investors may be losing confidence in the U.S. dollar. And a serious financial crisis in China is a possibility, although China is also trying to become a “cyber superpower“.
- One reason propaganda works is that even knowledgeable people are more likely to believe a statement the more often it is repeated.
Most hopeful stories:
- One large sprawling city could be roughly the economic equivalent of several small high-density cities. This could potentially be good news for the planet if you choose in favor of the latter, and preserve the spaces in between as some combination of natural land and farm land.
- The problems with free parking, and solutions to the problems, are well known. This could potentially be good news if anything were to be actually done about it. Self-parking cars could be really fantastic for cities.
- The coal industry continues to collapse, and even the other fossil fuels are saying they are a bunch of whining losers. And yes, I consider this positive. I hope there aren’t too many old ladies whose pensions depend on coal at this point.
Most interesting stories, that were not particularly frightening or hopeful, or perhaps were a mixture of both:
- Some people really do win the lottery more than they should.
- You can buy a computerized chicken coop or cider maker.
- You can do network analysis or call Matlab in R.
calling Matlab from R
There are at least three ways to call Matlab from R. Which probably doesn’t interest the vast majority of people, but could be useful in engineering where different disciplines and people from different backgrounds are trained in a variety of tools but still need to work together.
Calculating the RPI
If you wanted to calculate the RPI in R here is how you would do it.
https://www.r-bloggers.com/calculating-college-basketball-rankings-using-functional-programming-in-r/
2017 in Review
Most frightening stories of 2017:
- January: The U.S. government may be “planning to roll back or dilute many of the provisions of Dodd-Frank, particularly those that protect consumers from toxic financial products and those that impose restrictions on banks”.
- February: The Doomsday Clock was moved to 2.5 minutes to midnight. The worst it has ever been was 2 minutes to midnight in the early 1980s. In related news, the idea of a U.S.-China war is looking a bit more plausible. The U.S. military may be considering sending ground troops to Syria.
- March: La Paz, Bolivia, is in a serious crisis caused by loss of its glacier-fed water supply. At the same time we are losing glaciers and snowpack in important food-growing regions, the global groundwater situation is also looking bleak. And for those of us trying to do our little part for water conservation, investing in a residential graywater system can take around 15 years to break even at current costs and water rates.
- April: The U.S. health care market is screwed up seemingly beyond repair. Why can’t we have nice things? Oh right, because our politicians represent big business, not voters. Also, we have forgotten the difference between a dialog and an argument.
- May: We hit 410 ppm at Mauna Loa.
- June: The Onion shared this uncharacteristically unfunny observation: “MYTH: There is nothing mankind can do to prevent climate change. FACT: There is nothing mankind will do to prevent climate change”. It’s not funny because it’s probably true.
- July: Long term food security in Asia could be a problem.
- August: The U.S. construction industry has had negligible productivity gains in the past 40 years.
- September: During the Vietnam War the United States dropped approximately twice as many tons of bombs in Southeast Asia as the Allied forces combined used against both Germany and Japan in World War II. After the Cold War finally ended, Mikhail Gorbachev made some good suggestions for how to achieve a lasting peace. They were ignored. We may be witnessing the decline of the American Empire as a result.
- October: It is possible that a catastrophic loss of insects is occurring and that it may lead to ecological collapse. Also, there is new evidence that pollution is harming human health and even the global economy more than previously thought.
- November: I thought about war and peace in November. Well, mostly war. War is frightening. The United States of America appears to be flailing about militarily all over the world guided by no foreign policy. Big wars of the past have sometimes been started by overconfident leaders thinking they could get a quick military victory, only to find themselves bogged down in something much larger and more intractable than they imagined. But enemies are good to have – the Nazis understood that a scared population will believe what you tell them.
- December: A lot of people would probably agree that the United States government is becoming increasingly dysfunctional, but I don’t think many would question the long-term stability of our form of government itself. Maybe we should start to do that. The Consumer Financial Protection Bureau has been doing a decent job of protecting consumers and reducing the risk of another financial crisis. The person in charge of it now was put there specifically to ruin it. Something similar may be about to happen at the Census Bureau. A U.S. Constitutional Convention is actually a possibility, and might threaten the stability of the nation.
Most hopeful stories of 2017:
- January: The theory of island biodiversity gives us some clues on how to maximize the biodiversity that a given amount of natural land can support.
- February: You can take a class on how to not be fooled by the news.
- March: A new political survey says there is a chance that a majority of Americans are not bat-shit crazy. Which suggests they might not be too serious about Steve Bannon, who believes in some bat-shit crazy stuff. There are a number of apps and guides out there to help sane people pester our elected representatives when they fail to represent our interests.
- April: The value added tax is a boring but good idea. Why can’t the U.S. have nice things? Oh, right…
- May: Buzz Aldrin and NASA have plans for Mars colonization around the 2030s. Stephen Hawking thinks this is a good idea to hedge our bets against bad things that might happen here on Earth.
- June: On the education front: Finland achieves some of the world’s best educational outcomes with a lot of playtime and not a lot of homework. Musical training early in life is good for your brain later in life, even if you don’t continue it. There are lots of free philosophy and ethics courses online.
- July: A new cancer treatment genetically modifies a patient’s own immune system to attack cancer cells.
- August: The Aichi Biodiversity Targets are some very specific numerical targets that have incorporated in the 2015 Sustainable Goals.
- September: Utility-scale solar energy cost dropped 30% in one year.
- October: Supersonic (civilian) travel is almost back.
- November: Donald Trump does not appear to be trying to destroy the Federal Reserve.
- December: Macroeconomic modeling is improving. So, just to pick a random example, it might be possible to predict the effects on a change in tax policy on the economy. Now all we need is politicians who are responsive to logic and evidence, and we could accomplish something. At least a few economists think the imperfect tax plan the U.S. Congress just passed might actually stimulate business capital investment enough to move the dial on productivity. The deliberate defunding of health care included in the bill is going to hurt people, but maybe not all that dramatically.
Most interesting stories that weren’t particularly frightening or hopeful, or perhaps were a mixture of both:
- January: Apple, Google, and Facebook may destroy the telecom industry.
- February: The idea of growing human organs inside a pig, or even a viable human-pig hybrid, is getting very close. Tiny brains can also be grown on a microchip. Bringing back extinct animals is also getting very close.
- March: Bill Gates has proposed a “robot tax”. The basic idea is that if and when automation starts to increase productivity, you could tax the increase in profits and use the money to help any workers displaced by the automation. In related somewhat boring economic news, there are a variety of theories as to why a raise in the minimum wage does not appear to cause unemployment as classical economic theory would predict.
- April: I finished reading Rainbow’s End, a fantastic Vernor Vinge novel about augmented reality in the near future, among other things.
- May: The sex robots are here.
- June: “Fleur de lawn” is a mix of perennial rye, hard fescue, micro clover, yarrow, Achillea millefolium, sweet alyssum, Lobularia maritima, baby blue eyes, Nemophila menziesi, English daisy, Bellis perennis, and O’Connor’s strawberry clover, Trifolium fragiferum.
- July: Ecologists have some new ideas for measuring resilience of ecosystems. Technologists have some wild ideas to have robots directly counteract the effects of humans on ecosystems. I like ideas – how do I get a (well-compensated) job where I can just sit around and think up ideas?
- August: Elon Musk has thrown his energy into deep tunneling technology.
- September: I learned that the OECD Science, Technology and Innovation Outlook named “ten key emerging technology trends”: The Internet of Things, Big data analytics, Artificial intelligence, Neurotechnologies, Nano/microsatellites, Nanomaterials, Additive manufacturing / 3D printing, Advanced energy storage technologies, Synthetic biology, Blockchain
- October: Even if autonomous trucks are not ready for tricky urban situations, they could be autonomous on the highway with a small number of remote-control drivers guiding a large number of tricks through tricky urban maneuvers, not unlike the way ports or trainyards are run now. There is also new thinking on how to transition highways gradually through a mix of human and computer-controlled vehicles, and eventually to full computer control. New research shows that even a small number of autonomous vehicles mixed in with human drivers will be safer for everyone. While some reports predict autonomous taxis will be available in the 2020s, Google says that number is more like 2017.
- November: It’s possible that the kind of ideal planned economy envisioned by early Soviet economists (which never came to pass) could be realized with the computing power and algorithms just beginning to be available now.
- December: Microsoft is trying to one-up Google Scholar, which is good for researchers. More computing firepower is being focused on making sense of all the scientific papers out there.
I’ll keep this on the short side. Here are a few trends I see:
Risk of War. I think I said about a year ago that if we could through the next four years without a world war or nuclear detonation, we will be doing well. Well, one year down and three to go. That’s the bright side. The dark side is that it is time to acknowledge there is a regional war going on in the Middle East. It could escalate, it could go nuclear, and it could result in military confrontation between the United States and Russia. Likewise, the situation in North Korea could turn into a regional conflict, could go nuclear, and could lead to military confrontation between the United States and China.
Decline…and Fall? A question on my mind is whether the United States is a nation in decline, and I think the surprisingly obvious answer is yes. The more important question is whether it is a temporary dip, or the beginning of a decline and fall.
Risk of Financial Crisis. The risk of another serious financial crisis is even scarier that war in some ways, at least a limited, non-nuclear war. Surprisingly, the economic effects can be more severe, more widespread and longer lasting. We are seeing the continued weakening of regulations attempting to limit systemic risk-taking for short-term gain. Without a pickup in long-term productivity growth and with the demographic and ecological headwinds that we face, another crisis equal to or worse than the 2007 one could be the one that we don’t recover from.
Ecological Collapse? The story about vanishing insects was eye-opening to me. Could global ecosystems go into a freefall? Could populous regions of the world face a catastrophic food shortage? It is hard to imagine these things coming to a head in the near term, but the world needs to take these risks seriously since the consequences would be so great.
Technology. With everything else going on, technology just marches forward, of course. One technology I find particularly interesting is new approaches to research that mine and attempt to synthesize large bodies of scientific research.
Can the human species implement good ideas? Solutions exist. I would love to end on a positive note, but at the moment I find myself questioning whether our particular species of hairless ape can implement them.
But – how’s this for ending on a positive note – like I said at the beginning, the one thing about 2017 that definitely didn’t suck was that we didn’t get blown up!
R for Excel users
Here’s a new intro to R for Excel users.
https://www.r-bloggers.com/a-history-oriented-introduction-to-r-for-excel-users/
online dashboards
This R-bloggers post lists 8 types of online dashboards along with some alternatives for creating them.
alternatives to word clouds
I like this post on R bloggers proposing several alternatives to word clouds. I’ll list them below but really, you should look at the pictures because hey, this is about pictures.
- circle packing (basically this replaces the words with circles, dealing with the problem of bigger/longer words appearing to be more important in standard word clouds); there is a variation on this called the “horn of plenty” where the circles are arranged in order rather than randomly
- cartogram (in my ignorance, I have been calling this a “bubble map”. I have used these frequently to show engineering model results and find they work well for many people)
- chloropleth (these shade in geographic areas to convey data. I find these work well if the size of the geographic area is important information. If it is not, these tend to draw the viewer’s eye to larger areas, and in that case the bubbles are better. For example, per-person income of Luxembourg vs. China.)
- treemap (I’ve been calling these “packed rectangles” and I generally find them good for anything where conveying relative magnitudes of things to people is important)
- donuts (surpringly, the author concludes a donut is the best option for the data he is trying to show and I kind of agree, it gets the point across and leaves lots of room for labels)
The article has links to the specific packages and code used to create the graphics.