In the beginning there was R. Or, S? I’ve heard that R actually rests on a foundation of C++ or Java. Anyway, then there was the tidyverse, sort of another whole programming language that rests in R (or a metastasizing cancer that has grown to dominate R, if you ask certain people, but I personally am a big fan). Now within the tidyverse was always ggplot2, which I have grown to rely on almost exclusively for plotting. Now ggplot2 itself has grown into an “ecosystem” of related programs and extensions. Here is a useful guide. I’ve always been interested in finding the really good ones for things like interactive charts (plotly) and animations (gganimate). And awesome as ggplot2 is, there are some things that are just clunky, like scales and legends (seriously, legends are a big pain point for me – I hope there is an extension out there that really streamlines legends). But I am also wary of using extensions that might be buggy or not updated/supported long term, which could make my code obsolete sooner. So I usually try to do things with ggplot2 proper first, and if that doesn’t work with a reasonable effort I will try one of the extensions. So this guide seems timely and useful.
Tag Archives: computer programming
bibliometric analysis
Here is some R code and detailed explanation on how to thoroughly explore keywords within an academic field, in this case ecology.

those f-ed up spreadsheets
I’ve seen a lot of fucked up spreadsheets in my time. In fact, you could probably say most spreadsheets contain errors, and this article confirms it. Nonetheless, quality control procedures that apply to other types of software (and spreadsheets are a very useful kind of software because they are very powerful and have an easy learning curve, which is also their downfall) can and should be adapted to spreadsheets. One of the most important ones (which really applies to any program or project) is to begin the quality control process in the planning and concept development phase.
Spreadsheet quality assurance: a literature review
Spreadsheets are very common for information processing to support decision making by both professional developers and non-technical end users. Moreover, business intelligence and artificial intelligence are increasingly popular in the industry nowadays, where spreadsheets have been used as, or integrated into, intelligent or expert systems in various application domains. However, it has been repeatedly reported that faults often exist in operational spreadsheets, which could severely compromise the quality of conclusions and decisions based on the spreadsheets. With a view to systematically examining this problem via survey of existing work, we have conducted a comprehensive literature review on the quality issues and related techniques of spreadsheets over a 35.5-year period (from January 1987 to June 2022) for target journals and a 10.5-year period (from January 2012 to June 2022) for target conferences. Among other findings, two major ones are: (a) Spreadsheet quality is best addressed throughout the whole spreadsheet life cycle, rather than just focusing on a few specific stages of the life cycle. (b) Relatively more studies focus on spreadsheet testing and debugging (related to fault detection and removal) when compared with spreadsheet specification, modeling, and design (related to development). As prevention is better than cure, more research should be performed on the early stages of the spreadsheet life cycle. Enlightened by our comprehensive review, we have identified the major research gaps as well as highlighted key research directions for future work in the area.
Make/write/record music with R
Move over Garage Band, there is a new musical sheriff in town. For those of us who like to mess around in R – okay, realistically this gets added to my list of retirement projects a few decades down the road, assuming my body and brain manage to stick around for the next few decades.
gm: Generate Music Easily and Show Them Anywhere
Provides a simple and intuitive high-level language, with which you can create music easily. Takes care of all the dirty technical details in converting your music to musical scores and audio files. Works in ‘R Markdown’ documents <https://rmarkdown.rstudio.com/>, R ‘Jupyter Notebooks’ <https://jupyter.org/>, and ‘RStudio’ <https://www.rstudio.com/>, so you can embed generated music anywhere. Internally, uses ‘MusicXML’ <https://www.musicxml.com/> to represent musical scores, and ‘MuseScore’ <https://musescore.org/> to convert ‘MusicXML’.
CRAN
2018 in Review
Most frightening and/or depressing stories:
- JANUARY: Cape Town, South Africa looked to be in imminent danger of running out of water. They got lucky, but the question is whether this was a case of serious mismanagement or an early warning sign of water supply risk due to climate change. Probably a case of serious mismanagement of the water supply while ignoring the added risk due to climate change. Longer term, there are serious concerns about snowpack-dependent water supplies serving large urban populations in Asia and western North America.
- FEBRUARY: Cape Town will probably not be the last major city to run out of water. The other cities at risk mentioned in this article include Sao Paulo, Bangalore, Beijing, Cairo, Jakarta, Moscow, Istanbul, Mexico City, London, Tokyo, and Miami.
- MARCH: One reason propaganda works is that even knowledgeable people are more likely to believe a statement the more often it is repeated.
- APRIL: That big California earthquake is still coming.
- MAY: The idea of a soft landing where absolute dematerialization of the economy reduces our ecological footprint and sidesteps the consequences of climate change through innovation without serious pain may be wishful thinking.
- JUNE: The Trump administration is proposing to subsidize coal-burning power plants. Meanwhile the long-term economic damage expected from climate change appears to be substantial. For one thing, Hurricanes are slowing down, which means they can do more damage in any one place. The rate of melting in Antarctic ice sheets is accelerating.
- JULY: The UN is warning as many as 10 million people in Yemen could face starvation by the end of 2018 due to the military action by Saudi Arabia and the U.S. The U.S. military is involved in combat in at least 8 African countries. And Trump apparently wants to invade Venezuela.
- AUGUST: Noam Chomsky doesn’t love Trump, but points out that climate change and/or nuclear weapons are still existential threats and that more mainstream leaders and media outlets have failed just as miserably to address them as Trump has. In related news, the climate may be headed for a catastrophic tipping point and while attention is mostly elsewhere, a fundamentalist takeover of Pakistan’s nuclear arsenal is still one of the more serious risks out there.
- SEPTEMBER: A huge earthquake in the Pacific Northwest could be by far the worst natural disaster ever seen.
- OCTOBER: The Trump administration has slashed funding to help the U.S. prepare for the next pandemic.
- NOVEMBER: About half a million people have been killed in Iraq, Afghanistan, and Pakistan since the U.S. invasions starting in 2001. This includes only people killed directly by violence, not disease, hunger, thirst, etc.
- DECEMBER: Climate change is just bad, and the experts seem to keep revising their estimates from bad to worse. The Fourth National Climate Assessment produced by the U.S. government is not an uplifting publication. In addition to the impacts of droughts, storms, and fires, it casts some doubt on the long-term security of the food supply. An article in Nature was also not uplifting, arguing that climate change is happening faster than expected due to a combination of manmade and natural trends.
Climate change, nuclear weapons, and pandemics. If I go back and look at last year’s post, this list of existential threats is going to be pretty much the same. Add to this the depressing grind of permanent war which magnifies these risks and diverts resources that could be used to deal with them. True, we could say that we got through 2018 without a nuclear detonation, pandemic, or ecological collapse, and under the circumstances we should sit back, count our blessings, and wait for better leadership. And while our leadership is particularly inept at the moment, I think Noam Chomsky has a point that political administration after political administration has failed to solve these problems and this seems unlikely to improve. The earthquake risk is particularly troublesome. Think about the shock we felt over the inept response to Katrina, and now think about how essentially the same thing happened in Puerto Rico, we are not really dealing with it in an acceptable way, and the public and news media have essentially just shrugged it off and moved on. If the hurricanes, floods, fires and droughts just keep hitting harder and more often, and we don’t fully respond to one before the next hits, it could mean a slow downward spiral. And if that means we gradually lose our ability to bounce back fully from small and medium size disasters, a truly huge disaster like an epic earthquake on the west coast might be the one that pushes our society to a breaking point.
Most hopeful stories:
- JANUARY: There is new evidence that reading is really good for the developing brain.
- FEBRUARY: You can read a book or take a free course on why Buddhist meditation may be really good for your brain and life. Also good for your brain would be curing Alzheimer’s disease, which has now been done in mice (although it seems to have caused them other problems).
- MARCH: The coal industry continues to collapse, and even the other fossil fuels are saying they are a bunch of whining losers. And yes, I consider this positive. I hope there aren’t too many old ladies whose pensions depend on coal at this point.
- APRIL: There are free online resources to teach general systems theory in middle school.
- MAY: Psychedelics might produce similar benefits to meditation.
- JUNE: A new estimate indicates that “stranded fossil fuel assets” could mount up to global wealth losses on the order of $1-4 trillion. Oil companies are starting to look ahead to a possible peak and decline in demand. In some states, natural gas companies are fighting the nuclear power industry, which is already in a tailspin. Elsewhere, offshore wind power may now be cost-competitive with natural gas. And in absolutely shockingly hopeful news, the U.S. Congress may have a realistic, bipartisan plan for a carbon tax.
- JULY: Looking at basic economic and health data over about a 50-200 time frame reminds us that enormous progress has been made, even though the last 20 years or so seems like a reversal.
- AUGUST: Vancouver has successfully combined green street and complete street concepts. The American Society of Landscape Architects has also compiled some helpful resources on this topic.
- SEPTEMBER: The Suzuki and Kodaly methods are two ways of teaching music to young children that may actually help them think later in life. Training in jazz improvisation may also be good for young brains in a slightly different way.
- OCTOBER: Applying nitrogen fixing bacteria to plants that do not naturally have them may be a viable way to reduce nitrogen fertilizer use and water pollution.
- NOVEMBER: New drugs could mean very low transmission risk of AIDS, even for people engaging in previously high-risk behavior.
- DECEMBER: There is a strong link between Alzheimer’s disease and insulin resistance, suggesting that the average individual has some ability to avoid the disease. There is also a promising new treatment involving ultrasound.
I believe our children are our future…ya ya blahda blahda. It’s a huge cliche, and yet to be hopeful about our world I have to have some hope that future generations can be better system thinkers and problem solvers and ethical actors than recent generations have been. Because despite identifying problems and even potential solutions we are consistently failing to make choices as a society that could divert us from the current failure path. And so I highlighted a few stories above about ideas for better preparing future generations, ranging from traditional school subjects like reading and music, to more innovative ones like meditation and general system theory, and just maybe we should be open to the idea that the right amount of the right drugs can help.
Fossil fuels just might be on their way out, as alternatives start to become economical and public outrage slowly, almost imperceptibly continues to build.
There is real progress in the fight against disease, which alleviates enormous quantities of human suffering. I mention AIDS, diabetes, and Alzheimer’s disease above. We can be happy about that, of course. There are ideas about how to grow more food, which is going to be necessary to avoid enormous quantities of human suffering. Lest anyone think otherwise, my position is that we desperately need to reduce our ecological footprint, but human life is precious and nobody deserves to suffer illness or hunger.
Good street design that lets people get around using mostly their own muscle power. It might not be sexy, but it is one of the keys to physical and mental health, clean air and water, biodiversity, social and economic vibrancy in our cities. Come to think of it, I take that back, it can be sexy if done well.
Good street design and general systems theory – proof that solutions exist and we just don’t recognize or make use of them. Here’s where I want to insert a positive sentence about how 2019 is the year this all changes for the better. Well, sorry, you’ll have to find someone less cynical than me, and/or with much better powers of communication and persuasion than me to get the ball rolling. On the off chance I have persuaded you, and you have communication and/or persuasion super powers, let me know.
Most interesting stories, that were not particularly frightening or hopeful, or perhaps were a mixture of both:
- JANUARY: If you are a white supremacist, it’s okay to have an Asian girlfriend according to the New York Times. The unfortunate corollary would seem to be that if someone is accusing you of being a white supremacist, you can’t just get an Asian girlfriend and expect that to be a sufficient defense.
- FEBRUARY: SpaceX is planning to launch more than 4,000 small satellites.
- MARCH: You can do network analysis or call Matlab in R.
- APRIL: A space hotel could open as soon as 2020.
- MAY: Uber Air is looking a little more real. There are other ideas for autonomous urban helicopters too.
- JUNE: Explicit taxes to fund wars were the norm in the U.S. right up to the Vietnam war.
- JULY: Some physicists take the idea of creating a universe in a bottle seriously.
- AUGUST: Google Lens can identify a plant or animal from its picture, and the subway body scanners from Total Recall are now real.
- SEPTEMBER: In biotech news, gene editing is starting to be used for food crops, starting with soybean oil. Also, old mice can live longer if they are transfused with the blood of young mice. And there is a new (to me) book about de-extinction.
- OCTOBER: New tech roundup: Artificial spider silk is an alternative to carbon fiber. Certain types of science, like drug and DNA experiments, can be largely automated. A “quantum internet” could mean essentially unbreakable encryption.
- NOVEMBER: New tech roundup: People in Sweden are barely using cash at all, and some are paying with microchips embedded in their fingers. New technology may allow screening of multiple airport passengers from 25 feet away with minimal disruption. This is great for airline passengers who are already expecting to be screened intrusively, but of course raises some concerns about potential uses elsewhere in the public realm. Amazon is hiring about 100,000 seasonal workers this year, compared to about 120,000 in past years, and the difference may be explained by automation. There is a new ISO standard for toilets not connected to sewers systems (and not just your grandfather’s septic tank.)
- DECEMBER: New Zealand is trying to use gene drives to completely wipe out rats in short order. Google is trying to do something similar with mosquitoes, only for the entire earth.
Whatever else happens, technology and accumulation of human knowledge in general march on, of course. Computer, robotics, and surveillence technology march on. The human move into space is much slower and painful than many would have predicted half a century ago, and yet it is proceeding.
I’ll never drop the waterless sanitation thing, no matter how much others make fun of me. It’s going to happen, eventually. I don’t know whether we will colonize Mars or stop defecating in our water supply first, but both will happen.
The gene drive thing is really wild the more I think about it. This means we now have the ability to identify a species or group of species we don’t want to exist, then cause it not to exist in relatively short order. This seems like it could be terrifying in the wrong hands, doesn’t it? I’m not even sure I buy into the idea that rats and mosquitoes have no positive ecological functions at all. Aren’t there bats and birds that rely on mosquitoes as a food source? Okay, I’m really not sure what redeeming features rats have, although I did read a few years ago that in a serious food crunch farming rats would be a much more efficient way of turning very marginal materials into edible protein than chicken.
The universe in a bottle thing is mind blowing if you spend too much time thinking about it. It could just be bottles all the way down. It’s best not to spend too much time thinking about it.
That’s it, Happy 2019!
box plots!
One of my nerdy interests in box plots. And no, you can’t make great ones in Excel. Here is a blog about making fantastic ones in the R package ggplot2.
calling Matlab from R
There are at least three ways to call Matlab from R. Which probably doesn’t interest the vast majority of people, but could be useful in engineering where different disciplines and people from different backgrounds are trained in a variety of tools but still need to work together.
network analysis in R
I’ve played around with the igraph package a little and didn’t find it particularly user friendly. This r-bloggers post talks about 2 newer packages that sound like they might be a little easier to use, using a (somewhat out of season) Game of Thrones data set.
Scratch and Raspberry Pi
Here’s an article about how Scratch (a programming language supposedly any child can learn) and Raspberry Pi (a computer that costs less than $50) fit together.
We’re huge fans of Scratch here at the [Raspberry Pi] Foundation. It was designed to teach young people how to program but it’s a great learning tool at any age: you can build your first program in minutes and pick up fundamental concepts very quickly. Whilst having fun. Sneaky!
If you’ve never tried Scratch before then today is the day to boot up your Pi and have a play. If you are Pi-less then you can use it online, but you’ll be missing out on the best bit of all: physical computing with Scratch. It’s probably the easiest way to hook up sensors, LEDs, buttons and motors to the Pi
February 2015 in Review
This blog got 173 hits in February! Pretty cool, considering I really just meant it as a place to collect my own scattered thoughts and refer back to them later. If 173 out of the 6 billion people out there like it, I am flattered. Okay, I understand there may have been a few repeat visitors. Also, judging from the most popular posts, there is one thing I mention occasionally that people really like: robots!
Negative trends and predictions:
- Fresh Air had an interview with Elizabeth Kolbert, author of The Sixth Extinction. The idea here is that what humans are doing to other species is equivalent in scope to events that have killed off most life on Earth in the past.
- The drought in the western U.S. continues to grind on.
- There are some depressing new books out there about all the bad things that could happen to the world, from nuclear terrorism to pandemics. Also a “financial black hole”, a “major breakdown of the Internet”, “the underpopulation bomb”, the “death of death”, and more!
- Government fragmentation explains at least part of suburban sprawl and urban decline in U.S. states, with Pennsylvania among the worst.
Positive trends and predictions:
- Libraries are starting to go high-tech using warehouse robot technology.
- I had a rambling post on technologies to watch: carbon fiber, the internet of things, self-driving cars and trucks, biotechnology for everything from carbon sequestration to cancer treatment to agriculture, and of course more automation, robots, and artificial intelligence. And yes, Clark W. Griswold’s cereal varnish is a real thing!
- U.S. utility solar capacity is slowly ramping up.
- A new study suggests a sudden, catastrophic climate tipping point may not be too likely.
- Robots can independently develop new drugs.
- According to Google, self-driving taxis are only 2-5 years away.
- Complex ecosystems can be designed.
- Compost toilets may save the world…if we can get over the ick factor and the sawdust problem.
- There are lots of cheap new options for the aspiring high-tech handymen (and women and children) among us. Even better news, we may have reached the point where if you build a robot with your kid in the basement, and he then tells other kids about it, he might not get beat up on the playground.
- New York City has some good examples of green stormwater infrastructure integrated in sidewalk and street design.
One thing that strikes me is that we keep hearing about biotechnology, but we haven’t seen big, obvious impacts in most of our daily lives yet. I suspect biotechnology is like computers and robots in the 70s, 80s, and 90s – slow but steady progress was being made in the background, the pressure was building, and then the wave suddenly broke onto the commercial and public consciousness. I suspect biotechnology is the next big wave that is going to break.