- Under the Hood of App Inventor for Android — regular readers know I’m a big fan of visual programming language Scratch, and apparently Google are too. They’ve got twelve university classes testing App Inventor for Android, a visual connect-the-bits programming environment for Android. University classes probably because one of the co-creators is Hal Abelson, coauthor of the definitive programming textbook. Also found online: the PR-type announcement, a Professor using it, and @AppInv (nothing juicy on Twitter–it looks like might be a channel for tech support for the students). (via Hacker News)
- Google Web Optimizer Case Study (Four Hour Work Week) — GWO manages A/B tests for you, with a lot of statistical analysis. It’s a fascinating read to see how these should be done. Every equation may halve the readership of a book, but every table of numbers and relevancy analysis doubles the value of a post like this. (via Hacker News)
- Opening Up The BBC’s Natural History Archive — the BBC are releasing programme segments and a whole lot of metadata around their programming. Audio and video segmented, tagged with DBpedia terms, and aggregated into a URI structure based on natural history concepts: species, habitats, adaptations, etc. Gorgeous!
- Yahoo! Term Extraction API to Close — Internally, both services
share a backend data source that is closing down, so the publicly-facing YDN
services will be closing as well. I think it’s the most significant casualty of Y! outsourcing search to MSFT, as this API was used by a lot of projects. (via Simon Willison)
The Jawbone UP shows the promise available in all kinds of wearable sensors.
In a recent conversation, I described my phone as “everything that Compaq marketing promised the iPAQ was going to be.” It was the first device I really carried around and used as an extension of my normal computing activities. Of course, everything I did on the iPAQ can be done much more easily on a smartphone these days, so my iPAQ sits in a closet, hoping that one day I might notice and run Linux on it.
In the decade and a half since the iPAQ hit the market, battery capacity has improved and power consumption has gone down for many types of computing devices. In the Wi-Fi arena, we’ve turned phones into sensors to track motion throughout public spaces, and, in essence, “outsourced” the sensor to individual customers.
Phones, however, are relatively large devices, and the I/O capabilities of the phone aren’t needed in most sensor operations. A smartphone today can measure motion and acceleration, and even position through GPS. However, in many cases, display isn’t needed on the sensor itself, and the data to be collected might need another type of sensor. Many inexpensive sensors are available today to measure temperature, humidity, or even air quality. By moving the I/O from the sensor itself onto a centralized device, the battery power can be devoted almost entirely to collecting data.
Current tools make collection and visualization easier but don't reduce work
New tools are raining down on system administrators these days, attacking the “monitoring sucks” theme that was pervasive just a year ago. The new tools–both open source and commercial–may be more flexible and lightweight than earlier ones, as well as more suited for the kaleidoscopic churn of servers in the cloud, making it easier to log events and visualize them. But I look for more: a new level of data integration. What if the monitoring tools for different components could send messages to each other and take over from the administrator the job of tracing causes for events?
Hadoop, Sqoop, and ZooKeeper
Kathleen Ting (@kate_ting), Technical Account Manager at Cloudera, and our own Andy Oram (@praxagora) sat down to discuss how to work with structured and unstructured data as well as how to keep a system up and running that is crunching that data.
Key highlights include:
- Misconfigurations consist of almost half of the support issues that the team at Cloudera is seeing [Discussed at 0:22]
- ZooKeeper, the canary in the Hadoop coal mine [Discussed at 1:10]
- Leaky clients are often a problem ZooKeeper detects [Discussed at 2:10]
- Sqoop is a bulk data transfer tool [Discussed at 2:47]
- Sqoop helps to bring together structured and unstructured data [Discussed at 3:50]
- ZooKeep is not for storage, but coordination, reliability, availability [Discussed at 4:44]
You can view the full interview here:
The Havana release features metering and orchestration
I talked this week to Jonathan Bryce and Mark Collier of OpenStack to look at the motivations behind the enhancements in the Havana release announced today. We focused on the main event–official support for the Ceilometer metering/monitoring project and the Heat orchestration project–but covered a few small bullet items as well.
How to use SlideShare presentations for more than public speaking
For years, PowerPoint slidedecks dominated boardrooms and marketing meetings for companies around the globe. With the introduction of SlideShare six years ago, a whole new platform appeared, and with it the opportunity to share slideshows in a new way. Now, technologists, programmers and developers are using slidedecks to collaborate, demonstrate their professional knowledge, and move quickly in an open, agile workplace. For example, you can:
- connect with other professionals who are working on similar projects.
- quickly learn what is being presented at industry conferences.
- identify others who are working on technologies and projects that you’re interested in.
- learn how companies and organizations are using specific technologies, and follow their progress.
You Are Invited to Optimize Computer Infrastructure
Key highlights include:
- Find out about the Open Compute Project [Discussed at 0:13]
- It’s all about open hardware [Discussed at 1:10]
- Facebook data centers have been optimized [Discussed at 2:12]
- How do you get involved? [Discussed at 3:33]
You can view the full interview here:
Other than marry Carol Brady, that is...
When computers were small and programs short, writing programs for them was pretty much a solo enterprise. But as computers became more powerful, and importantly, interconnected, you started to see software development in teams, and with teams came the most unfortunate of byproducts, management.
In a modern software development project, there are in fact a lot of contributors who have nothing to do with actually generating code. You have a product owner, determining exactly what functions the software should enable. You may have user experience designers, who are turning those requirements into a cohesive and logical look and feel. You have UI designers, creating graphical assets and fleshing out the U/X designers. There are strictly overhead roles such as managers of different flavors (including scrum-masters, who are really just a specific breed of manager). There are the developers themselves, who may include team leads. And then there are the folks with the mysterious title of Architect. But just what do they do?
In my depressingly long career as a software engineer (the career isn’t depressing; the length of it is), I have seen no title as subject to variable interpretation as Architect. I’ve seen architects that were essentially the MVPs of the project, and I’ve seen ones that are speed-bumps on the road of life. Why they are placed onto projects, and how they are used, can drastically influence the overall happiness and productivity of a team.
Let’s begin our look at the role of architects with the most negative example. There exists a class of companies that have architects Because They’re Supposed To. In this kind of environment, architects can gum up the works in a number of ways. First, these tend to be the people who Read Things in Magazines (or online). They tend to be in love with concepts like elegance and abstraction (which are good in moderation), and to some extent try to justify their existence by making things more complex. They rarely code.
Let’s call this subspecies of architect the Formalists. You can tell them by the fact that they have more books on UML and Patterns on their bookshelves than on languages and operating systems. The Formalists believe in form over function. As an example of the kind of poison they can spread in a project, I once had an architect explaining to me that my clean, compact and well-tested Java code was unacceptable, because I did not implement a single Pattern in it. I added a Singleton (which neither helped nor hurt the code), and he became happy.
The Formalist frequently kills the enthusiasm of the developers with mandates that drastically increase the workload of the developer for little to no benefit. Formalists are the polar opposites of the agile process, because they will insist on building in levels of complexity far in advance of a proven need for the extensibility provided. If you have a simple application laden with Enterprise Service Buses and Factories, a Formalist probably had their hand in the design.
Let me be clear, when used correctly, architects are awesome. Let’s label the group of good architects the Pragmatists. The primary phenotype of a good architect is that they know how to code, and code well. Beyond that, a good architect serves a crucial role in the definition of the product requirements, not just drawing boxes at 30,000 feet.
To understand this role, you have to look at the lifecycle of a successful product. After the basic product requirements have been laid out, but before developers start to implement, there’s an important step that many teams miss. Someone, and by someone I mean a good architect, needs to flesh out the requirements into a sufficiently detailed set of high-level tasks. It is in vogue these days to do this during the “grooming” portion of the scrum process, but it really needs to happen much earlier, and by someone looking at the entire product and how this particular task fits into the overall picture. The architect needs to answer questions such as:
- Do the U/X, UI, and development requirements really cover the end user requirements (i.e., gap analysis)?
- Is this task being designed and developed in a consistent fashion to the remainder of the application (and, where appropriate, the way software is developed at the company in general)?
- Is the design flexible enough to allow future enhancements?
- Does the application implement appropriate levels of security and meet required performance standards?
Handle sensitive data, free memory, and more
It may seem that when dynamic memory has been deallocated, we are done with it. We have avoided a memory leak and can confidently move on to other issues. In some cases this may be the case. However, we may need to be concerned with issues such as how we handle sensitive data and whether we need to even worry about freeing memory if the application is about to terminate. In this discussion we will examine these issues and others.
The Heap and System Memory
The heap typically uses operating system functions to manage its memory. The heap’s size may be fixed when the program is created, or it may be allowed to grow. However, the heap manager does not necessarily return memory to the operating system when the
free function is called. The deallocated memory is simply made available for subsequent use by the application. Thus, when a program allocates and then frees up memory, the deallocation of memory is not normally reflected in the application’s memory usage as seen from the operating system perspective.
Memory is typically deallocated using the free function. One concern deals with what happens when we try to free the same memory twice. Freeing a block of memory twice is referred to as double free. The following illustrates how this can be done:
char *name = (char*)malloc(...); ... free(name); // First free ... free(name); // Double free
More subtle occurrences of double free occur when pointers are aliased:
char *name = (char*)malloc(...); char *tmp = name; ... free(name); // First free ... free(tmp); // Double free
In an earlier version of the zlib compression library, it was possible for a double-free operation to result in a denial of service attack or possibly to insert code into the program. However, this is extremely unlikely and the vulnerability has been addressed in newer releases of the library. More information about this vulnerability can be found at cert.org.
A simple technique to avoid this type of vulnerability is to always assign NULL to a pointer after it has been freed. Subsequent attempts to free a null pointer will be ignored by most heap managers.
char *name = (char*)malloc(...); ... free(name); name = NULL;
Stop standardizing HTML, persistence of plastic, social media's 2.0 moment, map of U.S. terror attacks 1970-2011
Stop standardizing HTML: Simon St. Laurent writes “HTML itself is still useful—many people and tools know how to read and write it—but there is less and less reason to let the HTML vocabulary be a cage limiting our possibilities.” His unique take on the issue prompted an uproar on Slashdot.
The persistence of plastic: Did you know that plastic production during the past decade equals that of the entire twentieth century?
Agile in name only: James Turner calls out companies that claim to embrace agile development, but don’t really understand it. Is agile really agile if you end up going over a waterfall at the end?
Social media’s 2.0 moment: Over on O’Reilly Radar Joshua-Michéle Ross wonders if apps like SnapChat and Poke are creating a massive acceleration in the traditional timeline needed to create branded content.
Visualization of the Week: Gain some perspective and cut through the jungle canopy that is the 24/7 news cycle. Using START Global Terrorism Database, the Guardian’s Simon Rogers mapped every U.S. terror attack between 1970 and 2011.