Me at “We Are Millions“

Assange

Berlinale: Sweet Thing

A film from a loving father, showing the faces of his beautiful daughter and son. If he were a painter, he would paint like August Macke painted his wife.

Apart from this love, the film didn't tell me much new.

Berlinale: Surge

A part of me seeks balance, avoids big changes. When big changes are inevitable, the system can get out of balance and remain in a screaming state for a while until a new balance is found.

This balancing movement is, for example, set to music in Gustav Mahler's First Symphony, 4th movement "Stormily agitated – Energetic" and also the subject of Rita Kalnejais' film "Surge". The increasingly wobbly camera work is intended to underline the movement. The leading actor was known to me as Grenouille from "The Perfume". I liked looking at him and at some point I sympathized with his transformation.

When I ride my bicycle home, next to the highway, the cars at walking pace like every working day at this time, I wonder why these compensatory movements are so rare.

Berlinale: Siberia

The plot leaves the real world when the gambler turns into a bear. The film ends with a talking fish.

For me, the film is an intense, partly disturbing firework of memories and feelings, as they are typically experienced by a man. It is about the soul, about life and death, about parents, about the women/wife, about reason and its limits.

Now and then there is an allusion to the "black arts", for me a Faustian motif. Abel Ferrara was unknown to me before; I would like to see more films by him. I will probably watch Siberia a second time.

Berlinale: FREM

The filmmaker took some beautiful pictures of Antarctica using drones. But instead of one more episode of "National Geographic" she decided to add an idea: How would it look, how would it feel if an A.I. starts getting interested in life?

The result is some analog visual effects and a nerve-racking sound, which in my opinion is almost a psychological torture. The sound reminds me of a song by Hüsker Dü from 1984. I, for one, suffered so much that I slid back and forth in my cinema chair and my mobile phone slipped out of my pocket unnoticed and got lost.

But after the movie I don't know anything new about A.I.. My feeling is that an A.I. would not simply collect images, but would interact with the environment. The researcher Karl Friston once joked that A.I. stands for active interference.

The film didn't teach me anything new about life either. Since in Antarctica, it doesn't distinguish between plants and animals.

What remains are some landscapes from the Antarctic, which show me that even non-life, relaxation processes in the thermodynamic sense, can lead to beauty. And there remains some envy of the filmmaker who made this trip to Antarctica. (My cell phone was found the next day.)

Berlinale: Los Lobos

For me the most disturbing film I have seen this year at the Berlinale. An autobiographical film about a mother who goes to the United States with her two sons. While she takes on several jobs, the sons remain alone in the apartment.

The film got under my skin because the truth slowly unfolded. At first I thought it was "only" the three people, a single incident, but later I realized that there are whole districts in which such precarious conditions prevail. And it takes a while before I understand that the at first playful climbing action of the two boys serves to urinate in the kitchen sink because the overstrained, exhausted mother has locked herself in the bathroom.

The role of the church is dubious. The gestures of gratitude have something compulsively sick about them, especially because the mother stands sceptically beside them and actually only wants to have food.

Berlinale: Petite Fille

Before the movie, I didn't know that there were boys at 7 or 8 who thought they were girls -- "gender dysphoria"

During the movie, I thought, how can you know what it means to have gender x or y at age 7 or 8? Isn't it more like being told: you're a boy, you have a penis. And the consequences, all the stuff with the dolls, the clothes, the long hair comes after, step by step? And why does the mother say "Madmoiselle" and not "Sasha"?

After the film, while listening to the director, a present, calm, sensitive man, I thought, maybe there really is such a thing such as a child becoming irreversibly attached to the opposite sex.

In the end, I believe that most of the problems of "gender dysphoria" would be obsolete if we would stop forcing gender stereotypes on children at an early age.

Berlinale: H is for Happiness

Australian

Book

Cinematographically

Developed.

Eventually,

Felt

Happiness.

Berlinale: One of these days

My first film at the Berlinale this year was the premiere of the panorama film "one of these days". The next morning, I was once again standing in the queue at Kudamm to buy new tickets in advance. It was drizzling, and I thought about the film.

The film is about a competition: Twenty people stand around a car and touch the car with at least one hand. The last person standing wins the car. Basically, it's an advertisement for the car salesman on whose premises the competition takes place.

It's about how much you really want something. It's a little bit like waiting in line at the Kudamm. After ten minutes in the drizzle, one or the other person has left. In front of me there were two who took turns: one in the queue, the other under the rain roof. Others had chairs with them.

I had hoped for more from the film. A cinematic representation of waiting, an interior perspective with changed colours for example. Something about the motives to participate and about the occasions to give up. I had firmly expected the sentence: "It's only a car". Despite the two hours the director gave away a lot. In this respect, the film was a disappointment to me.

One character from the film comes into contact with his fear of not being enough for his beautiful wife and fails because of this fear. A fear I can well understand. The beautiful woman is played by Callie Hernandez and looks a little bit like the ticket seller at the box office at Kudamm, where I've been three times already and who has this glow in her eyes. Since it was the premiere of the film, some of the people involved in the film came on stage afterwards. Besides the beautiful Callie, I also noticed "The Notwist", a band I heard 25 years ago when I was at school. Nice to know they are still around.

Missing number sense at the b.z.

On September 21st 2015, the Berlin local newspaper b.z. published an article on a so-called
"gebetomat" (pray-o-mat).

It is a small booth where you can listen to prayers and chants of the religion of your choice. Sounds like an interesting art project, I really want to have a look when I am near it.

However, the author exhibits a missing number sense when he begins his article with "Berlin's smallest house of prayers has only 83 square centimeters." He erred on an order of magnitude.

Smart Data Kick-off Conference

On April, 22nd, a kick-off conference for the German innovation program “Smart data” took place. I took part, here is what I learned.

Prescriptive Models

One speaker differenciated between explanatory models, predictive models, and prescriptive models. The last class of models makes very much sense to me. We are going to build more sophisticated controls in our devices, and this are prescriptive models.

Data as Business Case

Regarding the data source, the common ground is to use inhouse data. Smarter projects use data from similar sources outside the firm, and/or integrate data from various sources.

Conforming to law, that is, respecting privacy and copyright, is one challenge. Another one is the automatic integration of heterogenous data source. In some pilot projects presented at the conference use semantic models for integration.

Pilot Projects

From the programs that got funded by the BMWi, there were three programs that got my attention. SmartEnergyHub analyses energy use at the airport Stuttgart and introduces load management. SmartRegionalStrategy offers strategic analysis of local urban mass data. KDI (clinical data intelligence) develops a business model for its research database / study management system.

Highlights of Re:publica 15

Few weeks ago, the re:publica 2015 internet culture conference took place here in Berlin. Almost 200 of the talks are available on youtube. I watched several hours of this, here is what I learned from it.

Surveillance Everywhere

Former UK intelligence officer and whistleblower Annie Machon held a eye-opening talk on past and current activities and strategies of the intelligence agencies in several countries. Some intelligence agencies do kill people without giving them the chance to go to court. This is a big threat to the democratic state of law. See also Noam Chomsky's analysis (who was not at the re:publica).

M. C. McGrath from the Transparency Toolkit gave a great talk on what we can find out about secret surveillance programs in our country using open data and free software. He mentions that his organisation has about 27.000 profiles which may likely be connected with surveillance.

Addie Wagenknecht and Jillian York presents some artistic approaches to the surveillance topic in the post-Snowden age.

Privacy

Consultant Gemma Galdon Clavell gave several interesting examples how projects can fail if possible privacy issues are not adressed earlier and properly. For instance, the netherlands had to implement an opt-out option for energy smart-metering because a story like Anne Frank hidden in a house would not be possible in a smart-metered world.

Wolfie Christl showed in his talk that it may well be possible to predict Big 5 character traits from smartphone metadata or from facebook likes. His talk led me to a browser plugin Disconnect, which allows me to control ad-tracking and social media requests online.

In order to effectively protect privacy, Frank Rieger argues that it is better to regulate how data is used than to regulate how data is collected.

Frederike Kaltheuner and Kave Salamatian point out that Asimov's fictional science called “Psychohistory” lays ground under what circumstances predictions from social media data can work. I added his “Foundation” series to my mental reading list.

Innovations

My favorite talk was Stephan Porombka's on cuteness (in German). “Cute” in his analysis means something small with no obligations but at the same time heartily. He refers to Adorno's lectures and Walser's letters. I like his talk because his point of view is outside the ubiquitous technical-economical rational perspectives usually applied to explain what goes on in our digital age.

Thomas Andrae from 3M highlights upcoming applications of wearables. He points out that Google glasses – providing context to machines to maintenance teams for example – are a big success in industrial area. Andrae also mentions that sleep tracking software was minutes faster than other methods in predicting earthquakes.

Mostly not directly related to internet culture, Holm Friebe and Mads Pankow present interesting background stories and examples for the behavioural psychology technique called “nudging”.

For what it is worth, I made a simple collection of number of videos vs views. It seems that in 2013, the organization team decided to expand the number of presentations, doubling from about 75 to 150. The views per video hence the interest seem somewhat stable. Would be interesting to look for popular outliers in the videos.

Highlights of Re:publica 14

From May 6th to 8th 2014, Berlin was the host for the re:publica 14. It is a conference on internet culture. As the years before, I did not attend it, but I watched some talks online. Half a year late, here are my highlights (mostly in German, though):

Robot Journalism

An area of data analysis that is moving in focus: automated text generation. At the re:publica the term robot journalism was coined. In this setting, data analysis is used to identify and interpret information worth mentioning (some kind of outlier detection?). This information is then automatically redacted into output text or output multimedia, using grammatical rules, orthographical rules, and phrase databases.

Some actors in the field mentioned in the talk are NarrativeScience, automated insights, AX Semantic Technologies, and texton.

Hypertopical News

Lara Setrakian talked about her News site Syria Deeply. She sees a positive future for single subject news sites (“xx watch”, “xx bureau”), on global issues. The content of her sites is licensed as creative commons and she plans “Teach Syria Deeply”.

When it comes to financing, she recommends not to monetize traffic/content such as ad revenue or paywalls/subscriptions, but to monetize engagements/community such as live events, conference calls, merchandizing, crowd funding, markets and job boards.

In case of Syria Deeply, financing was provided through think tanks, institutions and universities who “loved the design” and paid for technology consulting. Setrakian states: “People who know it best, often don't have a platform”.

Bullshit

Quite a few talks circled around bullshit. Torsten Kleinz talked on bullshit as business model. Martin Giesler and Jannis Kucharz proposed 15 journalistic startup ideas based on clickbaiting.

Christiane Frohmann from Katersalon proclaims nonsense as means for performative enlightenment.

Miscellaneous

There was a talk about the Internet Archive, where I learned that you can proactively enforce the archiving process via the “Save Page now” feature. You go to the Internet Archive's site and capture a web page for use as a trusted citation in the future. The link will not die then.

What are the baby seals, what is the police dog attacking uninvolved afro-americans, in the area of internet policy, is asked in this somewhat funny talk by Felix Schwenzel. Other theses were: Intelligence service hide their work because they do not want expose their dilettantism. Surveillance is inherent in the system, given by the technology, such as road kills are a consequence of the automobile technology.

Berlinale 2015 -- the inexhaustible variety of film

Most of the movies which are screened at the Berlinale are not available afterwards in Germany*. Therefore I like to go to the Berlin International Film Festival: It brings new impressions and perceptions from all regions of the world I would otherwise hardly get.

Nearly 400 films premiere every year at the Berlinale, with nearly 100 short films. There is no overarching motto; due to this large number of films, every visitor selects his or her own Berlinale. With 15 films over one week, my Berlinale 2015 had several topics. Here is my roundup.

I watched some biographical films on special women: Annemarie Schwarzenbach, Yvonne Rainer, Gertrude Bell. The movie “Je suis Annemarie Schwarzenbach” was interesting to me, not only because of her shimmering personality, but also because of the concept of the movie. The film shows a casting for a (fictional? discarded?) movie on Schwarzenbach. Thus we see several actors each exposing a special perspective on that character. Fascinating.

Two documentaries, “Flotel Europa” and “Hotline” draw attention to the situation of refugees. I subsequently read about the Dublin Regulation, according to which a person seeking asylum needs to apply for it in the EU country where he/she arrived. Thus Malta is burdened with very many asylum applications and Italy tolerates that new arrivals travel unidentified to France and apply for asylum there. I also read about the so-called Anti Infiltration Law in Israel, which in its 2012 version violated UN human rights. Asylum policy is a new topic to me, and an inconvenient one. The German wikipedia article I have started on the Hotline film) was in its early stage listed for deletion, because it is a “niche film” and not “relevant” for the encyclopedia. I hope it will not remain a “niche film”.

Two movies dive into the world of the super-rich: “The Last Summer of the Rich” shows how unrestrained power may lead to loneliness and arbitrariness. And “The Money Complex” was one of my favorite movies this year. It is inspired by Franziska von Reventlow's epistolary novel. I felt relaxed after watching it, I liked the adoration of idleness, and the music in the movie. Here are some youtube links to the songs played, partly different versions though: Ya se que tienes novio, Schubert's Allegretto in D Minor, Schnberg's 6 kleine Klavierstcke Op 19, Zampullin y Somormujo by Jorge Arribas and Diego Galaz, Los Cuatro Muleros by Josephine Foster and the Victor Herrero, and La Golondrina by Ricardo Delgado & Daniel Madero .

Most movies I selected this year had a religious topic: “Story of Judas” (where Judas never betrayed Jesus), “Superworld” (where a joyless monotone life of a check-out clerk undergoes an unexpected turn when she starts hearing god) and “The Voice of Water” (where the front woman of a dubious sect turns from being manipulative into being faithful), All these movies were great, yet “Superworld” with its great panoramas and its touching plot is my favorite out of them.

The secret winner of this Berlinale according to the blogosphere is “Victoria”. It is a movie about love and crime and Berlin, and without a single cut. It was filmed in one shot. I hope it will find a distributor so I get a chance to see it later.

(* This is actually a conjecture. If time permits, I will look for evidence.)

My this year's Berlinale favorite: Shadow Days

This year I went to ten movies at the 64th Berlinale. My favorite movie “Shadow Days” was by Zhao Dayong on China's “one-child” policy and the remains of the cultural revolution.

I contributed to the German Wikipedia article on this movie.

A common ground for Quantified Self enthusiasts

Today I joined a Quantified Self meetup at the Google Office here in Berlin. I have to say I don't like the direction this “movement” is taking.

I think what is missing is a common ground, an inspiring basis everyone likes, trusts and participates. Something like the Raspberry Pi, Wordpress, Wikipedia or Apache, – something you can build on.

The talks today were quite the opposite. Someone showed an application which is now free but will likely become crippled when the user base is large enough. Someone else plans a web site for blood tests per mail. Thank you, not for me.

Highlights of Re:publica 13

From May 6th to 8th, Berlin was the host for the re:publica 13. I did not have time to attend it, but many of the talks of this internet culture conference are online. Here are my highlights (mostly in German, though):

• Matthias Schindler from Wikimedia promotes a political campaign for making governmental publications public domain. Very good argumentation, impressive work.
• Some social scientists observe that when copyright law can not be enforced, still some social norms define rules that work in some cases very well for the creators.
• Nice talk from the OKFN presenting show cases of open data (bicycle accidents in and open transport data for Berlin)
• The administrators of Wikipedia, imagine that some kind of voting system would be a big improvement to tackle vandalism. An astronomer from South Africa promotes some very interesting big data citizen science projects. The ironblogger club motivates to blog weekly, I like to join.
• Interesting art projects involve music roboters and interactive landscapes.
• Many successfull youtubers organize in networks like mediakraft. Podcasters, as it seems, suffer from public-funded radio and live as non-commercial projects.
• OwnCloud is the open source “reclaim your data on the web” project. Additionally, Sascha Lobo encourages to “reclaim social media” and sees Wordpress as the OS of the internet (I doubt that the Ruby and Django etc developers agree).
• There are some people thinking about why cat videos are so popular, and reflecting how they feel about funerals in their social network.

ScraperWiki in R

ScraperWiki describes itself as an online tool for gathering, cleaning and analysing data from the web. It is a programming oriented approach, users can implement ETL processes in Python, PHP or Ruby, share these processes among the community (or pay for privacy) and schedule automated runs. The software behind the service is open source, and there is also an offline version of the program library.

As far as I know, ScraperWiki has no R support yet. This is where the scrape method of my datamart package chimes in. It provides an S4 generic for storing scraped data into an offline database. It is not a wiki (however collaboration can take place using r-forge or CRAN), has no scheduling support (scheduling the start of R scripts should take place outside R) and is not intended as a web application.

Example use case — Berlin’s bathing water quality

Here is an example use case. It is (finally) summer in Berlin and the warm weather invites to go swimming. For the health-aware and/or data-affine people (or those with strong imagination) the public authorities provide the latest water quality assessments online for around 30 bathing areas as open data.

Only the last measurement is available. Now if we were interested in collecting the quality measurements, maybe to find a trend or to search for contributing factors, we would need a mechanism to regularly (in this case, weekly) scrape the data via the online API and save the data locally.

With the framework proposed in datamart, once the details are defined, the mechanism would be executed with just three lines

> ds <- datastore("path/to/local.db")
> bq <- be_bathing_water_quality()
> scrape(bq, ds) # get & save the data

The rest of this blog post is on defining the process, which boils down to defining the be_bathing_water_quality function.

ETL process — extract, transform, load

The task the be_bathing_water_quality function has to accomplish is to define an three-step process by passing the process details to the urldata function. In the database or data warehouse context this process is often refered to as ETL process. In our context, the steps are:

• After mapping a resource name to an URL the data is extracted, i.e. downloaded. If the network is not available, an authentication failed, or similar, the process ends here.
• The extracted data is then transformed into an object R can work with. Currently, only data.frame objects are supported, xts and igraph are envisioned. If the extracted data was not as expected and the tranformation failed, the process ends here.
• The tidy data is then stored (or loaded) int a local sqlite database.

The urldata function provides parameters for each of these steps.

Example: extract and load

In the bathing water quality example, there is on URL for all bathing areas. We map the resource BathingWaterQuality to this URL. the data is returned as JSON, which is why we use fromJSON as extraction function. The data is then transformed into a data.frame. (We do not go into into every detail of that transformation step.) Hence, the call to urldata is:

> be_bathing_water_quality <- function() urldata(
+   template="http://www.berlin.de/badegewaesser/baden-details/index.php/index/all.json?q=%s",
+   map.lst=list(BathingWaterQuality=""),
+   extract.fct=fromJSON,
+   transform.fct=function(x) {
+     tbl <- x[["index"]]
+     nm <- names(tbl[[1]])
+     dat <- as.data.frame(
+       matrix(NA, length(tbl), length(nm))
+     )
+     colnames(dat) <- nm
+     for(i in 1:length(tbl)) 
+        dat[i,nm] <- tbl[[i]][nm]
+     #some steps omitted...
+     return(dat)
+   }
+ )

Now, we can create an data object, inspect it using the queries method, and access the latest measurements form the web using query:

> bq <- be_bathing_water_quality()
> queries(bq)
[1] "BathingWaterQuality"
> nrow(query(bq, "BathingWaterQuality"))
[1] 38

The urldata function as part of the datamart package has been described in more detail in an earlier post on the Gapminder datasets.

Example: load

In order to save the scraped data, one more step is necessary. The urldata provides a parameter scrape.lst for specifying which resources to save locally, and how:

> be_bathing_water_quality <- function() urldata(
+   template="...", # see above
+   map.lst=list(BathingWaterQuality=""),
+   extract.fct=fromJSON,
+   #transform.fct= # see above
+   scrape.lst=list(
+     BathingWaterQuality=list(
+       uniq=c("id", "badname", "dat"), 
+       saveMode="u", 
+       dates="dat"
+     )
+   )
+ )

The name(s) of the scrape.lst argument must be a subset of the names of map.lst. The entries of the list are passed as arguments to dbWriteTable: - saveMode="u" indicates that existing observations should be replaced and new observations appended, uniq defines the columns that determine if an observation exists or not. In the example, an observation is identified by the name of the bathing area (id, badname) and the measurement date (dat). If a place/date combination already exists both in the database and in the newly scraped data, the row in the database gets overwritten. - There are other options to dbWriteTable such as specification of columns with dates or timestamps to enforce data type conversion.

Now with this extended function definition we can decide to use local or web data:

> bq <- be_bathing_water_quality()
> ds <- datastore(":memory:") # not persistent
> scrape(bq, ds)
> #
> #local:
> system.time(query(bq, "BathingWaterQuality", dbconn=ds))
user  system elapsed 
0       0       0 
> #
> #web:
> system.time(query(bq, "BathingWaterQuality"))
user  system elapsed 
0.04    0.00    0.23 

Usually, the local data is accessed faster. Additionally, it is possible to keep observations as long as we want to.

Conclusion

This blog post belongs to a series of articles on a S4 data concept I am currently prototyping. Currently, the framework allows to access and query “conventional” internal datasets, SPARQL endpoints and datasets on the web. Topic of the next blog post is combining data objects using the Mashup class.

On the other side, I am thinking of building on top of this infrastructure. Combining data objects with text templates similar to brew is one idea, interactive views using rpanelor similiar is another.

The datamart package is in its early stage, changes to the classes are likely to happen. I would appreciate feedback on the concept. Please leave a comment or send an email.

Berlin's children

Few years ago, a newspaper claimed the block I live in — Prenzlauer Berg in Berlin — is the most fertile region in Europe. It was a hoax, as this (German) newspaper article points out. (The article has become quite famous because it coined the term Bionade Biedermeier to describe the life style in this area.)

However, there are more children in my district than in the other parts of Berlin. Have a look at this map:

(The base map and population data come from the State’s statistical office. Data at block level is not readily available, though.)

The place I live is marked by a hair cross. Indeed, in this district there is a “higher exposure to kids” than in the other districts, one children per 1000 inhabitants more than in Friedrichshain-Kreuzberg, and twelve children per 1000 more than in Charlottenburg-Wilmersdorf. Exposure is of course different from fertility, maybe that is what I learned from playing with the map.

If you want to know how to draw this thematic map with R, and add the point and the legends to it, or if you are just looking for a shapefile of Berlin, then read on.