What is ‘Big Data’?

A new hype shakes up the business world. Everybody talks about “Big Data”. But what is that all about?

Basically it’s an utterly normal issue. More and more data is collected, increasingly larger amounts of data have to be analysed and made “usable”. And this is the crux of the matter, even NSA faces problems with. Because you can’t see the wood for the trees. And to call for the example of the Needle in the Haystack… As I showed years ago, that “problem” is none, but can be solved quickly with the right tool (a strong magnet)…

But what the “experts” try here, is to find from a randomly found pine tree needle from the forest, all the other needles that originally belonged to the same tree. And the forest grows!

The first, Big-Data-experts came up with, have been personal profiles, coming from a variety of different sources. That Google and Facebook still offer me young Russian ladies for marriage is a good sign that they are way off even that goal. It makes their life not any easier that I don’t work at German Mail in Bonn, neither being the treasurer of Kassel.

Big Data is simply a reality, no “hype”. But it remains a “problem”. To manage “big data”, I need the matching technology on the one side, on the other intelligent and fast fitering algorythms. And as the amount of data grows faster than the processing power, the real problem is predictable. Either I filter the large amount of data very rough, or I filter smaller amount of data very detailed. A question of width and depth of information. So what does NSA do with the data? First, filtering it by catch words and phrases. Then have a highly paid analyst try to separate the wheat from the chaff.

Two dangers:

1. False Positive. This defines information that is incorrectly assessed. I.e. in Spam an important message the computer considers Spam. Is it about people and the NSA, a “false positive” can have devastating repercussions, such as the denial for totally innocent people trying to enter the U.S. (such happening).

2. Privacy Protection. It’s being compromised and I keep lately quoting the former U.S. Presidents Franklin und Truman:

“Those who surrender freedom for security will not have, nor do they deserve, either one.” [Benjamin Franklin]

“we want no Gestapo or secret police. FBI is tending in that direction. They are dabbling in sex-life scandals and plain blackmail. J. Edgar Hoover would give his right eye to take over, and all congressmen and senators are afraid of him”. Ersetze FBI durch NSA… Und was, wenn die “Teaparty” tatsächlich an die Macht kommt (vergleichbar bei uns der NPD)? [Harry S. Truman]
Replace FBI by NSA… And what, if the “Tea Party” in the U.S. really get’s to power (comparably our NPD)? We call it something falling into the wrong hands.

In aviation, we also work daily with “big data” and not just British Airways develops ideas to merge their customer profiles with the public profiles of their clients from the Social Networks… We also have to think about how far we want to go. Or if there are other areas of “big data” of interest, such as operations management.

Food for Thought
comments welcome