Surveillance and privacy
Discussion of issues related to liberty and privacy, and especially how they are affected by and interrelated with data management and analytic technologies. Related subjects include:
Petabyte-scale data management
Privacy, censorship, and freedom (in The Monash Report)
Integrated internet system design
What are the central challenges in internet system design? We probably all have similar lists, comprising issues such as scale, scale-out, throughput, availability, security, programming ease, UI, or general cost-effectiveness. Screw those up, and you don’t have an internet business.
Much new technology addresses those challenges, with considerable success. But the success is usually one silo at a time — a short-request application here, an analytic database there. When it comes to integration, unsolved problems abound.
The top integration and integration-like challenges for me, from a practical standpoint, are:
- Integrating silos — a decades-old problem still with us in a big way.
- Dynamic schemas with joins.
- Low-latency business intelligence.
- Human real-time personalization.
Other concerns that get mentioned include:
- Geographical distribution due to privacy laws, which for some users is a major requirement for compliance.
- Logical data warehouse, a term that doesn’t actually mean anything real.
- In-memory data grids, which some day may no longer always be hand-coupled to the application and data stacks they accelerate.
Let’s skip those latter issues for now, focusing instead on the first four.
Where the privacy discussion needs to head
An Atlantic article suggests that the digital advertising industry is coalescing around the position “restrict data use if you must, but go easy on data collection and retention.”
There is a fascinating scrum over what “Do Not Track” tools should do and what orders websites will have to respect from users. The Digital Advertising Alliance (of which the NAI is a part), the Federal Trade Commission, W3C, the Internet Advertising Bureau (also part of the DAA), and privacy researchers at academic institutions are all involved. In November, the DAA put out a new set of principles that contain some good ideas like the prohibition of “collection, use or transfer of Internet surfing data across Websites for determination of a consumer’s eligibility for employment, credit standing, healthcare treatment and insurance.”
This week, the White House seemed to side with privacy advocates who want to limit collection, not just uses. Its Consumer Privacy Bill of Rights pushes companies to allow users to “exercise control over what personal data companies collect from them and how they use it.” The DAA heralded its own participation in the White House process, though even it noted this is the beginning of a long journey.
There has been a clear and real philosophical difference between the advertisers and regulators representing web users. On the one hand, as Stanford privacy researcher Jonathan Mayer put it, “Many stakeholders on online privacy, including U.S. and EU regulators, have repeatedly emphasized that effective consumer control necessitates restrictions on the collection of information, not just prohibitions on specific uses of information.” But advertisers want to keep collecting as much data as they can as long as they promise to not to use it to target advertising. That’s why the NAI opt-out program works like it does.
That’s a drum I’ve been beating for years, so to a first approximation I’m pleased. However:
- I don’t think currently proposed protections go nearly far enough, for reasons I previously stated plus others that keep coming to me. (For example, substantially all consumer privacy protections could be nuked simply by user agreements that compel you to “voluntarily” renounce most privacy rights in return for unfettered use of the internet.)
- If current trends are followed, it could end up that data use restrictions are too mild and data collection restrictions are too severe — and maybe that will all work out in a rough balance, at least for a while.
- In the not-so-near term, however, these rough political compromises may not work so well. That’s why I think next-generation digital advertising ecosystem design should start yesterday, or perhaps sooner.
So to sum up my views on consumer privacy:
- Focusing on data use is basically good.
- It is important to also focus on data collection, at least for a transitional period.
- For the whole thing to work out well, a major rethinking of systems is needed.
That’s the good news. The bad news is on the side of government data collection and use. As I wrote last year: Read more
| Categories: Surveillance and privacy | 10 Comments |
Translucent modeling, and the future of internet marketing
There’s a growing consensus that consumers require limits on the predictive modeling that is done about them. That’s a theme of the Obama Administration’s recent work on consumer data privacy; it’s central to other countries’ data retention regulations; and it’s specifically borne out by the recent Target-pursues-pregnant-women example. Whatever happens legally, I believe this also calls for a technical response, namely:
Consumers should be shown key factual and psychographic aspects of how they are modeled, and be given the chance to insist that marketers disregard any or all of those aspects.
I further believe that the resulting technology should be extended so that
information holders can collaborate by exchanging estimates for such key factors, rather than exchanging the underlying data itself.
To some extent this happens today, for example with attribution/de-anonymization or with credit scores; but I think it should be taken to another level of granularity.
My name for all this is translucent modeling, rather than “transparent”, the idea being that key points must be visible, but the finer details can be safely obscured.
Examples of dialog I think marketers should have with consumers include: Read more
| Categories: Predictive modeling and advanced analytics, Surveillance and privacy, Web analytics | Leave a Comment |
The latest privacy example — pregnant potential Target shoppers
Charles Duhigg of the New York Times wrote a very interesting article, based on a forthcoming book of his, on two related subjects:
- The force of habit on our lives, and how we can/do deal with it. (That’s the fascinating part.)
- A specific case of predictive modeling. (That’s the part that’s getting all the attention. It’s interesting too.)
The predictive modeling part is that Target determined:
- People only change their shopping habits occasionally
- One of those occasions is when they get pregnant
- Hence, it would be a Really Good Idea to market aggressively to pregnant women
and then built a marketing strategy around early indicators of a woman’s pregnancy. Read more
| Categories: Predictive modeling and advanced analytics, Specific users, Surveillance and privacy | Leave a Comment |
The Consumer Privacy Bill of Rights — OK but totally insufficient
The Obama Administration recently released a position paper on consumer data privacy. I have mixed feelings about it.
The document admirably says:
- Internet-related regulation should be informal, so as to maintain flexibility in the face of technological change (and, less clearly stated, government technological ignorance).
- Consumers should be given opt-ins and opt-outs regarding data retention, which should have good, clear user interfaces.
- If you don’t have good data security, then you’re not doing a good job of protecting privacy.
But it says less than it seems to about protecting citizens from privacy invasion by businesses. And it says nothing at all about protecting citizens from privacy invasion by government, which in the first footnote it says is beyond the scope of the document. On the whole, I think the document does much less than what is needed.
The core of the paper is a “Consumer Privacy Bill of Rights”, with seven provisions. Here goes: Read more
| Categories: Surveillance and privacy | 1 Comment |
Aster Data business trends
Last month, I reviewed with the Aster Data folks which markets they were targeting and selling into, subsequent to acquisition by their new orange overlords. The answers aren’t what they used to be. Aster no longer focuses much on what it used to call frontline (i.e., low-latency, operational) applications; those are of course a key strength for Teradata. Rather, Aster focuses on investigative analytics — they’ve long endorsed my use of the term — and on the batch run/scoring kinds of applications that inform operational systems.
| Categories: Analytic technologies, Application areas, Aster Data, Data warehousing, DataStax, RDF and graphs, Surveillance and privacy, Teradata, Web analytics | 1 Comment |
Application areas for SAS HPA
When I talked with SAS about its forthcoming in-memory parallel SAS HPA offering, we talked briefly about application areas. The three SAS cited were:
- Consumer financial services. The idea here is to combine information about customers’ use of all kinds of services — banking, credit cards, loans, etc. SAS believes this is both for marketing and risk analysis purposes.
- Insurance. We didn’t go into detail.
- Mobile communications. SAS’ customers aren’t giving it details, but they’re excited about geocoding/geospatial data.
Meanwhile, in another interview I heard about, SAS emphasized retailers. Indeed, that’s what spawned my recent post about logistic regression.
The mobile communications one is a bit scary. Your cell phone — and hence your cellular company — know where you are, pretty much from moment to moment. Even without advanced analytic technology applied to it, that’s a pretty direct privacy threat. Throw in some analytics, and your cell company might know, for example, who you hang out with (in person), where you shop, and how those things predict your future behavior. And so the government — or just your employer — might know those things too.
| Categories: Application areas, Predictive modeling and advanced analytics, SAS Institute, Surveillance and privacy, Telecommunications | 2 Comments |
So how many columns can a single table have anyway?
I have a client who is hitting a 1000 column-per-table limit in Oracle Standard Edition. As you might imagine, I’m encouraging them to consider columnar alternatives. Be that as it may, just what ARE the table width limits in various analytic or general-purpose DBMS products?
By the way — the answer SHOULD be “effectively unlimited.” Like it or not,* there are a bunch of multi-thousand-column marketing-prospect-data tables out there.
*Relational purists may dislike the idea for one reason, privacy-concerned folks for quite another.
| Categories: Data warehousing, Surveillance and privacy | 37 Comments |
The technology of privacy threats
This post is the second of a series. The first one was an overview of privacy dangers, replete with specific examples of kinds of data that are stored for good reasons, but can also be repurposed for more questionable uses. More on this subject may be found in my August, 2010 post Big Data is Watching You!
There are two technology trends driving electronic privacy threats. Taken together, these trends raise scenarios such as the following:
- Your web surfing behavior indicates you’re a sports car buff, and you further like to look at pictures of scantily-clad young women. A number of your Facebook friends are single women. As a result, you’re deemed a risk to have a mid-life crisis and divorce your wife, thus increasing the interest rate you have to pay when refinancing your house.
- Your cell phone GPS indicates that you drive everywhere, instead of walking. There is no evidence of you pursuing fitness activities, but forum posting activity suggests you’re highly interested in several TV series. Your credit card bills show that your taste in restaurant food tends to the fatty. Your online photos make you look fairly obese, and a couple have ashtrays in them. As a result, you’re judged a high risk of heart attack, and your medical insurance rates are jacked up accordingly.
- You did actually have that mid-life crisis and get divorced. At the child-custody hearing, your ex-spouse’s lawyer quotes a study showing that football-loving upper income Republicans are 27% more likely to beat their children than yoga-class-attending moderate Democrats, and the probability goes up another 8% if they ever bought a jersey featuring a defensive lineman. What’s more, several of the more influential people in your network of friends also fit angry-male patterns, taking the probability of abuse up another 13%. Because of the sound statistics behind such analyses, the judge listens.
Not all these stories are quite possible today, but they aren’t far off either.
| Categories: Facebook, Predictive modeling and advanced analytics, Surveillance and privacy, Telecommunications, Web analytics | 4 Comments |
Privacy dangers — an overview
This post is the first of a series. The second one delves into the technology behind the most serious electronic privacy threats.
The privacy discussion has gotten more active, and more complicated as well. A year ago, I still struggled to get people to pay attention to privacy concerns at all, at least in the United States, with my first public breakthrough coming at the end of January. But much has changed since then.
On the commercial side, Facebook modified its privacy policies, garnering great press attention and an intense user backlash, leading to a quick partial retreat. The Wall Street Journal then launched a long series of articles — 13 so far — recounting multiple kinds of privacy threats. Other media joined in, from Forbes to CNet. Various forms of US government rule-making to inhibit advertising-related tracking have been proposed as an apparent result.
In the US, the government had a lively year as well. The Transportation Security Administration (TSA) rolled out what have been dubbed “porn scanners,” and backed them up with “enhanced patdowns.” For somebody who is, for example, female, young, a sex abuse survivor, and/or a follower of certain religions, those can be highly unpleasant, if not traumatic. Meanwhile, the Wikileaks/Cablegate events have spawned a government reaction whose scope is only beginning to be seen. A couple of “highlights” so far are some very nasty laptop seizures, and the recent demand for information on over 600,000 Twitter accounts. (Christopher Soghoian provided a detailed, nuanced legal analysis of same.)
At this point, it’s fair to say there are at least six different kinds of legitimate privacy fear. Read more