March 3, 2006

BuzzMetrics.

Here's a WaPo article on the sophisticated software companies are using to track public opinion as it is expressed on the internet:
To capture the chatter, Nielsen BuzzMetrics, a giant in the industry, uses software that collects hundreds of thousands of comments a day. The technology can scan for specific companies, products, brands, people -- anything searchable. It can slice data into a range of categories to quantify the number of times a subject was discussed online, the individuals who mentioned it and the communities where it appeared.
Examples of knowledge acquired by this amazing new technology? People are going to be wanting more snacks in the future, and they prefer "American Idol" to the Olympics.

Or do you find this technology ominous?

Talk Left speculates that the government is using these techniques too. Is that bad? Is ascertaining the drift of the on-line conversation be any more threatening than than taking political polls? This process of aggregating large numbers of statements to read the general opinion is quite different from monitoring an individual. But in any case, the individuals generating the statements are writing in a completely open public space. We can't say the government is invading our privacy when we are inviting the entire world to read us. Yet maybe something terrible is happening, and we blithe bloggers will live to regret it.

25 comments:

Hoots said...

Technology is a wonderful thing. I remember our first TV when I was about ten years old. Only one or two channels had a picture, and even those were often full of snow. I remember thinking how television was better than radio because you could not only hear static, you sould also see it.

Today's technology extends the same phenomenon to public opinion. Thanks to comment threads mob behavior is no longer restricted to streets and fields. We get to participate from the comfort of our keyboards.

Likewise, corporations (just like people, you know, except bigger) are no longer limited to squeezing out the competition down the street, they can now do battle with the whole world as a playing field.

I have a flash of that scene in King Kong where the prehistoric monster is fighting a contemporary monster.

MadisonMan said...

Or do you find this technology ominous?

I expect it. But what if bloggers suddenly start throwing in words like Charmin Bath Tissue or Cheetos -- will that skew the Starbucks (Or as we call it, the church of St. Arbucks) demographics?

Osama Bin Laden.

L. Ron Halfelven said...

I'd like any corporate entity monitoring this blog to know that that my comments are now available for product-placement purposes. If you have a product or service that you'd like to see discreetly mentioned in the course of a comment about gay marriage or Social Security privatization, you'll find my rates surprisingly reasonable.

VW: izlloy. It's not just an alloy, it's Izlloy! Full metallurgical data and pricing available on request.

hoosthere said...

I think we are getting used to such intrusion, and are becoming more accepting than we may have been in years past.

For example, anybody who uses gmail (as I do) and perhaps other email services has become accustomed to Google analyzing contents of an email for the purposes of advertising related products. This can be a jarring experience, as it was last night when I learned that a friend of mine's father had committed suicide, and there was Google, telling me how to avoid suicide on the right-hand side.

But amazingly, I wasn't offended. I would have been 2 years ago.

Gaius Arbo said...

It's funny how people get massively worked up about the government invading their privacy but completely ignore the corporations who are doing much, much more privacy invasion every day.

Inquiring minds want to know about the properties of izlloy!

Joan said...

This is akin to the methods supermarkets use to spot buying trends among their customers. I know people who won't sign up for the "club cards", but all they are basically doing is opting out of the sale prices. There's just so much data out there, it's rather incomprehensible.

I used to work for the software company that wrote the database engine and applications that did all that data slicing and dicing, many many years ago. (There's probably more than one company with similar offerings now.) Multi-dimensional analysis is not as easy as it sounds. And while you certainly can use the software to apply filters and find everyone who bought a specific product within a certain time period, or everyone who commented on a particular issue, the idea that someone would secretly do something like that for nefarious purposes remains absurd to me. People don't realize how tedious and expensive this kind of analysis is. It's not the kind of thing an individual or even fringe group would easily have access to -- at least, it didn't used to be.

yetanotherjohn said...

I would really like to know how sophisticated the sifting algoithim is. Can it discern sarcasm? What does it make of slang? Imagine asking it to discern Bush job approval or disapproval out of comments.

I suspect this is similar to having representatives standing on a busy street corner, listening to passing conversations and trying to gauge how people felt about something. It might work. But it would leave out all those who don't comment on blogs, which is probably even a majority of blog readers.

Jacques Cuze said...

We can't say the government is invading our privacy when we are inviting the entire world to read us.

How do you feel about your email being read by the government? Your email is most likely transmitted in unencrypted cleartext from the time it leaves your computer to the time it arrives at its destination. Along the way, passes through an untold number of other computers and devices, all in the clear, and none under your control.

You have agreed to allow your email to travel unencrypted through these very public routes. Do you feel you have an expectation of privacy from the government regarding your email?

Your cellphone and home wireless calls (used to be and may still be) transmitted in the clear from transmitter to receiver. Do you feel you have an expectation of privacy from the government regarding those calls?

The times you leave your home and the routes you take to work are observable by all, do you feel you have an expectation of privacy from the government regarding this information?

The addresses of the USPS letters that you send and receive, do you have an expectation of privacy from the government regarding this information?

The heat signature of the house you reside in as transmitted through its walls, windows, and ceilings, do you feel you have an expectation of privacy from the government regarding this information?

The trash, its contents, its weight, your patterns as you throw it away, do you feel you have an expectation of privacy from the government regarding this information?

Your passage through streets and how it may be observed by cameras, do you feel you have an expectation of privacy from the government regarding this information?

How do you supposed George Washington, Benjamin Franklin, John Hancock and the rest of our rebellious forefathers would have answered?

Jacques Cuze said...

If you publish your blog under the Creative Commons, Attribution-NonCommercial-ShareAlike 2.5 license

Should buzzmetrics scan it? Should the government scan it?

Oh, how would you feel about toilets that monitor your urine to detect health conditions? This was once a Larry Niven science fiction device, but are now a real device. Would it be okay for a commercial operator to place those toilets in her building (perhaps to reduce her liability?) Would it be okay for the government to take those records and scan them and analyze them?

Are you sure we can't say that our privacy is being invaded when information and patterns about our everyday existence becomes digitizable and then becomes scanned?

hoosthere said...

All this talk about privacy over things that reasonable people would never really care if people knew or not.

The times you leave your home and the routes you take to work are observable by all, do you feel you have an expectation of privacy from the government regarding this information?

Who cares?

toilets that monitor your urine to detect health conditions

Similarly, who cares, and why would somebody need to hide this information?

All the fear over "PRIVACY INVASION!!!" is so surreal to me for two reasons: 1) What are people so ashamed of that they need to hide? Listen in on ALL of my frigging phone calls Uncle Sam...why should I give a rip?? Because...2) Contrary to the conspiracy fever swamp, it is highly inefficient to monitor everybody's actual phonce conversations (computers do it, and they aren't looking for details about my sex life or my stock tips) and there are far fewer government employees out there with nefarious impulses than we would be led to believe.

MadisonMan said...

...and there are far fewer government employees out there with nefarious impulses than we would be led to believe.

This begs two questions. What do you base that statement on, and how many nefarious government employees is too many, in your opinion?

I agree it is inefficient for the government to monitor citizenry. Why, then, should they do it?

Jacques Cuze said...

If the South Dakota law abortion law makes plan b illegal, if parental notification laws further restrict abortion, wouldn't it be a reasonable step for schools to install such toilets that could monitor chemicals, anti-biotics, hormones? Or not just schools, but say, any government building?

When price drops, what happens to demand?

If Griswold were overturned, what better way these days for the government to monitor contraception use than to install toilets and other sensors in public places? Perhaps cameras outside pharmacies.

What are computers but devices that transform the "highly inefficient" into the "cost effective?"

No tinfoil hats required, all cameras, smart toilets, and computers are all very 2005 and 2006. Along with warrantless wiretapping based on the AUMF and the emplacement of a Supreme Court Justice that writes a letter thanking an anti-abortion group for their help in getting him his seat.

hoosthere said...

to answer your questions,

1) I went to high school in Langley, VA; my father is a dreaded lobbyist (and just about the most ethical man I know); I know people who work at NSA or NGIC and they are highly trustworthy; and lastly how many nefarious people do YOU know in this culture? I mean, the kind of people who would surf terrabytes of information to look for the sort of information cited above (e.g. commuter patterns) for the purposes of evil ends. I bet not very many.

2) Sure, one nefarious government employee is too many, but that's why it's so frigging huge...the checks and balances which are in place to prevent abuse are the cause of all the dreaded red-tape we always moan about. Bloated bureacracies are good for SOMETHING after all (providing checks and balances).

And last question...um, to keep people from blowing stuff up works well enough for me.

and just read your post q-man...seriously, I can't believe anybody would actually believe that our government ENFORCE contraception use? The point here is that America will never be Maoist China, despite the black helicopters crowd. The very fact that we can have this convo is case in point.

PatCA said...

Buzzmetrics sounds like a lot of empty...buzz. The kind of narrow focus results in, for instance, the insipid, timid political rhetoric of today. FDR didn't have buzzmetrics, and he moved a population (yes, sometimes to rage, but he moved them).

I worked for the government for many years. Believe me, they are not capable of making any coherent use out of this mass of information they are supposedly secretly gathering any more than they are capable now of finding the guys who stole your TV, despite their having left fingerprints all over your house.

Terrorists, OTOH, with this information worries me greatly. All it takes is one lucky strike.

MadisonMan said...

sure, one nefarious government employee is too many, but that's why it's so frigging huge.

I ponder how J. Edgar Hoover would work in the computer age.

As for accepting snooping to counter terrorist threats, well that's been discussed earlier re: the "Patriot" Act. I am willing to live with a very low-level threat and be unmonitored. Others here prefer more government snooping for (what I believe to be) a false sense of security.

Semanticleo said...

Just as any new technology becomes
proven, or merely theoretical,
military minds seek ways to exploit
for weapons or intel.

Yes, it is ominous.

L. Ron Halfelven said...

Your marital status (Lines 1-5), number of dependents (Line 6), income from wages, taxes, and tips (Line 7), interest (Line 8), and dividends (Line 9).... do you feel you have an expectation of privacy from the government regarding this information?

Jacques Cuze said...

Your marital status (Lines 1-5), number of dependents (Line 6), income from wages, taxes, and tips (Line 7), interest (Line 8), and dividends (Line 9).... do you feel you have an expectation of privacy from the government regarding this information?

How they use that? Absolutely. Should congressman, senators, or random bureaucrats be able to pull that information up? No way.

Should specific authorized representatives of the IRS be able to use that information to determine the tax? Sadly, yes.

Jacques Cuze said...

I can't believe anybody would actually believe that our government ENFORCE contraception use?

I didn't say enforce, I said monitor ... Perhaps you have never heard of Griswold, Plan B, or South Dakota's anti-choice bill.

SippicanCottage said...
This comment has been removed by the author.
L. Ron Halfelven said...

"Privacy from the government", then, means only that the government agent who's spying on you has the proper credentials? Ben Franklin himself couldn't have put it better.

Jacques Cuze said...

"Privacy from the government", then, means only that the government agent who's spying on you has the proper credentials?

If by credentials you mean warrants or legal authorization than I guess yes. I would think that Franklin would be for warrants and statutes, but I am not the expert.

If you want to argue that the income tax itself neccessitates an unconstitutional invasion of our privacy you're on your own.

hoosthere said...

Sippican,

R. O. F. L.

Smilin' Jack said...

quxxo said...

No tinfoil hats required, all cameras, smart toilets, and computers are all very 2005 and 2006.


More like 1995-96. We've come a long way since then. Didn't you read Ann's post last month about the government's tiny robotic spy insects? There's one behind you right now, recording every move, every word, transmitting it all to Washington, even reading over your shoulder as you type your comments on this blog. I'd hate to be you when Karl Rove gets around to reading the report on you.

L. Ron Halfelven said...

I would think that Franklin would be for warrants and statutes, but I am not the expert.

All righty then! All the things in your list of horribles that aren't already legal can go into PATRIOT III, and old Ben can sleep soundly.