'Scrapers' Dig Deep for Data on Web
by Julia Angwin and Steve Stecklow
Wall Street Journal (October 12 2010)
At 1:00 am on May 7, the website PatientsLikeMe.com noticed suspicious activity on its "Mood" discussion board. There, people exchange highly personal stories about their emotional disorders, ranging from bipolar disease to a desire to cut themselves.
It was a break-in. A new member of the site, using sophisticated software, was "scraping", or copying, every single message off PatientsLikeMe's private online forums.
PatientsLikeMe managed to block and identify the intruder: The Nielsen Company, the privately held New York media-research firm. Nielsen monitors online "buzz" for clients, including major drug makers, which buy data gleaned from the Web to get insight from consumers about their products, Nielsen says.
"I felt totally violated", says Bilal Ahmed, a 33-year-old resident of Sydney, Australia, who used PatientsLikeMe to connect with other people suffering from depression. He used a pseudonym on the message boards, but his PatientsLikeMe profile linked to his blog, which contains his real name.
After PatientsLikeMe told users about the break-in, Mr Ahmed deleted all his posts, plus a list of drugs he uses. "It was very disturbing to know that your information is being sold", he says. Nielsen says it no longer scrapes sites requiring an individual account for access, unless it has permission.
The market for personal data about Internet users is booming, and in the vanguard is the practice of "scraping". Firms offer to harvest online conversations and collect personal details from social-networking sites, resume sites and online forums where people might discuss their lives.
The emerging business of web scraping provides some of the raw material for a rapidly expanding data economy. Marketers spent $7.8 billion on online and offline data in 2009, according to the New York management consulting firm Winterberry Group LLC. Spending on data from online sources is set to more than double, to $840 million in 2012 from $410 million in 2009.
The Wall Street Journal's examination of scraping - a trade that involves personal information as well as many other types of data - is part of the newspaper's investigation into the business of tracking people's activities online and selling details about their behavior and personal interests.
Some companies collect personal information for detailed background reports on individuals, such as email addresses, cell numbers, photographs and posts on social-network sites.
Others offer what are known as listening services, which monitor in real time hundreds or thousands of news sources, blogs and websites to see what people are saying about specific products or topics.
One such service is offered by Dow Jones & Company, publisher of the Journal. Dow Jones collects data from the Web - which may include personal information contained in news articles and blog postings - that help corporate clients monitor how they are portrayed. It says it doesn't gather information from password-protected parts of sites.
The competition for data is fierce. PatientsLikeMe also sells data about its users. PatientsLikeMe says the data it sells is anonymized, no names attached.
Nielsen spokesman Matt Anchin says the company's reports to its clients include publicly available information gleaned from the Internet, "so if someone decides to share personally identifiable information, it could be included".
Internet users often have little recourse if personally identifiable data is scraped: There is no national law requiring data companies to let people remove or change information about themselves, though some firms let users remove their profiles under certain circumstances.
California has a special protection for public officials, including politicians, sheriffs and district attorneys. It makes it easier for them to remove their home address and phone numbers from these databases, by filling out a special form stating they fear for their safety.
Data brokers long have scoured public records, such as real-estate transactions and courthouse documents, for information on individuals. Now, some are adding online information to people's profiles.
Many scrapers and data brokers argue that if information is available online, it is fair game, no matter how personal.
"Social networks are becoming the new public records", says Jim Adler, chief privacy officer of Intelius Incorporated, a leading paid people-search website. It offers services that include criminal background checks and "Date Check", which promises details about a prospective date for $14.95.
"This data is out there", Mr Adler says. "If we don't bring it to the consumer's attention, someone else will".
New York-based PeekYou LLC has applied for a patent for a method that, among other things, matches people's real names to the pseudonyms they use on blogs, Twitter and other social networks. PeekYou's people-search website offers records of about 250 million people, primarily in the US and Canada.
PeekYou says it also is starting to work with listening services to help them learn more about the people whose conversations they are monitoring. It says it hands over only demographic information, not names or addresses.
Employers, too, are trying to figure out how to use such data to screen job candidates. It's tricky: Employers legally can't discriminate based on gender, race and other factors they may glean from social-media profiles.
One company that screens job applicants for employers, InfoCheckUSA LLC in Florida, began offering limited social-networking data - some of it scraped - to employers about a year ago. "It's slowly starting to grow", says Chris Dugger, national account manager. He says he's particularly interested in things like whether people are "talking about how they just ripped off their last employer".
Scrapers operate in a legal gray area. Internationally, anti-scraping laws vary. In the US, court rulings have been contradictory. "Scraping is ubiquitous, but questionable", says Eric Goldman, a law professor at Santa Clara University. "Everyone does it, but it's not totally clear that anyone is allowed to do it without permission".
Scrapers and listening companies say what they're doing is no different from what any person does when gathering information online - they just do it on a much larger scale.
"We take an incomprehensible amount of information and make it intelligent", says Chase McMichael, chief executive of InfiniGraph, a Palo Alto, California, "listening service" that helps companies understand the likes and dislikes of online customers.
Scraping services range from dirt cheap to custom-built. Some outfits, such as 80Legs.com in Texas, will scrape a million Web pages for $101. One Utah company, screen-scraper.com, offers do-it-yourself scraping software for free. The top listening services can charge hundreds of thousands of dollars to monitor and analyze Web discussions.
Some scrapers-for-hire don't ask clients many questions.
"If we don't think they're going to use it for illegal purposes - they often don't tell us what they're going to use it for - generally, we'll err on the side of doing it", says Todd Wilson, owner of screen-scraper.com, a ten-person firm in Provo, Utah, that operates out of a two-room office. It is one of at least three firms in a scenic area known locally as "Happy Valley" that specialize in scraping.
Screen-scraper charges between $1,500 and $10,000 for most jobs. The company says it's often hired to conduct "business intelligence", working for companies who want to scrape competitors' websites.
One recent assignment: A major insurance company wanted to scrape the names of agents working for competitors. Why? "We don't know", says Scott Wilson, the owner's brother and vice president of sales. Another job: attempting to scrape Facebook for a multi-level marketing company that wanted email addresses of users who "like" the firm's page - as well as their friends - so they all could be pitched products.
Scraping often is a cat-and-mouse game between websites, which try to protect their data, and the scrapers, who try to outfox their defenses. Scraping itself isn't difficult: Nearly any talented computer programmer can do it. But penetrating a site's defenses can be tough.
One defense familiar to most Internet users involves "captchas", the squiggly letters that many websites require people to type to prove they're human and not a scraping robot. Scrapers sometimes fight back with software that deciphers captchas.
Some professional scrapers stage blitzkrieg raids, mounting around a dozen simultaneous attacks on a website to grab as much data as quickly as possible without being detected or crashing the site they're targeting.
Raids like these are on the rise. "Customers for whom we were regularly blocking about 1,000 to 2,000 scrapes a month are now seeing three times or in some cases ten times as much scraping", says Marino Zini, managing director of Sentor Anti Scraping System. The company's Stockholm team blocks scrapers on behalf of website clients.
At Monster.com, the jobs website that stores resumes for tens of millions of individuals, fighting scrapers is a full-time job, "every minute of every day of every week", says Patrick Manzo, global chief privacy officer of Monster Worldwide Incorporated. Facebook, with its trove of personal data on some 500 million users, says it takes legal and technical steps to deter scraping.
At PatientsLikeMe, there are forums where people discuss experiences with AIDS, supranuclear palsy, depression, organ transplants, post-traumatic stress disorder and self-mutilation. These are supposed to be viewable only by members who have agreed not to scrape, and not by intruders such as Nielsen.
"It was a bad legacy practice that we don't do anymore", says Dave Hudson, who in June took over as chief executive of the Nielsen unit that scraped PatientsLikeMe in May. "It's something that we decided is not acceptable, and we stopped".
Mr Hudson wouldn't say how often the practice occurred, and wouldn't identify its client.
The Nielsen unit that did the scraping is now part of a joint venture with McKinsey & Company called NM Incite. It traces its roots to a Cincinnati company called Intelliseek that was founded in 1997. One of its most successful early businesses was scraping message boards to find mentions of brand names for corporate clients.
In 2001, the venture-capital arm of the Central Intelligence Agency, In-Q-Tel Incorporated, was among a group of investors that put $8 million into the business.
Intelliseek struggled to set boundaries in the new business of monitoring individual conversations online, says Sundar Kadayam, Intelliseek's co-founder. The firm decided it wouldn't be ethical to use automated software to log into private message boards to scrape them.
But, he says, Intelliseek occasionally would ask employees to do that kind of scraping if clients requested it. "The human being can just sign in as who they are", he says. "They don't have to be deceitful".
In 2006, Nielsen bought Intelliseek, which had revenue of more than $10 million and had just become profitable, Mr Kadayam says. He left one year after the acquisition.
At the time, Nielsen, which provides television ratings and other media services, was looking to diversify into digital businesses. Nielsen combined Intelliseek with a New York startup it had bought called BuzzMetrics.
The new unit, Nielsen BuzzMetrics, quickly became a leader in the field of social-media monitoring. It collects data from 130 million blogs, 8,000 message boards, Twitter and social networks. It sells services such as "ThreatTracker", which alerts a company if its brand is being discussed in a negative light. Clients include more than a dozen of the biggest pharmaceutical companies, according to the company's marketing material.
Like many websites, PatientsLikeMe has software that detects unusual activity. On May 7, that software sounded an alarm about the "Mood" forum.
David Williams, the chief marketing officer, quickly determined that the "member" who had triggered the alert actually was an automated program scraping the forum. He shut down the account.
The next morning, the holder of that account e-mailed customer support to ask why the login and password weren't working. By the afternoon, PatientsLikeMe had located three other suspect accounts and shut them down. The site's investigators traced all of the accounts to Nielsen BuzzMetrics.
On May 18, PatientsLikeMe sent a cease-and-desist letter to Nielsen. Ten days later, Nielsen sent a letter agreeing to stop scraping. Nielsen says it was unable to remove the scraped data from its database, but a company spokesman later said Nielsen had found a way to quarantine the PatientsLikeMe data to prevent it from being included in its reports for clients.
PatientsLikeMe's president, Ben Heywood, disclosed the break-in to the site's 70,000 members in a blog post. He also reminded users that PatientsLikeMe also sells its data in an anonymous form, without attaching user's names to it. That sparked a lively debate on the site about the propriety of selling sensitive information. The company says most of the 350 responses to the blog post were supportive. But it says a total of 218 members quit.
In total, PatientsLikeMe estimates that the scraper obtained about five percent of the messages in the site's forums, primarily in "Mood" and "Multiple Sclerosis".
"We're a business, and the reality is that someone came in and stole from us", says PatientsLikeMe's chairman, Jamie Heywood.
http://online.wsj.com/article/SB10001424052748703358504575544381288117888.html?mod=WSJ_article_MoreIn_Health
Bill Totten http://www.ashisuto.co.jp/english/
Wall Street Journal (October 12 2010)
At 1:00 am on May 7, the website PatientsLikeMe.com noticed suspicious activity on its "Mood" discussion board. There, people exchange highly personal stories about their emotional disorders, ranging from bipolar disease to a desire to cut themselves.
It was a break-in. A new member of the site, using sophisticated software, was "scraping", or copying, every single message off PatientsLikeMe's private online forums.
PatientsLikeMe managed to block and identify the intruder: The Nielsen Company, the privately held New York media-research firm. Nielsen monitors online "buzz" for clients, including major drug makers, which buy data gleaned from the Web to get insight from consumers about their products, Nielsen says.
"I felt totally violated", says Bilal Ahmed, a 33-year-old resident of Sydney, Australia, who used PatientsLikeMe to connect with other people suffering from depression. He used a pseudonym on the message boards, but his PatientsLikeMe profile linked to his blog, which contains his real name.
After PatientsLikeMe told users about the break-in, Mr Ahmed deleted all his posts, plus a list of drugs he uses. "It was very disturbing to know that your information is being sold", he says. Nielsen says it no longer scrapes sites requiring an individual account for access, unless it has permission.
The market for personal data about Internet users is booming, and in the vanguard is the practice of "scraping". Firms offer to harvest online conversations and collect personal details from social-networking sites, resume sites and online forums where people might discuss their lives.
The emerging business of web scraping provides some of the raw material for a rapidly expanding data economy. Marketers spent $7.8 billion on online and offline data in 2009, according to the New York management consulting firm Winterberry Group LLC. Spending on data from online sources is set to more than double, to $840 million in 2012 from $410 million in 2009.
The Wall Street Journal's examination of scraping - a trade that involves personal information as well as many other types of data - is part of the newspaper's investigation into the business of tracking people's activities online and selling details about their behavior and personal interests.
Some companies collect personal information for detailed background reports on individuals, such as email addresses, cell numbers, photographs and posts on social-network sites.
Others offer what are known as listening services, which monitor in real time hundreds or thousands of news sources, blogs and websites to see what people are saying about specific products or topics.
One such service is offered by Dow Jones & Company, publisher of the Journal. Dow Jones collects data from the Web - which may include personal information contained in news articles and blog postings - that help corporate clients monitor how they are portrayed. It says it doesn't gather information from password-protected parts of sites.
The competition for data is fierce. PatientsLikeMe also sells data about its users. PatientsLikeMe says the data it sells is anonymized, no names attached.
Nielsen spokesman Matt Anchin says the company's reports to its clients include publicly available information gleaned from the Internet, "so if someone decides to share personally identifiable information, it could be included".
Internet users often have little recourse if personally identifiable data is scraped: There is no national law requiring data companies to let people remove or change information about themselves, though some firms let users remove their profiles under certain circumstances.
California has a special protection for public officials, including politicians, sheriffs and district attorneys. It makes it easier for them to remove their home address and phone numbers from these databases, by filling out a special form stating they fear for their safety.
Data brokers long have scoured public records, such as real-estate transactions and courthouse documents, for information on individuals. Now, some are adding online information to people's profiles.
Many scrapers and data brokers argue that if information is available online, it is fair game, no matter how personal.
"Social networks are becoming the new public records", says Jim Adler, chief privacy officer of Intelius Incorporated, a leading paid people-search website. It offers services that include criminal background checks and "Date Check", which promises details about a prospective date for $14.95.
"This data is out there", Mr Adler says. "If we don't bring it to the consumer's attention, someone else will".
New York-based PeekYou LLC has applied for a patent for a method that, among other things, matches people's real names to the pseudonyms they use on blogs, Twitter and other social networks. PeekYou's people-search website offers records of about 250 million people, primarily in the US and Canada.
PeekYou says it also is starting to work with listening services to help them learn more about the people whose conversations they are monitoring. It says it hands over only demographic information, not names or addresses.
Employers, too, are trying to figure out how to use such data to screen job candidates. It's tricky: Employers legally can't discriminate based on gender, race and other factors they may glean from social-media profiles.
One company that screens job applicants for employers, InfoCheckUSA LLC in Florida, began offering limited social-networking data - some of it scraped - to employers about a year ago. "It's slowly starting to grow", says Chris Dugger, national account manager. He says he's particularly interested in things like whether people are "talking about how they just ripped off their last employer".
Scrapers operate in a legal gray area. Internationally, anti-scraping laws vary. In the US, court rulings have been contradictory. "Scraping is ubiquitous, but questionable", says Eric Goldman, a law professor at Santa Clara University. "Everyone does it, but it's not totally clear that anyone is allowed to do it without permission".
Scrapers and listening companies say what they're doing is no different from what any person does when gathering information online - they just do it on a much larger scale.
"We take an incomprehensible amount of information and make it intelligent", says Chase McMichael, chief executive of InfiniGraph, a Palo Alto, California, "listening service" that helps companies understand the likes and dislikes of online customers.
Scraping services range from dirt cheap to custom-built. Some outfits, such as 80Legs.com in Texas, will scrape a million Web pages for $101. One Utah company, screen-scraper.com, offers do-it-yourself scraping software for free. The top listening services can charge hundreds of thousands of dollars to monitor and analyze Web discussions.
Some scrapers-for-hire don't ask clients many questions.
"If we don't think they're going to use it for illegal purposes - they often don't tell us what they're going to use it for - generally, we'll err on the side of doing it", says Todd Wilson, owner of screen-scraper.com, a ten-person firm in Provo, Utah, that operates out of a two-room office. It is one of at least three firms in a scenic area known locally as "Happy Valley" that specialize in scraping.
Screen-scraper charges between $1,500 and $10,000 for most jobs. The company says it's often hired to conduct "business intelligence", working for companies who want to scrape competitors' websites.
One recent assignment: A major insurance company wanted to scrape the names of agents working for competitors. Why? "We don't know", says Scott Wilson, the owner's brother and vice president of sales. Another job: attempting to scrape Facebook for a multi-level marketing company that wanted email addresses of users who "like" the firm's page - as well as their friends - so they all could be pitched products.
Scraping often is a cat-and-mouse game between websites, which try to protect their data, and the scrapers, who try to outfox their defenses. Scraping itself isn't difficult: Nearly any talented computer programmer can do it. But penetrating a site's defenses can be tough.
One defense familiar to most Internet users involves "captchas", the squiggly letters that many websites require people to type to prove they're human and not a scraping robot. Scrapers sometimes fight back with software that deciphers captchas.
Some professional scrapers stage blitzkrieg raids, mounting around a dozen simultaneous attacks on a website to grab as much data as quickly as possible without being detected or crashing the site they're targeting.
Raids like these are on the rise. "Customers for whom we were regularly blocking about 1,000 to 2,000 scrapes a month are now seeing three times or in some cases ten times as much scraping", says Marino Zini, managing director of Sentor Anti Scraping System. The company's Stockholm team blocks scrapers on behalf of website clients.
At Monster.com, the jobs website that stores resumes for tens of millions of individuals, fighting scrapers is a full-time job, "every minute of every day of every week", says Patrick Manzo, global chief privacy officer of Monster Worldwide Incorporated. Facebook, with its trove of personal data on some 500 million users, says it takes legal and technical steps to deter scraping.
At PatientsLikeMe, there are forums where people discuss experiences with AIDS, supranuclear palsy, depression, organ transplants, post-traumatic stress disorder and self-mutilation. These are supposed to be viewable only by members who have agreed not to scrape, and not by intruders such as Nielsen.
"It was a bad legacy practice that we don't do anymore", says Dave Hudson, who in June took over as chief executive of the Nielsen unit that scraped PatientsLikeMe in May. "It's something that we decided is not acceptable, and we stopped".
Mr Hudson wouldn't say how often the practice occurred, and wouldn't identify its client.
The Nielsen unit that did the scraping is now part of a joint venture with McKinsey & Company called NM Incite. It traces its roots to a Cincinnati company called Intelliseek that was founded in 1997. One of its most successful early businesses was scraping message boards to find mentions of brand names for corporate clients.
In 2001, the venture-capital arm of the Central Intelligence Agency, In-Q-Tel Incorporated, was among a group of investors that put $8 million into the business.
Intelliseek struggled to set boundaries in the new business of monitoring individual conversations online, says Sundar Kadayam, Intelliseek's co-founder. The firm decided it wouldn't be ethical to use automated software to log into private message boards to scrape them.
But, he says, Intelliseek occasionally would ask employees to do that kind of scraping if clients requested it. "The human being can just sign in as who they are", he says. "They don't have to be deceitful".
In 2006, Nielsen bought Intelliseek, which had revenue of more than $10 million and had just become profitable, Mr Kadayam says. He left one year after the acquisition.
At the time, Nielsen, which provides television ratings and other media services, was looking to diversify into digital businesses. Nielsen combined Intelliseek with a New York startup it had bought called BuzzMetrics.
The new unit, Nielsen BuzzMetrics, quickly became a leader in the field of social-media monitoring. It collects data from 130 million blogs, 8,000 message boards, Twitter and social networks. It sells services such as "ThreatTracker", which alerts a company if its brand is being discussed in a negative light. Clients include more than a dozen of the biggest pharmaceutical companies, according to the company's marketing material.
Like many websites, PatientsLikeMe has software that detects unusual activity. On May 7, that software sounded an alarm about the "Mood" forum.
David Williams, the chief marketing officer, quickly determined that the "member" who had triggered the alert actually was an automated program scraping the forum. He shut down the account.
The next morning, the holder of that account e-mailed customer support to ask why the login and password weren't working. By the afternoon, PatientsLikeMe had located three other suspect accounts and shut them down. The site's investigators traced all of the accounts to Nielsen BuzzMetrics.
On May 18, PatientsLikeMe sent a cease-and-desist letter to Nielsen. Ten days later, Nielsen sent a letter agreeing to stop scraping. Nielsen says it was unable to remove the scraped data from its database, but a company spokesman later said Nielsen had found a way to quarantine the PatientsLikeMe data to prevent it from being included in its reports for clients.
PatientsLikeMe's president, Ben Heywood, disclosed the break-in to the site's 70,000 members in a blog post. He also reminded users that PatientsLikeMe also sells its data in an anonymous form, without attaching user's names to it. That sparked a lively debate on the site about the propriety of selling sensitive information. The company says most of the 350 responses to the blog post were supportive. But it says a total of 218 members quit.
In total, PatientsLikeMe estimates that the scraper obtained about five percent of the messages in the site's forums, primarily in "Mood" and "Multiple Sclerosis".
"We're a business, and the reality is that someone came in and stole from us", says PatientsLikeMe's chairman, Jamie Heywood.
http://online.wsj.com/article/SB10001424052748703358504575544381288117888.html?mod=WSJ_article_MoreIn_Health
Bill Totten http://www.ashisuto.co.jp/english/
posted by Bill Totten at
5:25 PM
2 Comments:
I assume this mining of all information in order to misuse it is the way to silence everyone, which is an imposssibiity.
It will contribute the the element of world disgust for War empire and the United States of America. Many young here in Holland no longer even think of taking a vaction in America any more.
Wth the 20 year olds joining us geezers, and in the European Union we are 800 million, that is a bunch of literate, talkative souls, indeed. There must be millions of Japanese who agree with us?
Since American copies Israel doing all planned to do, eight hundred souls and their descendants have just been written off the rolls of the viable or possible. End of empire? Think it only the beginning. Suzanne
By
suzannedk, at 7:47 PM, November 02, 2010
Interesting post. I agree the WSJ story raises real concerns.
My view: http://ianbruce.blogspot.com/2010/11/secrets-and-lies.html
By
Unknown, at 7:26 AM, November 03, 2010
Post a Comment
<< Home