In Spam Nation, Brian Krebs covers a key portion of our cyber security and cyber crime history: 2007–2013, that period when we started to learn about the Russian Business Network, bulletproof-hosting providers, fast-flux obfuscation, criminal best business practices, underground cyber crime forums, and strange-sounding botnet names like Conficker, Rustock, Storm, and Waledac. This period just happens to coincide with Krebs’s rise in popularity as one of the leading cyber security journalists in the industry. His relationship with two competitive pharmaceutical spammers—Pavel Vrublevsky and Dimitry Nechvolod—is a big bag of crazy and is the key storyline throughout the book. The competition between Vrublevsky and Nechvolod escalated into something that Krebs calls the Pharma Wars and Krebs gives us a bird’s-eye view into the details of that escalation that eventually destroyed both men and the industry they helped to create. Krebs’s weird symbiotic relationship with Vrublevsky is worth the read by itself. Spam Nation is definitely a cyber security canon candidate, and you should have read this by now.
I have been a fan of Brian Krebs for many years. His blog, Krebs on Security, has been a mainstay of my recurring reading list since he started it in 2010 and even before when he was writing for The Washington Post. Since he struck out on his own, he has carved out a new kind of journalism that many reporters are watching to see how they might duplicate it themselves as journalism transitions from dead-tree printing to new media. Krebs’s beat is cyber security, and he is the leading journalistic authority on the underbelly of cyber crime. Spam Nation is a retelling— with more detail and more color—of some of the stories he covered from 2007 until about 2013 on a very specific sub-element of the cyber crime industry called pharmaceutical spam.
Many security practitioners will hear the phrase “pharmaceutical spam” and immediately start to nod off. Of all the problems they encounter on a daily basis, pharmaceutical spam is pretty low on the priority list. While that may be true, this subset of cyber crime is responsible for starting and maturing many of the trappings that we associate with cyber crime in general: botnet engines, fast-flux obfuscation, spamming, underground forums, cyber crime markets, good service as a distinguisher of criminal support services, and bulletproof-hosting providers.
The story really begins with Krebs’s weird symbiotic relationship with Vrublevsky (a.k.a. RedEye and Despduck). Vrublevsky was a Russian businessman and cofounder and former CEO of ChronoPay, the infamous credit card processing company that initially got started in the rogue anti-virus industry. I think it is safe to say that in his heyday, Vrublevsky was a bit of an extrovert. He followed Krebs’s blog religiously and would instigate long conversations with Krebs on stories that were fantastical, true, and everything in between. Vrublevsky would feed Krebs half-truths about what was going on in the industry and left it to Krebs to sort it out. Vrublevsky’s downfall was his deteriorating relationship with his former partner, Dimitry Nechvolod (a.k.a. Gugle).
Vrublevsky and Nechvolod founded ChronoPay together in 2003, but by 2006, Nechvolod had left the company to pursue his own interests. He started two pharmacy spam operations called GlavMed and SpamIT. Because of the competition between these two men, the situation escalated out of control to something that Krebs calls the Pharma Wars, which ultimately scuttled the entire pharmaceutical spam industry, not just Vrublevsky and Nechvolod’s operations, but everybody else’s too.
Krebs’s main sources of information for this book came from leaked customer and operational databases from these two men. Although Vrublevsky and Nechvolod never admitted it, they both stole the other’s data and leaked it to Krebs. Krebs had many conversations with both Vrublevsky and Nechvolod about their side of the story, and Krebs even traveled to Moscow to interview Vrublevsky personally. From these conversations and other research done by Krebs, we get an inside view of how cyber crime operates in the real world.
Krebs set himself seven research questions:
- Who is buying the stuff advertised in spam and why?
- Are the drugs real or fake?
- Who profits?
- Why does the legitimate pharmaceutical industry seem powerless to stop it?
- Why is it easy to pay for the drugs with credit cards?
- Do customers have their credit card accounts hacked after buying?
- What can consumers, policy makers, and law enforcement do [about this cybercrime]?
Spam Nation is about the Brian Krebs’s story too. Traditional journalists reading this book are going to hate the fact the he plays a key role in most everything that he talks about in this book. His original reporting on bulletproof-hosting providers operating in the US and elsewhere—the Russian Business Network (RBN), Atrivo, and McColo—became that catalyst that eventually got them shut down. This got him noticed by Vrublevsky and started that weird relationship that ultimately led to Krebs receiving the databases from Vrublevsky and Nechvolod. It also led him to leave The Washington Post and to start his Krebs on Security blog.
In the background, Krebs introduces us to the key players involved in the development and operations of some of the most infamous botnets that have hit the Internet community in recent history:
- Conficker worm (author: Severa; infected 9-15 million computers)
- Cutwail botnet (authors: Dimitry Nechvolod (Gugle) and Igor Vishnevsky; 125,000 infected computers; spewed 16 billion spam messages a day)
- Grum botnet (author: GeRA; spewed 18 billion e-mails a day)
- Festi botnet (operators: Artimovich brothers; delivered one-third of the total amount of worldwide spam)
- Rustock botnet (author: COSMA; infected 150,000 PCs; spewed 30 billion spam messages a day)
- Storm botnet (author: Severa).
- Waledac botnet (author: Severa; spewed 1.5 billion junk e-mails a day)
- June 2009: 15,000 illicit websites go dark at 3FN after the Federal Trade Commission convinced a northern California judge that 3FN was a black-hat service provider. NASA did the forensics work.
- November 2009: FireEye takes down the Mega-D botnet.
- January 2010: Neustar takes control of the Lethic spam botnet.
- March 2010: Microsoft takes down the Waledac botnet.
- October 2010: Armenian authorities take down the Bredolab botnet.
- March 2011: Microsoft takes down the Rustock botnet.
- July 2011: Microsoft offers a $250,000 reward for information leading to the arrest and conviction of the Rustock botmaster.
- July 2012: FireEye and Spamhaus take down the Grum botnet.
- July 2013: Microsoft and the FBI take down 1,400 botnets using the Citadel malware to control infected PCs.
- December 2013: Microsoft and the FBI take down the ZeroAccess botnet.
- June 2014: The FBI takes down of the Gameover Zeus botnet.
Cyber Crime Business Operations
For me, one of the most enjoyable parts of Spam Nation is the insight on how these criminal organizations operate. For example, Krebs highlights why pharmaceutical operations have great customer support: they want to avoid the penalty fees associated with a transaction when a buyer of illicit pills charges them with fraud. These are called chargebacks, and pharmaceutical customer support operations avoid them like the plague. These support operations require teams of software developers and technical support staff to be available 24/7.
Pharmaceutical operations have mature anti-fraud measures—equivalent to any legitimate bank’s anti-fraud measures—because they need to keep law enforcement and security researchers out of their business.
Most spammers do not make a lot of money. The top five do, but not everybody else. Krebs points out that it takes a multibillion dollar security industry to defend against a collection of criminals who are making a living wage.
In terms of botnet management, operators rent out top-earning botnets to other operators who do not have the skill to build a botnet themselves. Renters purchase installs and seed a prearranged number of bots with an additional malicious program that sends spam for the affiliate. They pay the rent by diverting a portion of their commissions on each pill sale from spam. Sometimes, that commission is as high as 50 percent. That is why the small-timers do not make any money.
Operators launder their money in a process called factoring. They map their client transactions into accounts on behalf of previously established shell companies. They tell the banks that the shell companies are the true customers. Then the operators pay the clients out of their own pockets.
Russian law allows FSB agents (Federal Security Service, the successor to the Soviet Union’s KGB), while remaining in the service, to be assigned to work at enterprises and organizations at the consent of their directors. Twenty percent of FSB officers are engaged in this protection business called “Krusha" in Russian, which means “roof” and pharmaceutical spam operations use them as much as possible.
Partnerships, called partnerkas, between spammers and dodgy advertisers that act as an intermediary for potential sponsors are essential. In this way, sponsors keep their distance from the illicit aspects of the spam business and can unplug from one partnerka in favor of another whenever they want. Some refer to this as organized crime (think The Godfather), but it is more like a loosely affiliated network of independent operators.
With all of these best business practices, you can see why the operators do not see themselves as criminals. They are just businesspeople trying to run a business.
Cyber crime runs on technology. In the pharmaceutical spam business, some tech is unique, and other tech is shared with other kinds of cyber crime operations. Unique to pharmaceutical spam is a technique called black search engine optimization (Black SEO). Pharmaceutical spammers hack legitimate websites and insert hidden pages (IFrames) with loads of pharmaceutical websites links. The more links that the common search engines like Google and Bing index, the higher the pharmaceutical sites get in the priority list when normal users search for pills online.
Also unique to the pharmaceutical spam business is a good spam ecosystem. It must have the ability to keep track of how many e-mails the system delivered and how many recipients clicked the link. It must scrub e-mail addresses that are no longer active or are obvious decoys and harvest new e-mail addresses for future operations.
Not unique to pharmaceutical spam are the forums. Forums are the glue that allows the loosely affiliated network of independent operators to communicate with each other. Forums are a place that allows newbies an opportunity to establish a reputation and lowers the barriers to entry for a life of cyber crime. There are forums for every language, but most are in English. Members enforce a strict code of ethics so that members who are caught cheating other members are quickly banned. Social networking rankings give members a way to evaluate potential partners. A single negative post may cost an individual thousands of dollars. Because of that, most amicably resolve issues. Sometimes newbies get labeled as a “deer,” members who unintentionally break one of the forum’s rules. More-serious infractions might find a member in the blacklist subforum defending himself or herself from fraud allegations.
New forums start all the time, but some have been in existence for more than a decade, indicating process maturity for self-policing, networking, and rapid information sharing. New forums allow open registration, but mature forums set up various hurdles for membership that are designed to screen out law enforcement and hangers-on. Most have sub-rooms for specialization such as the following:
- Cyber banking fraud
- Bank account cash-out schemes
- Malicious software development
- ID theft
- Credit card fraud
- Confidence scams
- Black SEO
In Spam Nation, Brian Krebs covers a key portion of our cyber security and cyber crime history: 2007– 2013, that period when we started to learn about the Russian Business Network, bulletproof-hosting providers, fast-flux obfuscation, criminal best business practices, underground cyber crime forums, and strange-sounding botnet names like Conficker, Cutwail, Grum, Festi, Rustock, Storm, and Waledac. This period just happens to coincide with Krebs’s rise in popularity as one of the leading cyber security journalists in the industry. His story, and the story of two competitive pharmaceutical spammers who eventually destroyed the lucrative moneymaking scheme for all players, is a fascinating read. It is definitely a cyber security canon candidate, and you should have read this by now.
“Spam Nation: The Inside Story of Organized Cybercrime - from Global Epidemic to Your Front Door,” by Brian Krebs, published by Brilliance Audio, 18 November 2014, last visited 13 November 2014,
“Blue Security folds under spammer's wrath,” by Robert Lemos, Security Focus, 17 May 2006, last visited 13 November 2014,
“Click Trajectories: End-to-End Analysis of the Spam Value Chain,” by Kirill Levchenko, Andreas Pitsillidis, Neha Chachra, Brandon Enright, Mark Felegyhazi, Chris Grier, Tristan Halvorson, Chris Kanich, Christian Kreibich, He Liu, Damon McCoy, Nicholas Weaver, Vern Paxson, Geoffrey M. Voelker, and Stefan Savage, last visited 13 November 2014,
“Experts Warn of New Windows Shortcut Flaw,” by Brian Krebs, Krebs on Security, 10 July 2010, last visited 13 November 2014
“Krebs on Security: In-depth security news and investigation,” by Brian Krebs, last visited 14 November 2014,
“PharmaLeaks: Understanding the Business of Online Pharmaceutical Affiliate Programs,” by Damon McCoy, Andreas Pitsillidis, Grant Jordan, Nicholas Weaver, Christian Kreibich, Brian Krebs, Geoffrey M. Voelker, Stefan Savage, and Kirill Levchenko, Usenix, August 2012, last visited 13 November 2014,
“Russian Business Network Study,” by David Bizeul, 11 November 2007, last visited 12 November 2014,
“Shadowy Russian Firm Seen as Conduit for Cybercrime,” by Brian Krebs, The Washington Post, 13 October 2007, last visited 12 November 2014,
“The Partnerka – What Is It, and Why Should You Care?” by Dmitry Samosseiko, Sophos, Virus Bulletin, September 2009, last visited 13 November 2014,
“The Sleazy Life and Nasty Death of Russia’s Spam King,” by Brett Forrest, Wired Magazine, August 2006, last visited 13 November 2014,
“The Underground Economy of Spam: A Botmaster’s Perspective of Coordinating Large-Scale Spam Campaigns,” by Brett Stone-Gross, Thorsten Holz, Gianluca Stringhini, and Giovanni Vigna, last visited 13 November 2014,
“Top Spam Botnets Exposed,” by Joe Stewart, SecureWorks, 8 April 2008, last visited 13 November 2014,