Saturday, February 7, 2015

Book Review: Countdown to Zero Day: Stuxnet and the Launch of the World's First Digital Weapon (2014) by Kim Zetter

Executive Summary

Operation Olympic Games is the US military code name that refers to the first ever act of real cyber warfare. Many journalists have told bits and pieces of the story since the attacks became public in 2010, but none have come close to telling the complete story. In Countdown to Zero Day: Stuxnet and the Launch of the World's First Digital Weapon, Kim Zetter changes that situation. She takes an extremely complicated subject in terms of technical detail, political fallout, and philosophical conundrums and makes it easy for the security practitioner to understand. It is a masterful bit of juggling and story telling. It is cyber-security-canon worthy, and you should have read it by now.


Kim Zetter has been at WIRED magazine since 2003 and has become one of the cyber security community’s go-to journalists to explain what is really happening within the space. When I heard that she was writing a book about the Stuxnet attacks, I was thrilled. I knew if anybody could take on this complicated subject, Zetter could. One of the annoying truisms of keeping up with cyber security events in the news is that journalists rarely go back and attempt to tell a complete story. When cyber security events occur —like the Target breach, the Sony breach, and the Home Depot breach to name three — news organization print the big headlines initially and then trickle out new information over the next days and weeks as it becomes available. For cyber security professionals trying to keep up to date on industry news, we rarely get the opportunity to see the big picture in one lump sum. We are not going to get that kind of story in a news article. You need a book to cover the detail, and there have been some good ones in the past. Mark Bowden’s Worm — about the Conficker worm and the cabal that tried to stop it — is one good example.[1] Another is The Cuckoo’s Egg, which is about the first publicly documented cyber espionage attack in the late 1980s.[2] Zetter’s book Countdown to Zero Day: Stuxnet and the Launch of the World's First Digital Weapon is the latest in the line, and it is really good.[3]

The Story

Operation Olympic Games is the US military code name that refers to the first ever act of real cyber warfare. Many journalists have told bits and pieces of the story since the attacks became public in 2010, but none have come close to telling the complete story. In June 2012, David E. Sanger published an article in The New York Times proclaiming for the first time that the United States, in conjunction with Israel, was indeed behind the infamous Stuxnet malware attacks that targeted the Iranian nuclear enrichment plant at Natanz.[4] Sanger followed that article, along with others, with his book Confront and Conceal: Obama’s Secret Wars and the Surprising Use of American Power.[5] In his articles and this book, he gave details about the cyber operation called Operation Olympic Games, which I consider to be the first act of cyber warfare in the world. Because the story was so new and so complicated, many of the technical details surrounding the attacks did not fully emerge until well after Sanger published his book. I have tried to keep up with the story myself over the years and even presented versions of it at DEF CON[6] and RSA,[7] but I do not have the journalistic chops to tell the complete story, and this is where Zetter’s book shines. Whereas Sanger’s book focused on the US foreign policy implications of offensive cyber warfare using government insiders as the main source, Zetter’s book fills in the technical story behind the attacks by interviewing everybody in the public space who was involved in unraveling the Stuxnet mystery. Zetter writes clearly and succinctly about the timing of key researchers discovering new facts, describes how the researchers determined when the attackers first used key pieces of the attack code, and then feathered those technical events with what was happening in the political arena at the same time. It is a masterful bit of juggling and storytelling.

The Code

Because of Countdown to Zero, we now have a complete picture of how the attack code worked. Zetter goes into great detail about how the malware proliferated within the Iranian power plant at Natanz and after it escaped into the wild. She puts to bed the question of how may zero-day exploits the attackers used in the complete code set, what they were, and how effective they all were. She covers all of the versions of the malware from Stuxnet, to DuQu, to Flame, and to Wiper. She even covers some of the tools of the trade that the researchers used to decipher the code base.


In Countdown to Zero, Zetter explains the significance of the critical and mostly unsecured supervisory control and data acquisition (SCADA) environments deployed in the United States today. These systems automatically control the flow of all power, water, and gas systems used within the United States and throughout most of the world. According to Zetter, 

“There are 2,800 power plants in the United States and 300,000 sites producing oil and natural gas. Another 170,000Bottom of Form facilities form the public water system in the United States, which includes reservoirs, dams, wells, treatment facilities, pumping stations, and pipelines. But 85 percent of these and other critical infrastructure facilities are in the hands of the private sector, which means that aside from a few government-regulated industries—such as the nuclear power industry—the government can do little to force companies to secure their systems.”[3]

In my experience, the SCADA industry has always been at least 10 to 15 years behind the rest of the commercial sector in adopting modern defensive techniques, and Zetter provides a possible explanation for this delay:

“Why spend money on security, they argued, when none of their competitors were doing it and no one was attacking them?”[3]

The significance of that statement becomes obvious when you realize that the same kinds of programmable logic controllers, or PLCs, that the United States exploited to attack Iran are deployed in droves to support the world’s own SCADA environments. The point is that if the United States can leverage the security weaknesses of these systems, then it is only a matter of time before other nation-states do the same thing and the rest of the world is no better defended against them than the Iranians were.

The Philosophical Conundrum

In a broader context, Countdown to Zero highlights some philosophical conundrums that the cyber security community is only now starting to wrestle with. We have known about these issues for years, but Zetter’s telling of the story makes us reconsider them. Operation Olympic Games proved to the world that cyber warfare is no longer just a theoretical construct. It is a living and breathing option in the utility belt for nation-states to use to exercise political power. With Operation Olympic Games, the United States proved to the world that it is possible to cause physical destruction of another nation-state’s critical infrastructure using nothing but a cyber weapon alone. With that comes a lot of baggage. 

The first is the intelligence dilemma. At what point do network defenders stop watching adversaries misbehave within their networks before they act to stop them? By acting, we tip our hand that we know what they are doing and how they are doing it. This will most likely cause the adversary team to change its tactics. Intelligence organizations want to watch adversaries as long as possible. Network defenders only want to stop the pain. This is an example of classic information theory. I first learned about information theory when I read about the code breakers at Bletchley Park during WWII. Because the allies had broken the Enigma cipher, the Bletchley Park code breakers collected German war plans before the German commanders in the field received them, but the Allies couldn’t act on all of the information because the Germans would suspect that the cipher had been broken. The Allies had to pick and choose what to act on. This is similar to what the Stuxnet researchers were wrestling with too. Many of them had discovered this amazing and dangerous new piece of malware. When do they tell the world about it?

The next conundrum involves the national government and vulnerability discovery. Zetter discusses the six zero-day exploits used by Operation Olympic Games in the attacks against Iran. That means that the US government knew about at least six high-impact vulnerabilities within common software that the entire nation depends upon and did nothing to warn the nation about them. If another attacker decided to leverage those vulnerabilities against the United States’ critical infrastructure in the same way that the United States leveraged them against Iran, the results could have been devastating. The nation’s ethical position here is murky at best and criminal at worst. Added to that is the well-known practice of the private sector selling zero-day exploits to the government. Should the government even be in the business of buying weapons-grade software from private parties? Zetter offers no solutions here, but she definitely gives us something to think about.


Zetter fills in a lot of holes in the Stuxnet story. In a way, it is a shame that it has taken five years to get to a point that the security community feels like it understands what actually happened. On the other hand, without Zetter putting the pieces together for us, we might never have gotten there. I have said for years that the Stuxnet story marked the beginning of a new era for the cyber security community. In the coming years, when it becomes common practice for nations-states to lob cyber attacks across borders with the intent to destroy another nation’s critical infrastructure, we will remember fondly how simple defending the Internet was before Stuxnet. Zetter’s book helps us understand that change. She takes a complicated subject and makes it easy to understand. Her book Countdown to Zero Day: Stuxnet and the Launch of the World's First Digital Weapon is cyber-security-canon worthy, and you should have read it by now.


[1] “The Cybersecurity Canon: Worm,” by Rick Howard, Unit 42, 4 February 2014, last visited 25 January 2015,

[2] “The Cybersecurity Canon: The Cuckoo’s Egg,” by Rick Howard, Unit 42, 24 December 2013, last visited 25 January 2015,

[3] “Countdown to Zero Day: Stuxnet and the Launch of the World's First Digital Weapon,” by Kim Zetter, Published by Crown, 11 November 2014, last visited 25 January 2015,

[4] “Obama Order Sped Up Wave of Cyberattacks Against Iran,” by David E. Sanger, The New York Times, 1 June 2012, last visited 25 January 2015,

[5] “The Cybersecurity Canon: Confront and Conceal,” by Rick Howard, Unit 42, 7 January 2014, last visited 25 January 2015,

[6] “Defcon-19-an-insiders-look-at-international-cyber-security-threats-and-trends,” by Rick Howard, DEF CON 19, 6 August 2011, last visited 25 January 2015,

[7] “Operation Olympic Games Is the Tom Clancy Spy Story that Changed Everything,” by Richard Howard, RSA Conference 2014, 28 February 2014, last visited 25 January 2015,

Tuesday, January 13, 2015

Book Review: Winning as a CISO (2005) by Rich Baich

Executive Summary

The latest candidate for the cyber security canon is Rich Baich’s Winning as a CISO. The roles of the chief information officer (CIO), the chief security officer (CSO), and the chief information security officer (CISO) in the modern enterprise have been constantly changing since we invented the need for such roles in the 1980s and 1990s. By the mid-2000s, the industry had settled on tucking the security function for an organization under the IT function of an organization. In other words, the CISO works for the CIO. But Baich is an innovative thinker. He has looked at how the CISO role has evolved over the years and makes a pretty good case for where it needs to go next. By asking questions about the appropriate supervisor for a CISO, a CISO’s needed skill set, and ways to approach the CISO job function, Baich breaks new ground on how the industry should views these topics. Our industry will be slow to adopt these new ideas, but with the rash of highly publicized and impactful data breaches to the retail sector in 2014, perhaps the industry is ready to start making a change. Reviewing Baich’s book is a good place to start. It is cyber-security-canon worthy, and you should have read it by now.


The roles of the CIO, the CSO, and the CISO in the modern enterprise have been constantly changing since we invented the need for such roles in the 1980s and 1990s. I picked up Winning as a CISO because my boss handed it to me after he met the author, Rich Baich, at a security event. He said that Baich was a smart guy and had some interesting ideas about the modern CISO’s role in today’s environments. In this book, Baich explains some innovative thinking about what today’s CISOs should be responsible for, how they should fit into the organization, and how they might accomplish their tasks once they are established. In order to understand where Baich is coming from, it is useful to review the history of the CIO, CSO, and CISO roles in modern business.

CIO, CSO, and CISO History

The idea of the C-suite did not really materialize until the 1920s when Alfred Sloan, the hugely successful chief executive officer (CEO) of General Motors, decided to distribute profit and loss (P&L) responsibility across his division managers in response to shareholder and regulator demand for more accountability.[1] 

Because of General Motors’ success with this new P&L model, business leaders across the world adopted it for their own organizations. That model lasted some 60 years until the 1980s when CEOs realized that in order to drive organizational change, they needed executives with technical and functional specialties.[1] CEOs began creating new C-level executive positions like chief marketing officers (CMOs), chief financial officers (CFOs), and, yes, CIOs. The idea of a C-level executive dedicated to security did not really emerge until the late 1990s, 10 years after the CIO position had become firmly established in modern business.

Steve Katz became the first CISO in 1995 when Citigroup created the role to respond to a highly publicized Russian malware incident.[2][3] Since then, the security industry specifically and business leadership in general have been thinking and rethinking the need and the responsibilities for such a person. 

The first practitioners came out of the technical ranks. Vendor solutions to mitigate the cyber threat ran on networks and workstations. In order to manage those solutions, it was helpful to have people who understood that world, but this was a new thing for the techies; trying to translate technical risk to a business leader did not always go very well. Security techies have always been, and still are, passionate about their responsibilities. The early trailblazers tended to say “no” to any new project because of the potential security risk. The business leaders did not want to deal with these people who wanted to make organizational decisions with no thought about the bottom line. It became convenient to tuck these kinds of people underneath the CIO organization. CISOs began working for the CIO because, from the C-suite perspective, all of that technical stuff belonged in one basket, and the security people did not know how to talk to the business people.

As business leaders began applying resources to mitigate cyber risk, other areas of security risk started to emerge: physical security, compliance, fraud prevention, business continuity, safety, ethics, privacy, brand protection, etc. The idea of the CSO role began to gain popularity with business leaders because they needed someone to look at the entire business, not just cyber security risk to the business, but general security risk to the business. CSO Magazine launched in 2002 to cater to that crowd.[4]. 

By the mid-2000s, the industry had settled on tucking the security function for an organization under the IT function for an organization. In other words, the CISO works for the CIO. This is not bad per se, and this arrangement works in many organizations. The IT folks generally handle the daily automation functions while the security teams have more of an oversight role in terms of security architecture, policy, risk assessment, and security operations. 

Since then, the industry has been in flux. Not every company is organized the same way. While the CIO role has made its way to the senior executive suite in some companies (Intel Corp. and McAfee to name two), that is by no means the norm. The CSO role is likewise lagging. Both tend to be lodged at the second tier of executives in many companies. And while it is not universal, the CISO tends to work for the CIO.

The Story

All of this history is essential background to the key messages in Baich’s book Winning as a CISO. He published it in 2005 and was quite rightly taking a look at where the CISO role was heading next. He organized the book as a fictional story about an established company in which the CEO had decided to hire his first CISO. His executive leadership team – the CIO, the general counsel, and the chief operating officer (COO) – had to decide what the new CISO’s responsibilities were and where this individual would fit in the organizational structure. Once the CEO made those decisions, the newly hired CISO had to decide how to execute this new role.

The Tech

The book is a quick read, with only 115 pages including the end credits, but it is a primer on what a CISO should do for any organization. In essence, any organization could use Baich’s book as a basic job description for a new CISO hire.

What Are a CISO’s Responsibilities?

When the story’s CEO brought his executive staff together to discuss the new position, he had them develop a list of responsibilities for the new hire. Here is the list:

  • Security Architecture
  • Incident Response
  • Security Awareness
  • Identity Management
  • Security Policy Development and Compliance
  • Due Diligence for Acquisitions and Mergers
  • Risk Management[5]
I think this is a pretty good list of high-level responsibilities. Anything that comes up later that we might want the CISO to do can be easily shoehorned into one of these broad categories. Once the staff agreed to the responsibilities, the next step was to determine which senior executive should own them. In other words, which senior executive should the CISO work for? 

To Whom Does the CISO Report?

All of the senior staff members had their perspectives. The CIO said, “The CISO should report to the IT Department because the focus of information security is related to technology. Information security solves technology related risks.”[5] The general counsel said, “The CISO should report through the legal structure. [The] focus can be placed on compliance.”[5] The COO said, “The CISO will have to collaborate with all departments, and everyone, including the sales team will benefit, but the team member who will need to utilize the resulting information the most will be the COO. A clear understanding of the operational risk factors will enable the successful CISO to present to the COO with a rubric of important options.”[5]

The CEO weighed each of these perspectives and had a few of his own. He said that he did not want the new CISO to have to wrestle with any artificial organizational conflicts because he chose to put the position under one senior executive as opposed to another.[5] He said that putting the CISO under the CIO had a number of problems, but the most important one was that it created a conflict of interest. “Reporting to the CIO would be like putting your boss on report.”[5] The CISO’s job is to make things more secure, and sometimes that job may be in direct conflict with the CIO’s job of making things more efficient. With the CISO under the CIO, the organization automatically weights efficiency needs over security needs, and that obviates the reason to hire the CISO in the first place.[5]

An opposing view comes from Forbes reporter Howard Baldwin. He complained in March 2014 that he did not like recent changes he was seeing within organizations that had broken out the security function to be a peer to the CIO. He says that these CIOs are highly paid executives who can handle competing priorities.[6] In other words, the CIO can handle making decisions between security and efficiency. That is what we pay a person in this position to do.

But that is not the point. In an interview by Jack Rosenberger, Eric Cole -- founder and chief scientist at Secure Anchor Consulting -- speculated on one of the reasons that may have contributed to the Target breach in 2014.[7] Cole said, “It is almost a guarantee that Target had an amazing security team, and they were screaming and yelling about all of the security issues, but there was no advocate who was listening to them and fighting for their cause with the executives.”[8] Cole is pointing out that of all the priorities the Target CIO had to juggle, security lost out. As Brian Krebs reported in the Guardian, “Virtually all aspects of retail operations are connected to the Internet these days: when the security breaks down, the technology breaks down – and if the technology breaks down, the business grinds to a halt.”[9] Before the breach, the pressure to keep the IT infrastructure up and running must have been immense for both the now-resigned CIO and the now-fired CEO. Krebs suggests that in hindsight, because of the breach’s devastating impact to the business, the Target CISO should not have worked for the CIO. It should have been the other way around.[9]

In Baich’s story, the CEO had reservations about putting the CISO under other staff organizations too. He said that putting the CISO under the general counsel “would potentially position the Information Security department as an arm of the audit department.”[5] According to Baich, auditing support is something the new CISO should help with, but based on the responsibilities the executive staff developed, the CISO’s role is much bigger.[5]

The CEO ultimately put the CISO under the COO. To him, it made sense that the CISO position be perfectly positioned to support the entire organization and not one specific staff element. I think this makes sense. If loss associated with security is something that will potentially materially affect the business, it makes total sense to raise the platform of the person in charge of it to have a view of the entire organization and the power to affect change. If that is the case, then what skill sets are needed for the person who takes on that responsibility?

What Skill Sets Does a CISO Need?

Once he decided whom the CISO should work for, the CEO turned again to his senior staff to determine what skill sets would be essential for success. Without fanfare, Baich lists these five attributes:

  • Must have an MBA
  • Prior budget or P&L experience
  • A proven ability to lead an effective information security organization
  • Experience and skill as a change agent
  • Ability to serve as an information security expert for the executive team[5]

The last three skills are fairly standard for many senior job positions in any organization. The first two are where Baich is providing some innovative thinking. Requiring an MBA and P&L experience for a CISO, as a mandatory requirement, is not the common thinking in the industry, but it is spot on for where the industry needs to go. As I said earlier, most CISOs have come up through the technical ranks and have little if any business experience. This is probably the main reason that security teams and business teams have a hard time communicating with each other. By requiring a CISO to have business experience first, Baich flips the typical experience equation on its head. Instead of training highly technical employees to be proficient in business concerns at the mid- to latter parts of their careers, he is suggesting that we take traditional business people and train them to be proficient in managing security operations. 

“If performing vulnerability assessments, configuring firewalls, and performing network forensics makes you happy then becoming Chief Information Security Officer may not be the right career choice for you.”[5]

Just like a traditional business person might find himself or herself as a general manager, product manager, finance officer, or marketing officer, Baich is suggesting we add security officer to the list, and I agree with him.

How Do You Be a CISO?

In Baich’s story, the CEO placed the CISO under the COO in order to give the position a matrixed view of the business. In that kind of environment, how does a CISO succeed? In spite of all the listed responsibilities this CISO has for the organization, Baich says that the most important implied responsibility for the CISO is running his or her organization like a business. The CISO needs to become the general manager of the security program.

“Ultimately, the success of any business, new or old, depends on a leader’s ability to build a team, market and sell the product, and run the business, still meeting the established measurements necessary to effectively operate the business.”[5]

Although the CISO in this story will bring in no revenue, this individual has to demonstrate to the business leadership the value of the position in other ways. The CISO must become a world-class internal marketing person for every aspect of the security program. It is not enough to make the organization more secure. The CISO’s efforts to do so must demonstrably show how the security program is helping the organization grow. 


Baich is an innovative thinker. He has looked at how the CISO role has evolved over the years and makes a pretty good case for where it needs to go next. By asking questions about the appropriate supervisor for a CISO, a CISO’s needed skill set, and ways to approach the CISO job function, Baich breaks new ground on how to think about these topics. Baich published the book in 2005. Back then, there was not a lot of impetus to change the current situation, and I do not see the industry adopting these ideas any time soon. But with the rash of highly publicized and impactful data breaches to the retail sector in 2014, perhaps the industry is ready to make a change. It is obvious that the way we are doing it now is not working. Because of Baich’s innovative thinking about the next step in the evolution of the CISO role, Winning as a CISO is cyber-security-canon worthy, and you should have read it by now.


[1] “The C-suite: Time for version 3.0?” by Eamonn Kelly, Deloitte University Press, 31 March 31 2014, last visited 6 January 2014,

[2] “EVOLUTION OF THE CISO And the Confluence of IT Security & Audit,” by Thomas Borton, ISACA, 13 March 2014, last visited 30 May 2014,

[3] “Prominent Information Security Executives Steve Katz and Michael Barrett Join Fortscale's Advisory Board,” by Fortscale, PR Newswire: Market Watch, 8 January 2014, last visited 30 May 2014,

[4] “Decade of the CSO: Looking back at 10 years of change and progress - and forward to what lies ahead,” by Derek Slater, CSO Magazine, 1 October 2012, last visited 30 May 2014,

[5] “Winning as a CISO,” by Rich Baich, Published by Executive Alliance Publishing House, 2005, last visited 6 December 2014,

[6] “Point/Counterpoint: Who Should The CSO Report To?” by Howard Baldwin, Forbes, 25 March 2014, last visited 28 May 2014, 

[7] “Reporting From the Web’s Underbelly,” by Nicole Perlroth, The New York Times, 16 February 2014, last visited 28 May 2014,

[8] “The Complicated Relationship Between CIOs and CSOs,” by Jack Rosenberger, CIO: Insight, 11 March 2014, last visited 28 May 2014,

[9] “What Target and Co aren't telling you: your credit card data is still out there,” by Brian Krebs, The Guardian, 6 May 2014, last visited 28 May 2014,

Monday, November 24, 2014

Book Review: Spam Nation: The Inside Story of Organized Cybercrime - from Global Epidemic to Your Front Door (2014) by Brian Krebs

Executive Summary

In Spam Nation, Brian Krebs covers a key portion of our cyber security and cyber crime history: 2007–2013, that period when we started to learn about the Russian Business Network, bulletproof-hosting providers, fast-flux obfuscation, criminal best business practices, underground cyber crime forums, and strange-sounding botnet names like Conficker, Rustock, Storm, and Waledac. This period just happens to coincide with Krebs’s rise in popularity as one of the leading cyber security journalists in the industry. His relationship with two competitive pharmaceutical spammers—Pavel Vrublevsky and Dimitry Nechvolod—is a big bag of crazy and is the key storyline throughout the book. The competition between Vrublevsky and Nechvolod escalated into something that Krebs calls the Pharma Wars and Krebs gives us a bird’s-eye view into the details of that escalation that eventually destroyed both men and the industry they helped to create. Krebs’s weird symbiotic relationship with Vrublevsky is worth the read by itself. Spam Nation is definitely a cyber security canon candidate, and you should have read this by now.


I have been a fan of Brian Krebs for many years. His blog, Krebs on Security, has been a mainstay of my recurring reading list since he started it in 2010 and even before when he was writing for The Washington Post. Since he struck out on his own, he has carved out a new kind of journalism that many reporters are watching to see how they might duplicate it themselves as journalism transitions from dead-tree printing to new media. Krebs’s beat is cyber security, and he is the leading journalistic authority on the underbelly of cyber crime. Spam Nation is a retelling— with more detail and more color—of some of the stories he covered from 2007 until about 2013 on a very specific sub-element of the cyber crime industry called pharmaceutical spam. 

Many security practitioners will hear the phrase “pharmaceutical spam” and immediately start to nod off. Of all the problems they encounter on a daily basis, pharmaceutical spam is pretty low on the priority list. While that may be true, this subset of cyber crime is responsible for starting and maturing many of the trappings that we associate with cyber crime in general: botnet engines, fast-flux obfuscation, spamming, underground forums, cyber crime markets, good service as a distinguisher of criminal support services, and bulletproof-hosting providers.

The Story

The story really begins with Krebs’s weird symbiotic relationship with Vrublevsky (a.k.a. RedEye and Despduck). Vrublevsky was a Russian businessman and cofounder and former CEO of ChronoPay, the infamous credit card processing company that initially got started in the rogue anti-virus industry. I think it is safe to say that in his heyday, Vrublevsky was a bit of an extrovert. He followed Krebs’s blog religiously and would instigate long conversations with Krebs on stories that were fantastical, true, and everything in between. Vrublevsky would feed Krebs half-truths about what was going on in the industry and left it to Krebs to sort it out. Vrublevsky’s downfall was his deteriorating relationship with his former partner, Dimitry Nechvolod (a.k.a. Gugle). 

Vrublevsky and Nechvolod founded ChronoPay together in 2003, but by 2006, Nechvolod had left the company to pursue his own interests. He started two pharmacy spam operations called GlavMed and SpamIT. Because of the competition between these two men, the situation escalated out of control to something that Krebs calls the Pharma Wars, which ultimately scuttled the entire pharmaceutical spam industry, not just Vrublevsky and Nechvolod’s operations, but everybody else’s too. 

Krebs’s main sources of information for this book came from leaked customer and operational databases from these two men. Although Vrublevsky and Nechvolod never admitted it, they both stole the other’s data and leaked it to Krebs. Krebs had many conversations with both Vrublevsky and Nechvolod about their side of the story, and Krebs even traveled to Moscow to interview Vrublevsky personally. From these conversations and other research done by Krebs, we get an inside view of how cyber crime operates in the real world. 

Krebs set himself seven research questions:

  • Who is buying the stuff advertised in spam and why?
  • Are the drugs real or fake?
  • Who profits?
  • Why does the legitimate pharmaceutical industry seem powerless to stop it?
  • Why is it easy to pay for the drugs with credit cards?
  • Do customers have their credit card accounts hacked after buying?
  • What can consumers, policy makers, and law enforcement do [about this cybercrime]?
For the most part, he answers all these questions. I will not spill the answers here, but I will tell you that I was surprised by every single one. I thought I knew this stuff, but Krebs provides the insight and research to make you re-evaluate what you think you know about illegal pharmaceutical spam operations.

Spam Nation is about the Brian Krebs’s story too. Traditional journalists reading this book are going to hate the fact the he plays a key role in most everything that he talks about in this book. His original reporting on bulletproof-hosting providers operating in the US and elsewhere—the Russian Business Network (RBN), Atrivo, and McColo—became that catalyst that eventually got them shut down. This got him noticed by Vrublevsky and started that weird relationship that ultimately led to Krebs receiving the databases from Vrublevsky and Nechvolod. It also led him to leave The Washington Post and to start his Krebs on Security blog.

In the background, Krebs introduces us to the key players involved in the development and operations of some of the most infamous botnets that have hit the Internet community in recent history:

  • Conficker worm (author: Severa; infected 9-15 million computers)
  • Cutwail botnet (authors: Dimitry Nechvolod (Gugle) and Igor Vishnevsky; 125,000 infected computers; spewed 16 billion spam messages a day)
  • Grum botnet (author: GeRA; spewed 18 billion e-mails a day)
  • Festi botnet (operators: Artimovich brothers; delivered one-third of the total amount of worldwide spam)
  • Rustock botnet (author: COSMA; infected 150,000 PCs; spewed 30 billion spam messages a day)
  • Storm botnet (author: Severa).
  • Waledac botnet (author: Severa; spewed 1.5 billion junk e-mails a day)
From my reading, Krebs’s unintentional hero of his story is Microsoft. While Vrublevsky and Nechvolod were tearing each other apart and Krebs was trying to sift through what was true and what was not, Microsoft and other commercial, academic, and government organizations were quietly dismantling the infrastructure that these and other illicit operations depended on:

  • June 2009: 15,000 illicit websites go dark at 3FN after the Federal Trade Commission convinced a northern California judge that 3FN was a black-hat service provider. NASA did the forensics work.
  • November 2009: FireEye takes down the Mega-D botnet.
  • January 2010: Neustar takes control of the Lethic spam botnet.
  • March 2010: Microsoft takes down the Waledac botnet.
  • October 2010: Armenian authorities take down the Bredolab botnet.
  • March 2011: Microsoft takes down the Rustock botnet.
  • July 2011: Microsoft offers a $250,000 reward for information leading to the arrest and conviction of the Rustock botmaster.
  • July 2012: FireEye and Spamhaus take down the Grum botnet.
  • July 2013: Microsoft and the FBI take down 1,400 botnets using the Citadel malware to control infected PCs.
  • December 2013: Microsoft and the FBI take down the ZeroAccess botnet.
  • June 2014: The FBI takes down of the Gameover Zeus botnet.
One takedown masterstroke came out of academia. George Mason University, the International Computer Science Institute, the University of California, San Diego, and Microsoft determined that 95 percent of all spam credit card processing was handled by three financial firms: one in Azerbaijan, one in Denmark, and one in Nevis (West Indies). They also pointed out that these financial firms were in violation of Visa’s own Global Brand Protection Program contract that required fines of $25,000 for transactions supporting the sale of Viagra, Cialis, and Levitra. Once Visa started levying fines, the financial firms stopped processing the transactions. The beauty of this takedown was that this was not a legal maneuver through the courts and law enforcement. It merely encouraged Visa to follow its own policy.

Cyber Crime Business Operations

For me, one of the most enjoyable parts of Spam Nation is the insight on how these criminal organizations operate. For example, Krebs highlights why pharmaceutical operations have great customer support: they want to avoid the penalty fees associated with a transaction when a buyer of illicit pills charges them with fraud. These are called chargebacks, and pharmaceutical customer support operations avoid them like the plague. These support operations require teams of software developers and technical support staff to be available 24/7.

Pharmaceutical operations have mature anti-fraud measures—equivalent to any legitimate bank’s anti-fraud measures—because they need to keep law enforcement and security researchers out of their business.

Most spammers do not make a lot of money. The top five do, but not everybody else. Krebs points out that it takes a multibillion dollar security industry to defend against a collection of criminals who are making a living wage.

In terms of botnet management, operators rent out top-earning botnets to other operators who do not have the skill to build a botnet themselves. Renters purchase installs and seed a prearranged number of bots with an additional malicious program that sends spam for the affiliate. They pay the rent by diverting a portion of their commissions on each pill sale from spam. Sometimes, that commission is as high as 50 percent. That is why the small-timers do not make any money.

Operators launder their money in a process called factoring. They map their client transactions into accounts on behalf of previously established shell companies. They tell the banks that the shell companies are the true customers. Then the operators pay the clients out of their own pockets.

Russian law allows FSB agents (Federal Security Service, the successor to the Soviet Union’s KGB), while remaining in the service, to be assigned to work at enterprises and organizations at the consent of their directors. Twenty percent of FSB officers are engaged in this protection business called “Krusha" in Russian, which means “roof” and pharmaceutical spam operations use them as much as possible.

Partnerships, called partnerkas, between spammers and dodgy advertisers that act as an intermediary for potential sponsors are essential. In this way, sponsors keep their distance from the illicit aspects of the spam business and can unplug from one partnerka in favor of another whenever they want. Some refer to this as organized crime (think The Godfather), but it is more like a loosely affiliated network of independent operators.

With all of these best business practices, you can see why the operators do not see themselves as criminals. They are just businesspeople trying to run a business.

The Tech

Cyber crime runs on technology. In the pharmaceutical spam business, some tech is unique, and other tech is shared with other kinds of cyber crime operations. Unique to pharmaceutical spam is a technique called black search engine optimization (Black SEO). Pharmaceutical spammers hack legitimate websites and insert hidden pages (IFrames) with loads of pharmaceutical websites links. The more links that the common search engines like Google and Bing index, the higher the pharmaceutical sites get in the priority list when normal users search for pills online.

Also unique to the pharmaceutical spam business is a good spam ecosystem. It must have the ability to keep track of how many e-mails the system delivered and how many recipients clicked the link. It must scrub e-mail addresses that are no longer active or are obvious decoys and harvest new e-mail addresses for future operations.

Not unique to pharmaceutical spam are the forums. Forums are the glue that allows the loosely affiliated network of independent operators to communicate with each other. Forums are a place that allows newbies an opportunity to establish a reputation and lowers the barriers to entry for a life of cyber crime. There are forums for every language, but most are in English. Members enforce a strict code of ethics so that members who are caught cheating other members are quickly banned. Social networking rankings give members a way to evaluate potential partners. A single negative post may cost an individual thousands of dollars. Because of that, most amicably resolve issues. Sometimes newbies get labeled as a “deer,” members who unintentionally break one of the forum’s rules. More-serious infractions might find a member in the blacklist subforum defending himself or herself from fraud allegations.

New forums start all the time, but some have been in existence for more than a decade, indicating process maturity for self-policing, networking, and rapid information sharing. New forums allow open registration, but mature forums set up various hurdles for membership that are designed to screen out law enforcement and hangers-on. Most have sub-rooms for specialization such as the following:

  • Spam
  • Cyber banking fraud
  • Bank account cash-out schemes
  • Malicious software development
  • ID theft
  • Credit card fraud
  • Confidence scams
  • Black SEO
Forums have many members (tens of thousands in some), but they exist to make money for the administrators. Admins offer additional services to improve the user experience. They offer escrow services—a small percentage of the transaction cost held until both sides agree that the other held up its end of the bargain—and stickies—ads that stay at the top of their sub-forums that range in price from $100 to $1,000 per month.


In Spam Nation, Brian Krebs covers a key portion of our cyber security and cyber crime history: 2007– 2013, that period when we started to learn about the Russian Business Network, bulletproof-hosting providers, fast-flux obfuscation, criminal best business practices, underground cyber crime forums, and strange-sounding botnet names like Conficker, Cutwail, Grum, Festi, Rustock, Storm, and Waledac. This period just happens to coincide with Krebs’s rise in popularity as one of the leading cyber security journalists in the industry. His story, and the story of two competitive pharmaceutical spammers who eventually destroyed the lucrative moneymaking scheme for all players, is a fascinating read. It is definitely a cyber security canon candidate, and you should have read this by now.


“Spam Nation: The Inside Story of Organized Cybercrime - from Global Epidemic to Your Front Door,” by Brian Krebs, published by Brilliance Audio, 18 November 2014, last visited 13 November 2014,


“Blue Security folds under spammer's wrath,” by Robert Lemos, Security Focus, 17 May 2006, last visited 13 November 2014,

“Click Trajectories: End-to-End Analysis of the Spam Value Chain,” by Kirill Levchenko, Andreas Pitsillidis, Neha Chachra, Brandon Enright, Mark Felegyhazi, Chris Grier, Tristan Halvorson, Chris Kanich, Christian Kreibich, He Liu, Damon McCoy, Nicholas Weaver, Vern Paxson, Geoffrey M. Voelker, and Stefan Savage, last visited 13 November 2014,

“Experts Warn of New Windows Shortcut Flaw,” by Brian Krebs, Krebs on Security, 10 July 2010, last visited 13 November 2014

“Krebs on Security: In-depth security news and investigation,” by Brian Krebs, last visited 14 November 2014,

“PharmaLeaks: Understanding the Business of Online Pharmaceutical Affiliate Programs,” by Damon McCoy, Andreas Pitsillidis, Grant Jordan, Nicholas Weaver, Christian Kreibich, Brian Krebs, Geoffrey M. Voelker, Stefan Savage, and Kirill Levchenko, Usenix, August 2012, last visited 13 November 2014,

“Russian Business Network Study,” by David Bizeul, 11 November 2007, last visited 12 November 2014,

“Shadowy Russian Firm Seen as Conduit for Cybercrime,” by Brian Krebs, The Washington Post, 13 October 2007, last visited 12 November 2014, 

“The Partnerka – What Is It, and Why Should You Care?” by Dmitry Samosseiko, Sophos, Virus Bulletin, September 2009, last visited 13 November 2014,

“The Sleazy Life and Nasty Death of Russia’s Spam King,” by Brett Forrest, Wired Magazine, August 2006, last visited 13 November 2014,

“The Underground Economy of Spam: A Botmaster’s Perspective of Coordinating Large-Scale Spam Campaigns,” by Brett Stone-Gross, Thorsten Holz, Gianluca Stringhini, and Giovanni Vigna, last visited 13 November 2014,

“Top Spam Botnets Exposed,” by Joe Stewart, SecureWorks, 8 April 2008, last visited 13 November 2014,

Tuesday, November 11, 2014

Book Review: The Practice of Network Security Monitoring: Understanding Incident Detection and Response (2013) by Richard Bejtlich

Executive Summary

Richard Bejtlich is one of the most respected security practitioners in the community. If he publishes something, we should all take notice. In The Practice of Network Security Monitoring, Bejtlich provides the theory and the hands-on tutorial on how to do network security monitoring the right way. The book is a primer on how to think about network security monitoring and incident response. 
For seasoned security practitioners, working through the examples in this book will only increase your understanding of the subject. For the beginners in the crowd, Bejtlich provides step-by-step instructions on how to install, configure, and use some of the best open-source tools available that will help any security program improve its network security monitoring capability. Newbies working through the examples in this book will demonstrate to themselves, once and for all, if they have what it takes to work in this field. This book is absolutely a Cybersecurity Canon Candidate and you should have read it by now.


I have been a fan of Bejtlich for a long time. He has been a cyber security book reviewer for many years and he was the inspiration for me to start doing my own book reviews. He is a no-nonsense kind of guy and has been practicing and advancing the craft of network security monitoring and incident response since he started in the industry as a US Air Force officer in 1998. Since then, he has risen in the ranks at some prominent security-minded companies—Foundstone, ManTech, and GE—and today he is the chief security strategist for FireEye. He knows a thing or two about network security monitoring and response. I happen to agree with his general philosophy of cyber security defense, and this book provides an introduction to that philosophy as well as an in-depth, hands-on look at the best open-source tools available. 

The book is a primer on how to think about network security monitoring and incident response, and for the beginners in the crowd, it provides step-by-step instructions on how to install, configure, and use some of the best open-source tools available that will help any security program improve its network security monitoring capability.

I am often asked what skills a wannabe cyber security analyst needs to get into the cyber security industry. My glib go-to answer, and the first question I ask any candidates asking to work for me is, can you install a Linux distribution on your home computer? If a newbie cannot get through that basic exercise, he or she should probably seek employment somewhere else. After reading this book though, I plan to up my game. My new question is, can you work through all of the examples in this book and make sense of it all? If you can, you may have a future in the cyber security industry as a SOC analyst or an incident responder. If you struggle with this book, then cybersecurity might not be for you.

The Network Security Monitoring Story

In my own career, I have routinely seen organizations buy and deploy every shiny and new cybersecurity tool that they could get their hands on and deploy them within the enterprise. Their leadership’s grand strategy seemed to be that shiny equals good. In my early days, I may have even subscribed to that theory. Today, I do not have the energy to chase every bright light that appears on the cyber security market. I mostly just want to see what I have already deployed work the way that I thought it should when I originally bought it. 

Network Security Monitoring Is More Than Just a Set Of Tools

Buying and deploying new technology is relatively easy compared to training the people and developing the processes necessary to fully use it. Organizations tend to forget this. They think that if they just buy the latest tool—pick your tool, it does not matter which one—that it will miraculously configure itself, monitor itself, and forcefully eject any intruders by itself. In the real world, this does not happen. Bejtlich agrees: 

“Products and technologies are not solutions. They are just tools. Defenders (and an organization’s management) need to understand this. No shiny silver bullet will solve the cybersecurity problem. Attacks have life cycles, and different phases of these life cycles leave different evidence in different data sources that are best exposed and understood using different analysis techniques. Building a team (even if it is just a team of one) that understands this and knows how to effectively position the team’s assets (including tools, people, and time) and how to move back and forth between the different data sources and tools is critical to creating an effective incident response capability.”[1] 

In a previous job, I had all of the best toys pumping mountains of data to a 24/7 security operations center, but finding an advanced adversary in all of that data was way too hard. The SOC analysts performed Herculean tasks, but we did not have the processes in place, nor the people trained to develop the processes, to fully use all of that advanced technology. It was frustrating. The bottom line is that if you buy the tool, make sure you spend some resources training your people and developing a plan to incorporate the tool into your overall security program.

Bejtlich also says that your traditional tools are not going to help much with our brand new cloud environments.[1] Customers of cloud environments just do not have access to the networks that a network security monitoring team needs. As we move more and more to the cloud, this can be either a liability or a major opportunity for a young entrepreneur to solve the problem.

Operate Like You Are Compromised: Kill Chain Analysis

In a previous blog, I said that kill chain analysis is one of the three great innovations that have come down the pipe from the security community this past decade.[2] Bejtlich says that Lockheed Martin’s paper on kill chain analysis[3] is unique because followers of the philosophy align their security program along the same lines that adversaries must use to penetrate their victim’s network. 

He confirms the notion that I have had for a few years now that the very old “defense-in-depth” model—which we all adopted in the early 1990s to keep the adversary out of our networks—is dead. It is simply not possible. On the other hand, it does not necessarily mean that you have a disaster on your hands just because one or more adversaries manage to work their way down a couple of links of your kill chain.[3] The idea is to detect these adversaries before they can accomplish their ultimate goal: crime, espionage, hacktivism, warfare, mischief, or whatever. Bejtlich says, 

“Prevention eventually fails … Rather than just trying to stop intruders, mature organizations now seek to rapidly detect attackers, efficiently respond by scoping the extent of incidents, and thoroughly contain intruders to limit the damage they might cause.”[1]

My own personal goal is early detection, quick eradication, and automatic prevention of those observed attacks going forward before these adversaries can claim victory. With the old defense-in-depth model, we were trying to prevent all penetrations into the network. Bejtlich says,

“It’s become smarter to operate as though your enterprise is always compromised.”[1]

Kelly Jackson Higgins interviewed Steve Adegbite, the director of cyber security for Lockheed Martin (LM), in 2013 regarding how LM used kill chain analysis to discover that the company’s RSA token deployment had been compromised.[4] Adegbite said that

"The goal of the Kill Chain is to make sure [the adversaries] don't get to step 7 [of the Kill Chain] and exfiltrate.”[4]

In other words, it is acceptable for adversaries to penetrate your networks as long as you have installed the processes to contain the damage they might cause. 

Network Security Monitoring as a Decision Tool, Not a Reaction Process

Bejtlich’s take on network security monitoring is subtly different than what I would expect from most other security practitioners who have not had a lot of experience actually doing it. According to Bejtlich, these practitioners use network security monitoring for forensics and troubleshooting.[1] His take is to use the discipline as a decision tool for how to contain the detected adversary. He also believes you have to measure your team’s effectiveness by measuring things like 
  • How long it takes to detect adversaries once they have entered your network
  • How long it takes to contain adversaries once you have detected them

In the 2014 Verizon Data Breach report,[5] researchers show that of the 1,367 known data breaches in 2013, security teams discovered less than 25 percent of them (341) within days of the initial compromise. Security teams discovered the rest (1,026) many days and weeks later. Bejtlich says that for a network security monitoring program do be effective, teams must measure how they reduce that time.[1]

Incident Response and Threat Intelligence Go Together

Bejtlich talks about the various approaches to handle a breach within your organization. Some incident response teams elect to identify the compromised asset, take it offline, maybe do some forensics on it, re-image it, and then put it back online so that they can wait for the next breach to happen. I call this the whack-a-mole approach to incident response. This process provides you no context about what the adversaries did and why. Other organizations engage their threat intelligence group and are able to understand the impact of what these adversaries are trying to accomplish. Bejtlich explains that incident response teams can frame the attacks from different perspectives: a threat-centric approach andBottom of Form an asset-centric approach.[1] He says that threat intelligence teams track adversaries by campaigns but that incident response teams respond to the adversary’s actions in waves.[1] He provides practical guidance about what kind of skills and capabilities an incident response team and intelligence team require.

So that’s the story: build a network security monitoring program by deploying the right tool, training your people how to use the tool properly, and developing the processes necessary to incorporate the tool into the overall program. Assume that your network is already compromised, and aggressively track adversaries down the kill chain. Remember, the network security monitoring team’s goal is to prevent adversaries from accomplishing their goals. Use the program to make decisions about how to contain the adversary quickly and efficiently, and use your intelligence team to understand the context of how and why the adversary is attacking your network.

Let’s talk about the tech.

The Network Security Monitoring Tech

This is where it gets really good. The theory is one thing—and I like the theory part—but the actual doing is what really matters. Bejtlich provides a hands-on tutorial on how to deploy the best open-source tools to do network security monitoring. If you are a young person thinking that you want to be a cyber security professional or if you are transitioning careers and you think cyber security is something you can handle, get this book and work through the examples. If you can do them, then I want to talk to you about a job. If you can’t, then maybe consider a less technically demanding career.

Bejtlich says that there are two types of network security monitoring data: full content and extracted content. He says that network security monitoring tools help analysts review these different data types and make a decision about containment based on an organization’s network security process. [1] He points practitioners to Doug Burks’ Security Onion (SO) distribution to get three types of tools: data collection, data presentation, and packet analysis.

Data Collection Tool: 

  • Argus

Data Presentation Tools:

  • Tcpdump
  • Tshark (the command line version of Wireshark)
  • Argus’s Ra client
  • Dumpcap in concert with Tshark

Packet Analysis Tools:
  • Wireshark
  • Xplico
  • NetworkMiner


Richard Bejtlich is one of the most respected security practitioners in the community. If he is speaking somewhere, take the time to hear what the man has to say. The same goes for his writing. If he publishes something, we should all take notice. In The Practice of Network Security Monitoring, Bejtlich provides the theory of and the hands-on tutorial on how to do network security monitoring the right way. He tells you why you should be doing it and how it should work together, and he gives you step-by-step instructions on how to deploy and use the best open-source tools available. If you are already a seasoned security practitioner, working through the examples in this book will only increase your understanding of the subject. If you are a newcomer to the subject, working through the examples will indicate once and for all if you have what it takes to work in this field. This book is absolutely a cyber security canon candidate, and you should have read it by now. 


[1] "The Practice of Network Security Monitoring: Understanding Incident Detection and Response, " by Richard Bejtlich, No Starch Press, 2 August 2013, last visited 29 September 2014,

[2] "Help Me Obi Wan – You’re My only Hope: Three Cyber Security Innovations to Give You Courage," by Rick Howard, Terebrate, 10 June 2013, last visited 30 September 2014,

[3] "Intelligence-Driven Computer Network Defense Informed by Analysis of Adversary Campaigns and Intrusion Kill Chains," by Hutchins, Cloppert & Amin, Lockheed Martin Corp., 2011, last visited 29 September 2014,

[4] "How Lockheed Martin's 'Kill Chain' Stopped SecurID Attack," by Kelly Jackson Higgins, DarkReading, 12 February 2013, last visited 30 September 2014,

[5] "2014 DATA BREACH INVESTIGATIONS REPORT," by Verizon, 2014, last visited 1 October 2014,