🏛️ Issues and Advocacy Training™
Courses › Large Organization Security Programs, Governance and Compliance™

Lesson 7 of 14 · about 60 minutes · Updated November 15, 2026

Security Testing: Penetration Tests, Red Teams and Vulnerability Disclosure

You'll be able to

  • Distinguish vulnerability scanning, penetration testing, red teaming, purple teaming and breach and attack simulation by the question each answers
  • State which rules require penetration testing and how often, including PCI DSS v4.0.1, FedRAMP, 23 NYCRR 500.5, the FTC Safeguards Rule, HIPAA, CMMC Level 3 and DORA
  • Explain the legal authority a test needs under the CFAA, Van Buren, the Justice Department's 2022 policy and state laws such as Idaho Code 18-2202, and write a sound authorization and rules of engagement
  • Choose and manage a testing firm, read its report, and run findings to closure with deadlines, retests and risk acceptance
  • Describe threat-led testing under CBEST, TIBER-EU and DORA and how U.S. practice differs
  • Set up a vulnerability disclosure policy and, where it fits, a bug bounty, with safe harbor language and safeguards against misuse

Start here: the short version

  • Different tests answer different questions. A vulnerability scan lists known weaknesses. A penetration test asks whether qualified people, working under written permission, can turn weaknesses into real access. A red team asks whether the organization would notice and stop a realistic, goal-driven adversary. A purple team and breach and attack simulation check whether specific defenses fire. Buying the wrong one wastes money and produces false comfort.12
  • Many rules now require testing on a schedule. PCI DSS v4.0.1 requires internal and external penetration tests at least every 12 months and after significant changes; New York's 23 NYCRR 500.5 requires penetration testing from inside and outside at least annually; the FTC Safeguards Rule requires annual penetration testing unless the institution has effective continuous monitoring; FedRAMP requires a third-party test before authorization and at least every 12 months; CMMC Level 3 requires penetration testing at least annually.34567
  • Written authorization is the line between a test and a crime. The federal Computer Fraud and Abuse Act and state laws such as Idaho Code 18-2202 punish access "without authorization." In 2019 two testers hired by Iowa's court system were arrested and charged with burglary at a county courthouse because the people who signed their contract may not have had authority over the building; the county paid them $600,000 in a settlement reported in January 2026.891011
  • Finding is cheap; fixing is the job. First American Financial's own test found a flaw in late 2018, staff logged it as low risk with a 90-day fix window, it was not fixed, and a journalist reported the exposure of tens of millions of document images in May 2019. The SEC fined the company $487,616 in 2021 and New York fined its title insurer $1 million in 2023.1213
  • Let outsiders report what they find. Federal civilian agencies have had to publish vulnerability disclosure policies since CISA's Binding Operational Directive 20-01 of Sept. 2, 2020. CISA's shared platform triaged more than 12,000 reports for 51 agency programs by the end of 2023. The Justice Department said on May 19, 2022 that it will not charge good-faith security research under the federal anti-hacking law.141516
  • A bug bounty is not a hush fund. Uber paid the people who stole data on 57 million riders and drivers $100,000 through its bug bounty program in 2016 and had them sign nondisclosure agreements. Its former chief security officer was convicted; the conviction was affirmed in 2025 and the Supreme Court declined review on June 29, 2026.171819
  • The board's questions are simple: what did we test this year, who did it, what did they find, what is still open past its deadline, who accepted the risk of what we did not fix, and can an outsider safely tell us about a flaw?

Rules and standards described: PCI DSS v4.0.1 (June 2024), Requirements 11.3 and 11.4, described and not reproduced; FedRAMP CSP Penetration Test Guidance Version 3.0 (June 30, 2022); New York 23 NYCRR Part 500 as amended effective Nov. 1, 2023, sections 500.1, 500.5 and 500.17; FTC Safeguards Rule, 16 CFR 314.4(d)(2); HIPAA Security Rule, 45 CFR 164.308(a)(8), and the proposed update of Jan. 6, 2025 (not final as of Nov. 15, 2026); 32 CFR 170.14 (CMMC, effective Dec. 16, 2024); NIST SP 800-53 Rev. 5 (Release 5.2.0, Aug. 27, 2025) and SP 800-115 (Sept. 2008); 18 U.S.C. 1030 as interpreted in Van Buren v. United States (June 3, 2021); the Justice Department's CFAA charging policy of May 19, 2022; 37 CFR 201.40(b)(18) as amended Oct. 28, 2024; CISA BOD 20-01 (Sept. 2, 2020), OMB M-20-32 (Sept. 2, 2020) and NIST SP 800-216 (May 2023); TIBER-EU as updated Feb. 11, 2025 and EU Regulation 2022/2554 (DORA, applying from Jan. 17, 2025); the EU Cyber Resilience Act reporting duties that began Sept. 11, 2026; Idaho Code 18-2202 and 74-105, read Nov. 15, 2026. Rules, prosecution policies and court decisions change; check the regulator's current text before relying on any point.

Before you change anything. Plan changes to production systems, test them first, keep a way back, and follow your change management process. For clinical, plant-floor or other safety systems, involve the vendor, your safety lead and qualified engineers (see modules 109 and 111).

A regional bank's chief information security officer gets two documents on the same Monday. The first is the annual penetration test report from an outside firm: 41 pages, three findings rated critical, and a note that the testers reached the loan system's database from a guest wireless network within two days. The second is an email from a stranger who says he found the bank's customer portal leaking account statements, attached a screenshot, and asked whether the bank has a bug bounty. By Friday the CISO has to decide which critical findings can be fixed before the quarterly board meeting, who will accept the risk of the ones that cannot, whether the stranger is a researcher acting in good faith or an extortionist, whether anyone may reply to him, and what the bank's own policy says it will do.

None of those are technical questions, and none of them can be answered well on the day they arrive. This lesson is about the management side of security testing: what the kinds of testing are and what each one proves, which laws and contracts require them and how often, the legal authority that separates a test from a break-in, how to scope a test and write its rules of engagement, how to pick a firm, how to read a report and make sure the findings get fixed, how banks in Europe and the United Kingdom run threat-led tests, and how to let people outside the organization report what they find through a vulnerability disclosure policy or a bug bounty program. It is written for executives, audit committees, the general counsel's office, compliance officers, CISOs, internal auditors, procurement teams, public officials and the reporters and advocates who ask about them.

This lesson is defensive and managerial. It explains how to commission, govern and act on testing. It does not explain how to test, and it names no testing tools, commands or techniques.

It builds on three lessons it does not repeat. Vulnerability scanning, patch deadlines, the CVSS, EPSS and KEV ways of ranking flaws and CISA's federal patch directives are in Foundations lesson 6, Network and Cloud Security. Detecting and responding to a real attack, including what to do when a "test" turns out to be an intrusion, is in Foundations lesson 10. The small business version, with website scanning and patching for a 20-person office, is in Vulnerabilities, Patching and Your Website. Inside this module, the CISO's duties and the Uber case's lesson for security leaders are in lesson 1, how test results become board metrics is in lesson 3, how penetration tests feed SOC 2, ISO 27001, PCI, FedRAMP and CMMC assessments is in lesson 5 and lesson 6, and how auditors treat open findings is in lesson 8.

Five kinds of testing and what each one answers

People use "pen test," "red team" and "scan" loosely, and vendors sell all three under shifting names. The National Institute of Standards and Technology (NIST), the federal agency that writes security standards for federal systems, separates them by purpose. Its Technical Guide to Information Security Testing and Assessment (SP 800-115, September 2008) groups assessment methods into review techniques (looking at documents, logs and settings), target identification and analysis (finding systems and the known weaknesses on them) and target vulnerability validation (confirming that a weakness can actually be used, which is where penetration testing sits).1 NIST's control catalog for federal systems, SP 800-53 Revision 5, treats them as separate controls: RA-5 for vulnerability monitoring and scanning, CA-8 for penetration testing, CA-8(1) for using an independent tester, CA-8(2) for red team exercises, and RA-5(11) for a public disclosure program that lets outsiders report flaws.2 A large organization usually needs all of them, at different frequencies, for different audiences.

Vulnerability assessment and scanning

A vulnerability assessment looks for known weaknesses: missing patches, unsupported software, weak settings, exposed services. Most of the work is done by automated scanners run on a schedule, with people reviewing the output and removing false alarms. It answers the question "what known problems do we have, and where?" It is broad and shallow: it checks thousands of systems against a list of known flaws, but it does not prove that any one of them can be used to cause harm, and it cannot find a flaw that is not on the list, such as a mistake in how the organization's own web application decides who may see which account.

Scanning is a continuous control, not an annual event. PCI DSS requires internal scans and external scans by an Approved Scanning Vendor at least once every three months; New York requires automated scans at a frequency set by the risk assessment and promptly after material changes; the FTC Safeguards Rule requires vulnerability assessments at least every six months unless the institution has continuous monitoring.345 How to run a scanning and patch program at scale, and how to rank what it finds, is covered in Foundations lesson 6.

Penetration testing

A penetration test is an authorized, time-limited attempt by qualified people to get past an organization's protections, within a written scope, to show what an attacker could actually do. New York's cybersecurity regulation defines it as "testing the security of information systems by attempting to circumvent" their security features, and requires it from both inside and outside the system boundary.4 It answers a narrower and more practical question than a scan: "given the weaknesses we have, how far can a capable person get, and what could they reach?" A good test chains small problems together the way real intruders do, and its report shows the path, the evidence and the fix.

Penetration tests come in several flavors. External tests start from the internet. Internal tests start from inside the network, as if an employee's laptop or a vendor connection had been taken over. Application tests focus on a web or mobile application and its programming interfaces. Segmentation tests check that walls between network zones (for example, around a payment card environment or a plant floor) actually hold. Cloud tests look at how cloud accounts and services are configured and connected. Testers may be given full information about the environment ("white box"), partial information ("gray box") or none ("black box"); more information usually produces more findings per dollar, because the client is not paying testers to rediscover what it already knows.

A penetration test is a sample, not a guarantee. It covers what was in scope, for the days the testers worked, with the skills they had. A clean report means the testers did not get in during that window, not that no one can.

Red teaming

A red team exercise is a longer, goal-driven simulation of a specific kind of adversary, usually run with only a small group inside the organization aware of it. Where a penetration test tries to find as many paths as possible in a defined scope, a red team picks an objective (for example, "show that you could move money out of the payments system" or "reach the plant's control network") and tries to reach it quietly, the way a real criminal group or state-backed team would. It answers the question "would our people, processes and tools notice and stop a realistic attack, and how fast?" It tests detection and response as much as prevention. NIST's catalog calls for red team exercises "to simulate attempts by adversaries to compromise organizational systems" as an enhancement for higher-risk federal systems.2

Red teaming is expensive, takes weeks or months, and only pays off when the organization already has a working security operations center, an incident response plan and basic hygiene. An organization still struggling to patch internet-facing systems will learn little from a red team that a penetration test would not have told it more cheaply.

Purple teaming

A purple team exercise puts the testers (the red side) and the defenders (the blue side) in the same room, or the same call, working together. The testers perform a planned activity, the defenders check whether their monitoring saw it and whether an alert fired, and both sides adjust until it does. It answers the question "for this specific kind of attacker behavior, do our detections work, and if not, what do we change?" It is a training and tuning exercise more than an assessment. Since the February 2025 update, the European TIBER-EU framework requires a purple teaming phase at the end of every threat-led test, because regulators concluded that the learning happens when the two sides compare notes.20

Breach and attack simulation

Breach and attack simulation (BAS) products run safe, scripted imitations of known attacker behaviors continuously or on a schedule, and report which ones were blocked or detected. They answer the question "are the defenses we paid for still working today?" Their strength is repetition: a setting changed last Tuesday that silently turned off a protection can show up in the next day's run. Their limit is that they only imitate behaviors someone has already written down; they do not think, improvise or chain new paths the way a human tester does. Many organizations use them between annual human tests, and as evidence for continuous monitoring.

Other assurance activities

Several related activities are often bought from the same firms: source code review and application security testing during development; social engineering assessments (simulated phishing, phone pretexting), covered as a people control in Foundations lesson 9; physical security tests of buildings and badge systems, the riskiest kind to authorize, as the Iowa case below shows; configuration reviews of cloud accounts and firewalls; and tabletop exercises that walk leaders through a scenario without touching any system, covered in lesson 12.

TypeQuestion it answersTypical frequencyWho knows it is happeningMain output
Vulnerability scanning and assessmentWhat known weaknesses do we have, and where?Continuous to quarterly; after material changesIT and security operationsA list of findings ranked by severity, fed into patch and fix queues
Penetration testHow far can a capable person get, and what could they reach?At least yearly and after significant change, under PCI, NYDFS, FTC, FedRAMP and CMMC Level 3Security leadership, system owners, often the operations centerA report showing attack paths, evidence, severity and fixes; a retest letter
Red teamWould we detect and stop a realistic adversary pursuing a goal?Every one to three years at mature organizations; every three years for designated EU financial entitiesA small control group onlyA narrative of the campaign, detection gaps, response timing
Purple teamDo our detections fire for these behaviors, and how do we tune them?Quarterly to yearly; after a red teamEveryone involvedImproved detections and a record of what was tested
Breach and attack simulationAre our controls still working today?Daily to monthly, automatedSecurity operationsPass and fail rates by control, trends

Sources for the table: NIST SP 800-115 and SP 800-53 Rev. 5; PCI DSS v4.0.1; 23 NYCRR 500.5; 16 CFR 314.4(d)(2); FedRAMP Penetration Test Guidance 3.0; 32 CFR 170.14; Regulation (EU) 2022/2554, Article 26. Frequencies at "mature organizations" are common practice, not rules.121

Five kinds of testing: what each one answers Type The question it answers How often Who knows More realistic, costlier, less often Vulnerability scanning What known weaknesses do we have, and where are they? Continuous to quarterly IT and security operations Breach and attack simulation Are the controls we paid for still working today? Daily to monthly, automated Security operations Penetration test How far can a capable person get, and what could they reach? At least yearly and after big changes Security leaders, system owners Purple team Do our detections fire for these behaviors? How do we tune them? Quarterly to yearly Everyone involved Red team Would we notice and stop a real adversary pursuing a goal? Every 1 to 3 years (3 under DORA) A small control group only Frequencies: PCI DSS v4.0.1, 23 NYCRR 500.5, FedRAMP guide 3.0, DORA Art. 26; others are common practice.
The five main kinds of security testing, from the broadest and most automated to the most realistic. The penetration test is the one most rules require by name. Original graphic for this module; frequencies from PCI DSS v4.0.1 (June 2024), 23 NYCRR 500.5 (Nov. 1, 2023), FedRAMP Penetration Test Guidance 3.0 (June 30, 2022) and Regulation (EU) 2022/2554, Art. 26; other frequencies are common practice, not rules.

Who requires testing, and how often

For most large organizations, testing is not optional. It is written into payment card contracts, state and federal regulations, federal cloud and defense contracting rules and, for European operations, EU law. The requirements differ in who must test, how independent the tester must be, how often, and what must happen after. A compliance officer running a multi-regime program usually builds one annual test plan that satisfies the strictest rule for each system, then maps each test to every rule it serves (the one-control-many-rules approach in lesson 4).

Payment cards: PCI DSS Requirement 11

The Payment Card Industry Data Security Standard (PCI DSS) is a contractual standard written by the PCI Security Standards Council, a body founded by the major card brands, and enforced through the contracts between merchants, their banks and the brands. Version 4.0.1, published in June 2024, is the only active version; requirements that had been "best practice" until then became mandatory on March 31, 2025.3 The standard is free to download after accepting a license and is described here, not reproduced.

Requirement 11 covers testing. In plain terms: quarterly internal vulnerability scans and quarterly external scans by an Approved Scanning Vendor (11.3); a written penetration testing method based on industry-accepted approaches that covers the whole perimeter of the cardholder data environment and critical systems, from inside and outside, at the network and application layers (11.4.1); internal and external penetration tests at least once every 12 months and after any significant infrastructure or application upgrade or change (11.4.2 and 11.4.3); correction of exploitable weaknesses and repeat testing to confirm the fixes (11.4.4); tests of segmentation controls, if segmentation is used to shrink the scope, at least every 12 months and after changes, and every six months for service providers (11.4.5 and 11.4.6).3 The tester can be a qualified internal person or an outside firm, but must be organizationally independent of the systems tested; the standard does not require a Qualified Security Assessor to do the testing. The Council's longer Penetration Testing Guidance information supplement (September 2017) explains what a good method, report and retest look like.22 How PCI assessments are validated (Report on Compliance, Self-Assessment Questionnaires) is in lesson 6.

Federal cloud: FedRAMP

Cloud services sold to federal agencies need a FedRAMP authorization, covered in lesson 6. FedRAMP's CSP Penetration Test Guidance, Version 3.0 of June 30, 2022, is the most detailed public government statement of what a penetration test must cover. It names six attack vectors that must be tested unless a documented reason excludes one: from the internet to the provider's corporate network, from the internet to the cloud system itself, from a tenant to the provider's management systems, from one tenant to another, from a mobile application to the system, and from client-side applications or agents to the system. It states that "All penetration test activities must be performed by a 3PAO that has demonstrated penetration testing proficiency" (a third-party assessment organization accredited for FedRAMP), requires rules of engagement approved before testing that tell the provider's leaders about critical findings as soon as they are found, and requires a test within six months before the assessment report for initial authorization and then "at least every 12 months."6 FedRAMP has been rebuilding its process since 2025 under its "20x" effort; whether a newer test guide has replaced Version 3.0 for every authorization path could not be confirmed for this lesson, so check FedRAMP.gov before relying on it.

New York financial services: 23 NYCRR 500.5

Banks, insurers, money transmitters and other companies licensed by New York's Department of Financial Services (DFS) must, under the regulation's vulnerability management section as amended effective Nov. 1, 2023, have written policies that require, at a minimum, penetration testing "from both inside and outside the information systems' boundaries by a qualified internal or external party at least annually"; automated scans, plus manual review of systems the scans do not cover, at a frequency set by the risk assessment and promptly after material changes; a way to learn promptly of new vulnerabilities; and timely remediation "based on the risk they pose."4 The scanning part took effect May 1, 2025. Each year by April 15, the company's highest-ranking executive and its CISO must sign and file a certification of material compliance, or an acknowledgment of noncompliance listing what was not met.4 A company that certifies compliance while its penetration test findings sit unfixed past its own deadlines is exposed twice: once for the gap and once for the certification.

Non-bank financial institutions: the FTC Safeguards Rule

Mortgage brokers, auto dealers that finance cars, tax preparers, payday lenders and other non-bank financial institutions under the Federal Trade Commission's jurisdiction must "regularly test or otherwise monitor" their safeguards. For information systems, the rule says testing must include either continuous monitoring or periodic testing; absent effective continuous monitoring, the institution "shall conduct" annual penetration testing "based on relevant identified risks in accordance with the risk assessment," and vulnerability assessments at least every six months, after material changes, and when the institution learns of circumstances that may have a material impact on its program.5 The rule's details for accounting and tax firms are in module 110, lesson 4.

Banks and credit unions: the FFIEC and the agencies

Federally regulated banks and credit unions answer to the Federal Financial Institutions Examination Council (FFIEC) handbooks used by examiners at the OCC, Federal Reserve, FDIC, NCUA and the Consumer Financial Protection Bureau. The Information Security booklet (September 2016) expects management to obtain independent assurance that controls work, using a mix of self-assessments, vulnerability scans, penetration tests and audits scaled to the institution's risk, and expects examiners to look at the scope and independence of tests and how findings were tracked to closure.23 The federal banking agencies' interagency information security guidelines require regular testing of key controls, at a frequency set by the bank's risk assessment, and say tests "should be conducted or reviewed by independent third parties or staff independent of those that develop or maintain the security programs."24 The FFIEC retired its voluntary Cybersecurity Assessment Tool on Aug. 31, 2025 and pointed institutions to NIST's Cybersecurity Framework 2.0 and CISA's Cross-Sector Cybersecurity Performance Goals instead; neither sets a penetration test frequency, so the frequency comes from the bank's risk assessment and its examiners.23

Health care: the HIPAA evaluation standard

The HIPAA Security Rule does not use the words "penetration test." Its evaluation standard, 45 CFR 164.308(a)(8), requires covered entities and business associates to "perform a periodic technical and nontechnical evaluation" of how well their security policies and procedures meet the rule, first against the rule's standards and then "in response to environmental or operational changes."25 Many hospitals meet the technical half with yearly penetration tests, and the HHS Office for Civil Rights has treated missing evaluations and risk analyses as violations in its resolution agreements (see module 109, lesson 8). The update HHS proposed on Jan. 6, 2025 would make it explicit: vulnerability scanning at least every six months and penetration testing at least every 12 months. It is not final; trade press reported in 2026 that the federal regulatory agenda shows a final rule no earlier than 2027.26 Testing a hospital means testing systems that keep patients alive, which is why module 109, lesson 7 on clinical technology belongs in every hospital's test scope discussion.

Defense contractors and federal agencies

The Cybersecurity Maturity Model Certification (CMMC) program rule, 32 CFR Part 170, effective Dec. 16, 2024, adds penetration testing only at its top level. Level 3 contractors, the small set handling the most sensitive controlled unclassified information and assessed by the Defense Department's own assessors, must conduct penetration testing "at least annually or when significant security changes are made to the system," using both automated scanning tools and ad hoc tests by subject matter experts (requirement CA.L3-3.12.1e, drawn from NIST SP 800-172).7 Levels 1 and 2 require scanning and security assessments but not penetration testing as such. The program's phases and dates are in lesson 6.

Federal agencies follow NIST SP 800-53: penetration testing (CA-8) is in the control baseline for high-impact systems, independent testers (CA-8(1)) are expected there, and agencies set the frequency in their system security plans.2 CISA, the Cybersecurity and Infrastructure Security Agency, also offers free assessments to federal agencies, state, local, tribal and territorial governments and critical infrastructure owners, including vulnerability scanning of internet-facing systems, remote penetration testing and longer risk and vulnerability assessments; demand usually exceeds capacity, so expect a wait.27

European operations: DORA

U.S. banks, insurers, investment firms and technology providers with regulated operations in the European Union fall under the Digital Operational Resilience Act, Regulation (EU) 2022/2554, which has applied since Jan. 17, 2025. It requires a testing program for all covered financial entities and, for entities designated by their supervisors as significant, advanced threat-led penetration testing at least every three years, covering critical functions running in production, with the supervisor involved and the results attested.21 How that works is covered in the threat-led testing section below; operational resilience under DORA more broadly is in lesson 12.

What public companies must say

The SEC does not require testing, but Regulation S-K Item 106 requires each public company's 10-K to describe its processes for assessing, identifying and managing material cyber risks, including "whether the registrant engages assessors, consultants, auditors, or other third parties in connection with any such processes."28 Many 10-K filings now say the company uses outside penetration testers and red teams. Those statements must be true. If the 10-K says the company tests yearly and the last test was three years ago, the problem is no longer a security gap; it is a disclosure problem, covered in Foundations lesson 4.

Rule (version, date)Who it coversPenetration testingWho may test
PCI DSS v4.0.1 (June 2024), 11.4Merchants and service providers handling card dataInternal and external at least every 12 months and after significant change; segmentation every 12 months (six for service providers)Qualified internal or external tester, organizationally independent
FedRAMP Penetration Test Guidance 3.0 (June 30, 2022)Cloud providers selling to federal agenciesBefore initial authorization (within six months) and at least every 12 months; six attack vectorsAn accredited 3PAO with penetration testing proficiency
23 NYCRR 500.5 (amended Nov. 1, 2023)DFS-licensed banks, insurers and othersAt least annually, from inside and outsideQualified internal or external party
16 CFR 314.4(d)(2)Non-bank financial institutions under the FTCAnnually unless effective continuous monitoringNot specified
12 CFR Part 30, App. B (and parallel rules)National banks and other federally supervised institutionsRegular testing of key controls; frequency by riskIndependent third parties or independent staff
45 CFR 164.308(a)(8)HIPAA covered entities and business associatesPeriodic technical evaluation (proposed: every 12 months, not final)Not specified
32 CFR 170.14, CA.L3-3.12.1eCMMC Level 3 defense contractorsAt least annually or after significant security changesNot specified; assessed by DoD
Regulation (EU) 2022/2554, Art. 26Designated significant EU financial entitiesThreat-led test at least every three yearsExternal testers, with limits on internal testers

Idaho public agencies

Idaho state agencies follow the Idaho Technology Authority's enterprise policies, adopted under Idaho Code 67-831 through 67-833. Policy P4520, Patching and Vulnerability Management (last revised June 30, 2021), requires agencies to fix vulnerabilities no later than 10 business days after a vendor announces a fix, or document the risk decision.29 The authority's published policy index, read Nov. 15, 2026, lists no separate penetration testing policy; testing frequency for state agencies is set through the risk management policy (P2045) and the agency's own program, and by any contract or federal rule that applies to the system (for example PCI DSS for a payment portal, or IRS Publication 1075 for federal tax data).29 Counties, cities, school districts and hospital districts set their own rules; many use CISA's free scanning and assessments. When an Idaho agency does commission a test, the report is probably exempt from public disclosure under Idaho Code 74-105, which protects vulnerability assessments of critical infrastructure and records about "the nature, location, or function of cybersecurity devices, programs, or systems," but not records of public spending, so the contract and invoices usually remain public.30

Authority: the law behind a permission slip

Everything a penetration tester does would be a crime if done without permission. That is not a figure of speech. The difference between a test and an intrusion is a written authorization from someone who has the legal power to give it, covering the systems and the activities actually performed. Executives who sign testing contracts need to understand the law well enough to know what they are authorizing, and what they cannot authorize.

The Computer Fraud and Abuse Act

The main federal anti-hacking law, the Computer Fraud and Abuse Act (CFAA), 18 U.S.C. 1030, makes it a crime to access a computer "without authorization" or to "exceed authorized access" and thereby obtain information, cause damage or commit related acts, and it lets people who suffer certain losses sue for damages.8 For decades, courts split over what "exceeds authorized access" meant: did it cover an employee who used a database he was allowed to use, but for a forbidden purpose, or someone who broke a website's terms of service?

The Supreme Court narrowed the reading in Van Buren v. United States, decided June 3, 2021, by a 6 to 3 vote. A Georgia police sergeant had run a license plate search, in a database he was allowed to use, in exchange for money. The Court, in an opinion by Justice Amy Coney Barrett, held that a person exceeds authorized access when he enters parts of a computer system, such as files, folders or databases, that are off limits to him, not when he misuses information he was entitled to reach. It described the test as "a gates-up-or-down inquiry": "one either can or cannot access a computer system, and one either can or cannot access certain areas within the system." The Court expressly left open whether the gates must be technical barriers or can also be limits written in contracts or policies.31

For managers, two practical points follow. First, Van Buren did not make testing without permission legal; it narrowed one phrase. Entering a system with no authorization at all is still covered. Second, because the question of whether a contract or policy can close a "gate" remains open, the written scope of a test matters: a tester who goes beyond the systems listed in the authorization may be outside the gate.

The Justice Department's 2022 charging policy

On May 19, 2022, the Justice Department replaced its 2014 CFAA charging policy with one that tells federal prosecutors not to charge good-faith security research. The policy defines it as "accessing a computer solely for purposes of good-faith testing, investigation, and/or correction of a security flaw or vulnerability, where such activity is carried out in a manner designed to avoid any harm to individuals or the public, and where the information derived from the activity is used primarily to promote the security or safety of the class of devices, machines, or online services to which the accessed computer belongs, or those who use such devices, machines, or online services." It also says that violating a website's terms of service, without more, is not a basis for federal charges. And it warns that calling something research does not make it so: finding vulnerabilities "in order to extort their owners" is not good faith.16

The policy has limits every manager should understand. It is a statement of how federal prosecutors will use their discretion, not a change to the statute. It does not bind state prosecutors. It does not stop a company from suing under the CFAA's civil provision. It protects researchers, not organizations that authorize tests badly. And it protects only good faith: someone who keeps data, demands payment or causes harm is outside it.

State computer crime laws, including Idaho's

Every state has its own computer crime law, and most have no research exception. Idaho Code 18-2202, unchanged since 1984, makes it a misdemeanor to knowingly "and without authorization" use, access or attempt to access any computer, system or network, and a felony to alter, damage or destroy one without authorization or to use one to defraud.9 The Justice Department's 2022 policy does not apply to an Ada County or Kootenai County prosecutor. A testing firm working in Idaho is protected by the written authorization of the system's owner and by nothing else. Physical tests add trespass and burglary laws to the list, as the Iowa case shows.

A second federal law reaches some testing: the Digital Millennium Copyright Act's ban on getting around technical protections on software (17 U.S.C. 1201). Every three years the Librarian of Congress grants exemptions. The current security research exemption, in 37 CFR 201.40(b)(18) as renewed on Oct. 28, 2024, covers circumvention on a lawfully acquired device, or on a computer or network with the owner's or operator's authorization, "solely for the purpose of good-faith security research," using nearly the same definition as the Justice Department's policy. The rule says plainly that such research "may nevertheless incur liability under other applicable laws, including without limitation the Computer Fraud and Abuse Act of 1986," and that the exemption "is not a safe harbor from, or defense to, liability under other applicable laws."32

The Iowa courthouse case

The clearest lesson in authority comes from Iowa. In 2019 the Iowa Judicial Branch's State Court Administration hired the security firm Coalfire to test the security of court facilities and systems, including physical access. On Sept. 11, 2019, two Coalfire employees, Gary DeMercurio and Justin Wynn, were found inside the Dallas County Courthouse in Adel. Dallas County Sheriff Chad Leonard took the position that the court system's contract could not authorize what they had done, and the two were arrested and charged with felony third-degree burglary.10

The Iowa Supreme Court commissioned an outside review by the firm Faegre Baker Daniels, which reported on Oct. 9, 2019 that local law enforcement in Dallas and Polk counties had not been told in advance, that it was unclear whether State Court Administration had authority to approve after-hours physical testing or what that testing could include, that the contract had not been reviewed by the branch's own legal staff before it was signed, and that its language, familiar to security professionals, could be misread by people without that background.10 The charges were reduced to trespass and dismissed in early 2020.10 The testers sued; news reports in late January 2026 said Dallas County had agreed to pay them $600,000 to settle.11

The testers did nothing outside their letter. What failed was the client's governance: no one confirmed who owned the buildings, the people with power to stop an arrest were not told, the scope was written in jargon, and no one with legal authority over the facilities signed. Every one of those failures is a management decision, and every one can be prevented with the steps below.

What a written authorization must cover

A testing authorization (often called the "permission to attack" letter, "authorization to test" or "get out of jail free letter," though it does no such thing if the signer lacks authority) should be a separate signed document, not a paragraph buried in a master services agreement. It should be signed by an officer with authority over every system and facility in scope, on the organization's letterhead, and carried by testers during any physical work. At a minimum it should state:

  1. The legal entity giving permission, and a statement that it owns or controls each asset in scope, or has the written consent of the party that does.
  2. The named testing firm and the individual testers, with a way to verify them (for example, a phone number answered around the clock by a named client contact).
  3. The exact scope: network ranges, domain names, applications, cloud accounts, buildings and rooms, listed by identifier, with anything excluded listed too.
  4. The activities permitted and forbidden (for example, whether social engineering of staff is allowed, whether denial of service testing is excluded, whether testers may access production data and how much).
  5. The dates and hours of testing.
  6. Who has been told: the operations center, the hosting provider or cloud provider if their terms require notice, building security, and for physical tests, local law enforcement or at least the building owner's security office.
  7. How to stop the test, and who can stop it.
  8. Data handling, confidentiality and destruction of copies at the end.
  9. The signer's name, title and statement of authority, and the date.

Third parties are the hard part. A large organization's systems sit in cloud services, managed data centers, software-as-a-service applications and vendor-hosted portals. The organization can authorize testing only of what it controls. Cloud providers publish testing policies that say what customers may test in their own accounts without asking and what requires advance approval; those policies change, so check each provider's current policy before every test and keep a copy with the authorization. For a vendor-hosted application, get the vendor's written consent or rely on the vendor's own test reports (see Foundations lesson 8). Shared buildings, leased space and public facilities need the owner's consent, which is exactly what was missing in Iowa.

A team at laptops during the capture the flag contest at DEF CON 17
A Japanese team in the capture the flag contest at DEF CON 17 in Las Vegas, Aug. 1, 2009. Contests like this are where many professional testers build their skills, under rules everyone agreed to. Photo: Nate Grigg. Wikimedia Commons, CC BY 2.0.

Scoping and rules of engagement

Rules of engagement are the operating manual for a test: what the testers may do, when, how they communicate, and what happens when something goes wrong. FedRAMP requires them in writing, based on NIST SP 800-115's appendix on rules of engagement, approved by the authorizing official before testing starts, and including a clause that the CIO, CISO and others are told of critical or high-impact findings "as soon as they are discovered."61 Even where no rule requires them, they are the best protection against an outage, a lawsuit or a misunderstanding.

Scoping the test

Scope should follow risk, not convenience. The most common failure in purchased tests is a scope chosen to pass: a few web servers that were just patched, excluding the legacy application everyone is worried about. Good scoping starts from the risk assessment (Foundations lesson 2) and the asset inventory, and asks:

  • What are the crown jewels? Payment systems, patient records, plant control systems, the financial reporting system, identity systems, backups. The test should show how close an attacker can get to each.
  • Where do attackers actually get in? Internet-facing remote access devices, file transfer systems and web applications have been the entry points in most large breaches in recent years; the Verizon 2026 Data Breach Investigations Report found exploitation of vulnerabilities was the leading way in, at 31 percent of breaches.33 Those belong in scope every year.
  • What changed? New applications, mergers, cloud migrations and network changes (PCI's "significant change" trigger) need testing whether or not the annual date has come.
  • What must be proven, and to whom? A PCI segmentation test, a FedRAMP annual test and a board's question about ransomware readiness are different tests and may need different firms.
  • What is fragile? Clinical devices, industrial controllers, mainframes and old systems may fail under ordinary testing traffic. For plant-floor and clinical systems, use the safety-first approach in module 111, lesson 5: passive review, test environments and vendor involvement, never active testing of live safety systems without the vendor, the safety lead and qualified controls engineers.

What the rules of engagement say

A complete rules of engagement document covers the authorization above plus the operating details: points of contact on both sides with phone numbers; the communication channel and how sensitive findings are sent; when testers must stop and call (for example, on finding evidence that someone else has already broken in, finding illegal content, or reaching regulated data such as patient records or card numbers); limits on data access (view and screenshot, do not copy, or copy only marked test records); limits on persistence and on changes to systems; how the operations center should treat alerts (respond normally, or confirm with a named contact before escalating); emergency stop procedures; how testers' own systems and the data they gather are protected and destroyed; reporting format and dates; and the retest. FedRAMP's guidance adds incident response coordination, physical testing constraints and acceptable social engineering pretexts.6

One rule deserves emphasis: a test is not cover for an intrusion. If the operations center sees suspicious activity during a test window, it must confirm with the named test contact whether the activity is the testers' before dismissing it. Real attackers have been active during tests, and a "that's just the pen testers" assumption can cost days. The incident response steps for that situation are in Foundations lesson 10.

When a test or its findings lead to changes in production systems, plan each change, test it first and keep a way back.

Choosing and managing a testing firm

Testing is a professional service bought on trust: the client gives a small team the knowledge and access to cause serious harm, and usually cannot judge their technical skill directly. Procurement teams should treat a testing firm like any other high-risk vendor, with the due diligence in Foundations lesson 8, plus questions specific to the work.

Credentials and accreditation

No U.S. government license is required to perform penetration tests, except for specific programs (FedRAMP's accredited assessors and, for CMMC Level 2 certification, accredited third-party assessment organizations, which assess controls rather than perform penetration tests). Clients therefore rely on private credentials of two kinds:

  • Company accreditation. CREST, a not-for-profit body founded in the United Kingdom and now international, accredits testing companies after reviewing their methods, quality management, data handling and complaints process, and examines individual testers at several levels. The Bank of England built its CBEST accreditation with CREST in 2014.3435
  • Individual certifications. Several widely recognized certifications require candidates to pass a hands-on practical exam in which they must actually compromise systems in a lab within a time limit, rather than answering multiple-choice questions; these are better evidence of skill than knowledge-only exams. Others test knowledge of method and reporting. FedRAMP requires the test team lead to hold an industry-recognized penetration testing credential.636

Credentials show minimum competence. The better evidence is a redacted sample report from a comparable engagement, references from clients in the same sector, and the résumés of the specific people who will do the work, named in the contract with a right to approve substitutes.

Questions for a testing firm

AreaWhat to askWhy
PeopleWho exactly will test? What are their credentials and experience with our kind of systems (cloud, mainframe, OT, clinical)? Are they employees or subcontractors? Have they passed background checks, and to what standard?The quality of a test depends on the individuals. Subcontracting and offshore staff raise data handling and, for federal work, eligibility questions.
MethodWhat published method do you follow? How much is manual versus automated? How do you rate severity? May we see a sample report?PCI and FedRAMP require industry-accepted methods and more than automated scanning.36
IndependenceDo you also sell us the products or managed services you will be testing? Did you design or build the systems in scope?A firm testing its own work has a conflict. PCI requires organizational independence; banking rules require independent testers or review.324
Data handlingWhere will our data and your findings be stored, encrypted and for how long? How are testers' laptops secured? How do you destroy data at the end?A test report is a map of how to break in. Testing firms are themselves targets.
Insurance and liabilityWhat professional liability and cyber insurance do you carry, with what limits? What happens if testing causes an outage?Contracts should allocate the risk of disruption. Cyber insurance is covered in lesson 9.
SafetyHow do you test fragile or safety systems? Will you stop if a system degrades?Essential for hospitals, utilities and plants.
RetestIs a retest of fixed findings included, and within what period?PCI requires repeat testing to confirm fixes.3

Rotation, cost and continuity

Many organizations rotate testing firms every two to three years, or rotate the individual testers, because a team that tests the same environment every year starts to see what it expects. Others keep one firm for continuity and add a different firm for a red team or a specialized test. Either approach can work; what matters is that the board knows which, and why.

Prices vary widely by scope, depth and firm, from a few thousand dollars for a narrow external test of a small site to several hundred thousand dollars for a multi-month red team at a large bank. No reliable public survey of prices was found for this lesson, so budget from quotes for your own scope. For context on cost against benefit, the Defense Department estimated in 2016 that hiring an outside firm for the security audit and vulnerability assessment it ran as its first bug bounty would have cost "more than $1 million," against the bounty's total cost of $150,000.37 Budgeting for testing as part of the security budget is covered in lesson 2.

Reading the report and fixing what it finds

The report is the product the organization paid for, and most of its value is lost if it goes into a folder. A typical penetration test report has an executive summary (overall risk, the most serious paths, a comparison with the last test), a description of scope and method (what was tested, when, from where, what was excluded), a narrative of the attack paths the testers found, a list of individual findings, and appendices of evidence. Each finding should give a title, a severity rating with the reasoning behind it, the affected systems, evidence (screenshots or records showing the finding is real), the business impact in plain terms, and a recommended fix.

What leaders should read first

  • Scope and exclusions. Before reading findings, read what was not tested. A clean report on a narrow scope is not good news.
  • The attack narrative. One chain of five "medium" findings that ends in control of the identity system is more serious than ten isolated "highs." Good reports say so; ask for the path, not just the list.
  • Repeat findings. Anything found last year and found again is a governance failure, not a technical one. Ask why the fix did not happen or did not hold.
  • Time to objective. How long did it take the testers to reach the crown jewels? Did the operations center notice, and when?
  • Root causes. Many findings share a few causes: no asset inventory, inconsistent configuration standards, excessive privileges, missing multifactor sign-in. Fixing a cause closes many findings at once.

Severity and deadlines

Severity ratings in reports usually start from the Common Vulnerability Scoring System (CVSS) and adjust for the organization's context, and the organization's own policy then sets how quickly each severity must be fixed. Ranking methods (CVSS, EPSS, CISA's Known Exploited Vulnerabilities catalog and the SSVC decision tree) and the federal deadlines in CISA's directives are covered in Foundations lesson 6.38 The point for managers is simpler: the deadlines must be written, they must be short enough to matter, the clock must start when the finding is reported (not when someone gets around to opening a ticket), and a finding must not be downgraded to buy time without a documented reason and a second signature.

That last point is the First American lesson. First American Financial Corporation, a title insurance company, runs a document-sharing application called EaglePro. An internal security test conducted between December 2018 and January 2019 found that documents could be viewed without signing in; the information security team finished its report on Jan. 11, 2019. Under the company's own policy, a medium-risk finding had to be fixed within 45 days. The team logged it as low risk, which gave it 90 days, and it was not fixed in 90 days either. On May 24, 2019 a journalist told the company that the flaw exposed tens of millions of document images, including Social Security numbers and financial information, and published his story the same day. When the company issued its public statements, its CEO and CFO had not been told that its own staff had found the flaw months earlier.12

The SEC's order of June 14, 2021 found that First American lacked disclosure controls that would have brought the earlier finding to the executives who signed public statements, and imposed a $487,616 penalty.12 New York's Department of Financial Services, which had charged the title insurance subsidiary in July 2020 in its first enforcement action under the cybersecurity regulation, settled on Nov. 28, 2023 for $1 million and remedial measures, finding failures in governance and classification, access controls and risk assessment.13 A routine test did its job. The organization did not.

Example finding deadlines (illustrative internal standard, not a rule) Severity CISO told Fix by Who may extend Critical Same day 15 days CISO and CIO, plus executive acceptance High Within 2 business days 30 days CISO, with risk acceptance Medium In the final report 60 days Security risk manager Low In the final report 180 days System owner, logged Reference points from real rules and cases First American's own policy: 45 days for a medium finding, 90 for a low one (SEC order, 2021). CISA BOD 20-01 guidance: acknowledge a report in 3 business days, resolve it in 90 days. Idaho ITA P4520: fix within 10 business days of a vendor's fix, or document the risk. PCI DSS 11.4.4: fix exploitable findings and retest. NYDFS 500.5: timely fixes, by risk. Known exploited vulnerabilities: use the faster federal timelines in Foundations lesson 6. Lowering a tester's severity rating should need a written reason and a second signature.
Turning severity into deadlines. The upper table is an illustrative internal standard, not a requirement; set your own from your risk assessment and the rules that apply. The lower lines show real reference points. Original graphic for this module; data from the SEC's First American order (June 14, 2021), CISA BOD 20-01 (Sept. 2, 2020), Idaho ITA Policy P4520 (revised June 30, 2021), PCI DSS v4.0.1 (June 2024) and 23 NYCRR 500.5 (Nov. 1, 2023).

Tracking remediation to closure

Every finding should become a tracked item with an owner (a person, not a team), a due date taken from policy, a status, and evidence of the fix. Large organizations put test findings in the same system that holds audit findings, regulatory findings and risk acceptances, so that the CISO, internal audit and the risk committee see one list (the GRC platforms described in lesson 4). A good tracking process:

  1. Records every finding the day the report (or the urgent call during testing) arrives, with the tester's severity and any adjustment, and who approved the adjustment.
  2. Assigns an owner and a due date by policy, and sends critical findings to the CISO the same day.
  3. Asks whether the same weakness exists on systems that were not tested, and fixes those too.
  4. Requires evidence of the fix and an independent retest, by the testing firm or by staff who did not make the fix. PCI DSS requires repeat testing to verify corrections.3
  5. Sends anything that will miss its deadline to the formal risk acceptance process: a written statement of the risk, the compensating controls, a new date, and a signature from an executive with authority to accept that level of risk (risk acceptance authority is in Foundations lesson 2).
  6. Reports open, overdue and accepted findings to the security committee monthly and to the board or its committee at least quarterly, with trends and repeat findings shown.
  7. Tells the people who sign public statements, regulatory certifications and SOC or ISO management assertions what is open, before they sign. This is the step First American missed.

NIST's Cybersecurity Framework 2.0 puts this loop in its Identify function under "Improvement," which includes the outcome that improvements are identified from security tests and exercises.39 When fixes require changes to production systems, plan them, test them and keep a way back.

1. Plan scope from risk and rules 2. Authorize owners sign; ROE agreed 3. Test urgent finds called in 4. Report paths, evidence, severity 5. Triage and assign owner, due date by policy 6. Fix plan, test, way back 7. Retest independent confirmation 8. Report up, learn board, metrics, next plan Cannot fix by the deadline? Written risk acceptance: compensating controls, a new date and an executive's signature, reported to the board No finding leaves the loop until it is retested or a named executive has accepted the risk.
The test-to-fix cycle. Most of the value of a test is in steps 5 to 8; First American's 2019 exposure began at step 5, when a finding was rated low enough to double its deadline. Original graphic for this module; process drawn from NIST SP 800-115 (Sept. 2008), PCI DSS v4.0.1 Requirement 11.4 (June 2024), NIST CSF 2.0 (Feb. 26, 2024) and the SEC's First American order (June 14, 2021).

Who sees the report

A penetration test report is one of the most sensitive documents an organization holds. Distribution should be limited by name; summaries rather than full reports should go to wider audiences; and external sharing (with customers, auditors or regulators) should use a summary or attestation letter from the testing firm rather than the full report wherever the recipient accepts it. Regulators and examiners can usually require the full report. Some organizations route tests through the general counsel's office in the hope of protecting reports from discovery in later lawsuits; whether that works depends on the facts and the court, and it should never delay fixing findings or telling the executives who need to know.

Public bodies have a specific question. In Idaho, a test report on a public agency's systems is likely exempt from disclosure under Idaho Code 74-105's cybersecurity and critical infrastructure exemptions, but the fact that a test was bought, from whom, for how much, and whether the governing board was briefed generally are not.30

Air Force cyber operators setting up computer stations at DEF CON 27
Air Force cyber operators set up before a capture the flag competition at DEF CON 27 in Las Vegas in August 2019. Government and industry both recruit from the security research community. Photo: U.S. Air Force photo by Tech. Sgt. Robert Biermann. Wikimedia Commons, Public domain.

Threat-led testing in finance: CBEST, TIBER-EU and DORA

Financial regulators in the United Kingdom and the European Union decided in the 2010s that ordinary penetration tests chosen and scoped by the banks themselves did not tell supervisors enough about resilience to serious attackers. They built programs in which the regulator is involved, the test is based on real threat intelligence about who targets that institution, and the test runs against live production systems.

CBEST in the United Kingdom

The Bank of England launched CBEST on June 10, 2014, in response to its Financial Policy Committee's recommendation on cyber resilience. Each test starts with a threat intelligence phase, using government and accredited commercial intelligence providers, to model the attackers most likely to target the firm, then has accredited testers simulate those attackers against the firm's critical systems, followed by workshops on the results. The accreditation standards for intelligence and testing providers were built with CREST.35 The Bank's Executive Director Andrew Gracie described the aim as bringing together "the best available threat intelligence from government and elsewhere," tailored to each firm.35

TIBER-EU and DORA's threat-led penetration testing

The European Central Bank and the national central banks of the Eurosystem published TIBER-EU (Threat Intelligence-Based Ethical Red Teaming) in 2018 as a common framework that each country could adopt. DORA then made threat-led penetration testing (TLPT) a legal requirement for financial entities their supervisors designate, at least every three years, covering critical or important functions, with the testers and the results subject to the supervisor's oversight and mutual recognition across member states.21 On Feb. 11, 2025 the Eurosystem updated TIBER-EU to match the European Commission's technical standards on TLPT: the process steps and timelines now match DORA's, purple teaming is mandatory, the internal group that knows about the test was renamed from the "White Team" to the "Control Team," and new guidance covers procurement of testing providers.20 Whether the Commission's TLPT technical standards are now in force in their final form was not confirmed for this lesson from the Official Journal; check the current text.

A TLPT has distinct phases: a preparation phase in which the entity and the authority agree the scope and the control team; a threat intelligence phase in which an independent provider builds scenarios; a red team phase in which testers carry out those scenarios against live systems; and a closure phase with a replay of what happened, purple teaming, a remediation plan and an attestation from the authority. The structure separates the people who know about the test (a few named people) from the defenders being tested, and it requires a way to stop the test immediately if it threatens live services.

The United States approach

The United States has no federal equivalent that requires regulator-led red teaming of banks. U.S. bank examiners rely on the FFIEC handbooks, the interagency guidelines' independent testing requirement and each bank's risk-based program, and they review the bank's own tests.2324 Large U.S. banks with UK or EU operations take part in CBEST or DORA testing through those subsidiaries, and many run voluntary intelligence-led red team programs modeled on them. Treasury, the Federal Reserve and the industry's information sharing groups also run sector-wide exercises, which are tabletop and coordination exercises rather than live testing; those are covered in lesson 12. Whether U.S. regulators should adopt a TLPT-style program is one of the hard questions below.

Vulnerability disclosure: letting outsiders report what they find

However much an organization tests, people outside it will find flaws it missed: security researchers, students, customers, journalists, other companies' engineers, and sometimes criminals. A vulnerability disclosure policy (VDP) is a public statement that tells them where to report, what kind of looking is authorized, what the organization will do with the report, and that it will not pursue them for good-faith research that follows the policy. Without one, a well-meaning person has no safe way to report, and the organization learns about its flaw from a news story, a regulator or an attacker.

Coordinated vulnerability disclosure is the practice behind these policies: the finder reports privately, the organization fixes the flaw, and the two agree when and how the flaw will be made public. Two international standards describe it: ISO/IEC 29147:2018, on receiving vulnerability reports and publishing advisories, and ISO/IEC 30111:2019, on the internal process for investigating and fixing reported vulnerabilities.4041 Both are sold by ISO and are described here, not reproduced; the free U.S. government documents below follow them closely.

Federal agencies: BOD 20-01 and OMB M-20-32

On Sept. 2, 2020, CISA issued Binding Operational Directive 20-01, which requires every federal civilian executive branch agency to publish a VDP, and the Office of Management and Budget issued memorandum M-20-32, "Improving Vulnerability Identification, Management, and Remediation," the same day.1442 The directive required agencies to list a monitored security contact for each .gov domain within 30 days; publish a VDP at a standard web address ("/vulnerability-disclosure-policy") and write handling procedures within 180 days; bring every newly launched internet-facing system into scope; expand the scope by at least one system every 90 days starting at 270 days; and cover all internet-accessible systems within two years, by Sept. 2, 2022. The policy must state what systems are in scope and what testing is allowed, tell finders how to report and that they may report anonymously, set expectations for acknowledging reports, and include a "commitment to not recommend or pursue legal action" against good-faith research the agency deems authorized. It may not require finders to give personal information or limit research to U.S. citizens or vetted people. CISA's implementation guidance recommends acknowledging a report within no more than three business days, an initial assessment within seven days and resolution within 90 days.14

OMB's memo called VDPs "among the most effective methods for obtaining new insights regarding security vulnerability information" and noted they "provide high return on investment," while telling agencies to weigh "the cost, organizational competence, and maturity required" before running paid bug bounties.42 Congress added a related requirement in the IoT Cybersecurity Improvement Act of 2020, which led NIST to publish SP 800-216, Recommendations for Federal Vulnerability Disclosure Guidelines, in May 2023, setting out how agencies should receive, assess and coordinate reports about software and devices they use or build.4344

What the federal VDP platform shows

CISA launched a shared VDP platform in 2021 so agencies would not have to build their own intake and triage. Its 2023 annual report, published September 2024, says that since launch the platform had "triaged over 12,000 submissions (over 7,000 in 2023) on behalf of 51 onboarded agency programs," that more than 2,400 unique valid vulnerabilities had been identified, of which "nearly 2,000 have been remediated," and that more than 3,200 researchers had taken part. It estimates that participating agencies saved an average of $4.45 million in potential remediation costs for critical and severe findings and that they validated submissions two days faster than agencies not on the platform.15 The 2022 report found that agencies received on average 67 more reports and 15 more valid reports in the quarter after joining than the quarter before, that the average time to fix a finding was 38 days, and that reports were screened against the KEV catalog, which "led to the detection of KEVs on federal systems that were not identified by existing scanning tools."45 Those are CISA's own figures and estimates about its own program; they have not been independently audited.

Private sector expectations

No general U.S. law requires a private company to have a VDP, but expectations have hardened:

  • CISA's Secure by Design pledge, which software makers sign voluntarily, includes a goal to publish within one year a VDP "that authorizes testing by members of the public on products offered by the manufacturer, commits to not recommending or pursuing legal action against anyone engaging in good faith efforts to follow the VDP," provides a clear reporting channel and allows public disclosure consistent with coordinated disclosure practice.46
  • The Justice Department's Criminal Division published A Framework for a Vulnerability Disclosure Program for Online Systems (Version 1.0, July 2017) so that organizations could write policies that "clearly describe authorized vulnerability disclosure and discovery conduct, thereby substantially reducing the likelihood that such described activities will result in a civil or criminal violation of law" under the CFAA.47
  • RFC 9116 (April 2022) defines "security.txt," a small machine-readable file placed at a standard location on a website that tells finders where to report and links to the policy. It costs nothing to publish.48
  • The EU Cyber Resilience Act now requires manufacturers of products with digital elements sold in the EU to report actively exploited vulnerabilities and severe incidents through ENISA's Single Reporting Platform; those reporting duties began Sept. 11, 2026, and most of the Act's other requirements, including a coordinated disclosure policy, apply from Dec. 11, 2027.49 U.S. companies selling connected products in Europe are covered.
  • Large customers increasingly ask suppliers in security questionnaires whether they have a VDP, and some state and sector rules (for example for medical devices, covered in module 109, lesson 7) expect manufacturers to have coordinated disclosure processes.

Handling reports inside the organization

Publishing a policy is the easy part. Behind it the organization needs an intake channel monitored every business day; a triage step that decides quickly whether a report is real and how severe it is; a route to the system owner and into the same remediation tracking used for test findings; a person who writes back to the finder; a decision maker for disputes, such as a finder who threatens to publish early; and coordination with the incident response team, because some reports are evidence of an intrusion already underway, and some "reporters" are criminals testing whether the organization will pay. ISO/IEC 30111 describes this internal process.41 The general counsel's office and communications staff should agree in advance on how to respond to a report that arrives from a journalist, and on what the organization will never do (threaten a good-faith finder, or pay for silence).

Bug bounty programs

A bug bounty program is a VDP that pays. The organization offers cash rewards, set in advance by severity, for valid reports about in-scope systems. Many organizations run bounties through commercial platforms that recruit and vet researchers, triage reports and handle payments. Bounties can be public (open to anyone) or private (invitation only), continuous or time-limited.

Hack the Pentagon and the federal experience

The Defense Department ran the first federal bug bounty, "Hack the Pentagon," from April 18 to May 12, 2016, on selected public Defense websites. Defense Secretary Ash Carter announced the results on June 17, 2016: more than 1,400 eligible people registered, more than 250 submitted at least one report, and 138 reports were found to be "legitimate, unique and eligible for a bounty." The pilot cost $150,000, about half of it paid in bounties; the department said hiring an outside firm for a comparable security audit and vulnerability assessment would have cost more than $1 million.37

The department followed with a standing VDP, run by the DoD Cyber Crime Center, in November 2016. By March 2024 it had processed 50,000 vulnerability reports; by the end of 2022, about 4,000 researchers had taken part, and of the close to 45,000 reports received by then, more than 25,000 had been judged valid enough to act on. A 12-month pilot extending the program to volunteer defense contractors, launched in 2021, handled more than 1,000 vulnerabilities and was estimated to have saved $61 million.50

When a bounty makes sense, and when it does not

A bounty adds the most when the organization already has a working VDP, a strong scanning and testing program and the staff to triage and fix quickly. Then a bounty buys the attention of many skilled people on the systems it chooses, paying only for results. It adds the least, and can do harm, when the organization cannot keep up with the reports it already gets, when scope is unclear, or when it is used in place of testing it is required to do. Bounty reports do not satisfy PCI, FedRAMP or NYDFS penetration testing requirements, because those require a planned test of a defined scope by a qualified tester. A bounty also creates new duties: paying promptly and fairly, handling disputes, and checking that payments do not go to people on government sanctions lists.

Safe harbor language

Both VDPs and bounties need "safe harbor" language: the organization's statement that it considers research conducted under the policy to be authorized, that it will not pursue or support legal action against good-faith researchers who follow the policy, and that if a third party brings an action, it will make clear that the research was authorized. Safe harbor language cannot bind a prosecutor or a third party whose systems were touched, and it should not promise what the organization cannot deliver. It should be paired with clear rules: stay in scope, do not access or keep more data than needed to show the flaw, do not degrade services, report promptly, give the organization reasonable time to fix before publishing, and never demand payment as a condition of reporting.4714

The Uber case: a bounty used as cover

In November 2016, Uber learned that intruders had used an access key an engineer had posted on a code-sharing site to download data from its cloud storage, including roughly 25 million names and email addresses, 22 million names and mobile phone numbers, and 600,000 driver's license numbers of drivers and riders. The intruders demanded money. Uber paid them $100,000 through its bug bounty program, which the FTC described as a program meant to reward the responsible disclosure of security vulnerabilities, and did not tell consumers or the FTC until November 2017, while the FTC was investigating an earlier Uber breach.17 Uber's then chief security officer, Joseph Sullivan, had the intruders sign nondisclosure agreements. He was convicted in October 2022 of obstructing the FTC's proceeding and concealing a felony, and sentenced in May 2023 to three years' probation and a $50,000 fine; the Ninth Circuit affirmed in 2025, and the Supreme Court declined to hear his appeal on June 29, 2026.185119

The lesson for program design is concrete. A real researcher reports a flaw; an extortionist has taken data and wants payment for silence. A bounty program must have rules that make the difference visible (no data kept, no payment demanded as a condition), a triage step that routes anything involving stolen data to incident response, and a policy that payments involving data theft are decided under the incident response plan, with the general counsel's office, sanctions screening and the legally required notices, never by the security team alone. The FTC's revised 2018 order required Uber to keep records of bug bounty reports about unauthorized access to consumer data.17 The implications for the CISO's personal accountability are in lesson 1, and breach notice duties are in Foundations lesson 11.

A heavy steel bank vault door in a museum
The vault door of the Davis County Bank, now in the Farmington Historical Museum in Utah. Testing asks whether the door holds, and whether anyone notices someone trying it. Photo: Ben P L from Provo, USA. Wikimedia Commons, CC BY-SA 2.0.

Governance: putting testing in the program

In a large organization, testing should be a planned, funded, reported program, not a set of separate purchases. The pieces:

  • A testing policy or standard approved by the security governance committee, stating what kinds of tests are required for which systems, how often, by whom, how findings are rated and how fast they must be fixed (the policy hierarchy is in lesson 4).
  • An annual test plan built from the risk assessment, the asset inventory and every rule and contract that requires testing, with a budget line. The plan should name the systems, the type of test, the quarter and the rule each test satisfies.
  • A procurement and authorization process that makes the authorization letter and rules of engagement a required step, with the owner of every asset signing.
  • A single findings register shared with internal audit and compliance, with owners and deadlines.
  • A published VDP with a monitored inbox and a security.txt file, and, if the program is mature enough, a bounty.
  • Reporting to the board or its committee: what was tested, the most serious paths found, open findings by age and severity, repeat findings, accepted risks, VDP volume and time to fix. Lesson 3 shows how to turn these into metrics. Internal audit should periodically test whether the program works as described (lesson 8).

Sample policies

Two samples follow: a public Vulnerability Disclosure Policy and a Penetration Test Rules of Engagement template. Both use [brackets] for blanks.

Sample: Vulnerability Disclosure Policy

Sample only. Adapt it to your organization's risks, systems and laws, have qualified people review it, and do not treat it as a substitute for a risk assessment.

Follows: CISA Binding Operational Directive 20-01 and its VDP template elements (Sept. 2, 2020); OMB M-20-32 (Sept. 2, 2020); U.S. Department of Justice, A Framework for a Vulnerability Disclosure Program for Online Systems, Version 1.0 (July 2017); the Justice Department's CFAA charging policy definition of good-faith security research (May 19, 2022); CISA Secure by Design pledge VDP goal; NIST SP 800-216 (May 2023); ISO/IEC 29147:2018 and 30111:2019 (described); RFC 9116 (April 2022).

1. Introduction

[Organization name] is committed to the security of the [customers, patients, members, residents] who rely on our systems. This policy explains how to report a security vulnerability to us, what research we authorize, and what you can expect from us. We want to hear from you, and we will not treat good-faith research under this policy as an attack.

2. Authorization

If you make a good-faith effort to follow this policy while researching our systems, we consider your research authorized. We will work with you to understand and fix the problem quickly, and we will not recommend or pursue legal action related to your research. If a third party brings legal action against you for research that followed this policy, we will make known that your research was authorized. This policy cannot bind other organizations, including [cloud providers or vendors that host in-scope systems] or any government agency.

3. Scope

This policy covers the following systems and services: [list domains, applications, product names and versions]. Any service not listed is out of scope, including [vendor-hosted systems, third-party services, physical facilities, employees and contractors]. If you are not sure whether a system is in scope, contact us at [address] before testing. We will expand this scope over time; the date of the latest change is at the bottom of this page.

4. Guidelines for research

  • Notify us as soon as possible after you discover a real or potential security issue.
  • Make every effort to avoid privacy violations, degradation of service, and destruction or alteration of data.
  • Use only the minimum access needed to show the vulnerability. If you reach personal, health, financial or other sensitive information, stop, do not keep, copy or share it, and tell us in your report.
  • Do not use the vulnerability to go further into our systems, keep access, or reach other systems.
  • The following are not authorized: [denial of service testing; physical testing of offices and facilities; social engineering of employees, contractors or customers; testing that uses others' accounts; automated testing that generates heavy traffic; any test of systems not listed in Section 3].
  • Give us a reasonable time to fix the issue before you disclose it publicly. We ask for [90] days from our acknowledgment, and we will agree a different date with you when needed.
  • Do not demand payment or anything else in exchange for information. A report that comes with a demand is not good-faith research under this policy and will be handled as a security incident.

5. How to report

Send reports to [security email address or web form], also listed in our security.txt file at [https://www.example.org/.well-known/security.txt]. You may report anonymously. Please include: the system or product and version affected; a description of the vulnerability and its potential impact; the steps we need to reproduce it; and any screenshots or other evidence. Do not send us sensitive data you came across. You may encrypt your report with [public key location].

6. What you can expect from us

  • We will acknowledge your report within [three] business days.
  • We will tell you our initial assessment within [seven] days, and keep you informed at least every [14] days until the issue is resolved.
  • We aim to resolve valid vulnerabilities within [90] days, and will explain if we need longer.
  • With your permission, we will credit you publicly when we disclose the issue.
  • [We do not offer payment for reports under this policy.] or [Eligible reports may qualify for a reward under our bug bounty program at [link].]

7. Internal handling (internal annex, not published)

The [CISO] owns this policy. The [security operations team] monitors the reporting channel every business day, triages each report within [two] business days, records valid reports in the [findings register] with an owner and a due date set by the [Vulnerability Management Standard], and sends any report that suggests data has been taken, systems are under active attack, or payment is demanded to the incident response team immediately. No one may threaten legal action against a reporter, offer payment outside the bug bounty program, or ask a reporter to sign a nondisclosure agreement without approval from [the general counsel's office and the CISO]. The [security committee] reviews metrics (reports received, valid reports, median time to acknowledge, median time to fix, overdue items) each quarter.

8. Document information

Issued [date]. Last updated [date]. Approved by [title].

Sample: Penetration Test Rules of Engagement template

Sample only. Adapt it to your organization's risks, systems and laws, have qualified people review it, and do not treat it as a substitute for a risk assessment.

Follows: NIST SP 800-115, Appendix B, rules of engagement (Sept. 2008); FedRAMP CSP Penetration Test Guidance Version 3.0, Section 5.0 and Appendix D (June 30, 2022); PCI DSS v4.0.1 Requirement 11.4 and the PCI SSC Penetration Testing Guidance (Sept. 2017), described; 23 NYCRR 500.5; NIST SP 800-53 Rev. 5, CA-8; the October 2019 independent review of the Iowa Judicial Branch's security testing program.

1. Parties and authority

Client: [legal name], [address]. Testing firm: [legal name], under [master services agreement and statement of work numbers]. The Client's signer, [name, title], states that the Client owns or controls every asset listed in Section 3 or has attached the written consent of the party that does (Attachment A). Facilities owners consenting to physical testing: [names, titles, signatures in Attachment B].

2. Purpose and type of test

Type: [external / internal / web application / cloud configuration / segmentation / red team / purple team / physical / social engineering]. Objectives: [for example, determine whether an attacker on the internet can reach cardholder data; validate PCI DSS segmentation; satisfy 23 NYCRR 500.5(a)(1); satisfy the FedRAMP annual test]. Rules served: [list].

3. Scope

In scope: [IP ranges; domain names; application names, URLs and environments; cloud account identifiers; buildings and rooms]. Explicitly out of scope: [list, including systems hosted by third parties without consent, safety and clinical systems, production databases beyond read-only proof]. Testing outside this list is not authorized. If testers believe an out-of-scope system is relevant, they will stop and ask in writing.

4. Schedule and windows

Start [date and time, time zone]. End [date and time]. Permitted hours: [for example, 24 hours for external testing; 7 p.m. to 6 a.m. for internal testing of [systems]; business hours only for any physical testing]. Blackout dates: [month-end close, peak season, scheduled changes].

5. Permitted and prohibited activities

Permitted: [list at the level of activity type, for example network and application testing, authenticated testing with provided test accounts, controlled social engineering of [named groups] using pretexts approved in Attachment C]. Prohibited: [denial of service; changes to production data; deletion of logs; installing persistent tools beyond the test window; accessing real customer, patient or employee records beyond what is needed to prove access; testing of [named fragile systems]].

6. Contacts and communications

Client test coordinator: [name, phone, available 24 hours]. Client alternate: [name, phone]. Testing firm lead: [name, credential, phone]. Testers: [names]. Secure channel: [method]. Daily status: [time and method]. Notification of law enforcement or facility security for physical tests: [who, when, by whom; confirmation attached].

7. Stop conditions and urgent findings

Testers will stop and call the Client test coordinator immediately if they: find a critical vulnerability that is exploitable from the internet; find evidence that another party has compromised a system; reach regulated data (payment card numbers, health information, Social Security numbers); see any sign of system instability; or are challenged by staff, security or police. The Client test coordinator or [CISO] may stop the test at any time by [method]. Critical and high findings will be reported to [CISO and system owner] within [24 hours] of discovery, not held for the final report.

8. Detection and response

The security operations center [will / will not] be informed. If informed, it will respond normally and confirm any suspected test activity with the Client test coordinator before closing an alert. Suspicious activity that the testing firm does not confirm as its own will be treated as a real incident under the [Incident Response Plan].

9. Data handling

Testers will collect only the evidence needed to document findings, store it encrypted on [firm-controlled systems located in [country]], share it only through the secure channel, and destroy all Client data within [30] days after the final report, with written certification. Reports are marked [classification] and distributed only to [names].

10. Reporting and retest

Draft report by [date]; final report by [date]; executive summary suitable for sharing with [auditors, customers] by [date]. Each finding will include severity with rationale, affected assets, evidence, business impact and recommended remediation. A retest of remediated findings is included within [90] days.

11. Signatures

Client authorized officer: [name, title, signature, date]. Asset and facility owners (if different): [Attachment B]. Testing firm officer: [name, title, signature, date]. Testers will carry a copy of this document and Attachment B during any physical testing.

Pictures, Numbers and Infographics

The key numbers in this lesson, with their sources. Dates are the date of the rule, report or event.

NumberWhat it isSource
12 monthsMaximum interval between internal and external penetration tests under PCI DSS v4.0.1 (also after significant change); six months for service provider segmentation testsPCI DSS v4.0.1, June 20243
6 and 12 monthsFedRAMP: test within six months before the initial assessment report, then at least every 12 monthsFedRAMP Penetration Test Guidance 3.0, June 30, 20226
6Attack vectors FedRAMP requires a test to cover unless justifiedFedRAMP Penetration Test Guidance 3.06
April 15Annual NYDFS certification deadline, signed by the top executive and the CISO23 NYCRR 500.174
Every 6 monthsFTC Safeguards Rule vulnerability assessments, absent continuous monitoring (penetration test annually)16 CFR 314.4(d)(2)5
3 yearsMinimum frequency of threat-led penetration tests for designated EU financial entitiesRegulation (EU) 2022/2554, Art. 2621
6 to 3Supreme Court vote in Van Buren v. United States, June 3, 2021593 U.S. 37431
$600,000Settlement paid to the two testers arrested at the Dallas County, Iowa courthouse in 2019, reported January 2026News reports11
45 versus 90 daysFirst American's required fix time for the flaw as medium risk, versus the time it got after being logged as low riskSEC order, June 14, 202112
$487,616 and $1 millionSEC penalty (2021) and NYDFS penalty (2023) in the First American mattersSEC; NYDFS1213
3, 7 and 90 daysCISA's recommended targets for acknowledging, assessing and resolving a vulnerability reportBOD 20-01 implementation guidance14
12,000+; 2,400+; nearly 2,000Reports triaged, unique valid vulnerabilities found and vulnerabilities fixed through CISA's VDP platform from 2021 through 2023, for 51 agency programsCISA VDP Platform 2023 Annual Report, Sept. 202415
38 daysAverage time for agencies to fix findings reported through the platform (2022 report)CISA VDP Platform Annual Report, Aug. 202345
1,400+; 138; $150,000Hack the Pentagon registrants, valid unique vulnerabilities and total cost, April to May 2016DoD, June 17, 201637
50,000Vulnerability reports processed by the DoD VDP from November 2016 to March 2024SecurityWeek, March 18, 202450
$100,000What Uber paid the 2016 intruders through its bug bounty programFTC, April 12, 201817
31 percentShare of breaches in which exploiting a vulnerability was the way inVerizon 2026 DBIR, executive summary33
Testing and disclosure milestones, 2014 to 2026 2014June 10: Bank of England launches CBEST threat-led testing 2016Apr. to May: Hack the Pentagon. Nov.: DoD disclosure program opens. Uber pays intruders via bounty 2017July: Justice Department framework for vulnerability disclosure programs 2018Eurosystem adopts TIBER-EU. Apr. 12: FTC expands its Uber order 2019May 24: First American exposure reported. Sept. 11: testers arrested at Iowa courthouse 2020Sept. 2: CISA BOD 20-01 and OMB M-20-32 require federal disclosure policies 2021June 3: Van Buren narrows the CFAA. June 14: SEC First American order 2022Apr.: RFC 9116 security.txt. May 19: DOJ good-faith research policy. June 30: FedRAMP guide 3.0 2023May: Sullivan sentenced; NIST SP 800-216. Nov. 28: NYDFS First American $1 million 2024Mar.: 50,000 DoD reports. June: PCI DSS v4.0.1. Oct. 28: copyright research exemption renewed 2025Jan. 17: DORA applies. Feb. 11: TIBER-EU update. Mar. 31: PCI future-dated rules in force 2026Jan.: $600,000 Iowa settlement reported. June 29: Supreme Court declines Sullivan appeal Sept. 11: EU Cyber Resilience Act vulnerability reporting begins Larger dots mark the years of the Iowa arrests, the DOJ policy and the Iowa settlement.
Twelve years of testing and disclosure milestones, from the first regulator-led bank tests to Europe's new reporting duty. Original graphic for this module; dates from the Bank of England (June 10, 2014), DoD (June 17, 2016), FTC (April 12, 2018), CISA, OMB, the Supreme Court, the Justice Department, FedRAMP, the SEC, NYDFS, the PCI SSC, the U.S. Copyright Office (89 FR 85446), the ECB (Feb. 11, 2025) and ENISA (Sept. 11, 2026).

Common mistakes

  • Downgrading a finding to fit the calendar. First American's team rated its own test's finding as low rather than medium risk, which doubled the fix deadline from 45 to 90 days, and then missed the 90 days too. The SEC and New York both penalized the company.1213 Require a second signature to lower any tester's severity rating.
  • Not telling the people who sign. First American's CEO and CFO made public statements about the exposure without knowing that the company's own staff had found the flaw months earlier; the SEC treated that as a disclosure controls failure.12 Open critical findings belong in the disclosure committee's packet and in the NYDFS and SOC management certification process.
  • Authorization signed by someone without authority over the asset. The Iowa court administration approved after-hours physical testing of a county courthouse without settling whether it had authority to do so, without telling local law enforcement and without legal review of the contract, and two testers were arrested and charged with felony burglary.10 Confirm ownership of every asset and building, and get every owner's signature.
  • Contract language only specialists can read. The Iowa review found the testing contract's wording could be misunderstood by people outside the security field.10 Rules of engagement should be readable by a sheriff's deputy at midnight.
  • Using a bug bounty to pay for silence. Uber routed a $100,000 payment to people who had stolen data through its bounty program and hid the breach for a year; the FTC expanded its order and the chief security officer was convicted.1718 Payments involving stolen data go through the incident response plan, never the bounty.
  • Relying on scanners alone. CISA reported that reports from outside researchers found known exploited vulnerabilities on federal systems that agencies' own scanning tools had missed.45 Scanning, testing and outside reports catch different things.
  • A scope chosen to pass. Leaving the systems everyone worries about out of scope produces a clean report and no information. PCI DSS requires the whole cardholder data perimeter and critical systems in scope and testing after significant changes, not just once a year.3
  • No retest. A finding marked "fixed" without independent confirmation is a claim, not evidence. PCI DSS requires repeat testing to verify corrections.3
  • Treating a test window as an excuse. Operations centers that dismiss alerts during a test without confirming with the test contact can miss a real intrusion. FedRAMP requires rules of engagement that tie testing to incident response.6
  • Publishing a disclosure policy with no one behind it. A policy whose inbox is not watched, or that promises acknowledgment in three days and takes three months, turns researchers into public critics. BOD 20-01 requires agencies to set and report on timelines.14

Questions people ask

Is a vulnerability scan enough to meet our testing requirement?

Usually not. PCI DSS, New York's 500.5, the FTC Safeguards Rule (absent continuous monitoring), FedRAMP and CMMC Level 3 all name penetration testing separately from scanning, and FedRAMP says testing should not be "strictly limited to automated scanning techniques."34567 Scanning is required too, on its own schedule.

How often should we run penetration tests?

At least yearly for any system under PCI, NYDFS, FTC, FedRAMP or CMMC Level 3, and after significant changes. Beyond the rules, test internet-facing systems and the paths to your most important data every year, test new applications before launch, and test after mergers and major cloud moves. Red teams every one to three years make sense only once detection and response are working.

Should our own staff do the testing, or an outside firm?

Both, for different purposes. A large organization benefits from an internal team that tests continuously and knows the environment. Several rules require independence: PCI requires organizational independence of the tester, the banking guidelines require tests to be conducted or reviewed by independent parties, and FedRAMP requires an accredited third-party assessor.3246 Many organizations use internal testers year-round and an outside firm for the annual test that regulators and customers rely on.

Do we have to give customers our penetration test report?

Not usually the full report. Customers, especially large ones, often ask for evidence of testing. Most organizations give a letter from the testing firm stating scope, dates and that critical and high findings were fixed and retested, or rely on the testing described in their SOC 2 report (lesson 5). Regulators and examiners can generally require the full report. Contracts should say what will be shared, and under what confidentiality terms.

Someone reported a flaw and is asking for a reward. Should we pay?

If you have a bounty program and the report qualifies, pay under its rules. If you do not, you may choose to thank or reward a good-faith reporter, but decide that under policy, not under pressure. If the person has taken data, threatens to publish unless paid, or asks for money before telling you what they found, treat it as a security incident and possible extortion under the incident response plan, with the general counsel's office, sanctions screening and law enforcement contact as the plan requires. The Justice Department's own policy says finding vulnerabilities "in order to extort their owners" is not good faith.16

Can our VDP protect researchers from prosecution?

Only partly. A clear authorization in your policy makes the research authorized as far as you are concerned, which matters under the CFAA and state laws such as Idaho Code 18-2202, and the Justice Department will not charge good-faith research federally. Your policy cannot bind state prosecutors or third parties whose systems are involved, which is why good policies limit scope to systems you control and say so.16947

Do we need a red team?

Not until the basics are in place. A red team measures detection and response against a realistic adversary; if the organization has no monitored logging, no incident response plan and unpatched internet-facing systems, a penetration test and a gap assessment will produce more useful work for less money. Large financial institutions designated under DORA have no choice: they must run threat-led tests at least every three years.21

What should a board member ask about testing?

What did we test this year and why those systems? Who tested, and are they independent? What was the most serious path the testers found? How many critical and high findings are open, how many are past due, and who accepted those risks? Did anything repeat from last year? Did our monitoring detect the testers? Do we have a public way for outsiders to report flaws, how many reports did we get, and how fast did we fix them? Is what our 10-K or certification says about testing still true?

Hard questions

Should U.S. regulators require threat-led red team tests of the largest banks?

The case for: the UK has run CBEST since 2014 and the EU made threat-led testing law under DORA, on the view that tests chosen and scoped by the banks themselves are too easy to pass and that supervisors need to see how institutions hold up against realistic attackers on live systems.3521 Large U.S. banks are already tested this way in their UK and EU arms, so the capacity exists. A common U.S. program would let supervisors compare institutions and find shared weaknesses.

The case against: live red teaming of critical financial systems carries a real risk of disruption; the supply of qualified testers and threat intelligence providers is limited; banks already face independent testing requirements, examinations and NYDFS rules; and supervisors holding detailed attack paths for the largest banks become an attractive target themselves. And most breaches start with unfixed known flaws that ordinary testing already finds.

What the evidence shows: there is no public study comparing outcomes for banks tested under CBEST or TIBER-EU with similar banks that were not. Regulators' own accounts describe the programs as producing lessons, which is why TIBER-EU was strengthened in 2025 rather than scaled back.20 The evidence that known, unfixed flaws drive breaches, such as Verizon's 31 percent exploitation figure, supports both sides: it argues for better basics and for tests that check whether the basics hold.33

Should good-faith security research be protected by statute rather than prosecutors' policy?

The case for a statute: the Justice Department's 2022 policy can be changed by any future administration, binds no state prosecutor and does not stop civil suits. The Iowa testers were charged under state law, and Idaho's computer crime law, like most, has no research exception.16910 Researchers say legal uncertainty keeps many people from reporting flaws at all.

The case against: a statutory exception could be used as a defense by people who were not acting in good faith, and the line between research and intrusion depends on facts that prosecutors and juries are better placed to weigh case by case. Van Buren already narrowed the law, and the copyright exemption shows that lawmakers have been willing to act case by case.3132

What the evidence shows: the 2022 policy has been in place for more than four years; this lesson found no published count of cases it changed. The copyright exemption has been renewed in each rulemaking since 2015 and now says it is not a defense under the CFAA, which shows how separate the two systems remain.32

Should private companies be required to publish a vulnerability disclosure policy?

The case for: federal agencies have had to since 2020, the federal platform's numbers show outsiders finding thousands of real flaws, including known exploited vulnerabilities scanners missed, and the EU now requires manufacturers to have coordinated disclosure processes.14154549 A policy costs little to publish.

The case against: a policy invites reports the organization must triage and answer; a small or immature organization may not be able to handle the volume or fix what it learns, and publishing a policy without the staff behind it creates frustrated researchers and public disclosures. On this view, voluntary pledges such as CISA's and pressure from large customers are enough.

What the evidence shows: CISA's figures show that VDPs find real problems at modest cost, but they come from CISA's own program and are not independently audited, and they cover internet-facing federal systems, not every kind of organization.15

Do bug bounties make organizations safer?

For: bounties bring many testers with different skills to bear and pay only for results; the Pentagon's first bounty found 138 valid flaws for $150,000, where the department estimated a comparable outside assessment at more than $1 million.37

Against: bounties reward the easiest findings rather than the most important ones, can flood a small team with low-quality reports, do not satisfy testing rules, and, as the Uber case shows, can be misused to disguise payments to criminals.17 Low payouts can also undervalue serious findings that could sell for more elsewhere.

What the evidence shows: government programs report large numbers of valid findings fixed, but no public study measures whether organizations with bounties suffer fewer breaches than those without. The evidence supports bounties as an addition to testing and a VDP, not a replacement.42

Should public agencies' test results be public?

For openness: taxpayers pay for the systems and the tests, and voters and legislators cannot judge whether an agency is protecting their data if everything is secret. Knowing that a test happened, how many serious findings it produced and whether they were fixed is enough for accountability without giving attackers a map.

For confidentiality: detailed findings tell attackers where to go, and agencies may be less willing to commission hard tests if the results become headlines. Idaho, like most states, exempts security-sensitive records from public disclosure.30

What the evidence shows: federal inspectors general already publish yearly evaluations of agencies' security programs that summarize testing results while leaving out technical details, and some state legislative auditors do the same, which suggests a middle path is workable; whether a specific agency follows it is a question for that agency.

For reporters and advocates

What public records exist.

  • SEC filings on EDGAR. 10-K Item 1C describes whether a company uses outside assessors and testers; compare it with what happened in the company's next incident, and with its Form 8-K Item 1.05 filings. EDGAR full-text search finds every filing that uses a phrase such as "penetration testing" or "red team" in a date range.2852
  • Enforcement orders. The SEC's First American order, the NYDFS consent orders and the FTC's Uber orders show how regulators treat findings that were known and not fixed or a bounty used to hide a breach.121317
  • Court records. Criminal and civil cases involving researchers or testers are on PACER, with many federal dockets mirrored free on CourtListener; the Sullivan case's opinions and the Supreme Court docket are public.535119
  • Federal agency VDPs. Every federal civilian agency's policy is at its main domain followed by /vulnerability-disclosure-policy; CISA publishes annual VDP platform reports.1415
  • security.txt files. Any organization's file, if it has one, is at /.well-known/security.txt on its website, and shows where it accepts reports.48
  • Idaho public records. Under the Idaho Public Records Act (Idaho Code Title 74, Chapter 1), ask a state agency, county, city, school district or hospital district: whether it has commissioned a penetration test or used CISA's free assessments in the past three years; the contract, statement of work and invoices; the dates its governing body was briefed on results; how many critical and high findings remain open and for how long; and whether it has a vulnerability disclosure policy. Expect the report itself to be withheld under Idaho Code 74-105's cybersecurity and critical infrastructure exemptions; the spending records are not exempt.3029

What to ask a company or agency.

  • When was your last external penetration test, who performed it, and was it independent? Were critical findings fixed and retested?
  • Does your 10-K, certification or website description of testing match what you actually did this year?
  • How do outsiders report security flaws to you? How many reports did you receive last year, and how quickly did you fix them?
  • Have you ever paid someone who reported a flaw, and under what program?
  • Who in your organization signs testing authorizations, and does that person have authority over every building and system tested?

Cautions. Do not test an organization's systems yourself to check a story; reading a public web page is different from probing a system, and the line under the CFAA and state laws is not always clear. If a source brings you a flaw, the safest path is to report it through the organization's VDP or, for federal systems, through CISA, and to report on the response. Do not publish details that would let others exploit an unfixed flaw.

Where to go next

Still being checked (8 items)

These points could not be fully confirmed against a primary source when this lesson was written and fact checked in November 2026. The text above states each one as uncertain.

  • The FFIEC Information Security booklet's testing expectations and the retirement of the FFIEC Cybersecurity Assessment Tool on Aug. 31, 2025 are stated from earlier reading; the FFIEC websites blocked access on Nov. 15, 2026.
  • Whether FedRAMP has replaced its Penetration Test Guidance Version 3.0 (June 30, 2022) for any authorization path under its 2025 to 2026 rebuilding could not be confirmed.
  • DORA Articles 24 to 27 (three-year threat-led testing for designated entities) and whether the Commission's TLPT technical standards are in force in final form were not re-read in the Official Journal; the EUR-Lex download failed. TIBER-EU's original 2018 adoption date was not re-confirmed.
  • PCI DSS v4.0.1 Requirement 11 details are described from the standard as read for module 108 on Nov. 13, 2026 and from general knowledge of the standard; the licensed document was not re-opened for this lesson.
  • The $600,000 Dallas County settlement is from news reports (Reason, Feb. 17, 2026; Ars Technica, Jan. 29, 2026); the settlement agreement and its exact date were not seen.
  • The SEC's First American order was read through a law firm summary because SEC.gov blocked scripted access; the description of the exposed data as including Social Security numbers and financial information is from that summary and the SEC's press release as reported.
  • The description of CISA's free services (scanning, remote penetration testing, risk and vulnerability assessments) and the IoT Cybersecurity Improvement Act provisions were not re-read for this lesson.
  • Testing firm price ranges are general statements; no public price survey was found.

Last verified November 15, 2026. General education, not legal, compliance, audit or other professional advice; rule versions and dates are stated in the text, and the sources used are listed in the Notes and Sources below. Check the regulator's current text before relying on any point.

Notes

  1. National Institute of Standards and Technology, SP 800-115, Technical Guide to Information Security Testing and Assessment, September 2008: Section 3 (review techniques), Section 4 (target identification and analysis techniques), Section 5 (target vulnerability validation techniques, including penetration testing) and Appendix B (rules of engagement template) (read Nov. 15, 2026). Read it online. ↩
  2. NIST SP 800-53 Rev. 5, Security and Privacy Controls for Information Systems and Organizations, Release 5.2.0 (Aug. 27, 2025), OSCAL catalog last modified Oct. 22, 2026: CA-8 Penetration Testing, CA-8(1) Independent Penetration Testing Agent or Team, CA-8(2) Red Team Exercises ("to simulate attempts by adversaries to compromise organizational systems"), RA-5 Vulnerability Monitoring and Scanning and RA-5(11) Public Disclosure Program; baselines in SP 800-53B (read Nov. 15, 2026). Read it online. ↩
  3. PCI Security Standards Council, PCI DSS v4.0.1 (June 2024), Requirement 11.3 (internal and Approved Scanning Vendor external scans every three months) and Requirement 11.4 (11.4.1 method, 11.4.2 and 11.4.3 internal and external penetration tests at least every 12 months and after significant change, 11.4.4 correction and repeat testing, 11.4.5 and 11.4.6 segmentation testing); future-dated requirements effective March 31, 2025. Free to download after accepting a license; described, not reproduced (read for module 108 on Nov. 13, 2026). Read it online. ↩
  4. New York State Department of Financial Services, 23 NYCRR Part 500, Second Amendment (effective Nov. 1, 2023): 500.1 definition of penetration testing, 500.5 vulnerability management (annual penetration testing from inside and outside by a qualified internal or external party; scans at a risk-based frequency and after material changes, effective May 1, 2025; timely remediation by risk), and 500.17(b) annual certification or acknowledgment due April 15 (read Nov. 15, 2026). Read it online. ↩
  5. Federal Trade Commission, Standards for Safeguarding Customer Information, 16 CFR 314.4(d)(1) and (d)(2), eCFR current text (annual penetration testing and vulnerability assessments at least every six months absent effective continuous monitoring) (read Nov. 15, 2026). Read it online. ↩
  6. FedRAMP, CSP Penetration Test Guidance, Version 3.0, June 30, 2022: Section 3.1 (six attack vectors), Section 5.0 and Appendix D (rules of engagement, notification of critical findings "as soon as they are discovered"), Section 7.0 (test within six months before the Security Assessment Report and "at least every 12 months"), the requirement that tests be performed by a 3PAO with demonstrated proficiency, and the statement that testing should not be "strictly limited to automated scanning techniques" (read Nov. 15, 2026). Read it online. ↩
  7. 32 CFR 170.14(c)(4), CMMC Level 3 security requirements, item (xx), CA.L3-3.12.1e ("Conduct penetration testing at least annually or when significant security changes are made to the system"; automated scanning tools and ad hoc tests by subject matter experts), drawn from NIST SP 800-172 (February 2021); Part 170 effective Dec. 16, 2024; eCFR current text (read Nov. 15, 2026). Read it online. ↩
  8. Computer Fraud and Abuse Act, 18 U.S.C. 1030, including subsection (a)(2) (accessing a computer without authorization or exceeding authorized access and obtaining information) and subsection (g) (civil action), Legal Information Institute text (checked Nov. 15, 2026). Read it online. ↩
  9. Idaho Code 18-2202, Computer crime (added 1984; subsection (3), knowing access "without authorization," is a misdemeanor; subsections (1) and (2) are felonies), Idaho Legislature website (read Nov. 15, 2026). Read it online. ↩
  10. "Trespassing Charges Dropped Against Penetration Testers," BankInfoSecurity, Feb. 3, 2020 (arrests of Gary DeMercurio and Justin Wynn on Sept. 11, 2019 at the Dallas County Courthouse in Adel, Iowa; felony third-degree burglary charges reduced to trespass and dropped; the Faegre Baker Daniels report of Oct. 9, 2019 for the Iowa Supreme Court on notice to law enforcement, unclear authority of State Court Administration, lack of legal review and contract language) (read Nov. 15, 2026). Read it online. ↩
  11. "Brickbat: Bad Boss," Reason, Feb. 17, 2026 (Dallas County, Iowa pays $600,000 to settle with the two testers hired by the Iowa Judicial Branch); also reported by Ars Technica on Jan. 29, 2026 under the headline "County pays $600,000 to pentesters it arrested for assessing courthouse security." The settlement agreement itself was not read (read Nov. 15, 2026). Read it online. ↩
  12. U.S. Securities and Exchange Commission, In the Matter of First American Financial Corporation, Securities Exchange Act Release No. 92176, June 14, 2021 (announced June 15, 2021 in press release 2021-102), as summarized by Sullivan and Cromwell LLP, "SEC Charges Issuer for Inadequate Cybersecurity Disclosure Controls" (internal test December 2018 to January 2019; report finalized Jan. 11, 2019; finding logged as level 2 with a 90-day deadline instead of level 3 with 45 days; journalist's report May 24, 2019; CEO and CFO not informed; penalty $487,616). SEC.gov blocked scripted access, so the order was read through the summary (read Nov. 15, 2026). Read it online. ↩
  13. New York State Department of Financial Services, press release, "DFS Announces $1 Million Cybersecurity Settlement with First American Title Insurance Company," Nov. 28, 2023 (EaglePro flaw; senior management aware in May 2019; failures in governance and classification, access controls and identity management, and risk assessment), following the Department's statement of charges of July 2020 (read Nov. 15, 2026). Read it online. ↩
  14. Cybersecurity and Infrastructure Security Agency, Binding Operational Directive 20-01, "Develop and Publish a Vulnerability Disclosure Policy," Sept. 2, 2020, with its implementation guidance and VDP template (deadlines of 30, 180 and 270 days and two years; required policy elements; recommended acknowledgment within three business days, initial assessment within seven days and resolution within 90 days) (read Nov. 15, 2026). Read it online. ↩
  15. CISA, Vulnerability Disclosure Policy (VDP) Platform 2023 Annual Report, published September 2024, Annual Report Highlights and Time and Cost Savings (12,000+ submissions triaged since 2021, 51 agency programs, 2,400+ valid unique vulnerabilities, nearly 2,000 remediated, 3,200+ researchers, $4.45 million estimated average savings, validation two days faster) (read Nov. 15, 2026). Read it online. ↩
  16. U.S. Department of Justice, Office of Public Affairs, "Department of Justice Announces New Policy for Charging Cases under the Computer Fraud and Abuse Act," May 19, 2022 (policy effective immediately, replacing the 2014 policy; definition of good-faith security research; terms-of-service violations alone not sufficient; extortion example), page last updated Feb. 6, 2025; the policy is in the Justice Manual at 9-48.000 (read Nov. 15, 2026). Read it online. ↩
  17. Federal Trade Commission, "Uber Agrees to Expanded Settlement with FTC Related to Privacy, Security Claims," April 12, 2018 (2016 breach discovered November 2016; more than 25 million names and email addresses, 22 million names and mobile phone numbers and 600,000 driver's license numbers; $100,000 paid through the bug bounty program; not disclosed until November 2017; revised order requires records of certain bug bounty reports) (read Nov. 15, 2026). Read it online. ↩
  18. U.S. Attorney's Office, Northern District of California, "Former Chief Security Officer Of Uber Sentenced To Three Years' Probation For Covering Up Data Breach Involving Millions Of Uber User Records," May 5, 2023 (57 million users and drivers; nondisclosure agreements; $50,000 fine; October 2022 jury verdict) (read for module 108 on Nov. 13, 2026). Read it online. ↩
  19. Supreme Court of the United States, Docket No. 25-1082, Joseph Sullivan v. United States, petition filed March 12, 2026, denied June 29, 2026 (read for module 108 on Nov. 13, 2026). Read it online. ↩
  20. European Central Bank, "Eurosystem updates TIBER-EU framework to align with DORA," Feb. 11, 2025 (alignment with the DORA regulatory technical standards on threat-led penetration testing; mandatory purple teaming; White Team renamed Control Team; new procurement guidance) (read Nov. 15, 2026). Read it online. ↩
  21. Regulation (EU) 2022/2554 of Dec. 14, 2022 on digital operational resilience for the financial sector (DORA), applying from Jan. 17, 2025, Articles 24 to 27 (testing program; advanced testing based on threat-led penetration testing at least every three years for designated entities; requirements for testers). The EUR-Lex text did not download for this lesson and these points are stated from the regulation as summarized in the ECB's Feb. 11, 2025 notice and earlier reading (checked Nov. 15, 2026). Read it online. ↩
  22. PCI Security Standards Council, Information Supplement, Penetration Testing Guidance, Version 1.1, September 2017, in the PCI SSC document library (described, not reproduced; not re-read for this lesson) (checked Nov. 15, 2026). Read it online. ↩
  23. Federal Financial Institutions Examination Council, IT Examination Handbook: Information Security booklet (September 2016), sections on assurance and testing; and the FFIEC's 2024 announcement that its Cybersecurity Assessment Tool would be retired on Aug. 31, 2025. The FFIEC sites blocked scripted access on Nov. 15, 2026, so these points are stated from earlier reading and are listed as unverified (checked Nov. 15, 2026). Read it online. ↩
  24. Interagency Guidelines Establishing Information Security Standards, 12 CFR Part 30, Appendix B, section III.C.3 (tests "should be conducted or reviewed by independent third parties or staff independent of those that develop or maintain the security programs"), eCFR current text (read Nov. 15, 2026). Read it online. ↩
  25. 45 CFR 164.308(a)(8), Standard: Evaluation, eCFR current text (read Nov. 15, 2026). Read it online. ↩
  26. U.S. Department of Health and Human Services, "HIPAA Security Rule To Strengthen the Cybersecurity of Electronic Protected Health Information," proposed rule, 90 FR 898 (Jan. 6, 2025) (vulnerability scanning at least every six months and penetration testing at least every 12 months); final rule status per module 108 lesson 6 sources (read for module 108 on Nov. 13, 2026). Read it online. ↩
  27. CISA, "Cyber Hygiene Services" and "Cyber Resource Hub" pages describing free vulnerability scanning, remote penetration testing and risk and vulnerability assessments for federal, state, local, tribal and territorial governments and critical infrastructure (not re-read for this lesson) (checked Nov. 15, 2026). Read it online. ↩
  28. 17 CFR 229.106 (Regulation S-K Item 106), paragraph (b)(1)(ii) ("whether the registrant engages assessors, consultants, auditors, or other third parties in connection with any such processes"), adopted in SEC Release No. 33-11216 (July 26, 2023) (read for module 108 on Nov. 13, 2026). Read it online. ↩
  29. Idaho Technology Authority, Enterprise Policy P4520, "Patching and Vulnerability Management" (last revised June 30, 2021), and the ITA policy index listing policies P4110 through P4590 and P2045 (IT Risk Management Program, revised November 2025) (read Nov. 15, 2026). Read it online. ↩
  30. Idaho Code 74-105 (Public Records Act exemptions), subsections (4)(b) (critical infrastructure records, including vulnerability assessments) and (21) (records on "the nature, location, or function of cybersecurity devices, programs, or systems"), both excluding public expenditure records (read for module 108 on Nov. 13, 2026). Read it online. ↩
  31. Van Buren v. United States, 593 U.S. 374 (2021), decided June 3, 2021 (Barrett, J., for the Court; Thomas, J., dissenting, joined by Roberts, C.J., and Alito, J.), including the "gates-up-or-down inquiry" and footnote 8, Legal Information Institute text (read Nov. 15, 2026). Read it online. ↩
  32. 37 CFR 201.40(b)(18), exemption for good-faith security research under 17 U.S.C. 1201, as amended at 89 FR 85446 (Oct. 28, 2024), eCFR current text, including paragraph (b)(18)(iii) on the CFAA (read Nov. 15, 2026). Read it online. ↩
  33. Verizon, 2026 Data Breach Investigations Report: Executive Summary (data from Oct. 2024 through Nov. 2025; exploitation of vulnerabilities 31 percent of breaches) (read for module 108 on Nov. 13, 2026). Read it online. ↩
  34. CREST, "About CREST" and certification pages (professional examinations at Practitioner, Registered and higher levels; Practitioner aimed at about 2,500 hours of experience; member company accreditation) (read Nov. 15, 2026). Read it online. ↩
  35. Bank of England, "Bank of England launches new framework to test for cyber vulnerabilities," June 10, 2014 (CBEST; threat intelligence from government and accredited providers; accreditation standards developed with CREST and Digital Shadows; quotation from Andrew Gracie) (read Nov. 15, 2026). Read it online. ↩
  36. OffSec, PEN-200 course and OSCP certification page (24-hour proctored, hands-on exam), cited as an example of a practical certification, not an endorsement (read Nov. 15, 2026). Read it online. ↩
  37. U.S. Department of Defense, "Carter Announces 'Hack the Pentagon' Program Results," release of June 17, 2016 (pilot April 18 to May 12, 2016; more than 1,400 registered; more than 250 submitted reports; 138 valid unique vulnerabilities; total cost $150,000, about half in bounties; outside firm estimate more than $1 million), read in the GlobalSecurity.org copy (read Nov. 15, 2026). Read it online. ↩
  38. CISA, Known Exploited Vulnerabilities Catalog (continuously updated) (checked Nov. 15, 2026). Read it online. ↩
  39. NIST, The NIST Cybersecurity Framework (CSF) 2.0, CSWP 29, Feb. 26, 2024, Identify function, Improvement category, outcome ID.IM-02 ("Improvements are identified from security tests and exercises...") (read Nov. 15, 2026). Read it online. ↩
  40. International Organization for Standardization, ISO/IEC 29147:2018, Information technology, Security techniques, Vulnerability disclosure (paid standard; described, not reproduced; ISO's page could not be loaded by script) (checked Nov. 15, 2026). Read it online. ↩
  41. ISO/IEC 30111:2019, Information technology, Security techniques, Vulnerability handling processes (paid standard; described, not reproduced) (checked Nov. 15, 2026). Read it online. ↩
  42. Office of Management and Budget, Memorandum M-20-32, "Improving Vulnerability Identification, Management, and Remediation," Sept. 2, 2020 (VDPs "among the most effective methods" with "high return on investment"; bug bounty cost, competence and maturity) (read Nov. 15, 2026). Read it online. ↩
  43. NIST SP 800-216, Recommendations for Federal Vulnerability Disclosure Guidelines, May 2023 (K. Schaffer, P. Mell, H. Trinh, I. Van Wyk) (read Nov. 15, 2026). Read it online. ↩
  44. IoT Cybersecurity Improvement Act of 2020, Pub. L. 116-207 (Dec. 4, 2020), sections on guidelines for the reporting, coordinating, publishing and receiving of information about security vulnerabilities, Congress.gov (not re-read for this lesson) (checked Nov. 15, 2026). Read it online. ↩
  45. CISA, VDP Platform Annual Report (2022 data), Aug. 21, 2023 (67 more reports and 15 more valid reports per quarter after onboarding; 38-day average remediation; KEVs found that existing scanning tools missed) (read Nov. 15, 2026). Read it online. ↩
  46. CISA, Secure by Design Pledge, goal on vulnerability disclosure policy (publish within one year of signing; authorize public testing; no legal action for good-faith efforts) (read Nov. 15, 2026). Read it online. ↩
  47. U.S. Department of Justice, Criminal Division, Computer Crime and Intellectual Property Section, Cybersecurity Unit, A Framework for a Vulnerability Disclosure Program for Online Systems, Version 1.0, July 2017 (read Nov. 15, 2026). Read it online. ↩
  48. E. Foudil and Y. Shafranovich, RFC 9116, "A File Format to Aid in Security Vulnerability Disclosure" (security.txt), Internet Engineering Task Force, April 2022, informational (read Nov. 15, 2026). Read it online. ↩
  49. European Union Agency for Cybersecurity (ENISA), "The CRA Single Reporting Platform is launched," Sept. 11, 2026 (manufacturers' reporting of actively exploited vulnerabilities and severe incidents applies from Sept. 11, 2026; main requirements from Dec. 11, 2027) (read Nov. 15, 2026). Read it online. ↩
  50. "Pentagon Received Over 50,000 Vulnerability Reports Since 2016," SecurityWeek, March 18, 2024 (DoD Cyber Crime Center VDP launched November 2016; about 4,000 researchers and close to 45,000 reports by end of 2022, more than 25,000 judged valid; 12-month defense industrial base pilot, more than 1,000 vulnerabilities, estimated $61 million saved) (read Nov. 15, 2026). Read it online. ↩
  51. United States v. Sullivan, No. 23-927 (9th Cir.), opinion filed March 13, 2025; order denying rehearing en banc and amended opinion filed Nov. 12, 2025 (read for module 108 on Nov. 13, 2026). Read it online. ↩
  52. U.S. Securities and Exchange Commission, EDGAR full-text search (filings from 2001 onward) (checked Nov. 15, 2026). Read it online. ↩
  53. Free Law Project, CourtListener (free federal and state court opinions and RECAP docket archive) (checked Nov. 15, 2026). Read it online. ↩

Check yourself

1. Which kind of testing is designed mainly to show whether an organization would detect and stop a realistic, goal-driven adversary?

2. Under PCI DSS v4.0.1, how often must internal and external penetration tests be performed?

3. What does 23 NYCRR 500.5 require of DFS-regulated companies?

4. In Van Buren v. United States (2021), what did the Supreme Court hold about 'exceeds authorized access'?

5. Which statement about the Justice Department's May 19, 2022 CFAA charging policy is correct?

6. What failure did the SEC identify in its June 14, 2021 order against First American Financial?

7. What did the October 2019 review of the Iowa courthouse arrests find?

8. Under CISA BOD 20-01, what must a federal agency's vulnerability disclosure policy include?

9. How did Uber misuse its bug bounty program in 2016, according to the FTC?

10. Since its February 2025 update, what does TIBER-EU require at the end of each threat-led test?

Terms in this lesson

Penetration test
An authorized, time-limited attempt by qualified people to get past an organization's protections within a written scope, to show what an attacker could actually reach.
Red team exercise
A longer, goal-driven simulation of a specific kind of adversary, known to only a small control group, that tests detection and response as well as prevention.
Purple team exercise
A cooperative exercise in which testers and defenders work together to check and tune whether specific attacker behaviors are detected.
Breach and attack simulation (BAS)
Software that runs safe, scripted imitations of known attacker behaviors on a schedule and reports which were blocked or detected.
Rules of engagement (ROE)
The written operating rules for a test: scope, permitted and forbidden activities, timing, contacts, stop conditions, data handling and reporting.
Authorization letter
A signed statement from someone with authority over every asset in scope that permits named testers to perform named activities during set dates.
Good-faith security research
In the Justice Department's 2022 policy and the copyright exemption, accessing a computer solely to test, investigate or correct a security flaw, in a way designed to avoid harm, with the information used primarily to promote security.
Vulnerability disclosure policy (VDP)
A public statement telling outsiders where and how to report security flaws, what research is authorized, what the organization will do, and that it will not pursue good-faith reporters.
Coordinated vulnerability disclosure
The practice in which a finder reports a flaw privately, the owner fixes it, and both agree when and how it becomes public.
Bug bounty program
A disclosure program that pays rewards, set in advance by severity, for valid reports about in-scope systems.
Safe harbor
Language in a VDP or bounty stating that research under the policy is authorized and that the organization will not pursue or support legal action against good-faith researchers who follow it.
Threat-led penetration testing (TLPT)
Regulator-supervised red team testing of live critical systems based on threat intelligence about who targets the institution, as in CBEST, TIBER-EU and DORA.
security.txt
A small machine-readable file at /.well-known/security.txt on a website, defined in RFC 9116 (2022), that tells finders where to report vulnerabilities.
Retest
An independent check, after a finding is reported fixed, that the fix works; PCI DSS requires repeat testing to verify corrections.

Full glossary

Sources