Introduction

Freedom of information, in general, refers to the inherent right of individuals to have access to information. In the context of government information, freedom of information refers to the rights of citizens to know what the government is doing and to view government records. That being said, in the federal government access to information is largely driven by politics. The president is responsible for setting policy regarding government secrecy and openness in the executive branch. Some presidents have argued that government secrecy is necessary to maintain national security, while others have tried to establish more liberal policies. Of course, politics plays an important role in government secrecy and openness, too. In addition, Congress and the courts have important influence when it comes to freedom of information legislation and enforcement of laws relating to open government. At the same time, access to information about the deliberations and behind-the-scenes decisionmaking in the judicial and legislative branches is practically non-existent.

Security Classification

Classification refers to the practice of designating security levels for types of information or individual documentation. Documents may be classified as confidential, secret, top secret, or restricted data (a classification used by the Department of Energy). Classification and declassification is ultimately determined by the president’s policies. Agencies have developed classification manuals to guide classification designations. Classified information is subject to declassification schedules that mandate automatic declassification after a certain number of years have elapsed. However, it is possible to circumvent the declassification schedule using security designations other than confidential, secret, and top secret. For example, the Bush administration used Sensitive but Unclassified (SBU) and the Obama administration used a category called Controlled Unclassified Information (CUI). Documents so designated are not technically classified, so they do not have to be declassified according to a schedule.

Declassification falls into three categories:

• Automatic declassification is driven by presidential Executive Order 13526 of 2009 and is triggered when documents reach an age threshold of 25 years. However, agencies can exempt or exclude documents from declassification.
• Systematic review occurs in those cases where agencies have excluded documents from automatic declassification.
• Discretionary declassification may be done in response to public interest or if an agency determines that a category of information no longer needs to be kept secret prior to becoming subject to automatic declassification.

Automatic declassification is something of a misnomer. In the automatic declassification process, material must still be reviewed to ensure that personally identifiable information (PII) is not revealed, that material that is still classified is not referred to, and that agreements with foreign governments about keeping information secret are maintained. This extensive review process results in enormous backlogs of material scheduled for declassification.

In order to speed up the declassification review process, the National Declassification Center (NDC), part of the National Archives and Records Administration (NARA), settled on a policy of selective declassification of the most requested records series rather than declassifying an entire category of records at once. NARA receives input on this from a public interest declassification board. Citizens are invited to provide input on declassification priorities.

The best evidence that automatic declassification is an ideal, not a reality, is that in 2011 the CIA released the last declassified documents dating back to World War I (that’s over 100 years old!) related to invisible ink technology.^[1] There are still hundreds of millions of pages of documents awaiting review from World War II and even earlier.

Once a document has been declassified, does it stay declassified? The public learned after September 11, 2001, that documents could be returned to classified status when information about nuclear arsenals was removed from public view. In most cases, though, once a document has been released it is likely to be made available by an organization that specializes in disseminating declassified documents. As a result, a declassified document could remain publicly available in spite of efforts to reverse classification decisions. To illustrate this point, many years ago depository libraries received a book about IRS criminal investigations. A recall was issued when it was discovered that the document was for internal use only because it revealed investigative techniques that might be of use to criminals. A librarian who filed a FOIA request for the document received a heavily redacted version. However, another individual scanned the entire unredacted document and it is now available in HathiTrust.^[2]

Another complicating factor in declassification is a law known as the Kyl-Lott Amendment to the National Defense Authorization Act of 1999. It required that agencies conduct page-by-page reviews of records eligible for automatic declassification, and even previously declassified material, to check for Restricted Data and Formerly Restricted Data related to information about nuclear weapons. Artificial intelligence (AI) can be of some use in streamlining review of documents, but AI is not capable of reading some documents with faint typescript, text in foreign languages, handwritten notes, or overstamping.

Sources of Declassified Documents

Electronic Sources

• Foreign Relations of the United States (FRUS) is published by the Historical Office of the U.S. State Department. The volumes are currently published about 30 years following the events they cover and each volume covers a specific time period and geographic area or subject. The volumes consist of unclassified and declassified diplomatic correspondence, cables, and other documents that pertain to foreign relations. The contents include documents from the State Department, the CIA, presidential libraries, and other government sources. FRUS is available from the State Department, HeinOnline, HathiTrust, and other online sources.
• WikiLeaks is a repository of mostly restricted material obtained by the public interest site through leaks and other means. Researchers can search the full text of documents or browse by categories.
• Digital National Security Archive (ProQuest) The National Security Archive, based at Georgetown University, obtains documents on selected topics through FOIA requests. Most documents date from 1945 to the present. The archive holds many unpublished documents that must be viewed in person in its reading room. Part of its collection of declassified documents is available on its website, but libraries must subscribe to the database available through ProQuest to access its curated topical collections, searchable through added metadata.
• ProQuest’s History Vault database includes confidential files from the State Department, CIA, FBI, and military intelligence files, documents from the British Foreign Office, and other sources covering the 19th and 20th centuries.
• Declassified Documents Online (Gale) began as a microfiche set published by Carrollton Press, which collected and indexed documents that were declassified as a result of President Nixon’s Executive Order 11652 in 1972, which established the General Declassification Schedule.^[3] It features selected declassified documents obtained from presidential libraries and federal agencies. Its editors monitor presidential libraries for the release of newly declassified documents appropriate for inclusion in the database. Most of the content is presidential documents but it also includes material from the State Department, Department of Defense, CIA, FBI, National Security Council, and other agencies.^[4]
• Archives Unbound (Gale) offers access to State Department and FBI documents among many other primary sources.
• One of EBSCO’s digital archive modules is United States Bureau of Investigation Case File Archives, a collection of files from the predecessor of the FBI.
• GovernmentAttic.org hosts declassified documents free of charge. Content includes FOIA logs, which document all of the FOIA requests received by an agency during a defined period; lists of inspector general decisions; and a variety of other documents. Coverage dates from the mid-2000s to the present.
• Electronic reading rooms maintained by federal agencies and presidential libraries provide access to selected declassified documents. Usually, there is a link for the electronic reading room on the agency’s home page. For example, the FBI’s electronic reading room is called The Vault. Researchers must visit FOIA reading rooms in person to view documents that have not been published online.

The following additional sources may be used to view records that are not available electronically:

National Archives

The National Declassification Center at NARA publishes lists of recently declassified documents, and citizens are invited to provide input on which groups of records should be prioritized for declassification. Some of the declassified documents are later published in the subscription databases listed above, but for access to the bulk of declassified material, a trip to the archives is necessary.

Microform Collections

The Center for Research Libraries (CRL) holds many microfilmed sets of documents such as Documents of the National Security Council that were originally published by University Publications of America (now owned by ProQuest and available electronically through its History Vault product mentioned above). Documents can be requested from CRL through interlibrary loan. Finding aids or indexes to these collections are usually available online even though the contents of the microfilm or microfiche are not.

Gale’s Primary Source Media offers microfilm collections of John F. Kennedy’s national security files, U.S. State Department records, and other collections. Searchable finding aids are available online.

Freedom of Information Act (FOIA)

Dating to 1966, FOIA gives citizens the right to request documents from federal agencies. It is important to remember that FOIA only applies to the executive branch. There are nine exemptions that agencies may use to decline FOIA requests:

1. Information that is classified to protect national security.
2. Information related solely to the internal personnel rules and practices of an agency.
3. Information that is prohibited from disclosure by another federal law.
4. Trade secrets or commercial or financial information that is confidential or privileged.
5. Privileged communications within or between agencies, including those protected by the:
   • Deliberative Process Privilege (provided the records were created less than 25 years   before the date on which they were requested)
   • Attorney-Work Product Privilege
   • Attorney-Client Privilege
6. Information that, if disclosed, would invade another individual’s personal privacy.
7. Information compiled for law enforcement purposes that:
   • Could reasonably be expected to interfere with enforcement proceedings
   • Would deprive a person of a right to a fair trial or an impartial adjudication
   • Could reasonably be expected to constitute an unwarranted invasion of personal privacy
   • Could reasonably be expected to disclose the identity of a confidential source
   • Would disclose techniques and procedures for law enforcement investigations or prosecutions, or would disclose guidelines for law enforcement investigations or prosecutions if such disclosure could reasonably be expected to risk circumvention of the law
   • Could reasonably be expected to endanger the life or physical safety of any individual.
8. Information that concerns the supervision of financial institutions.
9. Geological information on wells.^[5]

Although FOIA applies to agencies in the executive branch, courts have ruled that certain entities within the Executive Office of the President that perform solely administrative or advisory functions are not agencies within the meaning of FOIA, namely the Council of Economic Advisers, Office of Administration, and National Security Council.^[6]

 

Let us examine some examples of information that may be withheld. The applicable exemptions are indicated in parentheses.

Intelligence Budget (National Security)

Researchers sometimes ask how much the United States spends on covert operations or other intelligence activities. The national intelligence budget was secret until 2012, and there is still no detail in terms of how much is spent on each function or program. It is easy to disguise intelligence expenditures. For example, during the Cold War the CIA established phony foundations to funnel money to anti-Communist propaganda programs in the United States.^[7] The CIA is an expert in hiding funds so that they cannot be traced back to the agency. To complicate matters, much of the intelligence work of the nation is now performed by defense contractors. Although one can learn how much goes to each contractor in total, one cannot tell what portion of the funds paid to these contractors is for defense work vs. intelligence work.

Geospatial Data (National Security)

An important category of information that may be subject to secrecy is geospatial data. The National Geospatial-Intelligence Agency, or NGA, is the official government agency that collects and controls the release of geospatial intelligence. Examples of geospatial data that may be withheld or redacted include detailed maps or imagery of military installations that might reveal functions or activities such as munitions storage, chemical weapons, or weapons testing.

You may have heard of Area 51, a large Air Force base near Groom Lake in Nevada that was an open secret for many years. Some people believe that Area 51 was where extraterrestrials were taken for study in the 1940s, but the consensus of military experts is that it is used for testing experimental aircraft. Satellite imagery of Area 51 was withheld or redacted for many years, fueling speculation about its hidden activities. Historical information about Area 51 was finally released in 2013 in response to a FOIA request. Apparently, the intelligence agencies determined that there was so much information already publicly available that they might as well release some basic information.^[8]

Biosecurity and Environmental Hazards (National Security; Trade Secrets)

Biosecurity and hazardous chemicals constitute another category of information that may be withheld or restricted. Understandably, there is concern about releasing detailed information about biopreparedness, security systems, and the nation’s vulnerability to attack. The Emergency Planning and Community Right-to-Know Act of 1986 (EPCRA) requires manufacturers who use or manufacture hazardous chemicals to report the chemicals to states. Emergency responders need access to this information so that they can respond appropriately in an emergency. Large amounts of certain chemicals must also be reported to the EPA and risk management plans must be on file. However, the reports and plans are not posted online where they could be consulted by emergency planners or the general public. In addition, EPCRA permits reporting facilities to withhold information about a chemical if it is a trade secret. Similarly, risk assessments and emergency plans for nuclear power plants could be used by terrorists to learn of vulnerabilities and help them circumvent security measures. On the other hand, area residents may not be informed about how vulnerable these plants could be.

Personal Information (Privacy)

Names of confidential informants, names of people involved in human subjects research, and information like social security numbers may be redacted, or entire documents containing such information may be withheld. Agencies sometimes make the argument that even if names have been redacted, it may be possible for a reader to determine someone’s identity from the context, so agencies may decline to release an entire document or redact large sections, not just the individuals’ names.

Law Enforcement

Exemptions to FOIA related to law enforcement include procedure manuals, internal rules, and interagency communications. Even an agency’s internal procedures for handling FOIA or Privacy Act (PA) requests can be secret, so that the public has no way of understanding the thought processes or internal policies that guide the handling of these requests. In fact, some FOIA experts believe that agencies are deliberately secretive about their internal handling of FOIA or PA requests so that they don’t give any clues to potential requestors about how to construct their requests more effectively.

Political Secrecy

Many people believe that a lot of government information is withheld not because of the legitimate reasons that fall under the exceptions to FOIA, but because of political secrecy. An example of alleged political secrecy involves a book written by a former FBI agent claiming that if the FBI had done a better job of investigating terrorists prior to the attacks on U.S. targets on September 11, 2001, the attacks might not have occurred. The FBI forbade publication of the book, but it is hard to say whether the FBI’s objections were politically motivated or based on national security concerns.^[9] Government employees who work with classified information are typically subject to non-disclosure agreements. Therefore, if a former CIA agent were to write a memoir, it would be subject to review by the agency, and if the agency objected to certain sections, the author would have to omit them.

FOIA Improvement Act of 2016

The most recent update to FOIA occurred with the FOIA Improvement Act of 2016 (Public Law 114-185). Among other modifications that were designed to streamline the FOIA process and reduce backlogs, it mandated a “presumption of openness,” required agencies to publish released documents, and limited the applicability of Exemption 5 to documents produced less than 25 years earlier. Prior to the 2016 law, the Open Government Act of 2007 established the Office of Government Information Services (OGIS), designed to serve as the FOIA ombudsman to mediate disputes between agencies and requesters.

Using FOIA to Request Documents

Both FOIA.gov and the National Security Archive offer guidance and tips for FOIA requesters. Before submitting a FOIA request, it’s important to verify that the desired documents are not already publicly available by reviewing the documents available through each agency’s FOIA reading room and checking the other sources listed above. Sometimes, agencies can be of help by informing researchers directly that a FOIA request is necessary. Note that fees charged for research and copies may vary depending on the affiliation of the requester.

Even with the legislative improvements to FOIA, there is no centralized administration of FOIA requests, so there are inconsistencies between how agencies handle requests. These inconsistencies have sometimes resulted in the very same document being released by a presidential library but withheld by the CIA, for example. Redactions of the same document released by two different agencies may be wildly different due to differences in declassification policies or simply the personal characteristics of the declassifier. One strategy employed by successful FOIA requesters has been to request the same documents from multiple agencies, then use the releases of one agency to argue for the release of additional information from other agencies.

FOIA.gov allows you to generate a basic report telling how many FOIA requests each agency received and how many were processed, but it doesn’t reveal how many were denied. It is difficult to determine whether backlogs have decreased because agencies are getting through more requests or because they are simply saying “no” and not spending time looking for the information requested in the first place.

Tools for Filing FOIA Requests

• MuckRock, a non-profit collaborative news site, offers the FOIA Machine, which allows researchers to submit and track FOIA requests at both the federal and state/local levels. It can be viewed as a sort of collaborative research site for FOIA requests, where researchers can share their inquiries and results with other researchers.
• FOIA.gov provides an interface to submit FOIA requests to participating agencies. However, some agencies do not currently allow requesting through FOIA.gov. In those cases, FOIA.gov provides instructions for submitting requests directly to the agency.

Mandatory Declassification Review (MDR)

MDR is another method to request access to documents, specifically those that are classified. Under the provisions of Executive Order 12958, researchers can request that agencies declassify specified documents before they are subject to automatic declassification. The process can take longer than a FOIA request because an agency has up to one year to respond to an MDR request. In addition, MDR requests must be very narrowly constructed, whereas FOIA requests can be much broader. Furthermore, when an MDR request is denied, the requester cannot sue, but the decision can be appealed to a review board.

Leaks

In addition to documents being released through MDR or FOIA requests, a third avenue for the release of documents is leaks. The Pentagon Papers, leaked by Daniel Ellsberg, is one of the most famous examples of classified documents that were leaked to the news media. WikiLeaks has obtained and published over 10 million pages of restricted and unrestricted government and other documents by obtaining them from individuals who had access to them. News organizations occasionally obtain and republish leaked documents or compile summaries of them.

 

 

Privacy Act

The Privacy Act (PA), passed in 1974, is an attempt to document and make known the records systems under agencies’ control that collect personally identifiable information. It also establishes a process through which agencies must publish information about their systems of records in the Federal Register, and they must maintain lists of systems and notices of changes to these systems. PA issuances are also compiled every two years and are a collection in GovInfo. The PA gives individuals the right to request government records about themselves and allows individuals to request amendment or correction of records about them. If you were to request information about the IRS’s files on you, that would be a PA request, not a FOIA request. FOIA would cover requests for other types of information like systems, procedures, or statistics compiled or maintained by the IRS.

FOIA in Practice

Athan Theoharis is a researcher who has written several books about the FBI. This section draws on his book A Culture of Secrecy: The Government Versus the People’s Right to Know, which is a discussion of several researchers’ experiences using FOIA, mostly with the FBI.^[12] Theoharis interviewed several authors who had done research using FBI files. In addition to fighting crime, the FBI has investigation files on subjects like Prohibition, organized crime, investigations of people suspected of being disloyal (e.g., Germans, Italians, Japanese, Communists), campaigns to discredit people like Martin Luther King, Jr., and files on many celebrities. Theoharis enumerated reasons why FOIA requests to the FBI were unsuccessful:

• The volume of information was too great.
• It was too difficult to write a request that only resulted in files the requesters wanted.
• Requesters sometimes couldn’t afford fees charged by the FBI.
• Finding aids were not very helpful or were unavailable because they were classified or secret (for example, they contained names of informants).
• The FBI sometimes claimed not to have files that later turned up.
• In some cases, the FBI refused to release documents that had already been released by presidential libraries, even when this was pointed out to them.

One researcher profiled in Theoharis’s book wrote about trying to find the arguments the government devised to prevent John Lennon, a former member of the Beatles, from immigrating to the United States. After Lennon’s death in 1981, the FBI took ten years to release his file, claiming the national security exemption. It’s hard to imagine national security considerations in Lennon’s case, so the researcher concluded that the real motivation was probably political.

The CIA is especially notorious for not being forthcoming. Authors reported that they got the “Glomar response” from the agency. One example of the frustrations of seeking CIA documents relates to MK-ULTRA, which was a 1950s-1960s program of experiments related to mind control, including dosing unsuspecting people with LSD. The CIA destroyed files related to MK-ULTRA, so it was not possible to find out who in the CIA was involved or the identities of the unwitting subjects. Later, however, the CIA found 8,000 pages of documents related to the experiments, leading to questions about the agency’s credibility when they claimed not to have the documents. Like the FBI, the CIA has refused to release documents that were previously released by presidential libraries. Judges have tended to side with CIA when their practices have been challenged in court.

The FRUS series, mentioned above, is the diplomatic history of the U.S. Volumes are supposed to be published within 30 years of events they describe, but the time gap has increased in recent years, mainly because of the declassification process done by military intelligence agencies and especially the CIA. Subsequent releases of declassified documents have prompted revisions of certain volumes. A volume covering relations with Iran during 1951-1954 is a case in point. Initially published in 1989, the volume lacked key information about U.S. covert operations in Iran. The State Department subsequently produced a revised volume in 2016, but the Secretary of State at the time, John Kerry, did not permit its publication due to concern about negotiations with Iran related to its nuclear programs. Finally, the volume was released in 2017.^[13] Previously, volumes concerning Indonesia and relations between Cyprus, Greece, and Turkey had been withheld by the State Department due to political sensitivity surrounding U.S. covert operations. ^[14]

One need look no further than the National Security Agency (NSA) for examples of between-agency inconsistency in releasing documents. Most of the NSA’s documents are exempt from disclosure. The agency has withdrawn documents from presidential libraries, but later copies of some of them were found at the National Archives. Its fees can be prohibitive, and it can take up to five years to get any sort of response from the NSA. The NSA refused to release documents related to the Gulf of Tonkin incident in 1968, in which a CIA spy ship claimed it was fired upon by a North Vietnamese vessel. The incident was used as an excuse to escalate U.S. involvement in Vietnam. Even though some of the documents had been released in the Pentagon Papers, the NSA refused to release related documents.

Open Government

The Sunlight Foundation, an organization that promotes transparency and open government, identified the characteristics of open government as completeness, primacy, timeliness, ease of physical and electronic access, machine readability, non-discrimination, use of commonly owned standards, licensing, permanence and usage costs.^[15] President Obama promised to have the most transparent administration ever, and early in his presidency he issued a memorandum on transparency and open government that reversed some of the policies established by president George W. Bush. In 2013, this policy was further refined with an executive order, Making Open and Machine Readable the New Default for Government Information, designed to make government data more accessible. The goals of Obama’s policy were to:

• Publish government information online
• Improve the quality of government information
• Create and institutionalize a culture of open government
• Create an enabling policy framework for open government^[16]

Agencies were also required to create open government plans that detailed how they would open their records and data. In spite of his directives, President Obama was not successful in changing the institutional culture of the executive branch. In fact, experts agree that government secrecy is entrenched, and no president has managed to effect sweeping changes that would result in a truly open government. Scholars and journalists generally agree that the administration of President Donald J. Trump tended not to emphasize transparency. At the same time, the Trump administration did not make any significant policy decisions affecting transparency.^[17] The Biden administration has made statements supporting open government but similarly has not made any sweeping changes to existing policies.

Open Data

An important component of open government is open data. In addition to Data.gov, various agencies have open data portals to provide access to their data products. In 2013, the White House Office of Science and Technology Policy (OSTP) directed agencies with at least $100 million in annual research and development to create plans for providing public access to peer-reviewed scientific publications and data resulting from federally funded research. In response, agencies created public access plans detailing the means by which they would comply with the directive. For instance, the Department of Energy (DOE) unveiled DOE Pages, a database of journal articles and accepted papers funded by DOE.

Congress and Open Government

As we have seen, FOIA does not apply to Congress, but there are other laws that require the disclosure of certain information by members of Congress. The federal Lobbying Disclosure Act of 1995 (LDA) requires lobbyists to register and report contributions. Furthermore, former member of Congress who work as lobbyists must report their contacts with sitting members.

The House and Senate have rules on gift disclosures, financial disclosures, legal expenditures reporting, and reporting on foreign travel expenditures. Note that privately funded foreign travel is not reported on the travel expenditures report; the member must list the trip on a gift disclosure. Both the House and Senate provide access to recent disclosure reports on their websites. Historical financial disclosure reports are available in the U.S. Congressional Serial Set. Government-funded travel expenditures are reported in the Congressional Record and are also available as stand-alone reports. It is striking that members of Congress must report their own and their spouse’s income, banking and investment accounts, real property, debts, and assets, while the president is not required to do so.

Conclusion

As we have seen, even though there is a tremendous amount of information available online, it is only the tip of the iceberg. Our duty as librarians is to help patrons understand which types of information are readily available, what must be requested, and what is not likely to be released. Most patrons will be constrained by time or money, in which case our task is to help them access the available relevant information.

Insider’s Library

Ginsberg, W., Carey, M. P., Halchin, L. E., & Keegan, N. (2012). Government transparency and secrecy: An examination of meaning and its use in the executive branch. Congressional Research Service. https://fas.org/sgp/crs/secrecy/R42817.pdf

This report provides a synopsis of federal laws related to transparency and secrecy in the executive branch.

National Security Archive. (2008). Effective FOIA requesting for everyone. https://nsarchive2.gwu.edu//nsa/foia/foia_guide/foia_guide_full.pdf

The staff at National Security Archive has abundant experience in requesting information through FOIA. This publication is considered the most comprehensive guide to submitting FOIA requests, with tips for using the most effective techniques to produce the desired documents.

Barczewski, B. (2024). The Freedom of Information Act (FOIA): A legal overview. Congressional Research Service. https://crsreports.congress.gov/product/pdf/R/R46238

Written for members of Congress, this overview explains exemptions to FOIA, reviews FOIA-related litigation, and describes how FOIA does not preclude Congress’s special access to agency documents.

Syracuse University. A Guide to Accessing Public Records for the First Time. https://onlinegrad.syracuse.edu/blog/accessing-public-records-foia-guide/

Syracuse’s guide gives some background about FOIA and offers five tips for first-time requesters.