The Birth and Testing of a New AI Tool for Market Research
Artificial intelligence (AI) is showing promise for streamlining the pre-award and administration phases of the contracting lifecycle.
By Scott Simpson
With most acquisition attention focused on improving solicitation and award, streamlining the pre-award and administration phases of the contracting lifecycle has received less attention. Now, the Department of Homeland Security (DHS) Procurement Innovation Lab (PIL) is leading efforts to streamline these phases using emerging technology.
The PIL endeavors to streamline the procurement process to improve mission outcomes.1 In 2019, it partnered with the Office of Federal Procurement Policy (OFPP) to investigate whether artificial intelligence (AI) could be used to streamline the past-performance evaluation process.
Under the authority of the commercial solutions opening pilot program (CSOP),2 the PIL awarded contracts to nine vendors to build prototypes that would test the following:
➢ Can a Contracting Officer upload a solicitation or statement of work?
➢ Can that document be machine-read?
➢ Can pertinent information from that document be compared to the Contractor Performance Assessment Reporting System (CPARS) reports for specific vendors who responded to the related solicitation?
➢ Can the emerging technology identify the most relevant reports from each specific vendor for the Contracting Officer to review?
The goal of the initiative was to build a competitive marketplace of commercial solutions that support the acquisition workforce in the evaluation of past performance.3 After just four months, seven of the nine vendors were able to show working prototypes that demonstrated the idea’s feasibility.
Following this feasibility assessment, the PIL expanded the partnership with OFPP to include nine additional federal agencies and began building AI for Past Performance (CPARS AI) solutions with a subset of the vendors that had delivered viable working prototypes.
DHS formed this coalition of PIL partners on the principle that commercial solutions ideally will be built in collaboration with users and used for all federal agencies. That approach increases the value of the solutions due to diversity of end-user design, decreasing costs, and avoiding redundancy and duplication.
Barely a year later, four of the vendors had produced working versions of software that could ingest a solicitation or requirements document, use machine learning and natural language processing to compare it to a particular vendor’s CPARS reports, and accurately identify which CPARS reports were most relevant.
Building on AI for Past Performance
Throughout these first two phases, the PIL team used agile approaches to engage directly with the front-line acquisition staff (contracting
officers, contract specialists, program managers, and contracting officer representatives) – the software’s end users.
The team performed discovery, held user engagement sessions, and received user feedback on incrementally developed designs, prototypes, and software. During these engagements, end users continually commented that the same tools would be extremely valuable in supporting market research.
Accordingly, in 2021, DHS awarded follow-on orders to three of the vendors to develop an add-on tool that would use AI to help acquisition teams begin market research.
The AI for Market Research spin-off initiative had four stages: discovery, design, test, and launch.
From October 2021 to January 2022, acquisition staff discussed with vendors the strategies and reasoning for performing market research.
This information can be found in many sources, including the Federal Acquisition Regulations (FAR), but hearing the user journey firsthand was critical to understanding how the technology could add value to, and improve, the efficiency of the process.
The “why” of market research seems simple: because the FAR requires it. The reality is far more complicated.
Acquisition teams perform market research for many reasons: to find vendors that can perform requirements, to assess whether small businesses can perform the work, and to determine whether there are existing contract vehicles (e.g., the General Services Administration (GSA) schedules, government-wide acquisition contracts, department-wide vehicles, etc.) with which to hire capable contractors or buy suitable products and services.
Some agencies perform market research to determine whether a company has worked in a particular country or location. Others might use it to determine whether an idea is possible. All these use cases were raised and discussed by users during discovery sessions.
With so many different motivations for market research, it was important to create a backlog of requirements, identify where they most
overlapped, and begin building the tools to address the most common requirements.
After the vendors understood the “why,” they wanted to identify the “how” to see where innovations such as AI might be applied. One commonality emerged. The current process is manual, time-intensive, and labor-intensive. It can involve researching GSA Schedules, sorting through data on USASpending.gov, and releasing a request for information (RFI) via SAM.gov. On average, each DHS acquisition workforce member spends more than 130 hours performing market research annually.4 
In this stage, the vendors began to tailor their CPARS AI software to performing market research that aligned with acquisition workforce requirements. They reused two pieces of the original software: the platform and the AI.
During the CPARS AI effort, all three vendors worked extensively with users to obtain the important information they needed to evaluate past performance. They used this background as a starting point for the market research effort. During the AI for Market Research design stage, constant user feedback on design and development ranged from how users entered search criteria to how the results were displayed and sorted.
One challenge was determining which sources of data most closely met the needs identified during discovery. SAM.gov houses not only entity registration but also information about award data from Federal Procurement Data Service Next Generation (FPDS -NG) and exclusion records (formerly Federal Awardee Performance and Integrity Information System [FAPIIS], now SAM Responsibility/Qualification Reports).
The vendors identified SAM.gov as an important resource because of the desire within the acquisition workforce to know which vendors had previously received awards to perform similar work, what types of businesses the vendors represented, and whether the vendors were currently able to accept new awards.
Another notable data source is USASpending.gov, which includes data about awards and consolidated information about each vendor’s awards (such as under which NAICS codes they have received awards and which awards – including GSA Schedules and other GWACs – they have received). Each acquisition workforce member spends more than 10 hours finding potentially capable vendors per market research engagement, according to a survey of DHS staff.
The vendors also worked to determine how best to visualize the data. One of the biggest market research challenges is dealing with the large amount of data available to the procurement team (for instance, lists of hundreds or even thousands of vendors sorted by GWAC, size, experience, and so on).
To streamline this information, the vendors created simple, easily understood visualizations. The survey of DHS participants found that for each market research engagement, acquisition team members spend seven hours, on average, identifying contract vehicles for capable vendors.
One obstacle that was not present during the development of CPARS AI solutions was the sheer volume of potentially capable vendors. During a CPARS evaluation, the government knows exactly how many offers it has received and thus works from a relatively small data set.
The entire point of market research, however, is to determine which companies, from the entire pool of possible vendors, are potentially capable of performing a requirement.
More than 100,000 vendors receive federal contracts each year, and many more are registered in SAM.gov. Filtering them by experience still could result in thousands of vendors. The AI for Market Research contractors had to determine how to display the most relevant and the most experienced potentially capable vendors. Some decided users might want to begin with fewer potentially capable vendors for review and showed 30 to 50 of the most relevant and potentially capable vendors. Others displayed more than 100. Time and testing will determine the number of results preferred by users.
In May 2022, the vendors were ready to begin testing their rebranded, reconfigured AI for Market Research tools. Our government-wide team of dedicated users and volunteers tried and gave feedback on every step of each solution: sign-in, search terms, user interface, and data display.
Some vendors tested and iterated for a solid three months. Others tested for a month, then iterated on the design for a few weeks before re-testing for a month. All promptly informed users whether and how their feedback was implemented. After the testing, all three vendors felt that they had solid first versions of their products.
All three vendors received ordering contracts in September 2022 under the CSOP authority. The ordering contracts included simple terms and a ceiling amount of $9 million for each vendor, which was the amount remaining under the CSOP threshold.5 This process permitted DHS to streamline both the award process and the commercial terms and conditions.
DHS also worked with OFPP and the Government-Wide Category Manager for Information Technology to ensure that the ordering contracts are available for use by all federal agencies.
Launch of the AI for Market Research solution as a multiple-award contract (MAC) was the logical next step and supported the original goal of minimizing building costs and ensuring the federal contracting community had easy access to the solutions for purchase and use.
Most acquisition professionals perform market research more than 10 times per year. It takes them a few hours to a few days, on average, to find capable vendors. The AI for Market Research project is currently in that stage. According to Section 107 of the Fiscal Year 2022 Department of Homeland Security Appropriations Act, pilots and demonstration programs must be reported to Congress when they meet certain thresholds.
DHS submitted the Section 107 pilot report in September 2022, simultaneously awarding three multiple-award ordering contracts, and placing orders for a few hundred licenses. These licenses were shared across several DHS Components, including the Federal Emergency Management Agency (FEMA), U.S. Citizenship and Immigration Services (USCIS), Customs and Border Protection (CBP), Immigration and Customs Enforcement (ICE), and several DHS headquarters offices.
Prior to issuance of the licenses and commencement of the pilot launch at DHS, participants were asked to take a pre-use survey (referenced above) to measure how long it currently takes to manually perform market research and assess its quality. Responses from 52 of the 140 pilot participants showed most acquisition professionals perform market research more than 10 times per year. (Figure 1)
On average, it takes them a few hours to a few days to find potentially capable vendors for a single procurement. Combining the survey results shows that buyers spend more than 80 hours per year identifying potentially capable vendors during the market research phase. (Figures 2, 3)
After identifying potentially capable vendors, it typically takes approximately seven hours on average to identify contract vehicles. This totals more than 55 hours per year on average.
Finally, more than 90% of respondents felt that the quality of manual market research was satisfactory or good but there was room for improvement. (Figure 4)
The survey results provide a baseline and will be repeated with similar questions at the end of the pilot phase to measure the use of the AI for Market Research tools. The results of the pilot will be shared with the wider federal acquisition community. The PIL hopes this pilot program will demonstrate that AI for Market Research software will reduce time spent on market research and improve its quality.
The PIL also hopes that the AI for Market Research program demonstrates how useful AI can be in the acquisition process and that these results will inspire other agencies and vendors to communicate with users and find new ways for AI to reduce the time spent in the acquisition process, increase competition, lower entry barriers to new and non-traditional vendors, and, ultimately, increase the likelihood of successful mission outcomes. CM
Scott Simpson is the Digital Transformation Lead at the DHS Procurement Innovation Lab. He is a graduate of American University’s Washington College of Law and School of International Service.
ENDNOTES
1 The Procurement Innovation Lab (PIL) is a DHS framework aimed at experimenting with innovative acquisition techniques across the DHS enterprise. The PIL provides a safe space to test new ideas, share lessons learned, and promote best practices. It fosters cultural changes that promote innovation and managed risk-taking through a continuous feedback cycle. As of January 2023, the PIL had coached more than 150 teams from solicitation release to award. You can learn more about the PIL at www.dhs.gov/pil.
2 The Commercial Solutions Opening Program (CSOP), from Section 880 of the National Defense Authorization Act (NDAA) for Fiscal Year 2017 (Pub. L. 114-328), allows DHS, GSA, and DOD to procurement commercial solutions that will be used in an innovative way (such as a new technology or a new application of an existing technology). More information about this authority can be found at https://www.fai.gov/content/commercial-solutions-opening.
3 The first phase of this initiative was summarized in an article published in the December 2020 NCMA Magazine, “Artificial Intelligence for Past Performance: How Emerging Technology is Advancing Good Government.”
4 DHS surveyed 140 acquisition workforce members from five DHS components that are participating in the AI for Market Research pilot program. The survey included 10 questions that are discussed below in more detail and had a 37% response rate. This is an unpublished survey.
5 At the time that Ordering Contracts were executed the CSOP Authority required awards to be $10 million or less. Legislation has since been passed that increased this threshold.