Using Glasshouse

Glasshouse runs security tests against AI systems and applications and reports what it finds. Alongside the findings it keeps every prompt it sent, every response it got back, and what the testing cost.

The tests

Glasshouse offers these tests:

• LLM safety testing (garak). Sends adversarial prompts to a model's address — jailbreaks, prompt injection, encoding tricks and the like — and checks whether the model resists them.
• Application penetration testing (Strix). Explores an application and its code the way an attacker would, and reports the weaknesses it can actually demonstrate. It can test a running application at its web address, a GitHub repository from an account you have connected, or a .zip archive you upload (up to 200 MB, or less on some plans).
• Answer-quality evaluation. This exists, but is not yet offered from the screens.

Scans and single tests

A scan is one target and any number of tests, run side by side. Use it when you want a full picture of one system in one report.

A single test runs one tool on its own. You start one from the Single tests screen.

Before you start a test

Before you start, the form shows an estimate of how long the test will take, what it will cost, how many tokens it will use and how complex it is. Once you have run some tests, the estimate is based on those past runs; until then, it is based on starting assumptions.

Spending on application penetration testing is capped automatically, from the size of the code and from past runs. The cap is a limit, not a charge.

While a test runs

A running test's progress updates live, so there is no need to refresh the page. You can stop a test at any time.

A scan can be stopped, or deleted. Your own scans are yours, including the ones the assistant started for you. Scans requested by the planner agent are visible to everyone in your organisation, and cannot be deleted here.

Reading the report

Each scan ends in a report. It has:

• A headline verdict — the short answer.
• What the scan shows, and what it cannot show — so you know how far the result goes.
• Next steps — what to do about it.
• Confirmed vulnerabilities — each with its severity.
• How to fix — under each confirmed vulnerability, what to change and how to check the fix worked.
• What was tested, and coverage gaps — what was not.
• Results to review — things the scanners noticed that are not vulnerabilities until a person confirms them, with a note when one looks like a false positive.

Once the scan has ended you can download the report as a PDF, if your plan includes PDF reports. Stored evidence files can be downloaded too, with any secrets in them masked.

Conversations and costs

Each test has a Conversation tab showing every prompt sent and every reply received, in full, so you can see exactly what happened.

Costs list every call made to a model. When the price of a call is not known, it is shown as unknown — never as zero, so a total never looks smaller than it really was.

Connecting your code

To test code rather than a running application, give Glasshouse access to it in one of two ways:

• Connect GitHub. Either install the Glasshouse app on a GitHub account, or grant access by signing in with GitHub.
• Upload an archive. Upload your code as a .zip file.

Disconnecting a GitHub account makes Glasshouse forget it. If you installed the app, it stays installed on GitHub until you remove it there.

Using the assistant

Signing in opens the assistant. It lets you do everything the other screens do by chatting: ask what can be tested, start a scan, follow it, and read what it found. Write your first message in the box and the conversation opens; attach a .zip of your code to a message to have it scanned. New chat at the top of the sidebar starts another, and your recent conversations are listed at the bottom of it. Rename or delete a conversation from the menu above it. Scans in the sidebar lists your scans.

Nothing that spends money or deletes data happens on the assistant's word. Starting, stopping or deleting a scan, and deleting an upload, each show a card first. Read it, tick each box to confirm you understand it, and approve or reject it. A card that is not answered within 15 minutes expires; ask again for a fresh one.

When a model endpoint needs an API key, add it with Add an API key rather than typing it into the message. It is encrypted, never shown in the conversation, and can only be used in that conversation, for a day. A key typed into a message is masked and cannot be used.

Replies can include text quoted from what a scanned system said, so links in a reply are shown as their address rather than as something to click, and images are never shown.

If the assistant has not been switched on for your deployment, the first screen says so; everything it does can also be done from the other screens.

Your plan

Plan and usage shows which plan you are on, what you have used this month, and what the plan includes: the number of scans a month, how much can be spent on tests, how many tests can run at once, which tests are included, whether GitHub repositories and PDF reports are included, the largest archive you can upload, and how many messages the assistant can take a month. A test run on its own counts as a scan. What the assistant spends on its own replies is shown separately and does not count against your spending.

When your plan does not include something, the forms say so instead of offering it, and if a limit is reached the screen tells you which one. To raise a limit, upgrade your plan or contact support.