Metadata-Version: 2.4
Name: webtarpit
Version: 0.7.1
Summary: Scanner honeypot: hallucinate a static HTML site per URI, vhost, and device
Author: Tactical Data Concepts
License: MIT
Keywords: honeypot,tarpit,scanner,static-html
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Dynamic: license-file

# webtarpit

A **scanner honeypot**. It listens on HTTP, looks at the request path, and asks an OpenAI-compatible model (Ollama, Open WebUI, LiteLLM, …) to invent a **static HTML** page that looks like it belongs at that URI. Pages are saved on disk and reused. One listener can serve **several Host names**, each with its own topic and persistence mode. Optional **per-device** sites follow a client across IP changes using passive HTTP fingerprints (header order, User-Agent, Accept*, Client Hints) plus optional HttpOnly cookie and ETag stickiness — no JavaScript.

Generated pages are **sanitized**: no JavaScript, no event handlers, no forms, no iframes. The process never executes request bodies or generated markup. Graphics are inline SVG (or data-URI images). After new pages appear, older pages can gain extra links so the invented site stays a web, not a pile of orphans.

Point a Cloudflare Tunnel (or any reverse proxy) at the listener. Bind **`0.0.0.0`** so LAN origins are reachable.

Built by [Tactical Data Concepts](https://tacticaldataconcepts.com). Free download.

## Install

**Interactive (recommended):**

```bash
curl -fsSL https://tacticaldataconcepts.com/tools/webtarpit/install.sh | bash
```

The script prompts on `/dev/tty` (so `curl | bash` works). If you pick a directory you cannot write (for example `/opt/webtarpit`) or a system systemd unit, it asks to continue with **sudo** and re-runs itself — you do not have to restart the one-liner as root.

It asks:

1. Install directory and whether to run via systemd, a user systemd unit, cron `@reboot`, or not at all; as which Unix user (default `webtarpit`, create if missing when you are root).
2. Whether to walk through generating `webtarpit.yaml` now, or just drop a sample and let you edit later.
3. Whether auto-update from TDC.com should be on (default **off**).

Non-interactive example:

```bash
curl -fsSL https://tacticaldataconcepts.com/tools/webtarpit/install.sh | sudo bash -s -- \
  --yes --service systemd --onboard --no-auto-update --user webtarpit
```

**Manual venv:** the `webtarpit` command is inside the venv until you activate it or call it by path.

```bash
python3 -m venv .venv
.venv/bin/pip install webtarpit-0.7.1.tar.gz
source .venv/bin/activate          # puts `webtarpit` on PATH for this shell
webtarpit sample-config --out webtarpit.yaml
# edit webtarpit.yaml
webtarpit --config webtarpit.yaml serve
```

Without activating the venv:

```bash
.venv/bin/webtarpit sample-config --out webtarpit.yaml
.venv/bin/webtarpit --config webtarpit.yaml serve
```

From a checkout: `python3 -m venv .venv && .venv/bin/pip install -e .`

`webtarpit sample-config` writes the bundled example (same as `config.example.yaml` in the source tree / tarball).

## Serve (on-demand generation)

Prefer a config file (JSON or YAML). CLI flags still override the file; the file overrides environment variables.

```bash
webtarpit sample-config --out webtarpit.yaml
# or: cp config.example.yaml webtarpit.yaml
# edit bind / port / openai_base / model / api_key / topic / …
webtarpit --config webtarpit.yaml serve
```

Same settings as flags if you want them on the command line:

```bash
webtarpit serve \
  --bind 0.0.0.0 --port 8088 \
  --data ./webtarpit-data \
  --mode shared \
  --topic "defunct regional electronics distributor, est. 1998" \
  --openai-base http://127.0.0.1:11434 \
  --model llama3.2 \
  --api-key sk-your-litellm-key
```

LiteLLM (and any OpenAI-compatible proxy that requires a key) is supported the same way as Ollama. Set `openai_base` to the proxy URL (LiteLLM default `http://127.0.0.1:4000`, with or without `/v1`). Put the **master key or virtual key** in one of:

- YAML: `api_key: "sk-…"` (quoted so `#` and spaces survive)
- YAML: `api_key_file: /opt/webtarpit/api_key` (first non-comment line; `chmod 600`)
- CLI: `--api-key` / `--api-key-file`
- Env: `WEBTARPIT_API_KEY`, `LITELLM_API_KEY`, or `OPENAI_API_KEY`
- Optional systemd `EnvironmentFile=-/opt/webtarpit/webtarpit.env` (`WEBTARPIT_API_KEY=…`)

Requests send `Authorization: Bearer <key>` plus `x-api-key` / `api-key` (LiteLLM accepts those aliases). Ollama with no key still works: leave the field blank.

`WEBTARPIT_CONFIG` is an alternate way to point at the file. Env vars (`WEBTARPIT_OPENAI_BASE`, `WEBTARPIT_MODEL`, `WEBTARPIT_API_KEY`, `LITELLM_API_KEY`, `WEBTARPIT_CHANNEL`, `WEBTARPIT_NO_AUTO_UPDATE`) remain as fallbacks when a key is not in the file.

**Auto-update is off by default.** Opt in with `--auto-update`, `auto_update: true` in the config, or `WEBTARPIT_AUTO_UPDATE=1`. Then `serve` checks https://tacticaldataconcepts.com/tools/webtarpit/version.json every 6 hours, downloads the tarball only over HTTPS from that host, verifies SHA-256, then `pip install`s it and restarts. Production: pin the tarball version you installed (`0.7.1`) and leave auto-update off. `--no-auto-update` is the loud explicit opt-out. One-shot: `webtarpit check-update` / `webtarpit update`.

Dictionary blasts are capped (`max_gens_per_min`, `max_pages_per_tenant`); cache hits skip the model. Responses wait a random delay in `delay_min_ms`–`delay_max_ms`. Back-links only touch a few recent pages (`weave_sample` / `weave_recent`). Default model is a cheap local Ollama id (`llama3.2`).

`webtarpit report` prints top paths, unique IPs, and gen vs cache from `access.jsonl`. It loads `./webtarpit.yaml` (or `--config`) so the data directory matches `serve`. A missing log is an empty report, not a hard error; `serve` creates the file on start. Per-vhost `prompt:` is extra text injected into that domain’s LLM calls.

`--mode per-device` (recommended) generates a **persistent site per detected client**, so the same scanner keeps the same fiction after an IP change. Detection is **passive** (what the TCP/HTTP request already contains). Optional `device_cookie` / `device_etag` remember a known device without running script in the page. `--mode per-ip` keys only on IP. `--mode shared` uses one site for everyone.

Put a `domains:` map in the config file to host **multiple names** on one process. The `Host` header selects the vhost (topic, mode, model). Pages are stored separately per host.

Access log: `webtarpit-data/access.jsonl` (UTC JSON lines: IP, path, UA, host, device). Device map: `webtarpit-data/devices.json`.

## One-time pregenerate

```bash
# dictionary of paths, one per line
webtarpit pregenerate --paths-file paths.txt --mode shared --data ./webtarpit-data
# one client's site in per-ip mode
webtarpit pregenerate --paths-file paths.txt --mode per-ip --ip 203.0.113.9
# one vhost
webtarpit pregenerate --paths-file paths.txt --host tarpit.example --config webtarpit.yaml
```

On-demand `serve` still fills in paths that were not pregenerated.

## Safety

- Served HTML is static. CSP disallows scripts.
- Request bodies are discarded; they are never executed.
- **Operate webtarpit only on hosts you own or are authorized to run.** Do not point it at systems you do not operate.

## License

MIT.
