# Server Log Crawler Analyst

> Analyze credential-free public website monitoring evidence from server access logs, HTTP fetches, robots/llms/sitemap checks, and exported summaries while protecting raw IP and private query data. Use this skill when the user asks whether crawlers or AI bots reached a site, whether public SEO/LLM discovery files respond, or how to separate observed facts from unknown Search Console, rank, analyti…

- **Type:** Skill
- **Install:** `agentstack add skill-sergekostenchuk-ui-ux-agent-skill-system-server-log-crawler-analyst`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [sergekostenchuk](https://agentstack.voostack.com/s/sergekostenchuk)
- **Installs:** 0
- **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml)
- **Latest version:** 0.1.0
- **License:** Apache-2.0
- **Upstream author:** [sergekostenchuk](https://github.com/sergekostenchuk)
- **Source:** https://github.com/sergekostenchuk/ui-ux-agent-skill-system/tree/main/core/skills/server-log-crawler-analyst

## Install

```sh
agentstack add skill-sergekostenchuk-ui-ux-agent-skill-system-server-log-crawler-analyst
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Server Log Crawler Analyst

Use this skill when credential-free monitoring evidence is enough, or when credentialed
Search Console, rank tracking, analytics, or assistant citation tools are unavailable.

Read [references/data-source-tiers.md](references/data-source-tiers.md) before producing a monitoring report.

## Owns

- credential-free monitoring baseline;
- server access-log summary with IP privacy protection;
- crawler/user-agent observation reports;
- public HTTP checks for robots, llms.txt, sitemaps, RSS, and representative URLs;
- data-source tiering;
- monitoring limitations and unknowns;
- backlog handoff for Search Console, rank/SERP, analytics, and LLM citation monitoring.

## Does Not Own

- credentialed Search Console or analytics API work;
- scraping search engines;
- bypassing bot protections;
- live firewall or WAF changes;
- claims about rankings or assistant citations without direct evidence;
- storing raw IP logs in public artifacts.

## Workflow

1. Identify available evidence: public URL fetches, access-log snippet, exported summary, or synthetic fixture.
2. Label the evidence tier and collection time.
3. If logs are provided, summarize requests by path, status, user-agent class, and redacted IP hash.
4. Separate known crawler hits from unknown bot-like traffic.
5. Check public discovery endpoints when requested.
6. Record limitations: no Search Console, no rank data, no assistant citation data unless explicitly provided.
7. Produce a monitoring report using [assets/crawler-monitor-report.template.md](assets/crawler-monitor-report.template.md).
8. Hand off credentialed or citation tasks to the backlog.

## Non-Negotiables

- Do not publish raw IP addresses.
- Do not store credentials, cookies, API keys, or private analytics exports in skill files.
- Do not infer ranking, indexing, or citation success from server logs alone.
- Do not claim a user-agent is verified Google/OpenAI/Claude/Perplexity unless verification method is documented.
- Do not scrape SERPs or assistants against terms of service.
- Do not change firewall, WAF, robots, or server config from this skill.

## Safety And Privacy Boundaries

- Treat raw logs as private by default.
- Use aggregated counts and redacted identifiers.
- Keep exact timestamps only when needed for evidence; otherwise aggregate by hour/day.
- If the user asks to publish a report, remove raw IP, query strings, session IDs, auth paths, and admin URLs unless explicitly needed and safe.
- Public reports should say "observed in provided logs" rather than "definitely crawled by" unless bot verification is complete.

## Evidence Labels

- `Observed`: directly in log, HTTP response, or exported report.
- `Verified`: observed and checked by an accepted verification method.
- `Inferred`: likely based on user-agent/path/status, but not verified.
- `Open`: no evidence available in this run.

## Validation

Validate skill edits with:

```bash
python3 $CODEX_HOME/skills/senior-skill-architect/scripts/lint_production_skill.py $HOME/SKILL/skills/server-log-crawler-analyst
python3 $HOME/SKILL/skills/server-log-crawler-analyst/scripts/analyze_access_log.py $HOME/SKILL/skills/server-log-crawler-analyst/fixtures/sample-access.log --report /tmp/server-log-crawler-report.json
python3 $HOME/SKILL/plans/seo-llm-skill-cluster/scripts/lint_skill_cluster.py $HOME/SKILL
```

Forward tests live in [evals.json](evals.json).

## Output Shape

Return:

1. Evidence source and timestamp.
2. Data tier and privacy handling.
3. Observed crawler/user-agent classes.
4. Public endpoint status checks if performed.
5. Suspicious or irrelevant bot traffic summary.
6. Unknowns and claims refused.
7. Monitoring backlog and next evidence needed.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [sergekostenchuk](https://github.com/sergekostenchuk)
- **Source:** [sergekostenchuk/ui-ux-agent-skill-system](https://github.com/sergekostenchuk/ui-ux-agent-skill-system)
- **License:** Apache-2.0

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-sergekostenchuk-ui-ux-agent-skill-system-server-log-crawler-analyst
- Seller: https://agentstack.voostack.com/s/sergekostenchuk
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.