Israeli Chatbot Analytics

Israeli Chatbot Analytics

Analyze and optimize Hebrew chatbot performance. This skill covers conversation flow analytics, Hebrew-specific sentiment analysis, drop-off detection, user satisfaction scoring, A/B testing for Hebrew response variants, intent recognition accuracy tracking, anomaly alerting, and reporting dashboards. Use it to understand whether your Hebrew chatbot is actually helping users and where to focus improvements.

Instructions

Step 1: Collect and Structure Conversation Logs

Before analyzing, ensure conversation data is structured consistently. Each conversation session should include:

# Standard conversation log schema
conversation_log = {
    "session_id": "uuid-string",
    "user_id": "anonymous-or-identified",
    "channel": "whatsapp|telegram|web|app",
    "language": "he",           # Primary language detected
    "started_at": "ISO-8601",
    "ended_at": "ISO-8601",
    "messages": [
        {
            "timestamp": "ISO-8601",
            "sender": "user|bot",
            "text": "שלום, אני צריך עזרה",
            "intent": "greeting",           # Detected intent
            "intent_confidence": 0.92,       # Model confidence
            "entities": [],                  # Extracted entities
            "response_time_ms": 340,         # Bot response latency
        }
    ],
    "outcome": "resolved|escalated|abandoned|unknown",
    "satisfaction_score": null,   # CSAT score if collected
    "metadata": {
        "bot_version": "2.1.0",
        "ab_variant": "formal_he",
    }
}

If your platform does not export in this format, write a transformer to normalize logs before analysis. Common platforms and their export formats:

| Platform | Export Method | Format | |----------|-------------|--------| | Dialogflow CX | BigQuery export | JSON rows with session context. Use the he-il language code on new agents; iw is deprecated and frozen for new features (https://docs.cloud.google.com/dialogflow/cx/docs/reference/language). | | Rasa Pro / CALM | Analytics dashboard + tracker events | Flow-step events (Rasa Pro 3.x with CALM is dialogue-driven, not intent-driven, so legacy intent-accuracy metrics map differently). | | Rasa Open Source (legacy) | Tracker Store (SQL/Mongo) | Events list per conversation. Rasa OSS entered maintenance mode in 2025, see https://legacy-docs-oss.rasa.com/docs/rasa/. | | Botpress | Conversation export / DB | JSON. Hebrew is listed as a supported language but full RTL alignment in the default web webchat is still a community-reported gap as of 2026, verify message bubble alignment in your widget before reporting on dialect distribution. | | Custom bots | Application logs | Varies (normalize to schema above) | | WhatsApp Cloud API | Webhook logs | Message objects with metadata. See ## WhatsApp Business Platform pricing notes below for the per-message cost model that started July 2025. | | ManyChat | Audience + flow exports | CSV/JSON. WhatsApp send-out costs flow through Meta's per-message tariff. |

Step 2: Conversation Flow Analysis

Analyze session-level metrics to understand overall chatbot health:

Build a ConversationMetrics dataclass that tracks total_sessions, completed_sessions, escalated_sessions, abandoned_sessions, session_lengths (per-session message count), and session_durations (seconds). Derive rate properties (completion_rate, escalation_rate, abandonment_rate) as count / total_sessions, and avg_session_length / median_session_duration_seconds from the list fields.

compute_flow_metrics(conversations) iterates the structured logs once, increments the right outcome counter (resolved / escalated / abandoned), appends message count and (ended_at - started_at).total_seconds(), and returns the metrics object.

Key benchmarks for Hebrew chatbots (Israeli market, 2025-2026):

| Metric | Good | Average | Needs Improvement | |--------|------|---------|-------------------| | Completion rate | > 70% | 50-70% | 30% | | Abandonment rate | 35% | | Avg session length | 4-8 messages | 8-15 messages | > 15 messages | | First-contact resolution | > 65% | 45-65% | float: """Composite satisfaction (0.0-1.0). If csat_score is present, return (csat_score - 1) / 4 directly. Otherwise start at 0.5 (or 0.8/0.2 for thumbs up/down), then add: +0.15 resolved, -0.1 escalated, -0.2 abandoned, -0.15 repeatedfallbacks>2, -0.2 loopdetected, +/-0.1-0.15 finalsentiment, +/-0.05-0.1 sentimenttrend; clamp to [0, 1].""" ...

Provide `collect_post_chat_survey_he()` that returns a Hebrew post-chat survey: title `"נשמח לשמוע מה חשבת"`, a 1-5 rating on `"עד כמה הצ'אטבוט עזר לך?"`, a yes/no on `"האם הצ'אטבוט הבין את מה שרצית?"`, and an optional open `"רוצה לשתף עוד משהו?"` field. Use `"שלח משוב"` as the submit label.

### Step 7: A/B Testing for Hebrew Response Variants

Test different phrasings, formality levels, and gender handling strategies:

Build `HebrewABTestManager` with three responsibilities:

1. **Register a test.** `create_test(test_id, variants: {name: response_text}, traffic_split=None)`. Default split is uniform across variants. Store `{variants, traffic_split, created_at}` per test_id. Example variants:

```python
{"formal": "שלום וברוכים הבאים. כיצד נוכל לסייע לכם?",
 "casual": "היי! איך אפשר לעזור?",
 "gender_neutral": "שלום! ניתן לבחור מהאפשרויות הבאות:"}

1. Deterministic bucketing. assign_variant(test_id, user_id) hashes f"{user_id}:{test_id}" with hashlib.md5, maps to a bucket in [0, 1), and walks the cumulative traffic_split so the same user always gets the same variant. Use this in get_response(...) and increment an impressions counter at the same time.

1. Outcome tracking. record_outcome(test_id, variant, completed=False, satisfaction=None, escalated=False) and get_test_results(test_id) returning per-variant {impressions, completion_rate, avg_satisfaction, escalation_rate}.

Common Hebrew A/B test dimensions:

| Dimension | Variant A | Variant B | What to Measure | |-----------|-----------|-----------|-----------------| | Formality | "כיצד נוכל לסייע?" | "איך אפשר לעזור?" | Completion rate | | Gender | Slash notation ("את/ה") | Gender-neutral ("ניתן ל...") | Satisfaction score | | Length | Detailed explanation | Short, punchy response | Drop-off rate | | Emoji usage | With emoji | Without emoji | Engagement | | Error phrasing | "לא הצלחתי להבין" | "אפשר לנסח אחרת?" | Retry rate |

Step 8: Performance Dashboards and KPIs

Track these key metrics in your dashboard:

@dataclass
class ChatbotDashboard:
    """Key metrics for chatbot performance dashboard."""

    # Core metrics
    total_conversations: int = 0
    resolution_rate: float = 0.0        # % resolved without escalation
    first_contact_resolution: float = 0.0  # % resolved in first session
    avg_handle_time_seconds: float = 0.0
    escalation_rate: float = 0.0
    abandonment_rate: float = 0.0

    # User satisfaction
    avg_csat: float = 0.0               # 1-5 scale
    nps_score: float = 0.0              # -100 to 100
    thumbs_up_ratio: float = 0.0        # % positive

    # Intent accuracy
    intent_accuracy: float = 0.0        # % correctly classified
    fallback_rate: float = 0.0          # % of messages hitting fallback

    # Performance
    avg_response_time_ms: float = 0.0
    p95_response_time_ms: float = 0.0

    # Volume
    conversations_per_day: float = 0.0
    peak_hour: int = 0                  # 0-23
    busiest_day: str = ""               # "Sunday" etc.

    def to_report_dict(self) -> dict:
        """Group fields into core / satisfaction / accuracy / performance / volume
        sections for reporting (format rates as %, times as ms)."""
        ...

Implement build_dashboard(conversations, period_days=7) to populate the dataclass:

• Outcome rates from Counter(c["outcome"]) / n.
• avg_handle_time_seconds from (ended_at - started_at).total_seconds() per session.
• avg_csat from satisfaction_score where present.
• avg_response_time_ms / p95_response_time_ms from bot messages with response_time_ms (p95 via sorted_rts[int(len * 0.95)]).
• intent_accuracy = share of user messages with intent_confidence > 0.7. fallback_rate = share of user messages with intent == "fallback".
• conversations_per_day = n / period_days. peak_hour and busiest_day from Counter over started_at hour and weekday.

Israeli traffic patterns to expect:

• Peak hours are typically 10:00-12:00 and 19:00-22:00 (Israel Time, UTC+2/+3)
• Sunday is the busiest day (first workday of the Israeli week)
• Friday afternoon and Saturday see minimal traffic
• Holiday periods (Rosh Hashana, Pesach, Sukkot) show different patterns

Retention and Returning-User Metrics

Session-level metrics tell you how a single conversation went, but not whether the bot earns repeat use. Track these retention dimensions alongside the dashboard above (all require a stable user_id across sessions, pseudonymized per the Privacy and Consent section):

For each user_id, collect the set of distinct dates with a conversation. Then:

• D1 return rate = share whose first-date + 1 day is also in their set.
• D7 return rate = share whose first-date + 2..7 days intersects their set.
• Repeat-contact rate = share with > 1 distinct date.

• D1 / D7 return rate: share of users who start a new conversation the day after, or within a week of, their first contact. D7 is more stable than D1 for low-volume Israeli bots.
• Repeat-contact rate: share of users with more than one conversation. On a support bot this can be good (trust) or bad (unresolved issues), so read it with first-contact resolution.

Step 9: Hebrew-Specific Analytics Challenges

RTL Text in Charts and Visualizations

When rendering analytics dashboards that display Hebrew text, handle these RTL issues:

import matplotlib.pyplot as plt
import matplotlib

# Use a font that supports Hebrew
matplotlib.rcParams["font.family"] = ["DejaVu Sans", "Arial", "Heebo"]

# Tip: Use horizontal bar charts so Hebrew labels read naturally on the y-axis.
# For interactive dashboards, Plotly handles RTL better than matplotlib.
# Use font-family "Heebo, Arial, sans-serif" and add extra left margin for labels.

Hebrew Word Tokenization for Word Clouds

Standard whitespace tokenization does not work well for Hebrew due to prefix particles (ב, ה, ו, ל, מ, כ, ש):

# Standard whitespace tokenization fails for Hebrew due to prefix particles.
# Use YAP (https://github.com/OnlpLab/yap) for production, or strip common prefixes:
HEBREW_PREFIXES = ["ב", "ה", "ו", "ל", "מ", "כ", "ש", "וה", "של", "לה"]

# Strip prefixes only if word is long enough (>3 chars) and remainder >= 2 chars.
# For word clouds: use bidi algorithm to convert Hebrew for display,
# remove stopwords (של, את, על, עם, אני, זה, כי, גם, לא, יש, אין, מה).
# See references/hebrew-sentiment-guide.md for detailed tokenization code.

Mixed Hebrew-English Query Handling

Israeli users frequently mix languages. Track language distribution and handle accordingly:

import re

def detect_message_language(text: str) -> str:
    """Detect primary language by counting Hebrew vs English characters."""
    hebrew_chars = len(re.findall(r'[\u0590-\u05FF]', text))
    english_chars = len(re.findall(r'[a-zA-Z]', text))
    total = hebrew_chars + english_chars
    if total == 0:
        return "unknown"
    return "he" if hebrew_chars / total >= 0.5 else "en"

# Track mixed-language rate: messages where 20-80% is Hebrew.
# Israeli users frequently code-switch between Hebrew and English.

Step 10: Alerting and Anomaly Detection

Set up alerts to catch problems before they affect too many users:

from dataclasses import dataclass
from datetime import datetime, timedelta

@dataclass
class AlertRule:
    """Define an alerting rule for chatbot metrics."""
    name: str
    metric: str
    operator: str          # "gt" (greater than), "lt" (less than)
    threshold: float
    window_minutes: int    # Rolling window
    severity: str          # "critical", "warning", "info"
    description_he: str    # Hebrew description for ops team

# Recommended alert rules for Hebrew chatbots
# AlertRule(name, metric, operator, threshold, window_minutes, severity, description_he)
DEFAULT_ALERT_RULES = [
    AlertRule("high_escalation_rate", "escalation_rate", "gt", 0.35, 60, "warning",
              "שיעור הסלמה גבוה מ-35% בשעה האחרונה"),
    AlertRule("satisfaction_drop", "avg_csat", "lt", 3.0, 120, "critical",
              "שביעות רצון ממוצעת ירדה מתחת ל-3.0 בשעתיים האחרונות"),
    AlertRule("high_abandonment", "abandonment_rate", "gt", 0.40, 60, "critical",
              "שיעור נטישה גבוה מ-40% בשעה האחרונה"),
    AlertRule("high_fallback_rate", "fallback_rate", "gt", 0.25, 30, "warning",
              "שיעור fallback גבוה מ-25% בחצי שעה האחרונה"),
    AlertRule("slow_response", "p95_response_time_ms", "gt", 3000, 15, "warning",
              "זמן תגובה P95 חורג מ-3 שניות ברבע השעה האחרון"),
    AlertRule("new_unrecognized_intents", "new_unknown_intents_count", "gt", 20, 60,
              "info", "יותר מ-20 כוונות לא מזוהות חדשות בשעה האחרונה"),
]

AlertManager wraps the rule list. check_metrics(current_metrics: dict) walks every rule, skips when the metric is missing, and triggers when value > threshold (op gt) or value < threshold (op lt). Each triggered alert is a dict with rule_name, severity, metric, current_value, threshold, description_he, and triggered_at.

Step 11: Reporting Templates

Generate periodic reports summarizing chatbot performance:

Implement generate_weekly_report(dashboard, previous_dashboard=None, period_start, period_end):

• Helper trend_arrow(current, previous, higher_is_better): returns (ללא שינוי) for < 1% delta; otherwise emits [v] +X.X% (good direction) or [!] +X.X% (bad direction).
• Emit a # דוח ביצועי צ'אטבוט שבועי header, period subheader, and a | מדד | ערך | שינוי מהשבוע הקודם | markdown table over: שיחות, שיעור פתרון, CSAT, שיעור הסלמה (lower-is-better), שיעור נטישה (lower-is-better), דיוק זיהוי כוונות, זמן תגובה ממוצע (lower-is-better).
• Append a ## תנועה block with conversations_per_day, peak_hour, busiest_day.

Step 12: Integration with Chatbot Platforms

Dialogflow CX Analytics

Implement parse_dialogflow_cx_logs(bigquery_rows) to fold a Dialogflow CX BigQuery export into the standard conversations shape.

• Export query: SELECT * FROM project.dataset.dialogflow_cx_interactions WHERE DATE(request_time) BETWEEN @start AND @end.
• Group rows by session_id. For each session, track min/max request_time as started_at / ended_at.
• For each row, append a user message (text = query_text, intent = matched_intent, intent_confidence) and/or bot message (text = response_text). Sort each session's messages by timestamp. Set language = "he", outcome = "unknown" (derive from flow completion downstream).

Rasa Tracker Store Analytics

Note: Rasa Open Source is in maintenance mode. The intent-based tracker-store analytics below apply to existing Rasa OSS deployments; new Rasa builds use CALM (Conversational AI with Language Models), which is dialogue-driven rather than intent-driven, so intent-accuracy metrics map differently there. See the legacy OSS docs at https://legacy-docs-oss.rasa.com/docs/rasa/ for tracker-store details.

Implement parse_rasa_tracker_events(tracker_events) to fold a Rasa tracker-store stream into the standard conversations shape.

• Query: SELECT * FROM events WHERE sender_id = @sender_id ORDER BY timestamp.
• Iterate events. On session_started, flush the in-progress session and start a new one. On user, append a user message with intent.name and intent.confidence from `parse_dat

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: squadcodercom
• Source: squadcodercom/squadcoder
• License: MIT
• Homepage: https://squadcoder.com

Install and usage instructions live in the source repository linked above.