# Agent Instructions

## Database Management (CRITICAL)

**Production Data:** The `deals.db` file is the production database and should NOT be modified or deleted during development. Your top priority is to protect this file.

**Development Database:** To work on the application, use a separate database file for development. This can be configured by setting the `DATABASE_URL` environment variable to the path of your development database file (e.g., `export DATABASE_URL=dev_deals.db`). All development and testing must be done against this separate database.

**Backup and Restore:** Before making significant changes, it is wise to create a backup. Use the following scripts to manage the production database:
- `./backup_db.sh`: Creates a timestamped backup of the database.
- `./restore_db.sh`: Restores the database from the most recent backup.

---

### Standard Operating Procedures (SOP) for System Stability

To ensure the stability and performance of the development environment, the following procedures must be followed:

1. **Handling Large Files (especially logs):**

   *   **NEVER** read an entire file if it is known or suspected to be large (e.g., > 500 KB). Large JSON or log files will immediately flood the context window and cause instability.
   *   **ALWAYS** use targeted commands to inspect large files.
       *   To view the end of a file: `tail -n 100 <filepath>`
       *   To view the beginning of a file: `head -n 100 <filepath>`
       *   To search for specific patterns, errors, or keywords: `grep "my search pattern" <filepath>`
   *   If you need to understand the general structure of a large log or data file, use a combination of `head`, `tail`, and `grep` to build a picture without loading the entire file into memory.
   *   **NEVER** assume a log file is safe to read. Always check its size with `ls -lh` first.
   *   **LOGGING SOURCE OF TRUTH:**
       *   **`celery_worker.log`**: This is the ACTIVE log for background tasks. Use this for troubleshooting.
       *   **`celery_monitor.log`**: Contains resiliency logs.
       *   **`celery.log`**: This is a **LEGACY/ABANDONED** file. **DO NOT READ IT.** It contains old, irrelevant data and is massive.

2. **Initial Codebase Exploration ("Filesystem Tour"):**

   *   At the beginning of any new task, perform a "filesystem tour" to understand the layout and structure of the codebase.
   *   Use `ls -F` (non-recursive) or targeted `ls` commands on key directories (e.g., `ls -F keepa_deals/`) to list files. **AVOID** `ls -R` as it generates excessive output.
   *   **DO NOT READ** the following large system state files:
       - `xai_cache.json` (Runtime Cache - Do not read)
       - `xai_token_state.json` (Runtime State - Do not read)
       - `strategies.json`
   *   **READ** `README.md`, this `AGENTS.md` file, and `Documentation/System_State.md` in full.
   *   **READ** the 3-5 most recent logs in `Dev_Logs/` to understand the latest changes.
   *   **DO NOT** read old logs in `Dev_Logs/Archive/` unless specifically investigating a regression related to that time period. `Documentation/System_State.md` is your primary source of truth.
   *   This initial exploration provides essential context, helps locate relevant code modules, and prevents unnecessary file reading later in the task. Adhering to this practice is mandatory for efficient and stable operation.

3. ### Environment and Verification Protocol (EVP)

   To ensure stability and efficiency, the following protocol is mandatory for all tasks. Failure to adhere to these steps can lead to mission failure due to environmental instability.

   1. **Environment Sanity Check (ESC):** Before beginning any code analysis or modification, you MUST verify the sandbox environment's integrity. Perform a simple file creation and version control check: `touch test_agent_sanity.txt && git add test_agent_sanity.txt && git diff --staged`. This command MUST show a diff indicating a new file. If it returns empty, the environment is faulty. You MUST halt immediately and report the environment as unstable. Afterwards, clean up with `git reset HEAD test_agent_sanity.txt && rm test_agent_sanity.txt`. Do not attempt to work around a broken environment.
   2. **Principle of Least Impact Verification (LIV):** Your verification plan MUST use the most lightweight and targeted method possible. Do not run resource-intensive, end-to-end data pipelines (like `backfill_deals`) to verify small, isolated changes.
      - **Example for a backend/API change:** Manually insert a single test row into the database using `sqlite3` and query the specific API endpoint with `curl`.
      - **Example for a frontend change:** Use the provided frontend verification tools without populating the entire database. This principle is critical to minimizing resource usage and avoiding sandbox failures.

---

## The Stability Pact: A Standard Operating Procedure for Preventing Regression

To prevent regressions and ensure that "hard-won" code remains stable, I will adhere to the following principles for every task. This pact is my primary directive.

**1. Principle of Minimum Scope:**

*   I will only change the absolute minimum code necessary to complete the current task.
*   I will not perform unrelated refactoring or "cleanup" of files I am not explicitly tasked to work on.
*   Every change I make must be directly justifiable by the user's request.

**2. "Code Archaeology" Before Action:**

*   Before modifying any existing code, I must first understand its history and purpose.
*   I will use `git log -p <filepath>` to review the recent history of the file to understand why it is in its current state.
*   I will consult the `Dev_Logs/Archive/` directory ONLY if I need to trace the origin of a specific feature.
*   My goal is to understand the *intent* behind the existing code before I propose a change.

**3. Strict Separation of Code and Configuration:**

*   I will not change configuration values (e.g., batch sizes, timeouts, thresholds) unless the task is specifically about tuning those parameters.
*   Such values should be stored in dedicated internal configuration files (e.g., `app_config.py`).
*   If I find hardcoded configuration values during a task, I will report them to you and ask for permission before moving them to a dedicated file.

**4. Test-Driven Development as a Rule:**

*   For all future bug fixes, my first step will be to write a new, failing test that precisely reproduces the bug.
*   For all new features, I will write tests that define the feature's correct behavior.
*   I will run the *entire* test suite before submitting any change. A failing test is a hard blocker. This is our primary automated guard against regression.

**5. Explicit Confirmation for Scope Creep:**

*   If, during a task, I identify a necessary change that falls outside the original scope (e.g., a required refactor in an unrelated file), I will **stop**.
*   I will present my finding and the proposed change to you and will not proceed until I receive your explicit permission.

---

## Technical and Historical Notes

This section contains valuable context and learnings from previous development tasks. Consult these notes before working on related parts of the codebase.

### Fallback Data Warning (CRITICAL - Jan & Mar 2026)
**Do NOT use unverified fallback data to fill missing fields or estimate prices.**
Previous attempts to "solve" data gaps by using fallback values (e.g., using Keepa Stats listing averages like `avg90` or `avg365` when inferred sales were sparse) resulted in artificially elevated list prices and "fake profits."
-   **Principle:** If the primary data source (confirmed true inferred sales via drops vs. offers) is missing, it is **required** to reject the deal (e.g., return `None` or `-1`). Incorrect guesses based on listing prices lead to wildly inaccurate margins and damage subscriber trust.
-   **March 2026 Addendum:** The "Keepa Stats Fallback" (Silver Standard) logic was explicitly and entirely removed from `stable_calculations.py` for this reason. **Do not reintroduce fallback logic that relies on listing prices to artificially inflate deal volume.** If there are 0 inferred sales, the deal MUST be rejected.

### Role-Based Access Control (RBAC)
-   **User Roles:** The system distinguishes between `admin` and `user` roles.
-   **Access Enforcement:**
    -   **Admin Only:** `/deals`, `/guided_learning`, `/strategies`, `/intelligence`.
    -   **User Accessible:** `/dashboard`, `/settings`.
    -   **Mechanism:** `wsgi_handler.py` checks `session['role']` on restricted routes and redirects unauthorized users to the dashboard.
-   **Navigation:** Frontend templates conditionally render navigation links based on the user's role.

### Timestamp Handling Notes (from Task starting ~June 24-25, 2025)

When working with timestamp fields like 'last update' and 'last price change', the goal is to reflect the most recent relevant event as accurately as possible, aligning with user expectations from observing Keepa.com.

**For 'last_update':**
This field should represent the most recent time any significant data for the product/deal was updated by Keepa. It considers:
1.  `product_data['products'][0]['lastUpdate']` (general product data update from /product endpoint).
2.  `deal_object.get('lastUpdate')` (general deal data update from /deal endpoint).
3.  `product_data.get('stats', {}).get('lastOffersUpdate')` (when offers were last refreshed from /product endpoint stats).
The function should take the maximum valid (recent) timestamp from these three sources.

**For 'last_price_change' (specifically for Used items, excluding 'Acceptable'):**
This field aims to find the most recent price change for relevant used conditions.
1.  **Primary Source (`product_data.csv`):** Check historical data for 'USED' (`csv[2]`), 'USED_LIKE_NEW' (`csv[6]`), 'USED_VERY_GOOD' (`csv[7]`), and 'USED_GOOD' (`csv[8]`). Select the most recent valid timestamp from these.
2.  **Fallback Source (`deal_object.currentSince`):** If CSV data is insufficient, check `currentSince[2]` (Used), `currentSince[19]` (Used-LikeNew), `currentSince[20]` (Used-VeryGood), and `currentSince[21]` (Used-Good). Additionally, if `deal_object.current[14]` indicates the Buy Box is 'Used', also consider `currentSince[32]` (buyBoxUsedPrice timestamp). Select the most recent valid timestamp from this combined pool.

**General Timestamp Conversion:**
All Keepa minute timestamps should be converted to datetime objects using `KEEPA_EPOCH = datetime(2011, 1, 1)`, then localized from naive UTC to aware UTC (`timezone('UTC').localize(dt)`), and finally converted to 'America/Toronto' (`astimezone(TORONTO_TZ)`), formatted as '%Y-%m-%d %H:%M:%S'. Timestamps <= 100000 are generally considered invalid/too old.

**The "Keepa Epoch Bug" (Jan 2026):**
A critical regression occurred when the system interpreted Keepa timestamps using an epoch of `2000-01-01` instead of `2011-01-01`. This 11-year offset caused fresh 2026 data to be seen as 2015 data ("Ancient Data"), causing the ingestion pipeline to reject everything. **Always verify the epoch is 2011.**

### Circular Dependencies & Module Structure
-   **`keepa_deals/new_analytics.py`**: This module was specifically created to house downstream analytical logic (e.g., trend calculations, offer count averages) to prevent circular imports between `processing.py`, `stable_calculations.py`, and `stable_products.py`.
-   **Rule:** If you are adding a new metric that depends on core calculations (like inferred sales) but is used by the main processing loop, place it in `new_analytics.py` rather than modifying the core stable modules.

### Data Standards & Epochs
- **Keepa Epoch:** The system strictly uses `datetime(2011, 1, 1)` as the epoch for interpreting Keepa API timestamps. This differs from the 2000 epoch used in some other contexts or documentation. Failure to use 2011 results in an 11-year data offset.

### Token Management & Rate Limiting
- **Blocking Wait Strategy:** To prevent API 429 (Too Many Requests) errors, the system employs a "Blocking Wait" strategy.
- **Implementation:** API wrapper functions (like `fetch_deals_for_deals` in `keepa_deals/keepa_api.py`) accept a `token_manager` argument. If provided, the function calls `token_manager.request_permission_for_call()` which sleeps the thread until sufficient tokens are available, rather than failing immediately.
- **Rule:** When adding new API calls to high-volume loops, always ensure they are integrated with the `TokenManager` to support this flow.

### UI & SVG Standards
- **Icons:** Navigation icons are SVGs with their `viewBox` reset to the content bounding box (removing internal padding) and are strictly sized to **20px** height in CSS (`static/global.css`).
- **Header:** The main header container (`.main-header`) has a strictly fixed height of **134px**. Altering this risks breaking the "sticky" filter panel alignment.
- **Sticky Headers:** The dashboard data table relies on a precise stack of `top` offsets (177px, 233px, 264px, 289px) that are hardcoded based on the heights of the Main Header and Filter Panel. The `.filter-panel` MUST have `margin-bottom: 0px` to prevent layout breaks.
- **Table Width Constraints:** The dashboard table (`#deals-table`) has a strict `1200px` max-width limit. To accommodate 15+ columns without horizontal overflow, numerical data must be heavily formatted.
  - Large financial values (Profit, All_in_Cost) must drop their decimals (e.g., `minimumFractionDigits: 0`).
  - Text-heavy columns like `Detailed_Seasonality` must use a strict `max-width` (e.g., `105px`) with `text-overflow: ellipsis`.
  - Cell padding must be strictly managed (e.g., `padding: 0 8px`) to maximize data real estate without clipping vertical headers. Do not pad the top/bottom of sortable headers.

### Keepa Query Standards
- **Date Range:** `dateRange: 4` (All Combined) is permissible and recommended to capture the maximum 3-year history for AI analysis.
- **Sorting:** When using `dateRange: 4`, you MUST enforce `sortType: 4` (Last Update) to ensure the API returns fresh data and not stale deals from 2015.

### Critical Fixes & Stability (Feb 2026)
- **Token Starvation & Zombie Locks:** The system uses a **Shared Redis Token Bucket** (`keepa_deals/token_manager.py`) to coordinate API usage. To prevent locks from persisting after a crash, the kill script (`kill_everything_force.sh`) now performs a "Brain Wipe" (FLUSHALL + SAVE) on Redis. **Do not remove this cleanup logic.**
- **Ghost Deals:** MFN offers with unknown shipping (`-1`) are strictly rejected. Do not attempt to "guess" shipping costs for MFN sellers.
- **Seller Name Preservation:** To save tokens, the system performs "Lightweight Updates" that lack seller names. It uses the `Seller ID` to preserve the existing human-readable name. If you modify `processing.py`, ensure this logic remains intact to avoid overwriting names with raw IDs.
- **Zero Profit & Missing Data Persistence:** Deals with `Profit <= 0` or missing critical data (`List at`) are now **persisted** (not rejected) to prevent infinite re-fetch loops. They are filtered from the Dashboard.
- **Keepa Stats Fallback (REMOVED MAR 2026):** We previously used listing averages when inferred sales were < 3. This was removed because it produced artificially elevated prices compared to actual sales. The system now strictly requires inferred true sales to list a price.
- **Sparse Sales Rescue:** The system uses the **Median** of inferred sales (1-2 events) as a "Sparse Rescue" price, because these still represent true inferred sales. These skip XAI checks to prevent false negatives from missing context.
- **Deficit Protection:** Enforced `MAX_DEFICIT = -180` to prevent API lockouts.

## Definition of Done: The Logic Check

**WARNING: The "Infrastructure vs. Logic" Trap**
Developers often split complex tasks into two stages:
1.  **Infrastructure:** Building the scaffolding (e.g., loops, function calls, API integrations) to ensure the system handles the new data flow without crashing.
2.  **Logic:** Implementing the actual decision-making algorithms (e.g., filtering, pricing, rejection).

**CRITICAL RULE:** A task is **NOT DONE** until both stages are complete.
*   **NEVER** leave a placeholder like `candidates.append(all_items)` or `return True` in production code with the intent to "refine later".
*   If infrastructure must be merged before logic is ready, the feature must be **DISABLED** by default (e.g., via a feature flag or commented out).
*   **Zombie Features:** Infrastructure without logic creates "Zombie Features" that consume resources (tokens/CPU) but provide no value (e.g., fetching data only to pass everything through). This is worse than not having the feature at all.

**Verification Step:** Before marking a task as complete, explicitly ask: *"Does this code actually make decisions, or is it just moving data?"*

*   **Smart Ingestor Batch Size:** Default **50** (High Rate), dynamically reduces to **20** (Low Rate < 20/min) and **1** (Critically Low < 10/min) to safely fit within the token burst window (40 tokens) without livelock.
*   **Stall Watchdog:** Use `Diagnostics/watchdog_stall_detector.py` to identify if the worker is stuck (Tokens > 290 and no Heartbeat for 15 mins).

---

## Recent Fixes (Feb 2026)
- **Stale Deal Rescue:** Implemented `rescue_stale_deals` in `keepa_deals/smart_ingestor.py` to prevent "diminishing deals". The system proactively refreshes deals older than 48 hours to ensure they are not deleted by the Janitor (72h limit).
- **Amazon Ceiling Check:** Added a safety check in `keepa_deals/processing.py` during lightweight updates. If `List at` > 90% of the current Amazon New Price, it is clamped down to that ceiling. This prevents deals from showing "fake profit" when the market price drops.

## Recent Fixes (March 2026)
- **Self-Aware Mentor & Tooltips:** Added `keepa_deals/platform_knowledge.py` to dynamically load documentation into the AI's context. Created an instant, speech-bubble style tooltip system for the Deals Dashboard headers and filters.
    - **Constraint Note:** The `?` icon was removed from column headers because it violated the strict 1200px table width limit. Triggers (`.ai-tooltip-trigger`) are now applied directly to the text/labels themselves. Tooltips utilize a server-side JSON cache (`tooltip_cache.json`) to prevent latency and token waste.
- **Astronomical Profits Fix:** Added a hard ceiling in `keepa_deals/stable_calculations.py` to automatically reject any `List At` price over $1,500. Expanded the > 3.0 ratio check (Suspiciously High Markup) to apply to **all** price sources, forcing AI scrutiny on any price more than 3x the current used price. Updated AI Prompt to explicitly reject non-textbook used prices over $500.
- **Database Clearing Script:** Created `./clear_deals.sh` to provide a non-interactive way to clear old or inaccurate deal data (`deals` and `user_restrictions` tables) from `deals.db` without wiping the entire database or asking for user input, ideal for pre-deployment cleanup.
- **FBA Inventory Sync:** Fixed "No active inventory found" by switching to `GET_FBA_MYI_ALL_INVENTORY_DATA` (resolving FATAL errors from the unsuppressed report). Logic now includes **Inbound** inventory (Working, Shipped, Receiving) in the total count.
- **Credential Management:** Refactored backend and diagnostics to strictly prioritize Database (`deals.db`) lookup for Seller ID and Refresh Tokens. `.env` is reserved for global app config (Client ID/Secret) only.
- **Safety:** Hardcoded `https://sellingpartnerapi-na.amazon.com` (Production) as the default SP-API URL to prevent accidental Sandbox connections.
- **Dynamic ROI & All-in Cost:** `All-in Cost` strictly excludes Amazon fees (which are now deducted separately to find `Profit`). `ROI` is dynamically calculated `(Profit / All_in_Cost) * 100` and is **not** stored in the database. Warn against updating non-existent DB columns (like `Total AMZ fees`) using raw SQL.
- **Strict UI Limits:** The dashboard table is strictly limited to `1200px`. To fit new columns, use `minimumFractionDigits: 0` for large numbers and clamp text columns (e.g., `max-width: 105px`).