# Keepa Deals - API Development Log

**Note:** This log tracks the development of the Keepa Deals data extraction script. It has been revised for clarity, conciseness, and to accurately reflect the codebase as of early June 2025.

## 2025-06-07 (Assumed Date of Last Log Entry - Review Point)

*   **Timestamp Logic (`last_price_change`):** Reverted `last_price_change` in `stable_deals.py` to a balanced approach considering multiple used condition timestamps (CSV indices [2,6,7,8] and `currentSince` indices [2,19,20,21], plus Used Buy Box if applicable). This addressed issues where a too-strict focus on only 'USED' index 2 resulted in no timestamps.
    *   _Code Verified: `stable_deals.py` reflects this multi-source logic for `last_price_change`._
*   **Timestamp Logic (`last_update`):** Enhanced `last_update` in `stable_deals.py` to consider `product_data['products'][0]['lastUpdate']`, `deal_object.get('lastUpdate')`, and `product_data.get('stats', {}).get('lastOffersUpdate')`, selecting the maximum.
    *   _Code Verified: `stable_deals.py` reflects this._
*   **Package Dimensions:** Corrected `package_height`, `package_length`, `package_width` in `stable_products.py` to return "Missing" if dimensions are missing, null, or zero, preventing "0.0 cm" or TypeErrors.
    *   _Code Verified: `stable_products.py` reflects this logic for zero or missing (None/-1) values._
*   **Timezone Conversions:** Addressed issues in `stable_deals.py` for `deal_found`, `last_update`, and `last_price_change` to correctly localize naive Keepa epoch-based datetimes to UTC before converting to Toronto time.
    *   _Code Verified: Timestamp functions in `stable_deals.py` use `timezone('UTC').localize(dt).astimezone(TORONTO_TZ)`._

## 2025-06-0X (Assumed Date Range - FBM/FBA Lowest/Current Fixes)

*   **`New, 3rd Party FBM - Current`:**
    *   Guidance in `AGENTS.md` suggests this should strictly use `stats.current[7]`.
    *   **Discrepancy Noted:** The current `new_3rd_party_fbm_current` function in `stable_products.py` implements logic to parse the `product['offers']` array to find the lowest FBM offer. It does *not* currently use `stats.current[7]`. This log reflects the implemented offer parsing logic.
    *   _Code Verified: `new_3rd_party_fbm_current` in `stable_products.py` parses `offers`._
*   **`New, 3rd Party FBA - Lowest`:** Updated `new_3rd_party_fba_lowest` in `stable_products.py` to correctly use `product.stats.min[10][1]` for the historical minimum FBA price, as per `AGENTS.md` learnings.
    *   _Code Verified: `stable_products.py` reflects this._
*   **`New, 3rd Party FBA - Current`:** Updated `new_3rd_party_fba_current` in `stable_products.py` to use `stats.current[10]`, aligning with investigation and successful testing.
    *   _Code Verified: `stable_products.py` reflects this._

## 2025-06-03

*   **Buy Box Price (`buy_box_current` Logic & Testing):**
    *   Modified `buy_box_current` in `stable_products.py` to prioritize the `stats['buyBoxPrice']` field.
    *   If `buyBoxPrice` is missing or invalid (None, <=0), the function falls back to searching `product['offers']` using `product['buyBoxSellerId']` and `product['buyBoxCondition']`. If no matching offer is found, it returns "-".
    *   Introduced `test_keepa_deals.py` with unit tests for `buy_box_current`. These initial tests cover scenarios for `buyBoxPrice` (valid, missing, invalid, zero). The offer parsing fallback logic is not yet explicitly covered by these tests.
    *   Adjusted `Keepa_Deals.py` argparse handling (`parser.parse_args()` moved into `main()`) to prevent interference with unittest discovery.
    *   _Code Verified: `stable_products.py` (function logic) and `test_keepa_deals.py` (test content) reflect these changes. Test coverage for `buy_box_current` is partial, focusing on the primary field._

## Collaborator (Jules) Environment Issues (Late May - Early June 2025)

*   **Summary:** Documented persistent file system and permission errors encountered in the AI collaborator's (Jules) environment (`/app/keepa-deals/`). These included `git clone` failures, `venv` creation problems, and tools treating directories as files (e.g., `cat /app/keepa-deals: Is a directory`), blocking most operations.
*   **Impact & Resolution Efforts:** These platform-specific issues significantly hindered collaborative debugging and testing. Resolution attempts involved diagnostic commands and exploring alternative file/environment setup methods (e.g., `wget`, `virtualenv -p`). The issues pointed towards needing admin intervention or a fresh environment for the collaborator.

## 2025-05-27

*   **`Used, acceptable - Current` Fix Verification:** Investigated the "Used, acceptable - Current" column. Confirmed that the function `used_acceptable` in `stable_products.py` was already correctly implemented to use `stats.current[22]` directly, without complex fallbacks. This aligned with lessons documented in `AGENTS.md`. The previously reported issue of incorrect prices due to fallbacks was related to historical code that had since been corrected.
    *   _Code & AGENTS.md Verified: No code change was needed for this specific issue as the fix was pre-existing._

## 2025-05-25 - Environment and Dependency Management

*   **Python Environment Standardization:** Addressed confusion between Python 3.11 and 3.10. Standardized on Python 3.10.17 within the `keepa_venv` virtual environment, as it was observed to provide more consistent data capture for certain fields compared to 3.11 experiments.
*   **Virtual Environment Cleanup:** Discovered and removed numerous unexpected Python packages (e.g., `keepa`, `tqdm`, `aiohttp`) from `keepa_venv`. These were likely installed during ad-hoc testing and were not part of the declared `requirements.txt`. The environment was cleaned to include only essential dependencies (`pandas`, `pytz`, `requests`, `retrying`), resolving issues like unexpected `tqdm` progress bars.
*   **Backup Procedures:** Reviewed and corrected backup path issues to ensure reliable project backups to `/home/timscripts/keepa_api/Bak_May/`.

## 2025-05-22 - Field Debugging and API Client Strategy

*   **FBM/FBA Field Issues:** Investigated `New, 3rd Party FBM - Current`, `Buy Box Used - Current`, and `New, 3rd Party FBA - Current` showing `'-'` or missing data.
*   **API Client Experimentation:** Briefly experimented with switching `fetch_product` in `Keepa_Deals.py` to use the `keepa.query` (Python client). This was found to be unsuitable for ASINs originating from the `/deal` endpoint, as it omitted critical data like `offers` and specific `stats.current` indices (e.g., `current[9]`), and also broke `Sales Rank - Current`. Reverted to direct HTTP requests via the `requests` library.
*   **Strategy Confirmation:** Reinforced the project's strategy of using direct HTTP API calls for data fetching, finding it more reliable for accessing the full data spectrum needed, especially `offers` and complete `stats` arrays.
*   **`List Price - Current`:** Logged as non-functional (`stats.current[8]` returning '-'). Marked for future restoration and testing. _(Current `stable_products.py` uses `stats.current[4]` for `list_price`.)_

## 2025-05-20 - `Title` and `Amazon - Current` Column Fixes

*   **`Title` Column Population:** Resolved the issue of the `Title` column being empty. Modified `get_title` in `stable_products.py` to extract the title directly from the `product` object (sourced from `fetch_product`'s response: `product.get('title', '-')`) instead of making a redundant API call. This change integrated smoothly with the existing `FUNCTION_LIST` processing loop in `Keepa_Deals.py`.
*   **`Amazon - Current` Column Population:**
    *   **Initial Problem:** `Amazon - Current` was consistently `'-'` for some ASINs, even with live Amazon offers visible on Keepa.com (e.g., ASIN 150137012X showed `'-'` instead of $6.26). Debugging indicated `stats.current[10]` was -1.
    *   **Root Cause Analysis:** Identified that the `&buybox=1` parameter in the `fetch_product` URL (present since commit `7a98f95c`) could cause `stats.current[10]` (typically Amazon's price) to return -1 if Amazon was not the current Buy Box holder, even if an Amazon offer existed.
    *   **Initial Incorrect Fix Attempt (Logged Previously):** A prior log entry incorrectly stated the fix was removing `&buybox=1`. This was not the final implemented solution.
    *   **Actual Implemented Fix:** The `amazon_current` function in `stable_products.py` was modified to use `stats.get('current', [None] * 23)[0]` as the source for Amazon's current price. The `fetch_product` URL in `Keepa_Deals.py` *continues* to include the `&buybox=1` parameter.
    *   _Code Verified: `stable_products.py` uses `stats.current[0]` for `amazon_current`. `Keepa_Deals.py` fetch_product URL includes `buybox=1`._

## AI Assistant ("Grok") Interaction Challenges (Mid-May 2025)

*   **Summary of Issues:** Encountered significant and persistent difficulties with the AI assistant (Grok) correctly recognizing and reproducing filenames (e.g., repeatedly substituting "Keepa_api.py" for "Keepa_Deals.py").
*   **Impact:** These recognition failures led to considerable frustration, communication overhead, and delays in collaborative tasks that required precise file referencing.
*   **Relevance:** Documented as a notable tooling challenge within the development workflow, impacting efficiency. The issue was reportedly escalated to xAI by the assistant.

## Centralized Field Mappings with `field_mappings.py` (Mid-May 2025)

*   **Milestone:** Successfully implemented the `field_mappings.py` design around May 17, 2025. This centralized the mapping of CSV headers to their respective data extraction functions from `stable_products.py` and `stable_deals.py` via `FUNCTION_LIST`.
*   **Benefits:** Streamlined data processing in `Keepa_Deals.py`, eliminated redundant function calls, ensured correct header order, and resolved previous conflicts that led to missing or overwritten data in the CSV.
*   **Processing Logic:** `Keepa_Deals.py` iterates through `FUNCTION_LIST`, dynamically passing the `deal` object or `product` object as required by each function.

## Foundational Data Extraction & Debugging (Approx. April - Mid-May 2025 - Summarized)

*   **Core Focus:** Intensive efforts to populate `Keepa_Deals_Export.csv` by correctly mapping Keepa API responses.
*   **`stats.current[X]` Mapping:** Extensive trial-and-error to identify correct indices within `stats.current` for various price fields (Buy Box Used, general Used, condition-specific Used prices) and Sales Rank fields. Many initial mappings were incorrect (e.g., pulling sales rank instead of price).
*   **Timestamp Functions:** Initial development of `tracking_since`, `publication_date`, `listed_since`. Early challenges included distinguishing Keepa's epoch (minutes from 2011-01-01) from Unix epoch and implementing correct timezone conversions to 'America/Toronto'. `publication_date` development was particularly tricky due to API returning `0` or varied formats, eventually leading to its exclusion from active processing for a period.
*   **Product Attributes:** Implemented functions for `Manufacturer`, `Author`, `Binding`, `Categories`, and package dimensions. Package dimension functions were refined to handle missing/zero values gracefully.
*   **API Call Stability:** Addressed `fetch_product` timeouts (leading to retry mechanisms and timeout adjustments) and `NameError` issues.
*   **`/deal` Endpoint Issues:** Resolved `400 - Invalid selection parameter` errors from the `/deal` endpoint by refining URL encoding of the JSON query (`json.dumps` with `separators` and `quote_plus`).
*   **`get_stat_value` Utility:** Developed and refined this core helper in `stable_products.py` to handle price/number formatting and safe extraction from API statistics arrays, including those returning `[price, timestamp]` pairs.
*   **Initial Stable Columns:** Achieved stability for a foundational set of columns including `ASIN`, `Title`, `Sales Rank - Current`, and an early version of `Buy Box Used - Current` (later refined).
*   **API Client Choice:** Early experiments may have involved the `keepa` Python library, but the direction shifted to direct HTTP `requests` for more control and to access all necessary data fields, a decision reinforced by `README.md` and `AGENTS.md`.

## Early Development & API Exploration (Approx. Late March - April 2025 - Summarized)

*   **"Price Now" Source:** Initial investigations into the source for "Price Now", exploring `stats.current[3]` (Buy Box) and then various other used price indices from `stats.current` and the `offers` array.
*   **API Query Refinements:** Removed `to_csv` parameter from `api.query` as it was unsupported by the Keepa client/HTTP endpoint being used.
*   **`fetch_deals` Implementation:** Focused on correctly using the `/deal` endpoint with a structured JSON query to get initial ASINs based on specific criteria (e.g., used books, discount percentage).

## General Notes on SOW vs. Implemented Reality

*   **Project Focus:** The development to date primarily addresses Phase 1 (Data Extraction) of the "Agent Arbitrage" project outlined in `SOW.txt`.
*   **Script Name:** The main operational script is `Keepa_Deals.py`, not `Process500_Deals_v2.py` as mentioned in the SOW.
*   **CSV Columns:** The number of columns in `Keepa_Deals_Export.csv` is determined by `headers.json`, which currently defines 216 columns, differing from the 229 columns specified in the SOW.
*   **Python Version:** The project has standardized on Python 3.10.17, whereas the SOW initially mentioned Python 3.11.

## 2025-06-27: Fix for "Used - 365 days avg." Column

**Issue:** The "Used - 365 days avg." column in `Keepa_Deals_Export.csv` was not populating with data.

**Root Cause:**
*   No function was implemented in `stable_products.py` to calculate this specific average.
*   The corresponding entry in `field_mappings.py` for this header was set to `None`.

**Fix Implemented:**
1.  **Created `used_365_days_avg(product)` function in `stable_products.py`:**
    *   This function utilizes `get_stat_value(stats, 'avg365', 2, divisor=100, is_price=True)`.
    *   The key `'avg365'` fetches the 365-day average data, and index `2` corresponds to the general 'Used' price within Keepa's `stats` object.
2.  **Updated `field_mappings.py`:**
    *   Imported `used_365_days_avg` from `stable_products.py`.
    *   Mapped the "Used - 365 days avg." header in `FUNCTION_LIST` to the new `used_365_days_avg` function.

**Verification:**
*   Ran `Keepa_Deals.py` script successfully.
*   Confirmed that the "Used - 365 days avg." column in the output CSV is now populated with the correct 365-day average used prices.

**Note:** This confirms that `product.stats.avg365[2]` is the correct data point for the 365-day average of general used prices from the Keepa API product data.

**Date:** YYYY-MM-DD
**File(s) Touched:** `stable_calculations.py`
**Feature/Bug:** Percent Down 365 Column Symbol Rendering

**Issue:**
Initial implementation of the "Percent Down 365" column used Unicode arrow symbols (⇩ and ⇧) to indicate if the current price was below or above the 365-day average, respectively. 
During local testing by the user, these symbols did not render correctly and appeared as garbled characters (e.g., ‚á©80%).

**Solution:**
Modified `stable_calculations.py -> percent_down_365()`:
- Replaced the Unicode symbol "⇩" (U+21E9) with "-" (minus sign) when the current used price is less than the 365-day average.
- Replaced the Unicode symbol "⇧" (U+21E7) with "+" (plus sign) when the current used price is greater than the 365-day average.
- If the current price is equal to the average (0% difference), no symbol is prepended to the percentage.

**Examples of new output format:**
- Current < Average: "-15%"
- Current > Average: "+10%"
- Current == Average: "0%"

The calculation logic for the percentage itself remains unchanged.

**Reasoning:**
Standard ASCII characters like "+" and "-" have much broader compatibility across different terminals, file viewers, and operating systems, preventing rendering issues.

**Commit:** `feat: Update Percent Down 365 symbols for better compatibility` (Branch: `feat/percent-down-365`)
**Outcome:** Symbols now render correctly as per user feedback.

## Jules - 2025-06-28
**Task:** Investigate "Publication Date" column formatting. API providing "YYYY-MM" (e.g., "1985-06") was reportedly appearing as "Mon-YY" (e.g., "Jun-85") in the CSV.
**Investigation:**
- Reviewed `get_publication_date` function in `stable_products.py`.
- Confirmed that the existing string processing logic correctly identifies "YYYY-MM" string inputs and preserves this format for the CSV output.
- The main script `Keepa_Deals.py` uses `csv.writer`, which writes string data as-is.
**Conclusion:**
- The script correctly outputs "YYYY-MM" strings to the CSV when the API provides them in that format.
- The observed "Mon-YY" format is an artifact of how spreadsheet software (e.g., Microsoft Excel) interprets and displays date-like strings from CSV files. Excel may auto-convert "1985-06" to a date object (e.g., 1985-06-01) and then apply a default display format like "Jun-85".
**Action:**
- No code changes were made as the script's output is correct.
- Confirmed with the user that the raw CSV content (viewed in a text editor) shows the correct "YYYY-MM" format.
**Status:** Task completed.

Date: YYYY-MM-DD
Author: Jules
Task: Implement "Release Date" column functionality.

Issue:
- The "Release Date" column in `Keepa_Deals_Export.csv` was consistently showing "-" as no function was assigned to populate it.

Solution:
1.  **Created `get_release_date` function in `stable_products.py`:**
    *   Adapted logic from the existing `get_publication_date` function.
    *   Specifically targets `product_data.get('releaseDate')` as the primary source.
    *   Parses various date formats: Keepa Time Minutes (KTM), YYYYMMDD, YYYYMM, YYYY integers, and common date strings ('YYYY-MM-DD', 'YYYY-MM', 'MMM-YY', 'YYYY').
    *   Converts valid dates to 'YYYY-MM-DD', 'YYYY-MM', or 'YYYY' format.
    *   Returns '-' for invalid or missing dates.
    *   Includes comprehensive logging for traceability.
2.  **Updated `field_mappings.py`:**
    *   Imported `get_release_date` from `stable_products.py`.
    *   Assigned `get_release_date` to the "Release Date" field in the `FUNCTION_LIST`.

Outcome:
- The "Release Date" column should now populate with data when available and correctly parsed from the Keepa API.
- User will test locally to confirm.

Learnings:
- Reconfirmed the utility of adapting existing robust functions (like `get_publication_date`) for similar data types, saving development time and ensuring consistent parsing logic.
- The `releaseDate` field in Keepa product data can come in various formats (KTM, integer, string), requiring flexible parsing, similar to `publicationDate`.

## Dev Log Entry - YYYY-MM-DD

**Task:** Fix "Buy Box - 365 days avg." column showing all "-".

**Initial Approach:**
*   Identified that "Buy Box - 365 days avg." was mapped to `None` in `field_mappings.py`.
*   Created `buy_box_365_days_avg(product)` function in `stable_products.py`.
*   Initially assumed `stats.avg365[10]` was the correct data point for Buy Box 365-day average. Index 10 is often associated with Buy Box related data or New FBA.

**User Feedback (Round 1):**
*   User tested and reported that `stats.avg365[10]` was actually pulling "New, 3rd Party FBA - 365 days avg."

**Troubleshooting & Log Analysis:**
*   User provided debug logs including the raw `stats_raw` object from the Keepa API response for an ASIN (e.g., 0262611317).
*   The `stats_raw['avg365']` array was: `[-1, 9021, 2742, 1435763, 4999, -1, -1, 9257, -1, -1, 11460, 1, 9, -1, -1, -1, 40, 12, 6916, 9646, 3946, 3206, 1857, -1, -1, -1, -1, -1, -1, -1, -1, -1, 3683, -1]`
*   Analysis of this array against known Keepa data patterns and user-observed values on Keepa.com:
    *   `avg365[10]` (value 11460 or $114.60) matched "New, 3rd Party FBA - 365 days avg."
    *   `avg365[18]` (value 6916 or $69.16) was identified as the correct data point for "Buy Box - 365 days avg.", closely matching the user's observation of $68.31 on Keepa.com for the test ASIN.

**Revised Approach & Solution:**
*   Updated `buy_box_365_days_avg` function in `stable_products.py` to use `buy_box_avg_index = 18`.
*   The function now correctly targets `product.get('stats', {}).get('avg365', [])[18]`.

**Outcome:**
*   User confirmed the fix after the second round of testing. The "Buy Box - 365 days avg." column now populates with the correct data.

## Date: YYYY-MM-DD
**Task:** Implement "Amazon - 365 days avg." column.
**Issue:** Column was showing "-" for all entries.
**Solution:**
1.  **Created `amazon_365_days_avg(product)` function in `stable_products.py`:**
    *   Retrieves data from `product['stats']['avg365'][0]`.
    *   Formats the price as a currency string (e.g., "$12.34").
    *   Returns "-" if data is invalid or missing.
    *   Includes logging for value retrieval and formatting steps.
2.  **Updated `field_mappings.py`:**
    *   Imported `amazon_365_days_avg` from `stable_products.py`.
    *   Mapped `amazon_365_days_avg` to the "Amazon - 365 days avg." header in `FUNCTION_LIST`.
**Result:** The "Amazon - 365 days avg." column should now correctly populate with data. Awaiting user testing and confirmation.
**Files Modified:** `stable_products.py`, `field_mappings.py`.

Date: 2024-06-28
Version: 5.2 (Hypothetical next version)
User: Jules (AI Agent)
Branch: jules-fix-new-365-avg (example branch name)

Issue:
- The "New - 365 days avg." column in Keepa_Deals_Export.csv was consistently showing "-" (no data).

Investigation:
- Reviewed `headers.json` to confirm the header name.
- Examined `field_mappings.py` and found that the `FUNCTION_LIST` had a `None` entry for the "New - 365 days avg." header. This meant no function was being called to populate this data.
- Confirmed that the Keepa API, when called with `stats=365` (as it is in `Keepa_Deals.py`), should return 365-day average data in the `product['stats']['avg365']` array.
- Deduced that index 1 of the `avg365` array corresponds to the "New" price (similar to how index 0 is Amazon, index 2 is Used in `stats.current` and other `avg` arrays).

Resolution:
1.  **`stable_products.py`**:
    *   Added a new function `new_365_days_avg(product)`.
    *   This function retrieves `product['stats']['avg365'][1]`.
    *   It formats the price (dividing by 100) and handles cases where data is missing or invalid, returning '-' in such scenarios.
    *   Ensured standard field-order comments (`# New - 365 days avg. starts/ends`) were included.

2.  **`field_mappings.py`**:
    *   Imported `new_365_days_avg` from `stable_products.py` at the appropriate location within the import block, maintaining the commented-out structure for unimplemented fields.
    *   Updated the `FUNCTION_LIST` by replacing `None` with `new_365_days_avg` at the index corresponding to the "New - 365 days avg." header (index 87).

Testing:
- Code changes prepared for local testing by the user to verify the "New - 365 days avg." column populates correctly.

## Jules - 2024-07-30
*   **Issue**: The "New, 3rd Party FBA - 365 days avg." column in `Keepa_Deals_Export.csv` was showing all "-" (empty values).
*   **Investigation**:
    *   Confirmed the column was correctly defined in `headers.json`.
    *   Found that `field_mappings.py` had a `None` entry in `FUNCTION_LIST` for this column, meaning no data processing function was assigned.
    *   Cross-referenced `AGENTS.md` and `API_Dev_Log.txt`, which indicated that `stats.avg365[10]` from the Keepa product data object is the source for "New, 3rd Party FBA - 365 days avg.".
    *   The `Keepa_Documentation-official.md` was for the Python SDK, not direct API calls, so it didn't explicitly map `stats.avg365` indices. Relied on project-specific notes.
*   **Fix**:
    1.  Created a new function `new_3rd_party_fba_365_days_avg(product_data)` in `stable_products.py`.
        *   This function accesses `product_data['stats']['avg365'][10]`.
        *   It handles potential `KeyError`, `IndexError`, or `TypeError` if the data is missing or malformed, returning "-" in such cases.
        *   It converts the price from cents to a string formatted to two decimal places (e.g., "123.45"). If the value is -1 (Keepa's way of indicating no data), it returns "-".
    2.  Imported `new_3rd_party_fba_365_days_avg` into `field_mappings.py` within "Chunk 1 starts".
    3.  Replaced the `None` placeholder in `FUNCTION_LIST` (in `field_mappings.py`, "Chunk 2 starts") with the `new_3rd_party_fba_365_days_avg` function, ensuring the correct order was maintained.
*   **Outcome**: The column should now populate with the 365-day average price for New, 3rd Party FBA offers. Code provided to user for local testing with API tokens.

### Date: YYYY-MM-DD
**Issue**: The "New, 3rd Party FBM - 365 days avg." column in `Keepa_Deals_Export.csv` was consistently showing "-" (no data).
**Investigation**:
- Verified the target data point in Keepa API: `product.stats.avg365[7]` was identified as the 365-day average for New FBM offers (price in cents). This aligns with `stats.current[7]` used for "New, 3rd Party FBM - Current".
- Ensured `AGENTS.md` guidance on `stats` array indexing was considered.
**Solution**:
1.  Created a new function `new_3rd_party_fbm_365_days_avg(product_data)` in `stable_products.py`.
    - This function safely accesses `product_data.get('stats', {}).get('avg365', [])`.
    - It retrieves the value at index 7.
    - If valid, converts from cents to dollars, formats to "$XX.YY", and returns `{'New, 3rd Party FBM - 365 days avg.': formatted_price}`.
    - Returns `'-'` for invalid/missing data.
    - Includes standard logging.
2.  Updated `field_mappings.py`:
    - Imported `new_3rd_party_fbm_365_days_avg` from `stable_products.py`.
    - Mapped this function to the "New, 3rd Party FBM - 365 days avg." header in `FUNCTION_LIST`.
**Debugging Notes**:
- Encountered an `ImportError` during initial testing (`ImportError: cannot import name 'new_3rd_party_fbm_365_days_avg' from 'stable_deals'`).
- **Resolution**: Corrected `field_mappings.py` to ensure the new function was imported from `stable_products.py` and removed any erroneous import attempts from `stable_deals.py`. This involved carefully checking both the `stable_products` import block and ensuring no lingering references in the `stable_deals` import block for this specific function.
**Outcome**: The "New, 3rd Party FBM - 365 days avg." column should now populate correctly. User confirmed the fix.

## Dev Log Entry - YYYY-MM-DD

**Task:** Investigate and fix the "New, 3rd Party FBM - Current" column, which was reportedly empty ("-").

**Investigation:**
*   Reviewed the `new_3rd_party_fbm_current` function in `stable_products.py`. It was found to be parsing `product['offers']` to determine the price.
*   Consulted `AGENTS.md`, which has a specific entry: "Task: Fix "New, 3rd Party FBM - Current" Column (June 2025)". This entry states that `product['stats']['current'][7]` is the direct source for "New, 3rd Party FBM - Current price, including shipping".
*   `AGENTS.md` also emphasizes a "Strict Data Sourcing" convention: if the direct `stats` field is invalid, the column should output "-" and *not* fall back to parsing general offers for this specific, named column.

**Action Taken:**
*   Modified the `new_3rd_party_fbm_current` function in `stable_products.py` to align with the `AGENTS.md` guidance.
*   The function now first attempts to retrieve the price from `product['stats']['current'][7]`.
*   If `stats.current[7]` is valid (exists, is positive), it's used, and the price is formatted.
*   If `stats.current[7]` is invalid (missing, None, not positive, or index out of bounds), the function now correctly returns "-" for the column.
*   The previous logic of falling back to parsing `product['offers']` for this specific column was removed to adhere to the strict sourcing rule.
*   Enhanced logging within the function to show which data source is being used or why it's resulting in "-".

**Reasoning:**
The change ensures that the "New, 3rd Party FBM - Current" column strictly reflects the specific data point identified by Keepa for this metric (`stats.current[7]`). If this data point is unavailable from the API for a given product, the column will display "-", accurately representing the absence of this specific Keepa-aggregated value, rather than attempting to derive a potentially different value from general offers. This aligns with the established conventions for maintaining data integrity for columns that map to specific Keepa-provided fields. The user's report of the column being empty ("-") is consistent with `stats.current[7]` being unavailable for the ASINs they were testing, assuming a previous version of the code *was* correctly using `stats.current[7]`. This change brings the current code into compliance with documented best practices for this column.

Date: <YYYY-MM-DD>
Task: Implement 'Buy Box Used - 365 days avg.' column.
Developer: Jules
Branch: <branch_name_placeholder>

Summary:
- Added new function `buy_box_used_365_days_avg` to `stable_products.py`.
- The function targets `product_data.stats.avg365[32]`. This index was hypothesized based on `stats.current[32]` being used for 'Buy Box Used - Current' and was **confirmed by user's local testing** to provide correct data.
- The function converts price from cents to a formatted dollar string and includes error handling and logging.
- Updated `field_mappings.py` to import and use the new function in `FUNCTION_LIST`.
- User performed local testing, which confirmed the functionality and the correctness of index 32.

Next Steps:
- (None for this specific issue, resolved by user testing.)

## Jules - 2024-07-30

**Task:** Resolve persistent `ImportError` for `used_like_new_365_days_avg` in `field_mappings.py`.

**Issue:**
- `field_mappings.py` was attempting to import `used_like_new_365_days_avg` from `stable_deals.py` (where it does not exist) in addition to correctly importing it from `stable_products.py`.
- The `FUNCTION_LIST` in `field_mappings.py` also had a `None` placeholder for this function.

**Resolution:**
1.  **Corrected `field_mappings.py`:**
    - Ensured `used_like_new_365_days_avg` is imported *only* from `stable_products.py`. The erroneous import attempt from `stable_deals.py` (around line 230, specifically within the import block starting `from stable_deals import (`) was commented out.
    - Updated the `FUNCTION_LIST` at the corresponding index (149 for "Used, like new - 365 days avg.") to correctly reference the `used_like_new_365_days_avg` function.
2.  **Verification:**
    - Confirmed `used_like_new_365_days_avg` function definition exists in `stable_products.py`.
    - Confirmed `used_like_new_365_days_avg` function definition does NOT exist in `stable_deals.py`.
3.  **Iterative Debugging:**
    - The issue required multiple attempts to ensure the changes were correctly applied and reflected, particularly ensuring the erroneous import line was truly commented out. This highlighted the importance of careful verification of file state after modifications, especially when dealing with persistent import errors.

**Outcome:** `ImportError` resolved. Script proceeds past the previous error point.

## Dev Log Entry - YYYY-MM-DD

**Task:** Implement "Used, very good - 365 days avg." column.

**Issue:**
*   The "Used, very good - 365 days avg." column in `Keepa_Deals_Export.csv` was showing all "-" as no function was assigned to populate it.
*   Encountered an `ImportError: cannot import name 'FUNCTION_LIST' from 'field_mappings'` during development.

**Solution:**
1.  **Created `used_very_good_365_days_avg(product)` function in `stable_products.py`:**
    *   This function targets `product.stats.avg365[20]`. Index 20 corresponds to "Used - Very Good" average prices in Keepa's `stats.avg...` arrays (similar to how `stats.current[20]` is used for "Used, very good - Current").
    *   The function converts the price from cents to a formatted dollar string (e.g., "$XX.YY").
    *   Includes error handling for missing/invalid data (returns "-") and standard logging.
2.  **Updated `field_mappings.py`:**
    *   Imported `used_very_good_365_days_avg` from `stable_products.py`.
    *   Correctly mapped this function to the "Used, very good - 365 days avg." header in the `FUNCTION_LIST`.

**Debugging the `ImportError`:**
*   The `ImportError` for `FUNCTION_LIST` was transient.
*   Initial troubleshooting involved verifying the new function's syntax in `stable_products.py` and its import/usage in `field_mappings.py`.
*   A temporary reversion of the new code was performed as a diagnostic step.
*   The error resolved once the correct code was in place and the script was re-run, suggesting the issue might have been an intermittent environment state or a subtle syntax issue that was fixed during the iterative process of applying and checking the code. The core logic for the new field was sound.

**Outcome:**
*   The "Used, very good - 365 days avg." column now populates correctly with data. User confirmed with local testing.

**Files Modified:**
*   `stable_products.py`
*   `field_mappings.py`

## Dev Log Entry - YYYY-MM-DD

**Task:** Implement "Used, good - 365 days avg." column.

**Issue:**
*   The "Used, good - 365 days avg." column in `Keepa_Deals_Export.csv` was showing all "-" as no function was assigned to populate it.

**Solution:**
1.  **Created `used_good_365_days_avg(product_data)` function in `stable_products.py`:**
    *   This function targets `product_data['stats']['avg365'][21]`. Index 21 was determined based on the existing pattern where "Used, good - Current" uses `stats.current[21]`, and other condition-specific 365-day averages (like "Used, like new - 365 days avg." using `stats.avg365[19]` and "Used, very good - 365 days avg." using `stats.avg365[20]`) use the corresponding index in the `stats.avg365` array.
    *   The function converts the price from cents to a formatted dollar string (e.g., "$XX.YY").
    *   Includes error handling for missing/invalid data (returns "-") and standard logging.
2.  **Updated `field_mappings.py`:**
    *   Imported `used_good_365_days_avg` from `stable_products.py`.
    *   Correctly mapped this function to the "Used, good - 365 days avg." header in the `FUNCTION_LIST`.

**Outcome:**
*   The "Used, good - 365 days avg." column now populates correctly with data. User confirmed with local testing, and results were a perfect match.

**Learnings:**
*   Reinforced the pattern that `stats.avg365[X]` often corresponds to `stats.current[X]` for similar price types. In this case, `stats.current[21]` (Used, good - Current) successfully predicted that `stats.avg365[21]` would be "Used, good - 365 days avg.".

**Files Modified:**
*   `stable_products.py`
*   `field_mappings.py`

2025-0X-XX:
- Investigated and fixed "Used, acceptable - 365 days avg." column showing all "-".
- Problem: No function was assigned in `field_mappings.py` and the function `used_acceptable_365_days_avg` was missing in `stable_products.py`.
- Solution:
    - Created `used_acceptable_365_days_avg` in `stable_products.py` to fetch data from `stats.avg365[22]`.
    - Updated `field_mappings.py` to import and use this new function.
- Result: Column now populates correctly. Confirmed via user testing.
- Note: Identified `stats.avg365[22]` as the source for this average, consistent with `stats.current[22]` for the current price of the same condition. This pattern (index consistency between `current` and `avgX` arrays for specific conditions) seems to be a reliable heuristic for Keepa data.

Date: 2025-06-29
Task: Fix "New Offer Count - Current", "New Offer Count - 365 days avg.", "Used Offer Count - Current", and "Used Offer Count - 365 days avg." columns showing "-" or incorrect large numbers.
Jules Version: [Your Current Jules Version, e.g., Jules vX.Y]

Problem: Initial implementation used incorrect indices (stats.current[5] & [6], stats.avg365[5] & [6]) for offer counts, leading to wildly inaccurate data or hyphens.

Solution:
1.  Analyzed `stats_raw` object from `debug_log.txt` for ASIN 0804840385.
2.  Determined correct sources for offer counts:
    *   **New Offer Count - Current**: Sum of `product['stats'].get('offerCountFBA', 0)` and `product['stats'].get('offerCountFBM', 0)`.
    *   **Used Offer Count - Current**: Calculated as `product['stats'].get('totalOfferCount', 0) - (sum of new FBA & FBM counts)`. Includes checks for missing `totalOfferCount` and ensures the result is not negative.
    *   **New Offer Count - 365 days avg.**: `product['stats']['avg365'][11]`. This index aligns with `product.csv[11]` which is documented as `COUNT_NEW` history.
    *   **Used Offer Count - 365 days avg.**: `product['stats']['avg365'][12]`. This index aligns with `product.csv[12]` which is documented as `COUNT_USED` history.
3.  Updated functions in `stable_products.py`:
    *   `new_offer_count_current`
    *   `used_offer_count_current`
    *   `new_offer_count_365_days_avg`
    *   `used_offer_count_365_days_avg`
4.  Updated `field_mappings.py` to use the corrected functions.
5.  Updated `AGENTS.md` with these new data source details.

Result: User tested locally, and results are now an almost perfect match with Keepa.com data. Issue resolved.

Learnings:
*   Directly named fields in `product['stats']` (e.g., `offerCountFBA`, `totalOfferCount`) are reliable for current data points when available.
*   Indices in `stats.current` and `stats.avg...` arrays for aggregated counts (like average new/used offer counts) can often be inferred from the `product.csv` historical data array definitions (e.g., `csv[11]` for `COUNT_NEW` history mapping to `avg365[11]` for its average).
*   Initial assumptions about `stats` array indices based on simple ordering can be incorrect; detailed `stats_raw` review is crucial for complex or less common fields.


Implement FBA Pick & Pack Fee Collection

This commit updates the Keepa deals script to accurately collect and display the "FBA Pick & Pack Fee".

Key changes:
- Modified `Keepa_Deals.py` to log raw product data for a specific ASIN (1562243179) to help identify the correct data field.
- Analyzed the logged data, confirming the FBA Pick & Pack fee is located at `product_data['fbaFees']['pickAndPackFee']` and is provided in cents.
- Updated the `get_fba_pick_pack_fee` function in `stable_products.py` to correctly access this nested field and convert the value from cents to a dollar string.

The user has confirmed locally that this change correctly populates the "FBA Pick&Pack Fee" column in the `Keepa_Deals_Export.csv` output file.

YYYY-MM-DD (Jules): Referral Fee % Implementation & Precision Fix
- Task: Investigate "Referral Fee %" column showing all "-" and ensure accurate data collection.
- Initial Implementation: Added `get_referral_fee_percent` function in `stable_products.py` with speculative checks for `referralFeePercent` and `fbaFees.referralFeePercent` or `fbaFees.referralFee.percent` based on common API patterns, as the field was not explicitly in Keepa_Documentation-official.md.
- First User Feedback: Data was being pulled (mostly as 15%) but was not precise (e.g., Keepa showed 14.99%, CSV showed 15.00% after initial `.2f` formatting).
- Investigation: User provided RAW_PRODUCT_DATA for an ASIN (1562243179). Analysis of the JSON revealed two relevant keys at the root of the product object:
    - `"referralFeePercent": 15` (integer)
    - `"referralFeePercentage": 14.99` (decimal, more precise)
- Solution: Modified `get_referral_fee_percent` to prioritize `product_data.get('referralFeePercentage')` for extraction. If unavailable, it falls back to `product_data.get('referralFeePercent')`, and then to previously checked nested fbaFee locations. Ensured the final value is explicitly cast to float before formatting with `:.2f%`.
- Outcome: User confirmed a perfect match with Keepa.com data after the update.
- Learning: For fee percentages, the Keepa API might provide both a rounded integer (`referralFeePercent`) and a more precise decimal (`referralFeePercentage`). Always prioritize the field with greater precision if available. The precise field name might not always be obvious from general documentation and can require inspection of raw API responses.

Date: <2025 july 5>
User: Tim Emery
Jules: Yes
Task: Fix "Brand" field showing all "-" in Keepa_Deals_Export.csv.
Details:
- Added `get_brand` function to `stable_products.py` to extract product.get('brand', '-').
- Imported `get_brand` into `field_mappings.py`.
- Updated `FUNCTION_LIST` in `field_mappings.py` to use `get_brand` for the "Brand" column.
- User confirmed the fix locally; "Brand" column now populates correctly.
Outcome: Success.

# Development Log - Keepa API Rate Limit Investigation (Session v3.1 - Continued)

**Date:** [Current Date]
**Developer:** Jules (AI Assistant)
**Task Focus:** Analyzing and mitigating Keepa API 429 "Too Many Requests" errors in `Keepa_Deals.py`.

## Key Learnings & Decisions:

1.  **Confirmation of No Rate Limit Headers:** A critical piece of information was re-confirmed: the Keepa API **does not** provide standard rate limit headers (e.g., `x-rate-limit-limit`, `x-rate-limit-remaining`, `x-rate-limit-reset`). This was reportedly discovered in a previous session but was overlooked at the start of the current analysis.
    *   **Impact:** This means any client-side logic attempting to dynamically adjust request rates based on these specific headers is inherently flawed and will not function as intended. The script's `rate_limit_info` dictionary would always reflect null/None for these values.

2.  **Ineffectiveness of Previous Header Logging Attempts:** Efforts in the current session to add detailed logging for *all* response headers to identify these rate limit headers were, therefore, based on a misunderstanding. The logs provided for analysis did not contain these new header log lines, initially suggesting an issue with the logging modification itself. However, the root cause is the absence of the headers from Keepa's responses.

3.  **Strategy Pivot to Fixed Delays:** Given that dynamic adjustment based on server-sent headers is not an option, the strategy has pivoted entirely to a more conservative **fixed delay mechanism**.

## Actions Taken in This Session:

1.  **Initial Analysis (Based on Misunderstanding):**
    *   Reviewed logs which showed persistent 429 errors and `rate_limit_info` as consistently `None`.
    *   Attempted to enhance header logging in `Keepa_Deals.py` to capture raw headers, believing they might be present but differently named.

2.  **Revised Strategy (Post-Clarification on Headers):**
    *   **Modified `Keepa_Deals.py`:**
        *   Increased `MIN_TIME_SINCE_LAST_CALL_SECONDS` from 30 to **60 seconds** to significantly reduce request frequency.
        *   **Removed all logic** related to parsing `x-rate-limit-*` headers from API responses.
        *   **Removed dynamic token adjustment logic** that relied on these non-existent headers.
        *   Cleaned up associated logging to prevent misleading messages about missing headers or token discrepancies with server values.
        *   The script's local token quota system remains for client-side estimation but no longer attempts to reconcile with (non-existent) server-side header values.
    *   **Documentation:**
        *   Proposed an update for `AGENTS.md` to permanently record the finding that Keepa does not provide these rate limit headers, to prevent future redundant analysis.

## Next Steps:

1.  **User Testing:** Awaiting results from a user test run of the modified `Keepa_Deals.py` (with `MIN_TIME_SINCE_LAST_CALL_SECONDS = 60`). The primary goal is to observe if 429 errors are eliminated or significantly reduced.
2.  **Further Adjustments (If Necessary):** If 429 errors persist, `MIN_TIME_SINCE_LAST_CALL_SECONDS` may need to be further increased.
3.  **Finalization:** Once a stable configuration is achieved, revert `MAX_DEALS_TO_PROCESS_FOR_TESTING` to a production value and submit the changes.

## Reflections:

*   The re-emergence of the fact that Keepa doesn't send rate limit headers was crucial. Ensuring such key environmental/API constraints are well-documented and easily accessible (e.g., in `AGENTS.md`) is vital for efficient development and avoiding repeated troubleshooting cycles.

## Keepa API - Batch Product Query Details (Follow-up Research)

**Date of Research:** July 5, 2025 (via Grok, second query)
**Source:** Primarily Keepa API documentation and community forums, focusing on direct HTTP implications and Python library behavior.

**Key Verifications & New Details for Batch Queries:**

1.  **Token Cost Breakdown:**
    *   The base cost for an ASIN in a batch is **1 token**.
    *   **Crucially, additional parameters like `offers` (2 extra tokens/ASIN) or `buybox` still apply their costs *per ASIN within the batch*.** There's no reduction in token cost per ASIN for the *same data parameters* when batching versus individual calls. The primary benefit is reduced HTTP overhead and potentially different call rate treatment.
    *   Example: A request for 100 ASINs with parameters that cost X tokens per ASIN individually will still cost `100 * X` tokens in a batch (if X is the sum of base + parameter costs).

2.  **Response Structure (Confirmed):**
    *   The JSON response for a batch query includes a top-level `products` array. Each element in the array is a dictionary for an ASIN, containing details like `asin`, `title`, `data`, etc.
    *   The response may also contain top-level keys like `tokensLeft` and `refillIn` (milliseconds), which could provide direct feedback on token status if present in direct HTTP calls.
    *   Invalid ASINs in a batch return no data for that ASIN but still consume their token share.

3.  **Rate Limiting Behavior & Safe Frequency:**
    *   A batch request (e.g., for 100 ASINs) is treated as a **single HTTP request** by Keepa.
    *   The primary rate limit is the overall token quota and its refill rate (typically 5% of max tokens per hour).
    *   Explicit per-second/minute call limits are not documented, but 429 errors (`NOT_ENOUGH_TOKEN`) occur if the token bucket is empty.
    *   A **safe frequency** suggested for batch calls (especially when using the Python library's `wait=True` parameter, which handles some throttling) is around **1 to 2 batch requests per minute**. This implies a delay of 30-60 seconds *between batch calls* could be a good starting point.
    *   Standard advice is to use exponential backoff if 429s are encountered.

**Implications for `Keepa_Deals.py` Strategy:**
*   The major speed benefit of batching comes from processing up to 100 ASINs with the overhead of a single HTTP call, and potentially allowing a faster *effective* ASIN processing rate (e.g., 100 ASINs every 30-60 seconds vs. 1 ASIN every 60 seconds).
*   The total number of tokens consumed for the same dataset and parameters will likely remain similar to individual calls.
*   If direct HTTP batch calls also return `tokensLeft` and `refillIn`, this could allow for a much more accurate client-side token management and dynamic delay system than previously thought possible.
*   The `MIN_TIME_SINCE_LAST_CALL_SECONDS` in `Keepa_Deals.py` would need to be re-evaluated to be the delay *between batch calls*.

**Next Steps (Post-Current Test):**
1.  Confirm the outcome of the current test (individual calls with 60s delay).
2.  If proceeding with batch implementation: 
    *   Prioritize verifying the exact token cost per ASIN for the *specific parameters* we use (`stats`, `offers`, `stock`, `buybox`, etc.) when called in a batch via direct HTTP.
    *   Verify if `tokensLeft` and `refillIn` are present in the direct HTTP batch response.
    *   Plan a phased implementation, starting with a new function for batch fetching via direct HTTP.

## Date: 2025-07-08 (Evening)
**Developer:** Jules (AI Assistant) & Tim Emery
**Task Focus:** Major refactor of token management in `Keepa_Deals.py` based on new Keepa API documentation details obtained from Keepa Support.

**Key Discoveries from Keepa Documentation (re: `/product` endpoint with `offers` parameter):**

1.  **`tokensConsumed` Field is Authoritative:** The API response JSON (even for 429s, if a JSON body is returned) includes a `tokensConsumed` field. This is the actual cost and supersedes estimations.
2.  **Base ASIN Cost Waived with `offers`:** The standard "1 token per ASIN" base cost is **waived** when the `&offers=` parameter is used.
3.  **`&offers=N` Cost:** This is the primary driver. Cost is **6 tokens per found offer page** (an offer page contains up to 10 offers) *per ASIN*. This cost is variable.
4.  **`&buybox=1` Cost:** **0 tokens** (ignored) when `&offers=` is also used, as `offers` includes this data.
5.  **`&stats=DAYS` Cost:** **0 tokens.**
6.  **`&history=1` Cost:** **0 tokens.**
7.  **`&rating=1` Cost:** **0 or 1 token** per product (freshness dependent).
8.  **`&stock=1` Cost (with `offers`):** **0 or 2 tokens** per product (freshness dependent).

**Modifications to `Keepa_Deals.py` based on new understanding:**

1.  **`fetch_product_batch` Refactored:**
    *   Prioritizes parsing `tokensConsumed` from the API response (including 429 JSON bodies) to set `actual_batch_cost`.
    *   Removed static per-ASIN cost estimation for `actual_batch_cost` calculation *within* this function.
2.  **Token Deduction Logic Updated (Main Loop):**
    *   Deducts `actual_batch_cost` (from `tokensConsumed`) after successful `fetch_product_batch` calls.
    *   Crucially, if a failed call (e.g., 429) reports `tokensConsumed > 0` via the API response, these tokens are now also deducted.
3.  **Pre-Call Token Estimation Revised:**
    *   `TOKEN_COST_PER_ASIN` constant was replaced by `ESTIMATED_AVG_COST_PER_ASIN_IN_BATCH`, initialized to `15`. This is used *only* for the pre-call check to decide if a wait is needed. The actual deduction uses `tokensConsumed`.
4.  **Token Refill Simulation Overhauled (Previously done, but relevant context for this overall fix):**
    *   Uses `TOKENS_PER_MINUTE_REFILL = 5` and `REFILL_CALCULATION_INTERVAL_SECONDS = 60` for more dynamic and accurate local token simulation.
5.  **`NameError` for `HOURLY_REFILL_PERCENTAGE` Fixed:** Ensured all pre-call wait calculations use the new per-minute refill constants.
6.  **Default Test Values Updated:** `MAX_DEALS_TO_PROCESS_FOR_TESTING` to 150, `MAX_ASINS_PER_BATCH` to 50.
7.  **CSV ASIN Sanitization & Log Overwrite:** Implemented as per prior log entries.

**Next Steps:**
*   User to perform a controlled test run (e.g., 150 deals, 50/batch) with these comprehensive changes. The main goals are to verify that `tokensConsumed` is used correctly, the local token count aligns better with Keepa's actual balance, and the script handles API interactions (including 429s) robustly and efficiently.

## Date: 2025-07-11
**Task:** Correct Token Estimation Constant & Conduct Full Test Run of Keepa_Deals.py (Follow-up from Jules)
**Branch:** jules/correct-token-est-full-test-v1

**Summary of Actions & Findings (Task leading to debug_log-bigrun.txt):**
1.  Set `ESTIMATED_AVG_COST_PER_ASIN_IN_BATCH` to 4 in `Keepa_Deals.py`.
2.  Identified and fixed a `NameError` in `fetch_product_batch` related to `actual_token_cost` vs `actual_batch_cost`.
3.  User executed a full test run with these changes, processing 809 ASINs over approximately 25.75 hours (NoCache).
    - Output: `Keepa_Deals_Export.csv` (800 rows, ~10 partially missing), `debug_log-bigrun.txt`.
4.  Initial analysis of log excerpts showed `ESTIMATED_AVG_COST_PER_ASIN_IN_BATCH = 4` was significantly lower than actual batch costs (e.g., one batch of 50 ASINs cost 371 tokens, avg 7.42/ASIN). This caused the script to frequently enter token deficit, extending run time due to recovery pauses.
5.  The script completed the full run, indicating robust error handling and token management logic, despite the underestimation.
6.  Full analysis of `debug_log-bigrun.txt` is pending for the next task to determine a more accurate average token cost and investigate data integrity for a few reported missing rows/cells.

**Committed changes:**
- `ESTIMATED_AVG_COST_PER_ASIN_IN_BATCH = 4`
- `NameError` fix in `fetch_product_batch`.

## Date: 2025-07-12
**Developer:** Jules (AI Assistant)
**Task:** Analyze Full Test Run (`debug_log-bigrun.txt`) & Refine `Keepa_Deals.py` Token Management and Data Integrity.
**Branch:** (Will be created upon submission)

**Summary of Actions & Findings:**

1.  **`ESTIMATED_AVG_COST_PER_ASIN_IN_BATCH` Update:**
    *   Full analysis of `debug_log-bigrun.txt` (6MB) for token consumption was not feasible due to tool limitations with large file reads.
    *   Based on user (Tim) input, `ESTIMATED_AVG_COST_PER_ASIN_IN_BATCH` was updated from `4` to `6` in `Keepa_Deals.py`. This aims to provide a more realistic pre-call estimate to reduce token deficit pauses. Actual token deduction continues to use `tokensConsumed` from API responses.

2.  **Data Integrity Investigation (`Keepa_Deals_Export-missingcontent.csv` & `debug_log-bigrun.txt` grep):**
    *   Analyzed `Keepa_Deals_Export-missingcontent.csv` (10 rows with missing data) and used `grep` on `debug_log-bigrun.txt` for the affected ASINs.
    *   Identified two primary script errors causing missing data:
        *   **`TypeError: deal_found() takes from 1 to 3 positional arguments but 4 were given`**: Occurred during an initial processing pass for these ASINs.
        *   **`NameError: name 'deal' is not defined`**: Occurred during a subsequent processing pass for the same ASINs when functions like `deal_found`, `last_update`, and `last_price_change` were called.
    *   **Fixes Implemented in `Keepa_Deals.py`:**
        *   Removed a duplicated and erroneous code block responsible for iterating `FUNCTION_LIST` (this block was causing the `TypeError` by incorrectly calling `deal_found` with 4 arguments).
        *   In the remaining (intended) `FUNCTION_LIST` iteration block, corrected `input_data = deal` to `input_data = original_deal_obj`. This resolved the `NameError` as `original_deal_obj` holds the correct deal data in that scope.
        *   These changes ensure that `deal_found` is called correctly and that `original_deal_obj` is properly referenced.

3.  **CSV Row Construction Logic Review & Fix:**
    *   Identified a significant bug where the final `rows` list passed to `write_csv` was not correctly incorporating all processed data and placeholders.
    *   **Fix Implemented in `Keepa_Deals.py`:**
        *   Removed premature addition of placeholders to the main `rows` list.
        *   The script now uses `temp_rows_data` (which stores processed rows with their `original_index`) to build a `final_processed_rows` list.
        *   This `final_processed_rows` list is initialized to the full length of `deals_to_process` and then populated. Any remaining `None` slots (for ASINs skipped very early, e.g., due to `validate_asin` failure) are filled with placeholders.
        *   `write_csv` is now called with `final_processed_rows`, ensuring all deals are represented in the correct order.

4.  **Other Script Review:**
    *   Reviewed token refill logic and 429 error handling; these appear robust for the current batch processing flow.
    *   The `fetch_product` function (single ASIN fetch) is not actively used in the main deal processing loop, so its 429 handling is less critical but noted.

**Files Modified:**
*   `Keepa_Deals.py`: Updated `ESTIMATED_AVG_COST_PER_ASIN_IN_BATCH`, removed duplicate code block, fixed `NameError`, and corrected CSV row construction logic.
*   `API_Dev_Log.txt`: This entry added.

**Next Steps:**
*   User to test the modified script to confirm data integrity improvements and overall stability.
*   Submit changes after successful testing.

## Date: 2025-07-12
**Developer:** Jules (AI Assistant)
**Task:** Analyze Full Test Run (`debug_log-bigrun.txt`) & Refine `Keepa_Deals.py` Token Management and Data Integrity.
**Branch:** jules/fix-data-integrity-and-token-est-v2

**Summary of Actions & Findings:**

1.  **`ESTIMATED_AVG_COST_PER_ASIN_IN_BATCH` Update:**
    *   Full analysis of `debug_log-bigrun.txt` (6MB) for token consumption was not feasible due to my limitations with large file reads.
    *   Based on your (Tim) input, I updated `ESTIMATED_AVG_COST_PER_ASIN_IN_BATCH` from `4` to `6` in `Keepa_Deals.py`.

2.  **Data Integrity Investigation & Fixes:**
    *   Analysis of `debug_log-bigrun.txt` excerpts revealed a persistent `TypeError` for the `deal_found` function, which was the root cause for missing data in certain rows.
    *   The fix involved a deep-dive into `Keepa_Deals.py` to locate and correct the specific function call that was passing an incorrect number of arguments. This required multiple iterations, as the initial fix was incomplete.
    *   Also corrected a major structural bug in `Keepa_Deals.py` where the final list of rows for the CSV was not being assembled correctly, leading to missing rows. The logic now ensures all deals are represented in the final CSV in the correct order.
    *   Minor tweak made to `stable_products.py` to ensure `Manufacturer`, `Brand`, `Author`, and `Binding` fields correctly show `'-'` for empty string values.

**Outcome:**
*   You confirmed that subsequent test runs no longer produced the `TypeError` and that the previously missing data is now correctly populated in the CSV export. The core data integrity issues are resolved.
*   A new, minor issue (missing 'Deal found' column data) was identified and deferred to a future task.