# Dev Log: Fix Strategy Extraction Parsing & Update RBAC Documentation

**Date:** January 4, 2025 **Task:** Fix "Guided Learning" data persistence issue and update documentation for Role-Based Access Control (RBAC).

## 1. Task Overview

The primary objective was to investigate and fix a reported issue where strategies extracted via the "Guided Learning" tool were not appearing in the `strategies.json` database or the UI. The user suspected a silent failure during the save process.

Secondarily, the user requested a comprehensive update to the project documentation (`README.md`, `AGENTS.md`, etc.) to accurately reflect the application's current Role-Based Access Control (RBAC) policies, specifically clarifying which features are restricted to "Admin" vs. "User" roles.

## 2. Challenges & Diagnosis

### The "Missing Strategies" Bug

- **Symptom:** Content submitted to Guided Learning was processed, but the resulting strategies were not persisted to `strategies.json`.

- **Root Cause:** The `extract_strategies` and `extract_conceptual_ideas` functions rely on the xAI API (`grok-4-fast-reasoning`). It was discovered that the LLM frequently wraps its JSON output in Markdown code blocks (e.g., ````json [ ... ] ````). The existing code passed this raw string directly to `json.loads` in the `/approve` route (or the frontend display), causing a `json.JSONDecodeError` which was either silently caught or resulted in the data being discarded.

- Complexity:



  The issue occurred at two points:

  1. **Extraction:** The raw text returned to the UI contained the markdown artifacts.
  2. **Approval:** When the user clicked "Approve", the server attempted to parse this dirty string again. The fallback logic for "legacy text" (splitting by newline) was also at risk of including the markdown backticks as valid strategy text.

### Documentation Drift

- **Issue:** The codebase has evolved to include strict access control (Admin vs. User), but the core documentation files (`System_Architecture.md`, `README.md`) treated the system as having a single user type. This created ambiguity for future developers regarding the intended security model.

### File Timestamp Confusion

- **Issue:** The user noted that `strategies_structured.json` had an old timestamp (Jan 1st) despite new strategies being added (Jan 4th).
- **Finding:** Analysis confirmed that `strategies.json` is the active "live" database used by the application (`wsgi_handler.py` and `ava_advisor.py`). `strategies_structured.json` is merely a static artifact generated by the manual `migrate_strategies.py` utility and is not updated by the runtime application.

## 3. Actions Taken

### A. Code Fixes (`wsgi_handler.py`)

1. **Implemented Markdown Stripping:**

   - Added a regex sanitation step to both



     ```
     extract_strategies
     ```



     and



     ```
     extract_conceptual_ideas
     ```



     functions



     before



     returning the payload:

     ```
     content = re.sub(r'^```json\s*|\s*```$', '', content.strip(), flags=re.MULTILINE)
     ```

   - This ensures the UI receives clean, raw JSON ready for rendering.

2. **Robust Approval Logic:**

   - Updated the `/approve` route to re-apply this regex sanitation on the submitted form data before attempting `json.loads`.
   - Improved the fallback logic (used if JSON parsing fails) to explicitly filter out lines starting with backticks (`````), preventing markdown artifacts from being saved as "text strategies."

3. **Verification:** Created and ran a standalone script (`verify_markdown_strip.py`) to confirm the regex correctly handles various edge cases (with/without `json` label, leading/trailing whitespace).

### B. Documentation Updates

Updated the following files to explicitly define the Admin vs. User permissions:

- **`README.md`**: Marked features like "Guided Learning" and "Deals Config" as **(Admin Only)**.
- **`AGENTS.md`**: Added a "Role-Based Access Control (RBAC)" section to the Technical Notes.
- **`Documents_Dev_Logs/System_Architecture.md`**: Added a dedicated "User Roles & Access Control" section defining the specific routes accessible to each role.
- **`Documents_Dev_Logs/Feature_Deals_Dashboard.md`**: Clarified that "Deals Query Configuration" is restricted.
- **`Documents_Dev_Logs/Feature_Guided_Learning_Strategies_Brain.md`**: Added a prominent "Access Control" header stating these tools are strictly for Admins.

## 4. Outcome

- **Status:** **Successful**.
- Verification:
  - The parsing logic now robustly handles LLM outputs regardless of markdown wrapping.
  - Documentation accurately reflects the current security architecture.
  - The file timestamp discrepancy was verified as "working as designed" (manual artifact vs. live DB).