o Rgi=@sddlmZddlZddlZddlZddlZddlmZddlmZ ddl m Z ee Z ejedZdZe jd d d d ZdS) ) getLoggerN)datetime) celery_app) query_xai_apizintelligence.jsonhomogenization_statusz:keepa_deals.maintenance_tasks.homogenize_intelligence_task)namec Cstjtjj}|tt ddddt dt t dtt jtsEdt}t ||tt d|d dSzttd d d  }t|}Wd n1s^wY|st|tt dddWdSi}|D]&}t|trd|vr|d|t|d<qxt|trd |t|<qxg}|D] }t|trd|vr|t|dq|t|qd}t|}g} t d|dtd||D]} || | |} | |d} ||d|} |tt dd| d| ddddt | d}dddd|dgdd d!d"}t|}d#|vr;t d$| d%|d#| | qzI|d&dd'd}tjd(d)|tj d*}t!|}t|t"ru| |t d+| d,t| d-t|nt d.| d/| | Wqtj#t$t%fy}zt d0| d%|| | WYd }~qd }~wwg}t&'(d1}| D]}t|}||}|||r|n|d2qt|}||}t d3|d4|d5||dkr5t d6tz'ttd7d d }tj)||d8d9Wd n 1swYt d:Wnt*y4}z t d;||d }~wwt d<|tt d|d=|d>d?|WSt*yr}zt d@||tt dt|d |d }~ww)Az:Background task to homogenize intelligence.json using LLM.Runningz Starting...r)statusprogress removed_countz"Homogenization Task Started. CWD: zTarget Intelligence File: zFile not found: Error)r messagerzutf-8)encodingNComplete)r r content date_addedizStarting processing for z items...rzProcessing batch z of z...a9 You are a strict data cleaner. Below is a JSON list of "intelligence" items. **CRITICAL INSTRUCTIONS:** 1. Aggressively identify concepts that mean the same thing, even if phrased differently. 2. Merge them into a SINGLE, concise entry. 3. If two items share >50% conceptual overlap, KEEP ONLY THE BEST ONE. 4. Your goal is to REDUCE the list size by removing redundancy. 5. Return ONLY the final JSON list of strings. No markdown, no intro. **Input List:** z systemzYou are a data cleaner.)roleruserzgrok-4-fast-reasoningFg?)messagesmodelstream temperatureerrorz"xAI Error in homogenization chunk z: choicesrz^```json\s*|\s*```$)flagszChunk z : Reduced z -> z0Homogenization returned non-list JSON for chunk .z0Error parsing homogenization response for chunk z%Y-%m-%d)rrz#Homogenization complete. Original: z , Final: z , Removed: zWriting updated list to file: w)indentzFile write successful.z#Failed to write intelligence file: z#No items removed. File not updated.zComplete! Semantically merged z duplicate ideas.)r r rzError in homogenization task: )+redisRedisfrom_urlceleryconf broker_urlsetHOMOGENIZATION_STATUS_KEYjsondumpsloggerinfoosgetcwdINTELLIGENCE_FILEpathexistsropenload isinstancedictgetstrstripappendlenrangerextendresub MULTILINEloadslistJSONDecodeErrorKeyError IndexErrorrnowstrftimedump Exception) redis_client error_msgf intelligencecontent_date_mapitemintelligence_content_list CHUNK_SIZEtotal_originalall_cleaned_stringsichunkcurrent_chunk_num total_chunkspromptpayloadresultr cleaned_chunkefinal_objects_list today_strss_clean original_date final_countremoved write_errrf8/var/www/agentarbitrage/keepa_deals/maintenance_tasks.pyhomogenize_intelligence_tasks                &          rh)loggingrr/r+r#r?rworkerrr& ava_advisorr__name__r-r2joinr0r1r*taskrhrfrfrfrgs