ads fetch machanism change

[aliaslany]

divar had multiple changes in the way it gets/posts ads , so i applied changes to main.py and some other field.
its not completely verified, please double check it and tell me if something is wrong.

[qodo-code-review]

PR Summary by Qodo

Update Divar ad fetch to new postlist search API + scheduled GitHub Actions run

✨ Enhancement ⚙️ Configuration changes 🕐 20-40 Minutes

AI Description

• Switch Divar search from URL-based web-search to JSON POST postlist endpoint.
• Add city/category search inputs via env/secrets and harden request handling.
• Run bot on a 10-minute GitHub Actions schedule and persist seen tokens in-repo.

Diagram

graph TD
  A(["Schedule / Manual trigger"]) --> B(["GitHub Actions: run-bot"]) --> C["main.py"] --> D{{"Divar Search API\nPOST /postlist"}}
  C --> E{{"Divar Post API\nGET /posts-v2"}} --> F{{"Telegram Bot API"}}
  C --> G[("tokens.json\nseen tokens")]
  B --> H["git commit & push"] --> I["GitHub repo"]
  subgraph Legend
    direction LR
    _job(["Job/Step"]) ~~~ _code["Code"] ~~~ _ext{{"External"}} ~~~ _db[("State file")]
  end

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Store seen tokens outside the repo (Artifacts/Cache/KV store)

• ➕ Avoids committing to the main branch every 10 minutes
• ➕ Eliminates potential push/permission issues in Actions
• ➕ Keeps repo history clean
• ➖ Requires additional setup (artifact retention, cache keying, or external storage)
• ➖ Harder to inspect state directly in the repo

2. Implement cursor/pagination support for postlist results

• ➕ Reduces risk of missing ads when volume is high
• ➕ More robust than relying on a single first-page fetch
• ➖ More complexity (cursor handling, dedupe, throttling)
• ➖ May increase API load and runtime

3. Remove SEARCH_CONDITIONS remnants and validate env at startup

• ➕ Avoids confusion between old/new search mechanisms
• ➕ Fails fast when secrets are misconfigured
• ➖ Small breaking change if any deployments still rely on SEARCH_CONDITIONS

Recommendation: The PR’s approach (moving to the new postlist search API with city/category inputs) is the right direction given Divar’s API change. Consider switching token persistence away from repo commits and adding minimal startup validation (and removing SEARCH_CONDITIONS from the workflow/debug step) to reduce operational risk and confusion.

Files changed (3) +125 / -29

Enhancement (1) +83 / -29

main.pySwitch Divar search to POST /postlist with city/category body and new response shape +83/-29

Switch Divar search to POST /postlist with city/category body and new response shape

• Replaces the old URL-based web-search fetch with a JSON POST request to Divar’s postlist search endpoint, including new SEARCH_CITY_ID/SEARCH_CATEGORY env parameters and request headers. Updates token extraction to read from widget_list and simplifies the main loop to process a single (newest-first) page, while hardening tokens.json loading and HTTP error handling.

main.py

Other (2) +42 / -0

run-bot.ymlAdd scheduled GitHub Actions job to run bot and commit tokens.json +41/-0

Add scheduled GitHub Actions job to run bot and commit tokens.json

• Introduces a workflow triggered every 10 minutes (and manually) that checks out the repo, installs Python deps, runs main.py with secrets-provided env vars, then commits and pushes updates to tokens.json.

.github/workflows/run-bot.yml

tokens.jsonAdd initial persisted seen-ads state file +1/-0

Add initial persisted seen-ads state file

• Adds an empty JSON array file used to persist already-processed ad tokens between workflow runs.

tokens.json

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (3) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Unstable token file churn 🐞 Bug ☼ Reliability

Description

tokens is rebuilt via set() and then saved, which produces nondeterministic ordering and can
change tokens.json even when no new ads exist. Because the workflow commits/pushes tokens.json,
this can create unnecessary commits and increase the chance of push conflicts.

Code

main.py[R219-227]

+    # get new tokens list (single page - newest ads sorted by date)
+    tokens_list = get_tokens_page()
+    # remove repeated tokens
+    tokens_list = list(filter(lambda t: t not in tokens, tokens_list))
+    tokens = list(set(tokens_list + tokens))
+    asyncio.run(process_data(tokens_list))
+
+    # save new tokens
+    save_tokns(tokens)

Evidence

The code uses set() when merging tokens and then persists them; sets are unordered, so
serialization order can vary across runs. The workflow then commits and pushes tokens.json,
turning ordering-only diffs into real commits.

main.py[219-227]
.github/workflows/run-bot.yml[35-41]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`tokens = list(set(...))` produces nondeterministic ordering, causing `tokens.json` to change unpredictably and be re-committed even when token contents are unchanged.

## Issue Context
The scheduled workflow commits/pushes `tokens.json`, so ordering churn translates directly into repo churn and higher likelihood of push conflicts.

## Fix Focus Areas
- main.py[219-227]
- .github/workflows/run-bot.yml[35-41]

## Suggested fix
- Replace the `set()` merge with an order-preserving de-duplication (e.g., `list(dict.fromkeys(tokens_list + tokens))`), and/or sort tokens before saving.
- Optionally, only write/commit when the *content* actually changes (compare loaded tokens vs new tokens before writing).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. HTTP calls lack timeouts 🐞 Bug ☼ Reliability

Description

get_data() and fetch_ad_data() make requests calls without any timeout, so a stalled
connection can hang the workflow run. fetch_ad_data() also parses .json() without checking HTTP
status, so non-2xx responses can raise exceptions and abort the run before saving updated tokens.

Code

main.py[R81-104]

+def get_data():
+    body = build_search_body()
+    response = requests.post(URL, headers=REQUEST_HEADERS, json=body)
+    print(
+        "{} - Got response: {}".format(
+            datetime.datetime.now(), response.status_code
+        )
+    )
+    response.raise_for_status()
    return response.json()


def get_ads_list(data):
-    return data["web_widgets"]["post_list"]
+    # New API returns posts directly under "widget_list"
+    return data.get("widget_list", [])


def fetch_ad_data(token: str) -> AD:
    # send request
    data = requests.get(f"https://api.divar.ir/v8/posts-v2/web/{token}").json()
    images = []
    # check post exists
-    if not "sections" in data:
+    if "sections" not in data:
        return None

Evidence

Both the new search POST and the per-ad GET omit timeout=, and only get_data() calls
raise_for_status(). fetch_ad_data() directly calls .json() on the response which can fail on
non-JSON bodies or HTTP error responses.

main.py[81-90]
main.py[98-104]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Network requests to Divar have no timeouts, and `fetch_ad_data()` parses JSON without validating response status.

## Issue Context
On GitHub Actions, a hung request can waste runner time and prevent state persistence; transient API errors can crash the script mid-run.

## Fix Focus Areas
- main.py[81-90]
- main.py[98-104]

## Suggested fix
- Add explicit `timeout=` to all `requests` calls (e.g., `timeout=(5, 20)`).
- In `fetch_ad_data()`, store the response, call `raise_for_status()`, and handle `requests.RequestException`/`ValueError` (JSON decode) by returning `None` (or logging and skipping that token).
- Optionally use a shared `requests.Session()` for connection reuse.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Scheduled push can race ✓ Resolved 🐞 Bug ☼ Reliability

Description

The workflow commits and pushes directly on a 10-minute schedule without any concurrency control or
pull/rebase, so overlapping runs (or an intervening commit) can cause non-fast-forward push failures
and lose the updated state for that run. This is especially likely when a run takes longer than the
schedule interval or manual dispatch overlaps with the schedule.

Code

.github/workflows/run-bot.yml[R35-41]

+      - name: Commit updated state file
+        run: |
+          git config --global user.name "github-actions"
+          git config --global user.email "actions@github.com"
+          git add tokens.json || true
+          git diff --staged --quiet || git commit -m "Update seen ads state"
+          git push

Evidence

The workflow is scheduled every 10 minutes and always attempts to git push after committing
tokens.json without any concurrency guard or syncing with remote first, which is a classic cause
of push races and non-fast-forward failures.

.github/workflows/run-bot.yml[4-6]
.github/workflows/run-bot.yml[35-41]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The workflow pushes commits from a scheduled job without guarding against concurrent executions or remote updates.

## Issue Context
A scheduled workflow can overlap with itself or a manual run; `git push` may fail with non-fast-forward and the updated `tokens.json` won't be persisted.

## Fix Focus Areas
- .github/workflows/run-bot.yml[3-6]
- .github/workflows/run-bot.yml[35-41]

## Suggested fix
- Add a workflow-level `concurrency` group (e.g., `divar-bot`) to serialize runs.
- Before pushing, run `git pull --rebase` (or equivalent) and retry push on failure.
- Consider pushing state to a dedicated branch to avoid direct default-branch churn.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

4. Env/docs mismatch persists 🐞 Bug ⚙ Maintainability

Description

The code switches search configuration to SEARCH_CITY_ID/SEARCH_CATEGORY and no longer uses
SEARCH_CONDITIONS, but README and .env.sample still instruct users to set SEARCH_CONDITIONS.
The workflow also checks SEARCH_CONDITIONS in a debug step that doesn’t receive that env var, so
it reports an empty value regardless of the secret configured for the run step.

Code

main.py[R18-21]

+# New search parameters (replace the old SEARCH_CONDITIONS URL-path string)
+SEARCH_CITY_ID = os.environ.get("SEARCH_CITY_ID", "897")  # e.g. "897" for Golestan province
+SEARCH_CATEGORY = os.environ.get("SEARCH_CATEGORY", "real-estate")
+

Evidence

main.py introduces new env vars for search, while README and .env.sample still document
SEARCH_CONDITIONS; the workflow also references it in both a debug echo and env injection even
though the code no longer reads it.

main.py[18-21]
README.md[29-44]
.env.sample[1-5]
.github/workflows/run-bot.yml[22-33]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Docs and workflow still reference `SEARCH_CONDITIONS` even though the code now uses `SEARCH_CITY_ID`/`SEARCH_CATEGORY`.

## Issue Context
This is likely to confuse operators and make debugging harder (the debug step checks a variable that isn't exported for that step).

## Fix Focus Areas
- main.py[18-21]
- README.md[29-44]
- .env.sample[1-5]
- .github/workflows/run-bot.yml[22-33]

## Suggested fix
- Update README and `.env.sample` to describe the new configuration variables.
- Update the workflow to remove `SEARCH_CONDITIONS` entirely (or pass the relevant env vars to the debug step and validate those instead).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[qodo-code-review]

1. Unstable token file churn 🐞 Bug ☼ Reliability

tokens is rebuilt via set() and then saved, which produces nondeterministic ordering and can
change tokens.json even when no new ads exist. Because the workflow commits/pushes tokens.json,
this can create unnecessary commits and increase the chance of push conflicts.

Agent Prompt

## Issue description
`tokens = list(set(...))` produces nondeterministic ordering, causing `tokens.json` to change unpredictably and be re-committed even when token contents are unchanged.

## Issue Context
The scheduled workflow commits/pushes `tokens.json`, so ordering churn translates directly into repo churn and higher likelihood of push conflicts.

## Fix Focus Areas
- main.py[219-227]
- .github/workflows/run-bot.yml[35-41]

## Suggested fix
- Replace the `set()` merge with an order-preserving de-duplication (e.g., `list(dict.fromkeys(tokens_list + tokens))`), and/or sort tokens before saving.
- Optionally, only write/commit when the *content* actually changes (compare loaded tokens vs new tokens before writing).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

[qodo-code-review]

3. Env/docs mismatch persists 🐞 Bug ⚙ Maintainability

The code switches search configuration to SEARCH_CITY_ID/SEARCH_CATEGORY and no longer uses
SEARCH_CONDITIONS, but README and .env.sample still instruct users to set SEARCH_CONDITIONS.
The workflow also checks SEARCH_CONDITIONS in a debug step that doesn’t receive that env var, so
it reports an empty value regardless of the secret configured for the run step.

Agent Prompt

## Issue description
Docs and workflow still reference `SEARCH_CONDITIONS` even though the code now uses `SEARCH_CITY_ID`/`SEARCH_CATEGORY`.

## Issue Context
This is likely to confuse operators and make debugging harder (the debug step checks a variable that isn't exported for that step).

## Fix Focus Areas
- main.py[18-21]
- README.md[29-44]
- .env.sample[1-5]
- .github/workflows/run-bot.yml[22-33]

## Suggested fix
- Update README and `.env.sample` to describe the new configuration variables.
- Update the workflow to remove `SEARCH_CONDITIONS` entirely (or pass the relevant env vars to the debug step and validate those instead).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

[qodo-code-review]

4. Http calls lack timeouts 🐞 Bug ☼ Reliability

get_data() and fetch_ad_data() make requests calls without any timeout, so a stalled
connection can hang the workflow run. fetch_ad_data() also parses .json() without checking HTTP
status, so non-2xx responses can raise exceptions and abort the run before saving updated tokens.

Agent Prompt

## Issue description
Network requests to Divar have no timeouts, and `fetch_ad_data()` parses JSON without validating response status.

## Issue Context
On GitHub Actions, a hung request can waste runner time and prevent state persistence; transient API errors can crash the script mid-run.

## Fix Focus Areas
- main.py[81-90]
- main.py[98-104]

## Suggested fix
- Add explicit `timeout=` to all `requests` calls (e.g., `timeout=(5, 20)`).
- In `fetch_ad_data()`, store the response, call `raise_for_status()`, and handle `requests.RequestException`/`ValueError` (JSON decode) by returning `None` (or logging and skipping that token).
- Optionally use a shared `requests.Session()` for connection reuse.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools