[Ingest] Route scraping through an optional V2Ray/Xray proxy (Telegram in Iran)

Telegram and some sources are filtered in Iran. .NET cannot speak vmess/vless/trojan, so add an Xray sidecar (compose service 'xray', behind the 'proxy' profile) that converts the admin's config into a local SOCKS5 proxy (xray:10808). New ScrapeHttpClients provider builds a proxied or direct HttpClient (WebProxy supports socks5/socks4/http) cached per proxy URL; all five ingestion sources (Telegram/Bale/Divar/Medjobs/Websites) now use it. Admin settings gain IngestProxyEnabled + IngestProxyUrl (migration; UI under sources). Added deploy/xray/config.json template + README with vmess/vless/trojan examples. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-04 17:53:17 +03:30
parent 698565c460
commit cea27c8684
17 changed files with 1411 additions and 20 deletions
@@ -0,0 +1,63 @@
+# Ingestion proxy (Xray / V2Ray) — for scanning Telegram etc. from Iran
+
+The app's HttpClient can't speak `vmess` / `vless` / `trojan` directly. Instead, the **Xray
+sidecar** (compose service `xray`) reads your config and exposes a plain **SOCKS5 proxy at
+`xray:10808`** (and HTTP at `xray:10809`) on the internal compose network. The app is then
+pointed at that proxy from the admin panel, and only ingestion traffic goes through it.
+
+```
+[app ingestion] → socks5://xray:10808 → [Xray client] → vmess/vless/trojan → server → Telegram
+```
+
+## Setup
+
+1. **Put your config** in `deploy/xray/config.json`. Replace the `proxy` outbound with your
+   own vmess / vless / trojan outbound (templates below). Keep the `inbounds` and `routing`
+   sections as-is so the local SOCKS/HTTP ports stay the same.
+
+2. **Start the sidecar** (it's behind a compose profile so normal deploys don't run it):
+   ```bash
+   docker compose --profile proxy up -d xray
+   docker logs hamkadr_xray --tail 30      # should show it listening, no errors
+   ```
+
+3. **Point the app at it**: open `/Admin/Settings` → «کانال‌ها/منابع» →
+   - tick **«ارسال جمع‌آوری از طریق پروکسی»**
+   - set the proxy URL to **`socks5://xray:10808`**
+   - Save, then run ingestion (Telegram source enabled). 
+
+4. **Quick test** the proxy reaches Telegram:
+   ```bash
+   docker exec hamkadr_api sh -c "wget -q -O- --timeout=15 -e use_proxy=yes -e http_proxy=http://xray:10809 https://t.me/s/telegram | head -c 200" || true
+   ```
+
+## Where to get the config values
+If you have a share link (`vmess://…`, `vless://…`, `trojan://…`), import it into the Xray/v2rayN
+client and **export the JSON config**, or decode it and fill the templates below.
+
+### vless + ws + tls (matches the default template in config.json)
+```json
+{ "tag":"proxy","protocol":"vless","settings":{"vnext":[{"address":"HOST","port":443,
+  "users":[{"id":"UUID","encryption":"none"}]}]},
+  "streamSettings":{"network":"ws","security":"tls","tlsSettings":{"serverName":"SNI"},
+  "wsSettings":{"path":"/PATH","headers":{"Host":"SNI"}}} }
+```
+
+### vmess + ws + tls
+```json
+{ "tag":"proxy","protocol":"vmess","settings":{"vnext":[{"address":"HOST","port":443,
+  "users":[{"id":"UUID","alterId":0,"security":"auto"}]}]},
+  "streamSettings":{"network":"ws","security":"tls","tlsSettings":{"serverName":"SNI"},
+  "wsSettings":{"path":"/PATH","headers":{"Host":"SNI"}}} }
+```
+
+### trojan + tls
+```json
+{ "tag":"proxy","protocol":"trojan","settings":{"servers":[{"address":"HOST","port":443,
+  "password":"PASSWORD"}]},
+  "streamSettings":{"network":"tcp","security":"tls","tlsSettings":{"serverName":"SNI"}} }
+```
+
+> Security note: `config.json` contains your VPN credentials. It's mounted read-only into the
+> container. Do **not** commit a real config — keep the committed file as a placeholder and
+> drop the real one on the server only (or add it to `.gitignore` if you keep it locally).
@@ -0,0 +1,53 @@
+{
+  "log": { "loglevel": "warning" },
+  "inbounds": [
+    {
+      "tag": "socks-in",
+      "listen": "0.0.0.0",
+      "port": 10808,
+      "protocol": "socks",
+      "settings": { "udp": true, "auth": "noauth" }
+    },
+    {
+      "tag": "http-in",
+      "listen": "0.0.0.0",
+      "port": 10809,
+      "protocol": "http"
+    }
+  ],
+  "outbounds": [
+    {
+      "//": "REPLACE this whole outbound with YOUR vmess / vless / trojan config.",
+      "//vmess-example": "see deploy/xray/README.md for vmess & trojan templates",
+      "tag": "proxy",
+      "protocol": "vless",
+      "settings": {
+        "vnext": [
+          {
+            "address": "YOUR_SERVER_ADDRESS",
+            "port": 443,
+            "users": [
+              {
+                "id": "YOUR_UUID",
+                "encryption": "none",
+                "flow": ""
+              }
+            ]
+          }
+        ]
+      },
+      "streamSettings": {
+        "network": "ws",
+        "security": "tls",
+        "tlsSettings": { "serverName": "YOUR_SNI" },
+        "wsSettings": { "path": "/", "headers": { "Host": "YOUR_SNI" } }
+      }
+    },
+    { "tag": "direct", "protocol": "freedom" }
+  ],
+  "routing": {
+    "rules": [
+      { "type": "field", "inboundTag": ["socks-in", "http-in"], "outboundTag": "proxy" }
+    ]
+  }
+}