[Ingest] Route scraping through an optional V2Ray/Xray proxy (Telegram in Iran)
CI/CD / CI · dotnet build (push) Successful in 53s
CI/CD / Deploy · hamkadr (push) Successful in 1m12s

Telegram and some sources are filtered in Iran. .NET cannot speak vmess/vless/trojan, so add an Xray sidecar (compose service 'xray', behind the 'proxy' profile) that converts the admin's config into a local SOCKS5 proxy (xray:10808). New ScrapeHttpClients provider builds a proxied or direct HttpClient (WebProxy supports socks5/socks4/http) cached per proxy URL; all five ingestion sources (Telegram/Bale/Divar/Medjobs/Websites) now use it. Admin settings gain IngestProxyEnabled + IngestProxyUrl (migration; UI under sources). Added deploy/xray/config.json template + README with vmess/vless/trojan examples.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
soroush.asadi
2026-06-04 17:53:17 +03:30
parent 698565c460
commit cea27c8684
17 changed files with 1411 additions and 20 deletions
+7
View File
@@ -51,6 +51,13 @@ public class AppSetting
/// <summary>Generic web pages to scrape, one URL per line.</summary>
[MaxLength(4000)] public string? WebsiteUrls { get; set; }
/// <summary>Route ingestion fetches through a proxy (needed in Iran for Telegram etc.).</summary>
public bool IngestProxyEnabled { get; set; } = false;
/// <summary>Local proxy an Xray/V2Ray client sidecar exposes, e.g. socks5://xray:10808
/// (also accepts socks4:// or http://). The app cannot read vmess/vless/trojan directly;
/// the sidecar converts that config into this local proxy.</summary>
[MaxLength(200)] public string? IngestProxyUrl { get; set; }
public bool DivarEnabled { get; set; } = false;
[MaxLength(60)] public string? DivarCity { get; set; } = "tehran";
/// <summary>Divar search terms, one per line or comma-separated.</summary>