[Ingest] Route scraping through an optional V2Ray/Xray proxy (Telegram in Iran)
CI/CD / CI · dotnet build (push) Successful in 53s
CI/CD / Deploy · hamkadr (push) Successful in 1m12s

Telegram and some sources are filtered in Iran. .NET cannot speak vmess/vless/trojan, so add an Xray sidecar (compose service 'xray', behind the 'proxy' profile) that converts the admin's config into a local SOCKS5 proxy (xray:10808). New ScrapeHttpClients provider builds a proxied or direct HttpClient (WebProxy supports socks5/socks4/http) cached per proxy URL; all five ingestion sources (Telegram/Bale/Divar/Medjobs/Websites) now use it. Admin settings gain IngestProxyEnabled + IngestProxyUrl (migration; UI under sources). Added deploy/xray/config.json template + README with vmess/vless/trojan examples.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
soroush.asadi
2026-06-04 17:53:17 +03:30
parent 698565c460
commit cea27c8684
17 changed files with 1411 additions and 20 deletions
@@ -12,12 +12,12 @@ namespace JobsMedical.Web.Services.Scraping;
public class DivarListingSource : IListingSource
{
private const string BaseUrl = "https://api.divar.ir/v8/web-search";
private readonly IHttpClientFactory _http;
private readonly ScrapeHttpClients _clients;
private readonly ILogger<DivarListingSource> _log;
public DivarListingSource(IHttpClientFactory http, ILogger<DivarListingSource> log)
public DivarListingSource(ScrapeHttpClients clients, ILogger<DivarListingSource> log)
{
_http = http;
_clients = clients;
_log = log;
}
@@ -29,7 +29,7 @@ public class DivarListingSource : IListingSource
if (!s.DivarEnabled || queries.Count == 0) return Array.Empty<ScrapedItem>();
var city = string.IsNullOrWhiteSpace(s.DivarCity) ? "tehran" : s.DivarCity.Trim();
var client = _http.CreateClient("scrape");
var client = _clients.For(s);
var items = new List<ScrapedItem>();
foreach (var q in queries)
{