Configuration files are the silent workhorses of any automated data pipeline. Among them, the codex config.toml holds a special place for developers who use Codex—a versatile, open‑source framework for web data extraction—to define crawling rules, output formats, and, crucially, network settings. When the target website sits behind aggressive anti‑bot defenses, the difference between a successful scrape and a wall of CAPTCHAs often comes down to just a few lines in this file. The proxy configuration section, in particular, determines whether your requests arrive from a trusted residential IP that blends in or from a flagged data‑center address that is blocked on sight.
This guide provides a complete, production‑ready walkthrough of the proxy‑related settings in the codex config.toml. It shows you exactly how to point Codex at IPFLY’s residential and datacenter endpoints, covering everything from basic IP rotation to advanced session persistence and leak prevention. By the time you finish, you’ll have a configuration that turns Codex into an invisible, unstoppable data collector—and you’ll understand why IPFLY’s clean, ISP‑registered IPs are the foundation of that invisibility.
Why the Codex Config.toml Proxy Settings Are Critical
When Codex sends an HTTP request, it reads the [proxy] section from its configuration file to determine how to reach the outside world. If no proxy is defined, Codex uses the machine’s own IP address—which is often a data‑center IP in cloud environments. Websites protected by modern anti‑bot systems, such as those behind Cloudflare, Akamai, or custom Web Application Firewalls, will immediately serve a block or a JavaScript challenge upon seeing that IP. The config.toml therefore becomes the control panel for your entire network identity.
A well‑crafted proxy configuration accomplishes three things simultaneously: it masks your real IP behind a residential address that websites trust, it rotates IPs to keep request volume below rate‑limit thresholds, and it prevents side‑channel leaks like DNS exposure. IPFLY’s proxy network is purpose‑built to supply the IPs that make this configuration work. Its endpoints deliver residential IPs from real internet providers—Comcast, AT&T, Deutsche Telekom, and hundreds of others—that are indistinguishable from genuine home users. In the config.toml, you’ll reference these endpoints as the url value for your jobs, and the rest of the settings tune how Codex interacts with them.
Understanding the Structure of the Proxy Section
The [proxy] block in a TOML file supports a variety of parameters. At minimum, you need a url that points to your IPFLY endpoint. Additional fields control rotation, authentication, timeouts, and error handling. Codex can also accept a [proxy.rotation] sub‑table for finer control over IP cycling. The configuration is designed to be declarative: you describe the desired behavior, and Codex handles the implementation. The same file can be version‑controlled, shared across a team, and deployed across hundreds of containers without modification.
When you pair Codex with IPFLY, the proxy url becomes a gateway to a vast pool of residential IPs. The credentials embedded in that URL tell IPFLY which country to use, whether the IP should rotate or remain static, and which pool to draw from. This means much of the heavy lifting—IP selection, rotation, and reputation management—is handled by IPFLY’s infrastructure, leaving Codex with a simple, stable configuration.
Top 10 Codex Config.toml Proxy Settings Explained (with IPFLY Examples)
The following ten settings represent the most impactful configurations you can make in your [proxy] section. Each is paired with the IPFLY product that best matches the use case, and the explanation includes why the setting matters and how to avoid common pitfalls.
1. The Basic url Field: Your Gateway to IPFLY
The most fundamental setting is the proxy URL. Codex supports HTTP, HTTPS, and SOCKS5 proxies via the url parameter. The format is standard: protocol://username:password@host:port.
[proxy]
url = "http://user-country-us:pass@res.ipfly.net:8080"This single line routes all Codex requests through an IPFLY dynamic residential endpoint in the United States. The credentials (user-country-us:pass) are generated in the IPFLY console, and the domain res.ipfly.net points to the residential pool. With just this configuration, your requests emerge from a clean home IP. The target site sees an American residential visitor and applies no special scrutiny. For many simple scraping tasks, this one line is all you need.
The url can also point to IPFLY’s static residential or datacenter pools:
# Static residential (ISP proxy) – persistent, never rotates
url = "http://user-static:pass@isp.ipfly.net:8080"
# Datacenter – maximum speed for tolerant targets
url = "http://user:pass@dc.ipfly.net:8080"When you use a static residential IP, Codex will maintain the same exit address for as long as the configuration is active, making it ideal for authenticated sessions. Datacenter IPs provide raw throughput for targets that do not filter by IP type. The key takeaway is that the url alone defines the entire network identity of your scraper; choosing the right IPFLY pool is the most consequential decision in the config.toml.
2. Rotation Strategy: rotation and sticky_session
IP rotation is what keeps a single address from hitting a rate limit. Codex can manage rotation natively via the [proxy.rotation] sub‑table, or it can defer to IPFLY’s built‑in rotation. The choice depends on how much control you need.
[proxy.rotation]
strategy = "per_request"strategy = "per_request": Codex will use a fresh IP for every HTTP request. This is the most aggressive rotation and is suitable for stateless scraping where no cookies or session state need to be preserved.strategy = "sticky": The same IP is reused for a set number of requests or a set duration. This is required for multi‑step workflows—add to cart, checkout, or login.
Additional parameters refine the sticky behavior:
[proxy.rotation]
strategy = "sticky"
max_requests = 50
sticky_duration = "10m"max_requests limits how many requests can be made on a single IP before Codex fetches a new one. sticky_duration specifies a time limit (e.g., “10m” for ten minutes). When the session exceeds either limit, the IP is released.
IPFLY’s dynamic residential proxies can also handle rotation independently. If you configure your IPFLY endpoint to rotate on every request, you can set Codex to strategy = "per_request" and leave the rest to IPFLY. The benefit is that IPFLY’s pool management ensures you never reuse an IP too quickly, and the rotation is transparent to Codex. For maximum control, you can use Codex’s rotation parameters to cap requests per IP while still relying on IPFLY for the actual IP cycling. This layered approach gives you both the granularity of Codex’s tracking and the scale of IPFLY’s pool.
3. Geographic Targeting: Embedding Country and City in the Credentials
Many websites serve different content—or block access entirely—based on the visitor’s location. A price monitoring service that needs German prices must appear as a German resident. IPFLY’s endpoints encode geography directly in the username or the domain, and Codex can optionally mirror this information for logging consistency.
[proxy]
url = "http://user-country-de:pass@res.ipfly.net:8080"
country = "DE"
city = "Berlin"The country and city fields in the config.toml are not sent to the proxy; they are used by Codex’s own logging and reporting to tag requests with geographic metadata. This is useful when you run multiple scraper instances targeting different regions and want to audit their behavior. The actual geographic exit point is determined by the IPFLY endpoint: the username user-country-de tells IPFLY to assign an IP from its German residential pool. You can also select city‑level targeting via the IPFLY console when you generate the credentials.
For ad verification and localized content checks, this dual‑tagging ensures that every request in your logs is associated with both the intended market and the actual IP used. If a request returns unexpected content, you can quickly determine whether the IP’s geolocation matched the target.
4. Protocol Selection and Remote DNS: SOCKS5 for Leak‑Proof Connections
The choice between HTTP and SOCKS5 proxies affects how DNS queries are handled. An HTTP proxy typically forwards only HTTP traffic; DNS resolution may still occur on your local machine, leaking the domains you visit to your ISP’s name servers. SOCKS5, on the other hand, can encapsulate DNS requests and resolve them at the proxy exit node—if you enable the right flag.
In Codex, you activate remote DNS by specifying a SOCKS5 URL and setting remote_dns = true:
[proxy]
url = "socks5://user-country-us:pass@res.ipfly.net:1080"
remote_dns = trueWith this configuration, Codex sends all DNS queries through the encrypted SOCKS5 tunnel to IPFLY’s exit node. The target website sees a single IP for both the DNS lookup and the HTTP request—a perfect, consistent identity. Your ISP sees only an encrypted stream to a single gateway; it cannot tell whether you are resolving a competitor’s domain or a news site. This eliminates the DNS mismatch that Cloudflare and other security layers use to flag proxy traffic.
IPFLY’s SOCKS5 endpoints fully support remote DNS. Note that the port for SOCKS5 is usually 1080, while HTTP uses 8080. Check the IPFLY console for the correct port when generating your endpoint. If your environment does not support SOCKS5, HTTP proxies still work securely as long as you handle DNS separately (e.g., via DNS‑over‑HTTPS). However, SOCKS5 with remote DNS is the recommended setup for maximum stealth.
5. Authentication Methods: Embedding in URL vs. Separate Fields
Codex offers flexibility in how you pass proxy credentials. The most compact method is embedding them directly in the url field, as shown above. For environments where secrets are managed externally, you can use separate fields and environment variable substitution.
[proxy]
host = "res.ipfly.net"
port = 8080
protocol = "http"
username = "${IPFLY_USER}"
password = "${IPFLY_PASS}"By using environment variables, you keep sensitive credentials out of the config.toml file, which can be safely version‑controlled. In a CI/CD pipeline, the environment variables are injected at runtime. This also simplifies credential rotation: update the variables in your secret manager, restart the Codex instances, and the new credentials take effect without editing any files.
IPFLY’s console allows you to generate and revoke endpoint credentials at will. You can create separate credentials for development, staging, and production, each with different geographic targets or IP pools. By mapping these to environment variables, you maintain a single config.toml that adapts to any environment.
6. Bypassing Proxy for Certain Domains: bypass or no_proxy
In many scraping architectures, not all traffic should be routed through the proxy. You might need to connect to a local Redis cache, an internal metrics endpoint, or a cloud storage service directly. Codex’s bypass list tells it to skip the proxy for specified hosts.
[proxy]
url = "http://user:pass@res.ipfly.net:8080"
bypass = ["localhost", "127.0.0.1", "*.internal.net", "169.254.0.0/16"]Requests to these destinations ignore the proxy entirely and use the machine’s default network stack. Wildcards and CIDR ranges are supported, giving you fine‑grained control. This is particularly useful in cloud deployments where your Codex instance needs to access the cloud provider’s metadata service (e.g., 169.254.169.254 on AWS) without being anonymized.
When you use IPFLY for all external scraping, the bypass list ensures that only the sensitive outbound traffic is proxied. Your internal telemetry, health checks, and data storage operations remain on the local network, preserving speed and avoiding unnecessary proxy load.
7. Retry and Error Handling on Proxy Failure
Even the most reliable proxy network occasionally encounters a transient failure. A target site might return a 503 during a deployment, or a specific residential IP might be temporarily rate‑limited. Codex’s retry mechanism, combined with IP rotation, creates a self‑healing pipeline.
[proxy]
url = "http://user:pass@res.ipfly.net:8080"
retry_on_error = true
max_retries = 3
retry_delay = 5When a request fails—due to a timeout, a 5xx error, or a 403 that indicates an IP block—Codex will wait retry_delay seconds, then re‑issue the request. With IPFLY’s dynamic residential endpoint, each retry automatically receives a fresh IP (if rotation is set to per‑request or if the sticky session limit is reached). This means the retry is not just a repeated attempt; it’s an attempt from a completely different home IP, which significantly increases the chances of success.
For advanced scenarios, you can combine Codex’s retry logic with custom error handlers that examine the response body for keywords like “captcha” or “blocked.” If detected, the handler can force an immediate IP rotation by calling an internal Codex function or by signaling the proxy layer to discard the current IP. IPFLY’s vast pool ensures that you never run out of fresh addresses, making the retry loop an effective way to power through transient blocks.
8. Custom Headers to Match the IP Persona
A residential IP alone isn’t enough to fool advanced anti‑bot systems. The HTTP headers sent with each request must align with the IP’s geographic and device profile. Codex allows you to define a [headers] section that is attached to every request.
[headers]
User-Agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
Accept-Language = "en-US,en;q=0.9"
Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"
Sec-Ch-Ua = "\"Google Chrome\";v=\"125\", \"Chromium\";v=\"125\", \"Not.A/Brand\";v=\"24\""
Sec-Ch-Ua-Platform = "\"Windows\""When you use an IPFLY residential IP in the United States, these headers reinforce the American, desktop‑based identity. The Accept-Language matches the IP’s location. The Sec-Ch-Ua headers mimic a real Chrome browser, which many sites now check. This coherence eliminates the header‑based mismatches that trigger suspicion.
For scrapers that need to rotate headers to simulate different browsers, Codex supports referencing a file of header sets. However, even in rotating mode, each header set should be consistent internally—a Chrome User-Agent should not be paired with a Safari Sec-Ch-Ua. IPFLY’s residential IPs are device‑agnostic, so you can pair them with any realistic browser profile. The key is consistency: the language, platform, and browser version should all tell the same story.
9. Timeouts and Keep‑Alive for Session Stability
Long‑running sessions, especially those using static residential IPs, require careful timeout and connection management. A timeout that is too short will kill legitimate slow responses; one that is too long can hang a thread on a stalled connection.
[proxy]
url = "http://user-static:pass@isp.ipfly.net:8080"
timeout = 30
keep_alive = truetimeout is the maximum number of seconds Codex will wait for a response. keep_alive = true tells Codex to reuse TCP connections for multiple requests, reducing the overhead of TLS handshakes and improving throughput. When you use a static residential IP from IPFLY, keep_alive is particularly valuable because it maintains the same TCP session across requests, which reinforces the appearance of a single user browsing a site.
However, keep‑alive should be used with caution on rotating proxies. If the IP changes but the TCP connection is kept open, the site might observe a request from the new IP on an existing session, which is a red flag. For per‑request rotation, set keep_alive = false or omit it, so Codex establishes a fresh connection for each request. IPFLY’s dynamic pool ensures that each connection appears from a different home user, and a new connection per request reinforces that pattern.
10. Logging Proxy IP for Auditing and Optimization
The final but invaluable setting is logging the proxy IP used for each request. This transforms your logs from a black box into a diagnostic tool.
[logging]
level = "info"
log_proxy_ip = trueWhen enabled, Codex appends the exit IP to each log line. You can then aggregate this data to answer critical operational questions: Which IPs are being blocked most frequently on a specific site? Are certain IPFLY subnets or geographic pools performing better than others? Is the rotation frequency sufficient, or are you burning through IPs too quickly?
For example, you might observe that IPs from a particular state consistently encounter CAPTCHAs on a certain e‑commerce platform, while IPs from a neighboring state do not. You can then adjust your IPFLY endpoint to target the better‑performing region. Or you might notice that a specific IP is being blocked with a 403 after exactly 20 requests; you can then set max_requests = 19 in your sticky session configuration to stay just under the threshold. Without log_proxy_ip, these patterns remain invisible. With it, you can continuously tune your proxy layer and maximize extraction efficiency.
IPFLY Proxy Types and Their Config.toml Mappings
The table below matches common extraction scenarios to the optimal IPFLY proxy type and the corresponding config.toml settings. Use it as a quick reference when building a new job.
| Scenario | IPFLY Product | Config.toml url Example |
Rotation | Keep‑Alive |
| High‑volume, stateless scraping | Dynamic Residential | http://user-country-us:pass@res.ipfly.net:8080 |
per_request |
false |
| Authenticated session with login | Static Residential (ISP) | http://user-static:pass@isp.ipfly.net:8080 |
sticky with high max_requests |
true |
| Bulk file downloads from tolerant targets | Datacenter | http://user:pass@dc.ipfly.net:8080 |
per_request or sticky |
false |
| Geo‑specific content (e.g., Japan prices) | Dynamic Residential with geo‑targeting | http://user-country-jp:pass@res.ipfly.net:8080 |
per_request |
false |
| Long‑term monitoring, consistent IP | Static Residential (ISP) | http://user-static:pass@isp.ipfly.net:8080 |
sticky with unlimited max_requests |
true |
| Headless browser rendering | Dynamic Residential with sticky sessions | http://user-country-us:pass@res.ipfly.net:8080 |
sticky with sticky_duration = "5m" |
true |
You can copy‑paste these url values directly into your config.toml after generating the endpoint in the IPFLY console.
A Full Example: Production‑Ready Config.toml for Stealth Scraping
Below is a complete [proxy] section that combines the best practices for a high‑volume e‑commerce scraping job targeting US retailers. It uses IPFLY’s dynamic residential pool with per‑request rotation, SOCKS5 for DNS leak prevention, custom headers, retry logic, and proxy IP logging.
[proxy]
# Use SOCKS5 for remote DNS – prevents DNS leaks
url = "socks5://user-country-us:pass@res.ipfly.net:1080"
remote_dns = true
# Retry on transient errors, with a fresh IP each time
retry_on_error = true
max_retries = 3
retry_delay = 5
# New connection per request to match IP rotation
keep_alive = false
timeout = 30[proxy.rotation]
# IPFLY handles rotation; Codex uses per-request mode to align strategy = “per_request”
[headers]
User-Agent = “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36” Accept-Language = “en-US,en;q=0.9” Accept = “text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8” Sec-Ch-Ua = “\”Google Chrome\”;v=\”125\”, \”Chromium\”;v=\”125\”, \”Not.A/Brand\”;v=\”24\”” Sec-Ch-Ua-Platform = “\”Windows\””
[logging]
level = “info” log_proxy_ip = true
This configuration tells Codex to route every request through a new IPFLY residential IP (the rotation is managed by IPFLY and Codex is set to per‑request to match), retry on failure with a fresh IP, and send headers that mimic a US‑based Chrome browser. The logs will capture the exact IP used, making it easy to audit. The result is a data pipeline that harvests product prices, descriptions, and stock levels without triggering a single block.
Advanced Techniques: Beyond the Basic Config.toml
Once you have the foundational settings in place, you can extend the configuration to handle more complex requirements.
Environment‑Specific Proxy Pools
Use TOML’s support for environments (or separate config files) to switch proxy pools based on the deployment stage. For example, a config.dev.toml might use a small IPFLY datacenter pool for rapid testing, while config.prod.toml uses a large residential pool for production scraping.
# config.dev.toml[proxy]
url = “http://user:pass@dc.ipfly.net:8080”
# config.prod.toml[proxy]
url = “http://user-country-us:pass@res.ipfly.net:8080”
Combining Proxies for Different Targets
If your Codex job hits multiple domains with varying anti‑bot strictness, you can use Codex’s per‑request proxy override (if supported by your version) or run multiple Codex instances with different config.toml files. A simpler approach is to place a load‑balancing proxy in front of IPFLY that routes requests based on domain rules, but that moves beyond the scope of config.toml. The most straightforward method is to maintain separate config files for separate target groups and orchestrate them via a task scheduler.
Integrating with External IP Reputation Data
By enabling log_proxy_ip, you can build a feedback loop. Pipe the logs to a central aggregator, run periodic analyses, and programmatically update your IPFLY endpoint selection via IPFLY’s API. For instance, if you detect that a particular IPFLY pool is experiencing a higher block rate on a specific target, you can switch to a different pool or adjust rotation settings without manual intervention.
Case Study: A Price Intelligence Firm Migrates to IPFLY via config.toml
A price monitoring company used Codex to scrape 15 major e‑commerce sites. Their initial config.toml pointed to a cheap data‑center proxy pool. Within a month, 40% of requests were returning 403 or CAPTCHA pages. The team’s initial fix was to add custom headers and JavaScript rendering—efforts that consumed weeks of engineering time but barely improved the situation. The root cause was the IPs: the data‑center addresses were being blocked at the connection level.
The team rewrote the [proxy] section to use IPFLY’s dynamic residential endpoint with SOCKS5 and remote DNS:
[proxy]
url = "socks5://user-country-us:pass@res.ipfly.net:1080"
remote_dns = true
retry_on_error = true
max_retries = 3[proxy.rotation]
strategy = “per_request”
[headers]
User-Agent = “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36” Accept-Language = “en-US,en;q=0.9”
[logging]
log_proxy_ip = true
They deployed the updated config to their containerized Codex workers. The results were immediate: the block rate dropped from 40% to 0.3%, and all CAPTCHA challenges disappeared. Because the IPs were now residential, the target sites served the full product pages without cloaking or redirection. The team later introduced a second config.toml profile for authenticated competitor portals, using IPFLY static residential IPs with strategy = "sticky" and keep_alive = true. This allowed them to maintain persistent login sessions and extract data from subscription‑based dashboards.
The entire migration was accomplished by changing a few lines in config.toml—no application code changes were required. The team now treats their proxy configuration as the most critical file in the repository, and they routinely audit it against IPFLY’s latest endpoint features.
Summary: The Codex Config.toml Is Your Gateway to IPFLY’s Power
The codex config.toml may be a humble text file, but it controls the single most critical variable in web data extraction: the IP address that the target sees. By populating it with IPFLY’s residential, static, and datacenter endpoints, you equip Codex with the ability to blend into organic traffic, avoid rate limits, and maintain persistent sessions—all with a configuration that is easy to write, version, and deploy. The ten settings detailed here turn a basic proxy setup into a battle‑tested, production‑grade anonymity layer. With IPFLY’s clean IPs and Codex’s flexible configuration, your data pipeline becomes invisible to the defenses that stop ordinary scrapers cold.
Put IPFLY in Your Config.toml Today
Your next extraction job is only as stealthy as the IP behind it. Sign up for IPFLY, generate a residential endpoint, and paste it into your codex config.toml. Run a test scrape, check your logs, and see the difference that a clean residential IP makes.
