Can My Laptop Handle One Million Concurrent HTTP Requests Using Go?
Dec 01, 2024 pm 05:01 PM
Can Your Laptop Handle a Million Concurrent HTTP Requests?
Imagine sending 1,000,000 simultaneous HTTP requests to a REST API service, pushing your computer's limits to maximize concurrency. While tools exist for this task, let's dive into how to do it in Go using goroutines.
package main
import (
"fmt"
"net/http"
"runtime"
"time"
)
func main() {
runtime.GOMAXPROCS(runtime.NumCPU()) // Use all available CPU cores
transport := &http.Transport{} // Create an HTTP transport
for i := 0; i < 1000000; i++ {
go func() { // Start a goroutine for each HTTP request
req, _ := http.NewRequest("GET", "http://myapi.com", nil)
req.Header.Set("User-Agent", "custom-agent")
req.SetBasicAuth("xxx", "xxx")
resp, err := transport.RoundTrip(req)
if err != nil {
panic("HTTP request failed.")
}
defer resp.Body.Close()
if resp.StatusCode != 302 {
panic("Unexpected response returned.")
}
location := resp.Header.Get("Location")
if location == "" {
panic("No location header returned.")
}
fmt.Println("Location Header Value:", location)
}()
}
time.Sleep(60 * time.Second) // Sleep for 60 seconds
}But Wait, There's a Problem!
Running this script results in errors due to file descriptor limits. The system simply can't handle so many simultaneous connections.
An Improved Solution
To overcome these limitations, we need to use a more sophisticated approach.
Using a Dispatcher and Worker Pool
This solution involves creating a dispatcher goroutine that pushes requests onto a channel. A worker pool of goroutines pulls requests from the channel, processes them, and sends them to a response channel. A consumer goroutine then processes the responses.
// Dispatcher function
func dispatcher(reqChan chan *http.Request, reqs int) {
defer close(reqChan)
for i := 0; i < reqs; i++ {
req, err := http.NewRequest("GET", "http://localhost/", nil)
if err != nil {
log.Println(err)
}
reqChan <- req
}
}
// Worker function
func worker(t *http.Transport, reqChan chan *http.Request, respChan chan Response) {
for req := range reqChan {
resp, err := t.RoundTrip(req)
r := Response{resp, err}
respChan <- r
}
}
// Consumer function
func consumer(respChan chan Response, reqs int) (int64, int64) {
var (
conns int64
size int64
)
for conns < int64(reqs) {
select {
case r, ok := <-respChan:
if ok {
if r.err != nil {
log.Println(r.err)
} else {
size += r.ContentLength
if err := r.Body.Close(); err != nil {
log.Println(r.err)
}
}
conns++
}
}
}
return conns, size
}Results
Running this improved script generates impressive results, such as:
Connections: 1000000
Concurrent: 200
Total size: 15000000 bytes
Total time: 38m20.3012317s
Average time: 2.280131ms
Optimizing for Performance
However, tweaking the number of concurrent requests and total requests will help you push your system's limits and stress-test its capabilities. Remember, this is an extreme test and can rapidly consume system resources.
The above is the detailed content of Can My Laptop Handle One Million Concurrent HTTP Requests Using Go?. For more information, please follow other related articles on the PHP Chinese website!
Hot AI Tools
Undress AI Tool
Undress images for free
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
How can you handle JSON encoding and decoding effectively in Go?
Jun 11, 2025 am 12:02 AM
Effective handling of JSON in Go requires attention to structural labels, optional fields and dynamic analysis. Use the struct tag to customize the JSON key name, such as json:"name"; make sure the fields are exported for access by the json package. Use pointers or omitempty tags when processing optional fields to distinguish between unprovided values ??from explicit zeros. When parsing unknown JSON, map[string]interface{} can be used to extract data with type assertions. The default number will be parsed as float64. json.MarshalIndent can be used to beautify the output during debugging, but the production environment should avoid unnecessary formatting. Mastering these techniques can improve the robustness and ability of your code
How can Go programs interact with C code using Cgo? What are the trade-offs?
Jun 10, 2025 am 12:14 AM
Go programs can indeed interact with C code through Cgo, which allows Go to call C functions directly. When using Cgo, just import the pseudo-package "C" and embed C code in the comments above the import line, such as including C function definitions and calling them. In addition, external C library can be linked by specifying link flags such as #cgoLDFLAGS. However, there are many issues to pay attention to when using Cgo: 1. Memory management needs to be processed manually and cannot rely on Go garbage collection; 2. Go types may not match C types, and types such as C.int should be used to ensure consistency; 3. Multiple goroutine calls to non-thread-safe C libraries may cause concurrency problems; 4. There is performance overhead for calling C code, and the number of calls across language boundaries should be reduced. Cgo's lack
How can Go applications be cross-compiled for different operating systems and architectures?
Jun 11, 2025 am 12:12 AM
Yes,Goapplicationscanbecross-compiledfordifferentoperatingsystemsandarchitectures.Todothis,firstsettheGOOSandGOARCHenvironmentvariablestospecifythetargetOSandarchitecture,suchasGOOS=linuxGOARCH=amd64foraLinuxbinaryorGOOS=windowsGOARCH=arm64foraWindow
How does Go handle pointers, and how do they differ from pointers in C/C ?
Jun 10, 2025 am 12:13 AM
Go simplifies the use of pointers and improves security. 1. It does not support pointer arithmetic to prevent memory errors; 2. Automatic garbage collection and management of memory without manual allocation or release; 3. The structure method can seamlessly use values ??or pointers, and the syntax is more concise; 4. Default safe pointers to reduce the risk of hanging pointers and memory leakage. These designs make Go easier to use and safer than C/C, but sacrifice some of the underlying control capabilities.
What are the implications of Go's static linking by default?
Jun 19, 2025 am 01:08 AM
Go compiles the program into a standalone binary by default, the main reason is static linking. 1. Simpler deployment: no additional installation of dependency libraries, can be run directly across Linux distributions; 2. Larger binary size: Including all dependencies causes file size to increase, but can be optimized through building flags or compression tools; 3. Higher predictability and security: avoid risks brought about by changes in external library versions and enhance stability; 4. Limited operation flexibility: cannot hot update of shared libraries, and recompile and deployment are required to fix dependency vulnerabilities. These features make Go suitable for CLI tools, microservices and other scenarios, but trade-offs are needed in environments where storage is restricted or relies on centralized management.
How does Go ensure memory safety without manual memory management like in C?
Jun 19, 2025 am 01:11 AM
Goensuresmemorysafetywithoutmanualmanagementthroughautomaticgarbagecollection,nopointerarithmetic,safeconcurrency,andruntimechecks.First,Go’sgarbagecollectorautomaticallyreclaimsunusedmemory,preventingleaksanddanglingpointers.Second,itdisallowspointe
How do I create a buffered channel in Go? (e.g., make(chan int, 10))
Jun 20, 2025 am 01:07 AM
To create a buffer channel in Go, just specify the capacity parameters in the make function. The buffer channel allows the sending operation to temporarily store data when there is no receiver, as long as the specified capacity is not exceeded. For example, ch:=make(chanint,10) creates a buffer channel that can store up to 10 integer values; unlike unbuffered channels, data will not be blocked immediately when sending, but the data will be temporarily stored in the buffer until it is taken away by the receiver; when using it, please note: 1. The capacity setting should be reasonable to avoid memory waste or frequent blocking; 2. The buffer needs to prevent memory problems from being accumulated indefinitely in the buffer; 3. The signal can be passed by the chanstruct{} type to save resources; common scenarios include controlling the number of concurrency, producer-consumer models and differentiation
How can you use Go for system programming tasks?
Jun 19, 2025 am 01:10 AM
Go is ideal for system programming because it combines the performance of compiled languages ??such as C with the ease of use and security of modern languages. 1. In terms of file and directory operations, Go's os package supports creation, deletion, renaming and checking whether files and directories exist. Use os.ReadFile to read the entire file in one line of code, which is suitable for writing backup scripts or log processing tools; 2. In terms of process management, the exec.Command function of the os/exec package can execute external commands, capture output, set environment variables, redirect input and output flows, and control process life cycles, which are suitable for automation tools and deployment scripts; 3. In terms of network and concurrency, the net package supports TCP/UDP programming, DNS query and original sets.
