Web proxy has many purposes, to name a few, some people use it for anonymity purposes, and some people want bypass network censorship, and another common usage is for CDN. Here at CoinGecko, we use web proxies to protect ourselves while consuming and sharing large data by staying anonymous. Let’s go through the steps to create high availability proxy servers with zero maintenance using Amazon Web Services and Squid.
Step 1: Launch an EC2 instance 
Step 2: Reserve Elastic IPs (static ip) from AWS
In this example, We reserved 2 static ip(s) from AWS, specifically 3.12.240.78 and 3.12.195.16. You can reserved as many ip(s) you want depending on your need, each ip represents a proxy node.
Step 3: Creating a golden image (AMI) of EC2 with squid installed (EC2 > Instances)
Now, we are going to prepare a golden image for auto scaling. Now ssh into the ec2 instance created in step 1 and do the following:
sudo yum update -y sudo yum install docker -y sudo service docker start sudo usermod -a -G docker ec2-user sudo systemctl enable docker sudo amazon-linux-extras install python3 -y sudo docker run --name squid -d -p 3128:3128 --restart always --volume ~/squid.conf:/etc/squid/squid.conf datadog/squid
vim ~/squid.conf and and copy the following into the file (feel free to use your own configuration)
acl SSL_ports port 443 acl Safe_ports port 80 acl Safe_ports port 21 acl Safe_ports port 443 acl Safe_ports port 70 acl Safe_ports port 210 acl Safe_ports port 1025-65535 acl Safe_ports port 280 acl Safe_ports port 488 acl Safe_ports port 591 acl Safe_ports port 777 acl CONNECT method CONNECT #http_access deny !Safe_ports #http_access deny CONNECT !SSL_ports #http_access allow localhost manager #http_access deny manager #http_access allow localhost #http_access deny all http_access allow all http_port 3128 coredump_dir /var/spool/squid3 refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|?) 0 0% 0 refresh_pattern . 0 20% 4320
Now we have our ec2 instance setup to be a proxy, it is time to create an AMI.
Step 4: Create an appropriate IAM Role to allow address association. (IAM > Roles)
Step 5: Create AWS Security Group (EC2 > Security Group)
In this tutorial, I will be creating a security group that allows ALL inbound and outbound traffic for simplicity sake. Please use the appropriate security group when you are creating your own proxy cluster.
Step 6: Create AWS Launch Template (EC2 > Launch Templates)
Things to take note while creating launch template:
- For AMI, use the AMI created in Step 3
- For Security group, use the security group created in Step 5
- in Advanced tab > User data put in the following (for the valid ips, put in your comma-separated ip(s) created in Step 2
#!/bin/bash sudo pip3 install aws-ec2-assign-elastic-ip sudo /usr/local/bin/aws-ec2-assign-elastic-ip --region us-east-2 --valid-ips 3.12.240.78,3.12.195.16 sudo /usr/local/bin/aws-ec2-assign-elastic-ip --region us-east-2 --valid-ips 3.12.240.78,3.12.195.16 sudo docker container prune -f sudo docker run --name squid -d -p 3128:3128 --restart unless-stopped --volume /home/ec2-user/squid.conf:/etc/squid/squid.conf datadog/squid
Step 7: Create Auto Scaling Group (EC2 > Auto Scaling Group)
Use the launch template as the recipe to create the Auto Scaling Group. In this example, we configure the desired number of instance to be 2 since we reserved 2 ips in Step 2. Now click done and you should be able to see 2 EC2 instances being created in the dashboard
Step 8: Verify if proxy is working
In order to verify if the proxy is working, you can use https://www.proxy-checker.org/
With the above setup, auto scaling will take spawn new instance(s) using the static ip(s) we reserved and the configuration we setup from the above steps, if the node is down or terminated. Hence zero maintenance required.
The post How to create a self-healing web proxy cluster with AWS and Squid in 8 steps appeared first on CoinGecko Blog.
