Navigating the intricate landscape of the web often involves encountering HTTP redirects. These are crucial mechanisms that guide browsers and web crawlers from one URL to another, whether a page has moved permanently, temporarily, or for load balancing. For developers, SEO specialists, and data analysts, understanding how to effectively handle these redirects and, more specifically, how to get final URL after curl is redirected, is not merely a technicality but a fundamental skill. Without this capability, critical tasks like web scraping, API testing, and link validation can yield incomplete or incorrect data, leading to flawed insights and inefficient workflows. This guide will delve into the methods and best practices for tracking the true destination of a redirected link using the powerful curl command-line tool and various programming approaches.

Understanding HTTP Redirects and Their Impact

HTTP redirects are server responses that tell a client (like a browser or curl) that the requested resource has moved to a different location. The most common types include 301 (Moved Permanently), 302 (Found, or Moved Temporarily), 307 (Temporary Redirect), and 308 (Permanent Redirect). Each type signals a slightly different intent, impacting how clients and search engines should treat the original URL versus the new one. For instance, a 301 redirect passes most of the link equity to the new URL, which is vital for SEO when migrating content.

Websites utilize redirects for a multitude of reasons: consolidating duplicate content, site redesigns, domain migrations, A/B testing, or simply maintaining clean URLs. However, failing to follow these redirects can lead to significant problems. Imagine trying to scrape data from a product page that has moved; if your script doesn’t follow the redirect, it will fetch an outdated or non-existent page, resulting in missing data. Similarly, SEO auditing tools need to identify the canonical URL to prevent issues with duplicate content penalties and ensure accurate site mapping.

The “Location” header in an HTTP response is the key to redirects. When a server sends a 3xx status code, it typically includes a Location: [new_URL] header, indicating where the client should go next. Modern browsers handle this automatically, but when working with command-line tools or programming libraries, you often need to explicitly instruct them to follow these directives. Understanding this fundamental mechanism is the first step towards mastering how to effectively get the final URL after curl is redirected, ensuring your applications always reach the intended destination.

How to Get Final URL After Curl is Redirected: Command Line Basics

For those working directly in the terminal, curl is an indispensable tool for network requests. By default, curl does not follow HTTP redirects. To instruct curl to follow redirect chains until it reaches the final destination, you must use the -L or --location flag. This simple addition transforms curl into a powerful redirect-following utility, essential for tasks like validating links or checking the true path of a shortened URL.

To identify the final URL, you can combine the -L flag with other options. The -s (silent) flag suppresses progress meters and error messages, while -o /dev/null discards the actual body content, which is useful when you only care about the URL and not the page itself. Finally, -w "%{url_effective}\n" is a format string that tells curl to output the effective URL after all redirects have been followed. This combination is highly efficient for programmatically extracting the target URL.

When you need to get the final URL after curl is redirected, the most efficient method is to use the --location (or -L) flag in conjunction with the --write-out "%{url_effective}\n" option. This command tells curl to follow all subsequent redirect headers and then print the ultimate URL that was reached. For example, running curl -sL -w "%{url_effective}\n" -o /dev/null "http://bit.ly/example" will quickly return the resolved URL without downloading any content.

1. Specify Redirect Following: Use the -L or --location flag to enable curl to follow any 3xx HTTP redirect responses.
2. Suppress Output (Optional): Add -s or --silent to prevent curl from showing progress meters or error messages, keeping the output clean.
3. Discard Body Content (Optional): Include -o /dev/null to ensure that the actual HTML content of the final page is not downloaded or saved, which is crucial for speed when you only need the URL.
4. Print Effective URL: Use -w "%{url_effective}\n" or --write-out "%{url_effective}\n". This instructs curl to output the final URL reached after all redirects have been processed, followed by a newline character for easy parsing.
5. Execute the Command: Combine these flags with the initial URL you want to test. For example: curl -sL -w "%{url_effective}\n" -o /dev/null "https://example.com/old-page".

Question & Answer :

I need to get the final URL after a page redirect preferably with curl or wget.

For example http://google.com may redirect to http://www.google.com.

The contents are easy to get(ex. curl --max-redirs 10 http://google.com -L), but I’m only interested in the final url (in the former case http://www.google.com).

Is there any way of doing this by using only Linux built-in tools? (command line only)

curl’s -w option and the sub variable url_effective is what you are looking for.

Something like

curl -Ls -o /dev/null -w %{url_effective} https://example.com 

More info

-L Follow redirects -s Silent mode. Don't output anything -o FILE Write output to <file> instead of stdout -w FORMAT What to output after completion 

More

You might want to add -I (that is an uppercase i) as well, which will make the command not download any “body”, but it then also uses the HEAD method, which is not what the question included and risk changing what the server does. Sometimes servers don’t respond well to HEAD even when they respond fine to GET.