118

dnspython will do my DNS lookups very nicely, but it entirely ignores the contents of /etc/hosts.

Is there a python library call which will do the right thing? ie check first in etc/hosts, and only fall back to DNS lookups otherwise?

3
  • I created an issue for that: github.com/rthalley/dnspython/issues/149 Mar 31, 2016 at 15:07
  • 1
    dnspython will not implement this. For simple forward lookups, use the proposed socket.gethostbyname, for more complex queries, use dnspython.
    – sebix
    Jul 3, 2016 at 9:08
  • If you're on Linux with systemd, have dnspython use 127.0.0.53 as the dns server which should respect /etc/hosts. Aug 12, 2022 at 18:59

7 Answers 7

129

I'm not really sure if you want to do DNS lookups yourself or if you just want a host's ip. In case you want the latter,

/!\ socket.gethostbyname is deprecated, prefer socket.getaddrinfo

from man gethostbyname:

The gethostbyname*(), gethostbyaddr*(), [...] functions are obsolete. Applications should use getaddrinfo(3), getnameinfo(3),

import socket
print(socket.gethostbyname('localhost')) # result from hosts file
print(socket.gethostbyname('google.com')) # your os sends out a dns query
7
  • 1
    Does anyone know at which level this lookup is cached? Within Python? Or OS? Or DNS server?
    – Simon East
    Sep 18, 2011 at 9:36
  • @Simon Not cached by Python, nor the OS. It depends on any DNS server involved if it caches or not. – Generally speaking: DNS is cached only by the application itself, or by resolving-DNS-servers tucked in the resolving chain. May 18, 2013 at 6:07
  • @Jochen if “localhost” comes from the hosts file or not depends on the configuration! May 18, 2013 at 6:07
  • @RobertSiemer Sorry for the late comment: the result may be cached by the local resolver. nscd and nslcd on Unix boxes can do this. It could also be cached by a local name server configured for caching (a common setup, once upon a time. Probably not so much now). It's not a straightforward ‘no’ answer, unfortunately. These things rarely are. :)
    – Alexios
    Aug 14, 2014 at 17:35
  • 4
    For the record socket.gethostbyname in Python is not deprecated (as of 3.10 and 3.11-dev). You're referencing the man pages of Linux binaries, not Python libraries.
    – Tyler_1
    Mar 15, 2022 at 16:07
102

The normal name resolution in Python works fine. Why do you need DNSpython for that. Just use socket's getaddrinfo which follows the rules configured for your operating system (on Debian, it follows /etc/nsswitch.conf):

>>> print(socket.getaddrinfo('google.com', 80))
[(10, 1, 6, '', ('2a00:1450:8006::63', 80, 0, 0)), (10, 2, 17, '', ('2a00:1450:8006::63', 80, 0, 0)), (10, 3, 0, '', ('2a00:1450:8006::63', 80, 0, 0)), (10, 1, 6, '', ('2a00:1450:8006::68', 80, 0, 0)), (10, 2, 17, '', ('2a00:1450:8006::68', 80, 0, 0)), (10, 3, 0, '', ('2a00:1450:8006::68', 80, 0, 0)), (10, 1, 6, '', ('2a00:1450:8006::93', 80, 0, 0)), (10, 2, 17, '', ('2a00:1450:8006::93', 80, 0, 0)), (10, 3, 0, '', ('2a00:1450:8006::93', 80, 0, 0)), (2, 1, 6, '', ('209.85.229.104', 80)), (2, 2, 17, '', ('209.85.229.104', 80)), (2, 3, 0, '', ('209.85.229.104', 80)), (2, 1, 6, '', ('209.85.229.99', 80)), (2, 2, 17, '', ('209.85.229.99', 80)), (2, 3, 0, '', ('209.85.229.99', 80)), (2, 1, 6, '', ('209.85.229.147', 80)), (2, 2, 17, '', ('209.85.229.147', 80)), (2, 3, 0, '', ('209.85.229.147', 80))]
2
  • 6
    would be nice to add the transformation step. addrs = [ str(i[4][0]) for i in socket.getaddrinfo(name, 80) ] gives me the list of ips. Nov 26, 2015 at 15:43
  • my request result returns the same ip address multiple times; we can fix that with a simple change to alex's suggestion: addrs = { str(i[4][0]:i for i in socket.getaddrinfo(name, 80) } returns a dict that includes the unique ips as keys with the rest of the result paired.
    – tk421storm
    Sep 25, 2020 at 14:58
9

It sounds like you don't want to resolve DNS yourself. dnspython is a standalone DNS client that will understandably ignore your operating system because it's bypassing the operating system's utilities.

We can look at a shell utility named getent to understand how the (Debian 11-like) operating system resolves DNS for programs. This is likely the standard for all *nix like systems that use a socket implementation.

See man getent's "hosts" section, which mentions the use of getaddrinfo, which we can see as man getaddrinfo.

To use it in Python, we have to extract some info from the data structures:

import socket

def get_ipv4_by_hostname(hostname):
    # see `man getent` `/ hosts `
    # see `man getaddrinfo`

    return list(
        i        # raw socket structure
            [4]  # internet protocol info
            [0]  # address
        for i in 
        socket.getaddrinfo(
            hostname,
            0  # port, required
        )
        if i[0] is socket.AddressFamily.AF_INET  # ipv4

        # ignore duplicate addresses with other socket types
        and i[1] is socket.SocketKind.SOCK_RAW  
    )

print(get_ipv4_by_hostname('localhost'))
print(get_ipv4_by_hostname('google.com'))
5
list( map( lambda x: x[4][0], socket.getaddrinfo( \
     'www.example.com.',22,type=socket.SOCK_STREAM)))

gives you a list of the addresses for www.example.com. (ipv4 and ipv6)

3

This code works well for returning all of the IP addresses that might belong to a particular URI. Since many systems are now in a hosted environment (AWS/Akamai/etc.), systems may return several IP addresses. The lambda was "borrowed" from @Peter Silva.

def get_ips_by_dns_lookup(target, port=None):
    '''
        this function takes the passed target and optional port and does a dns
        lookup. it returns the ips that it finds to the caller.

        :param target:  the URI that you'd like to get the ip address(es) for
        :type target:   string
        :param port:    which port do you want to do the lookup against?
        :type port:     integer
        :returns ips:   all of the discovered ips for the target
        :rtype ips:     list of strings

    '''
    import socket

    if not port:
        port = 443

    return list(map(lambda x: x[4][0], socket.getaddrinfo('{}.'.format(target),port,type=socket.SOCK_STREAM)))

ips = get_ips_by_dns_lookup(target='google.com')
2
  • What's the '{}.'.format(target) for? It seems like both example.com and example.com. work fine. Mar 2, 2021 at 2:41
  • @MatthewD.Scholefield The '{}.'.format(target) construct could just as easily be replaced with {target} but I wanted to leave the original solution, proposed by @Peter Silver intact.
    – eatsfood
    Mar 2, 2021 at 21:41
2

You can dns lookup with python dns mudule.

import urllib.parse
import dns.resolver

try:
    parsed_url = urllib.parse.urlparse(url)
    hostname = parsed_url.hostname
    answers = dns.resolver.query(hostname, 'A')
    for rdata in answers:
        print(rdata.address)
except dns.resolver.NXDOMAIN:
    print('ip not found.') 
-1

I found this way to expand a DNS RR hostname that expands into a list of IPs, into the list of member hostnames:

#!/usr/bin/python

def expand_dnsname(dnsname):
    from socket import getaddrinfo
    from dns import reversename, resolver
    namelist = [ ]
    # expand hostname into dict of ip addresses
    iplist = dict()
    for answer in getaddrinfo(dnsname, 80):
        ipa = str(answer[4][0])
        iplist[ipa] = 0
    # run through the list of IP addresses to get hostnames
    for ipaddr in sorted(iplist):
        rev_name = reversename.from_address(ipaddr)
        # run through all the hostnames returned, ignoring the dnsname
        for answer in resolver.query(rev_name, "PTR"):
            name = str(answer)
            if name != dnsname:
                # add it to the list of answers
                namelist.append(name)
                break
    # if no other choice, return the dnsname
    if len(namelist) == 0:
        namelist.append(dnsname)
    # return the sorted namelist
    namelist = sorted(namelist)
    return namelist

namelist = expand_dnsname('google.com.')
for name in namelist:
    print name

Which, when I run it, lists a few 1e100.net hostnames:

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.