I have a function that parses the content of a given URL (usually the root of a domain) and returns a list of all links which point somewhere else within that domain –it doesn't include external links and it doesn't allow repeated links.
What would be the most simple way to run that function for each of the links that are already in the list, and then for each of the links that will be found in the next level, and so on, again and again, no matter how deep, no matter how long it takes, until an exhaustive, complete list of links for that specific domain is created?
The method should be reasonably efficient, and it should not use any libraries or modules, and not even list comprehensions. Just basic instructions in the most obvious and explicit way. Also, it's not important to know the structure of the tree. Just a list of links.
Thank you!