Subfinder is taking long time

[DharmX1]

Subfinder is taking approximately 2min for subdomain enumeration on latest version.
Currently I'm using
subfinder -dL domain.txt -o output.txt
[

](url)

[x-stp]

nice usage of a mobile device. 🥇

this behaviour occurs because you input a file with several domains, and the project will not handle them concurrently.
either wrap around it, and take the loss on rate limiting or be patient :)

might add a feature branch to add some optional backpressure, backoff handled concurrency with -dL some day.
@dogancanbakir is ^ something pd wants to go for or is it much noise on the sources?

[ehsandeep]

@x-stp concurrency won’t help since all sources have rate limits that Subfinder try to respects by default. Increasing concurrency leads to more rate-limiting and fewer results.

[x-stp]

yeah thats what i thought. not going to write code for it.

alternatively this could be done though:

           [ List of All Domains ]
      [ "google.com", "github.com", "tesla.com", ... ]
                     |
                     V
        [ Partitioner (hash(domain) % 3) ]
                     |
     +---------------------+--------------------+--------------------+
     |      Partition 0    |     Partition 1    |     Partition 2    |
     | [ "google.com" ]    | [ "github.com" ]   | [ "tesla.com" ]    |
     | [ "apple.com"  ]    | [ "netflix.com"]   | [ "microsoft.com"] |
     |      ...            |      ...           |      ...           |
     +---------------------+--------------------+--------------------+
              |                      |                    |
              V                      V                    V
      [ Worker 0 ]           [ Worker 1 ]         [ Worker 2 ]

(Only ever reads from P0) (Only ever reads from P1) (Only ever reads from P2) 

I could implement something like it; but I think would introduce a lot of overhead.

[dogancanbakir]

+1 for the overhead. The current design manages both concurrency and rate limiting. Of course, if there are specific cases that you think can be improved, we can look into it.

[x-stp]

If I have way too much spare time left (no), might draft something. Don't think so heh

[ehsandeep]

@DharmX1 You can use the -stats option to see the time taken by each source. If you’re aiming to reduce runtime over result count, you can exclude the sources (-exclude-sources) that take the most time.

   -stats  report source statistics