Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE REQUEST] Case insensitive sessions #334

Closed
mzpqnxow opened this issue Aug 19, 2021 · 2 comments
Closed

[FEATURE REQUEST] Case insensitive sessions #334

mzpqnxow opened this issue Aug 19, 2021 · 2 comments
Labels
enhancement New feature or request for-discussion

Comments

@mzpqnxow
Copy link
Sponsor

Will your endless flood of issues ever stop

Maybe. Probably not. Please feel free to close / decline to address / defer indefinitely any of them, or ask that I batch issues up with very condensed descriptions to save you time in reading them :)

Is your feature request related to a problem? Please describe.

It's difficult to say if this is truly a problem. If it is a problem, I don't yet know the scale of it as I need to review the data I have

The Behavior

In some configurations, you will see something like this (the 3 numbers you see there are line count, word count and count, respectively, a shortened output I use sometimes when sanity-checking/testing my post-processor)

...
http://redacted.co.za/views/admin.html 24 45 2
...
http://redacted.co.za/Views/admin.html 24 45 2         <--- Effectively a duplicate, if the server/app ignores case
...

I included the full JSON rows for these at the end in case you wanted to be sure I wasn't misreading the output

In some server/application configurations of course, case sensitivity is "a thing" so this behavior is certainly not wrong- it's correct, and is very possibly by design. However, it's not always desirable

Describe the solution you'd like

Perhaps provide an option to avoid making an "identical" request more than once- where identical is determined by ignoring the case of the URL paths

Ideally, this would mean that http://redacted.co.za/views/admin.html is only requested once, rather than twice with slightly differing case

It's probably fair to say this is a silly nitpick and/or a non-issue. I need to measure how frequently this actually occurs as it's not something I've noticed many times before. The best course of action may be for me to collect more data before even considering taking any action, unless you feel that it would be simple (and desirable) to make changes

Describe alternatives you've considered

  • Ensure your word-list is in a uniform case
    • This doesn't seem to help in this particular scenario, it occured with a wordlist that is tolower()'d. If this is not expected behavior, I included my full configuration below. I considered it may be related to --extract-links, where the word-list caused the first request (with lowercase /views) and extraction of the links caused the second one (with uppercase /Views. This would explain not seeing it happen too frequently
  • Remove the duplicates in post-processing (with an app, or a shell utility like tr or jq, to normalize the case of the URL path in the results)
    • This is a trivial solution for sure and provides the desired output; the deficiency with that solution is that it doesn't prevent the requests from occurring, it just acts as a filter. If this occurs often, at scale, it adds up to a lot of unnecessary requests- I'm not too concerned about this happening when used for 1, 2, 10 or 30 endpoints

Additional context

I don't consider this a significant issue, and won't until I have time to quantify the frequency/volume of it. I wanted to at least bring it to your attention though. I wonder if it may be more of an edge-case than anything else and may only occur when --extract-links is enabled, as mentioned above

In case this was unexpected behavior and you're curious about the details (or want to verify I'm not misinterpreting the data), full configuration and unabridged JSON rows follow...

*** Configuration ***

depth = 5
extensions = ["php", "asp"]
extract_links = true
filter_regex = [ "^<html><head><title>Request Rejected", "<HTML><HEAD>\n<TITLE>Access Denied</TITLE>\n</HEAD><BODY>\n<H1>Access Denied</H1>", "Reference&#32"]
filter_word_count = [6455]
insecure = true
json = true
no_recursion = false
quiet = false
redirects = false
save_state = false
scan_limit = 45
silent = false
status_codes = [200, 204, 301, 307, 308, 401, 403, 405]
time_limit = "5m"
timeout = 8

Unabridged Output Rows



{

"type": "response",

"url": "http://redacted.co.za/views/admin.html",

"path": "/views/admin.html",

"wildcard": false,

"status": 200,

"content_length": 830,

"line_count": 24,

"word_count": 45,

"headers": {

"accept-ranges": "bytes",

"server": "Microsoft-IIS/10.0",

"x-powered-by": "ASP.NET",

"date": "Thu, 19 Aug 2021 05:13:51 GMT",

"content-length": "830",

"last-modified": "Sun, 12 Jul 2020 11:39:47 GMT",

"content-type": "text/html",

"etag": ""265539254158d61:0""

}

}

{

"type": "response",

"url": "http://redacted.co.za/Views/admin.html",

"path": "/Views/admin.html",

"wildcard": false,

"status": 200,

"content_length": 830,

"line_count": 24,

"word_count": 45,

"headers": {

"content-type": "text/html",

"last-modified": "Sun, 12 Jul 2020 11:39:47 GMT",

"x-powered-by": "ASP.NET",

"server": "Microsoft-IIS/10.0",

"etag": ""265539254158d61:0"",

"accept-ranges": "bytes",

"date": "Thu, 19 Aug 2021 05:14:14 GMT",

"content-length": "830"

}

}



      		
    

      
  





        


            

              
            

        

      


      
    








      

    



      

  
          

  

    
  

  


    

        @mzpqnxow
mzpqnxow




        added
  the 

  enhancement

  New feature or request
 label


    Aug 19, 2021

  










      

  
      



  

  @epi052





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      

      

  
    Owner


      

  


  

    

      

      
          epi052
  

      

      

      commented


        Aug 20, 2021


      
    


  





      


        


  
    

      
          
Thank you for the report, as always!


A few quick comments:


    

  • i'm pretty sure i already prevent duplicate requests (the code i'm thinking of was there at the start, so it's been a minute). Duplicate here implies case-sensitivity tho.s
    • 

  • I think it's a fair assumption that --extract-links is the culprit, if your wordlist was all lowercase
    • 



I think I'm pretty much ok with this behavior as it is, based on what you've told me.  My reasoning is that ferox isn't requesting anything you haven't explicitly asked for.  You've asked for the words in your wordlist. Additionally, you've opted into the unknown grab-bag of extracting links from source. I feel like this should probably be chalked up to expected behavior.


I'm going to leave this open for discussion, and for any other data you find/wish to share (i'd be interested to know of any further analysis you perform). I'm not necessarily opposed to the suggestion, but don't see it as a need.

      		
    

  





        


            

              
            

        

      


      

            
  

    
  
  

  

  



    












      

  
          







      

  
      



  

  @mzpqnxow





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      

  Sponsor


      



      

  Author


  


  

    

  





      


        


  
    

      
          


i'm pretty sure i already prevent duplicate requests (the code i'm thinking of was there at the start, so it's been a minute). Duplicate here implies case-sensitivity tho.s

I think it's a fair assumption that --extract-links is the culprit, if your wordlist was all lowercase




I figured as much- I'm considering this a non-issue for now and will close it.


If I end up seeing this in large numbers later (I'm not expecting this) I'll look a little more closely and reopen with actual measurements rather than an anecdote


Thanks for taking the time

      		
    

  





        


            

              
            

        

      


      

            
  

    
  
  

  

  



    












      

  
          

  











  
  

    

      

  





  




  
  

              

  
    Sign up for free
    to join this conversation on GitHub.
    Already have an account?
    Sign in to comment


  



  






  
                  




    

  


      
  

    Assignees
  



        

    No one assigned







      


  


  

    Labels
  



    

      

    enhancement

  New feature or request
  

    for-discussion









      

  

    

        

    Projects
  


        




    None yet




  



      

  

    
  

    Milestone
  


      No milestone





    
      
          


  

    
      

        
    

    Development
  




              
No branches or pull requests







    
  




        
      

    

  
      

  

    

      2 participants
    

    

        
          @mzpqnxow 
        
          @epi052