Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add filtering of english words from entropy (and keyword) plugins #241

Merged
merged 26 commits into from
Sep 24, 2019
Merged

Add filtering of english words from entropy (and keyword) plugins #241

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
d93e9bd
:mortar_board: Standardize `do not` -> `don't`
KevinHock Sep 19, 2019
5b66a26
:snake: Refactor awkward [0] to a short-circuit
KevinHock Sep 19, 2019
bb543c5
:bug: We were doing e.g. '.svg' == 'svg'
KevinHock Sep 19, 2019
f10cce8
:snake: Refactor `for...if...return` with `any`
KevinHock Sep 19, 2019
bc4f922
:snake: Remove is_false_positive from RegexBasedDetector
KevinHock Sep 19, 2019
0b2c0e1
:mortar_board: Standardize comments
KevinHock Sep 19, 2019
f8cb31f
:tada: Add --word-list option
KevinHock Sep 19, 2019
7bdf06f
:100: Coverage for jwt.py
KevinHock Sep 19, 2019
e4494ea
:snake: Separate not :100: covered files
KevinHock Sep 19, 2019
d3c9583
:snake: Make automaton case-insensitive
KevinHock Sep 19, 2019
9e3669d
:snake: Slightly better condition for is_verified = True
KevinHock Sep 19, 2019
7ce4b85
:telescope: Don't put words less than 4 chars in automaton
KevinHock Sep 20, 2019
de7fbd2
:100: Coverage for high_entropy_strings.py
KevinHock Sep 20, 2019
139c64a
:snake: Refactor `audit --display-results` code
KevinHock Sep 21, 2019
6c6e028
:mortar_board: Use plural variable name for list
KevinHock Sep 21, 2019
977c4fb
:tada: Add verification for Mailchimp API keys
KevinHock Sep 21, 2019
9cabbe0
:tada: Add verification for Stripe secret API keys
KevinHock Sep 21, 2019
20d1921
:100: Coverage for baseline.py initialize.py usage.py
KevinHock Sep 21, 2019
433b75e
:snake: Add stats to `audit --display-results` code
KevinHock Sep 23, 2019
a096600
:performing_arts: Add more to `IGNORED_FILE_EXTENSIONS`
KevinHock Sep 23, 2019
ef5d000
:bug: Fix auditing a baseline with MailChimp in it
KevinHock Sep 23, 2019
a493356
:bug: Fix TypeError thrown when Yaml was all comments
KevinHock Sep 23, 2019
171eeca
:bug: Fix scanning files that don't exist
KevinHock Sep 23, 2019
69a3aac
:snake: Use hash.update to be more succinct
KevinHock Sep 24, 2019
19d22da
:snake: Inline leading . in constants.py
KevinHock Sep 24, 2019
b440623
:snake: Remove useless WordListSupportedDetector class
KevinHock Sep 24, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
💯 Coverage for high_entropy_strings.py 
- 🐍 Refactor hadouken code
- 🐍 Remove high_entropy_strings.py from the uncovered files list in tox
  • Loading branch information
@KevinHock
KevinHock committed Sep 20, 2019
commit de7fbd265942b1d404a3dc7292db386312a9b63d
50 changes: 24 additions & 26 deletions detect_secrets/plugins/high_entropy_strings.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -196,36 +196,34 @@ def _analyze_yaml_file(self, file, filename):
while len(to_search) > 0:
item = to_search.pop()

try:
if '__line__' in item and item['__line__'] not in ignored_lines:
# An isinstance check doesn't work in py2
# so we need the __is_binary__ field.
string_to_scan = (
self.decode_binary(item['__value__'])
if item['__is_binary__']
else item['__value__']
)

secrets = self.analyze_string(
string_to_scan,
item['__line__'],
filename,
)

if item['__is_binary__']:
secrets = self._encode_yaml_binary_secrets(secrets)

potential_secrets.update(secrets)

if '__line__' in item:
continue

if '__line__' not in item:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice refactor!

for key in item:
obj = item[key] if isinstance(item, dict) else key
if isinstance(obj, dict):
to_search.append(obj)
except TypeError:
pass
continue

if item['__line__'] in ignored_lines:
continue

# An isinstance check doesn't work in py2
# so we need the __is_binary__ field.
string_to_scan = (
self.decode_binary(item['__value__'])
if item['__is_binary__']
else item['__value__']
)

secrets = self.analyze_string(
string_to_scan,
item['__line__'],
filename,
)

if item['__is_binary__']:
secrets = self._encode_yaml_binary_secrets(secrets)

potential_secrets.update(secrets)

return potential_secrets

Expand Down
4 changes: 2 additions & 2 deletions tox.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ commands =
coverage run -m pytest tests -v
coverage report --show-missing --include=tests/* --fail-under 100
# This is so that we do not regress unintentionally
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️

coverage report --show-missing --include=detect_secrets/* --omit=detect_secrets/core/audit.py,detect_secrets/core/baseline.py,detect_secrets/core/secrets_collection.py,detect_secrets/core/usage.py,detect_secrets/main.py,detect_secrets/plugins/common/ini_file_parser.py,detect_secrets/plugins/common/initialize.py,detect_secrets/plugins/high_entropy_strings.py --fail-under 100
coverage report --show-missing --include=detect_secrets/core/audit.py,detect_secrets/core/baseline.py,detect_secrets/core/secrets_collection.py,detect_secrets/core/usage.py,detect_secrets/main.py,detect_secrets/plugins/common/ini_file_parser.py,detect_secrets/plugins/common/initialize.py,detect_secrets/plugins/high_entropy_strings.py --fail-under 97
coverage report --show-missing --include=detect_secrets/* --omit=detect_secrets/core/audit.py,detect_secrets/core/baseline.py,detect_secrets/core/secrets_collection.py,detect_secrets/core/usage.py,detect_secrets/main.py,detect_secrets/plugins/common/ini_file_parser.py,detect_secrets/plugins/common/initialize.py --fail-under 100
coverage report --show-missing --include=detect_secrets/core/audit.py,detect_secrets/core/baseline.py,detect_secrets/core/secrets_collection.py,detect_secrets/core/usage.py,detect_secrets/main.py,detect_secrets/plugins/common/ini_file_parser.py,detect_secrets/plugins/common/initialize.py --fail-under 96
pre-commit run --all-files --show-diff-on-failure

[testenv:venv]
Expand Down