Question: interpretation of results

[madig]

Hi!
I'm trying to profile a program and drill down into the underlying library (--profile-all --reduced-profile) and am somewhat stumped by the results.

1. The "Memory usage" at the top of the report swings between 300 and 800MB... It's a different value each run. Is this expected?
2. What does "Mem % Python" mean?
3. Does "Time % native" mean time spent in non-Python code?

[emeryberger]

Hi!

1. Scalene is a statistical profiler, meaning that it does sampling, and variance can certainly happen. A longer-running program that allocates and frees more memory will have more stable results.
2. "Mem % Python" means how much of the memory consumption is due to Python (vs. non-Python code)
3. Yes, "Time % native" means time spent in non-Python code.

If you have suggestions on how to make this more clear, please let me know!

[madig]

Hm, strange. The program ran for 5 mins on one input, 10 mins on the other and still had these swings. It turned out to be accidentally quadratic. One string concatenation was profiled to generate net 1.9GB of allocations :o

I think for now a simple listing in the Readme will help, gonna send a PR.

[emeryberger]

Sounds like a success story! I'd greatly appreciate it if you could write it up here (with whatever details you feel comfortable including): https://github.com/emeryberger/scalene/issues/58

I am surprised there are such large swings for such substantial allocations. If you can share your code (privately is fine), I'd like to see what's going on. Thanks!

[madig]

It was half a success story maybe, as I also used py-spy and my problem needed a flamegraph I think, to nail down who was calling a certain function so often it made up 90% of all activity. Just looking at memory allocation numbers sent me down the wrong path at first, until I understood why a string concatenation generated 1.9GB of memory traffic.

You can replay my findings by pip install psautohint==2.2.0 in a venv, downloading https://github.com/adobe-fonts/source-sans-pro/releases/tag/3.028R and running scalene venv/bin/psautohint source-sans-3v028R/VAR/SourceSans3VF-Italic.otf -o /tmp/out.otf. Set aside 10+ mins or so for it to finish. My fix is in adobe-type-tools/psautohint#289. Previously, the 3000 glyphs in the font would result in 3000 useless calls to CFF2.desuroutinize() that would then iterate over all 3000 glyphs to desubroutinze them. It's an expensive function that is already called once on init.